Preview build on GitHub Pages. The open registry will live at e2er.org.
Skip to content
Demonstrator · items marked Example are invented · what exists today
E2ER

An evaluation finds where a building block breaks

A check tells whether one run was faithful to its inputs. An evaluation tells whether a building block can be trusted, under which conditions, and at which point it fails. Developing and proposing this protocol is the centrepiece of the development stage.

Planned, Stage 2 (January–May 2027)

Benchmark tests

Compare a step or template with known answers, for instance an interview corpus that human coders have already coded.

Reproduction tests

Run a template on a published study and compare its results with the original.

Stress tests

Vary document length, corpus, data quality or model to locate the point at which output degrades.

Expert assessments

Researchers rate a building block against a shared rubric and report failures. RISE already scores research systems against a published rubric.

Real records

1 real record
A run with its checks switched to record only5 August 2026 · empirical template · 53 of 110 table values differed from their source, 14 had none; a faulty prose check fell from 284 flags to 2 once fixed.

Evaluation records

Example with invented data

The records below concern hypothetical building blocks. They show what a researcher would read before relying on a block.

qualitative-coding@0.3.1 · stressExample Agreement with human coders stays above the declared threshold of 0.75 up to 11,000 words per transcript. template36 runs, 4 failed, 1 aborted· Demo contributor qualitative-coding@0.2.0 · stressExample Agreement with human coders stays above 0.75 up to 6,000 words per transcript. template36 runs, 7 failed, 2 aborted· Demo contributor systematic-review@0.1.0 · benchmarkExample Recall of included studies 0.91; precision 0.62. Most misses were conference papers without abstracts. template10 runs, 1 failed, 0 aborted· Noor Haddad codebook-development@0.2.0 · expert assessmentExample Mean rubric score 3.8 of 5. Raters reported two failures: overlapping codes and missing exclusion criteria. skill9 runs, 0 failed, 0 aborted· Ines Varga

Compare two versions of a block

Example

The same stress test on two versions of the hypothetical qualitative-coding template. A researcher can see where the new version moved the breaking point before switching.

VersionStatusBreaking pointRuns (failed)Record
0.3.1in reviewAgreement falls below 0.75 at about 12,000 words.36 (4)ev-0001
0.2.0listedAgreement falls below 0.75 at about 7,000 words.36 (7)ev-0002

How a stress test reports a breaking point

Example with invented data

A hypothetical coding template, evaluated on transcripts of increasing length. The evaluation record states the length at which agreement with human coders falls below the threshold the template declares.

0.75

An evaluation record

Example format
block:        qualitative-coding@0.3.1
test:         stress / document length
material:     interview corpus, 240 transcripts, coded by two humans
model:        named model and version
runs:         all runs, incl. failed and aborted ones
result:       agreement by length band; breaking point where it falls below threshold
evaluator:    name, ORCID, contributor role "validation"
scope:        valid for this material; other corpora need their own record

Records include failed and aborted runs, because reporting only successful runs overstates performance.

Evaluate a building block

Evaluators author a citable record with the contributor role "validation". Reporting a failure earns credit in the block's change history and in the evaluation it triggers.

Submit an evaluationReport a failureEvaluation protocol