Validate retrieval runs before computing metrics

necs-validate is a dependency-light structural preflight for TREC-style qrels and run files. It is useful in CI before trec_eval, pytrec_eval, or a custom metric script.

It detects or reports:

This is not byte-for-byte NIST input certification or benchmark certification. A passing report does not establish dataset provenance, correct gains, model quality, or a sound experimental design.

Found a run this preflight rejects but trec_eval accepts, or a corruption it misses? Please open an issue with a reproducible case — that is the most useful bug report this tool can get.

Validate in your browser

Prefer not to install anything? The in-browser validator runs the exact v0.3.1 release wheel client-side through Pyodide: drop a qrels and a run file (or load a bundled valid/broken example), optionally toggle the strict-mode flags, and read the same pass/warn/fail report the CLI produces — then copy it as Markdown to share. Files are read in the page and never uploaded; there is no server, tracking, or analytics. The CLI below remains the canonical entry point for CI.

Run the CLI

From a checkout:

python -m pip install -e .
necs-validate \
  --qrels examples/validation/sample.qrels \
  --run examples/validation/sample.run \
  --expected-task task1_ranking

Add --format json for machine-readable output. Query-set differences, advisory rank anomalies, and unjudged documents are visible warnings by default. Use --require-query-coverage, --strict-ranks, or --require-judged to promote the corresponding diagnostics to errors when a fully matched evaluation contract requires them.

The command exits with 0 when no integrity errors remain and 1 when the report fails. Invalid command-line arguments use argparse’s standard nonzero exit. That makes the same command safe to place directly in CI.

Compatibility defaults

The defaults follow common IR tooling instead of enforcing one exporter:

The accepted whitespace-separated formats are:

# optional metadata header
# task: task1_ranking

# qrels
query_id iteration document_id relevance

# run
query_id Q0 document_id rank score run_tag

Float relevance grades

TREC-style qrels grade relevance with integers. Integer-only evaluators — including NIST trec_eval, respected reference software — parse the grade as an integer, so a fractional grade is silently truncated toward zero: a 1.5 judgement is scored as 1 and a 0.3 judgement as 0. The value changes with no error or warning, so a corrupted export (for example a float column written straight from a dataframe) can quietly shift every metric.

As defense in depth before evaluators that assume integer grades, this preflight treats a non-integer qrels grade as a structural error by default. A grade whose value is an integer is accepted regardless of how it is written, so 2 and 2.0 both pass; only a genuinely fractional value such as 1.5 fails.

$ necs-validate --qrels qrels.txt --run run.txt
NECS run validation: FAIL
qrels=1 queries/1 judgements; run=1 queries/1 entries
errors=1 warnings=0
[ERROR] non_integer_relevance (qrels.txt, line 1): query=1 document=d1 Relevance grade must be an integer, found '1.5'; integer-only evaluators silently truncate non-integer grades

Some graded-relevance workflows use fractional gains deliberately — NECS’s own NDCG uses ESCI gains of 1.0, 0.1, 0.01, and 0.0. Pass --allow-float-grades to keep those runs valid while still surfacing each fractional grade as an advisory warning instead of an error.

Use the GitHub Action

name: Validate retrieval evidence
on: [push, pull_request]

permissions:
  contents: read

jobs:
  validate-run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
      - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
        with:
          python-version: "3.11"
      - uses: Madhvansh/Neural-E-Commerce-Search@81e73c9ed4f3b0ad45b63204b20d82eba308ecc6 # v0.3.1 action code
        with:
          qrels: evaluation/qrels.txt
          run: evaluation/run.txt
          expected-task: task1_ranking
          # Optional strict contracts:
          # require-query-coverage: "true"
          # require-judged: "true"
          # strict-ranks: "true"

The full hardened action commit SHA is the strongest immutable pin. Replace it with v0.3.1 only if your update policy deliberately follows the readable release tag. The action requires Bash and Python 3.9 or newer; the example provisions Python explicitly. It runs the validator bundled with the referenced revision in Python isolated mode, uses no token or secrets, and does not install the project or download model assets.

The two paths are required. expected-task defaults to empty, while require-query-coverage, require-judged, and strict-ranks each default to "false". Pass boolean values as the exact quoted strings "true" or "false".

If you adopt it in a public repository, share the workflow or any failure it caught through the validator compatibility report.

Published compatibility evidence

Each report distinguishes maintainer-run compatibility testing from downstream adoption or upstream endorsement.

Further reading