Evaluation & validation

How RafiHive answers are tested

Before a release, every answer is run against a golden set of regulatory questions with known expected sources. The numbers below come straight from the last harness run on 2026-09-07; nothing here is typed in by hand.

A case passes only when every expected source and required concept is present, no citation is fabricated and no forbidden claim appears. One missing expected citation fails the whole case, which is why this count sits below the share of statements that are correct.

What the golden set covers

16 questions across variations, dossier structure, labelling, ICH quality and blue box topics. Retrieval is measured with top-8 results; answers are generated by eu.anthropic.claude-sonnet-4-6.

What counts as a pass

An answer passes only if every expected source is cited, no citation is invented, the required concepts are present and no forbidden claim appears. A single missing citation fails the whole case, which is why the answer pass rate sits below the retrieval hit rate.

Regulatory review status

The benchmark above is an automated regression set that runs before every release. Regulatory review of the benchmark cases is recorded internally with reviewer role and review date and is being completed separately from the harness; 0 of 16 cases carry a recorded review today. A release gate blocks the claim of full regulatory review until all cases do.

Limits

A golden set is a regression guard, not proof of correctness on your question. Every RafiHive output still requires qualified RA review before it is used. See AI governance for the human-review boundary and sample output for what a reviewed answer looks like.