Paper replication triage: parallel audit of five sources and conflict reconciliation

Scenario

Typical scenario: a research group plans to replicate a paper and first has to answer a few questions: is the official code available, can the data splits be obtained, do the paper's reported metrics match the community's replications, and how much compute is needed. The paper's own claims, the GitHub repository, the dataset page and replication notes in later citations are four or five sources that each tell their own story; going through them one by one and cross-checking them by hand takes time and misses contradictions.

This example gives each source to an agent that sees only that source, reviewing them in parallel without interfering with each other, and has a downstream planner agent reconcile them all. The sample data plants four conflicts: the paper says the official code is public, but the repository URL returns 404; the paper says all splits are public, but benchmark_holdout requires an access request; the paper reports macro-F1, but replication notes only come close to the original score with micro-F1; the paper says a single 24GB GPU is enough to reproduce, but the training command found launches on 4 GPUs by default.

How it works

The models in the table are the example repository's defaults; the actual choice is made per scenario at the design stage.

The model is selected by the LLM_PROVIDER environment variable, defaulting to anthropic / claude-sonnet-4-6, with openai, gemini, groq and openrouter also supported; none of the six agents declares tools, so they only make model calls. The team sets maxConcurrency: 5, the five audit tasks start at once, and the planning task starts after all five finish.

Every agent has a zod outputSchema: each of the five audit agents has its own report structure, and the planner outputs fields including decision (reproduce / reproduce_with_caution / do_not_reproduce_yet), risk_score, artifact_inventory and evidence_conflicts. The framework validates output in agent.ts and, on a first failure, retries once with the error message attached.

The team turns on sharedMemory: true; the planning task sets no memoryScope, and its prompt is injected with the outputs of the five prerequisite tasks (the ## Context from prerequisite tasks section). Before the run, validateMockBundle() checks that the snapshots contain all four planted signals and throws if one is missing.

The default SOURCE_MODE=mock reads the five JSON snapshots in the repository; SOURCE_MODE=live switches to fetching live data through Asta MCP and the GitHub API, and needs ASTA_API_KEY.

Result

Runnable output and how to verify it:

It needs an API key. The default is LLM_PROVIDER=anthropic with ANTHROPIC_API_KEY as the prerequisite; without it the script prints [skip] and exits with code 0. Run it with npx tsx packages/core/examples/cookbook/paper-replication-triage.ts, optionally followed by a paper title or arXiv ID.

The parallelism is observable. The progress callback prints [START] / [DONE] for each task; after the run, verifySourceTasksStartedTogether() computes the spread in start times across the five audit tasks and prints Source task start skew, throwing if it exceeds 2000 milliseconds.

The planning result is asserted. If the planner fails or returns no structured output, the script prints the raw output and exits with code 1; in mock mode, assertPlannerDetectedConflicts() requires at least 2 evidence_conflicts with at least one whose topic is code or dataset, and once it passes the script prints REPLICATION TRIAGE REPORT (JSON) and "Done."

The five snapshots in the example are fictional MOCK data: the paper SparseCheck, its authors and the dataset SciQA-LongTail are all invented and point to no real paper; the output is model-generated text, used to demonstrate and verify the workflow.

Sources

How this relates to YuanASI

The material on this page is a cookbook example from the official open-multi-agent repository, maintained by YuanASI.

If you run a real literature or technical due-diligence process and want papers, code repositories, datasets and citation feedback split across agents for parallel checking, rolled up into a structured verdict with a conflict list, and connected to your own literature library and search tools, the matching YuanASI service is Custom AI Agent Development →