Does the cited paper actually support the claim?

A paper citation can look decisive while supporting something much narrower than the sentence attached to it. A title matches the topic, the abstract sounds promising, and several scholarly indexes return the work. None of that yet answers the useful question: what part of which version of the paper tested the claim, and what did the authors actually find?

Consider three statements that a research agent might write after finding a relevant paper:

“Half of AI citations are fabricated.”

“Language models cannot use information in the middle of a long document.”

“AlphaFold predicts every protein structure with experimental accuracy.”

Each sentence points toward a real and important research paper. Each goes further than its source warrants. This article follows the work back through Semantic Scholar, Valyu, and OpenAlex, then reads the underlying publications. The providers do different jobs here: one helps resolve a paper and its citation graph, one returns searchable paper passages, and one ties a work to a DOI and publication locations. They are discovery routes, not three independent replications of any finding.

Disclosure and method. Search for Agents is produced by Valyu. On 25 September 2026, we looked up the three named works through the providers described below and checked the reported claims against the original publications. The provider lookups were different, task-specific operations; this is not a matched-query comparison of coverage, ranking, latency, or price. Search indexes and citation counts can change. We cite the paper version used for each check.

First, turn a headline into a testable claim

“The paper supports it” can mean at least three different things. It may simply mention the topic. It may report a result related to the claim, but under narrower conditions. Or it may directly establish the statement as written. A reviewer should decide which of those is true before adding the citation to a sentence.

Write down the exact proposition before searching. Name the subject, outcome, population or task, time, and strength of inference. “A model sometimes misses a fact placed in the middle of a test context” can be investigated. “All models cannot read the middle” silently adds a universal quantifier. “Predicted structures were accurate on most targets in a blind assessment” is different from “all protein structures are solved.” The extra words are often where a citation stops supporting the sentence.

Next, separate the bibliographic problem from the evidence problem. An arXiv identifier, DOI, or OpenAlex work ID is useful for finding the right work. A citation edge tells you that another work refers to it. An abstract previews the authors' account. None identifies a paragraph that establishes every proposed conclusion. Finding that paragraph requires opening the original and checking how the result was produced.

The table below is our working hypothesis before reading the three papers:

Claim to check	Work located	Question for the original paper
“Half of AI citations are fabricated.”	The ALCE citation-evaluation study	Which outputs and dataset were measured, and what counted as incomplete support?
“Models cannot use information in the middle.”	Lost in the Middle	Which models and tasks showed a position effect, and were there exceptions?
“AlphaFold predicts every protein perfectly.”	The 2021 AlphaFold structure-prediction paper	Which structures were tested, what metric defined accuracy, and where did accuracy fall?

Case one: citation support is not the same as a made-up reference

We used the Semantic Scholar Academic Graph API to retrieve Enabling Large Language Models to Generate Text with Citations by its arXiv identifier 2305.14627. The returned record connected a Semantic Scholar paper ID to external identifiers including that arXiv ID and its arXiv DOI. It also offered a citation count and an open-access PDF link. Those fields made the paper easy to locate; the count did not tell us whether the headline claim was correct.

The original ALCE paper, also available in the EMNLP 2023 proceedings, evaluates long-form answers with citations on three datasets. Its authors distinguish fluency, answer correctness, and citation quality. On their ELI5 question set, the abstract reports that even the best evaluated systems lacked complete citation support about 50% of the time. The revised full text explains that the relevant evaluated baselines produced generations not fully supported by their cited passages in roughly half of those cases.

That is not a count of fabricated paper titles. A citation may lead to a real page and still fail to support the sentence next to it. The paper distinguishes citation recall, which asks whether cited passages support a generated statement, from citation precision, which identifies irrelevant citations. Its automatic support assessment uses a natural-language inference model, and the authors also report human evaluations. In the paper's limitations, they acknowledge that an automatic evaluator can miss partial support and that the chosen datasets do not cover every harder task.

The denominator matters as much as the percentage. The researchers sampled 1,000 development examples per dataset; ELI5 is a particular long-form question-answering setting, with its own retrieval corpus and tested systems. The defensible sentence is: “In ALCE's ELI5 evaluation, even the best evaluated systems lacked complete support from their cited passages about half the time.” The study does not establish a universal failure rate for every research assistant, all scientific citations, or every citation on the web. It certainly does not show that half of their linked papers were invented.

This is the first rule for a paper-search agent: report what was measured, by whom, and on which set of cases. Semantic Scholar's graph made identification fast. It could not convert an evaluation metric into a broader statement the researchers did not test.

Case two: five results can be one paper, and one date can be wrong

For Lost in the Middle: How Language Models Use Long Contexts, we ran a Valyu paper search for “Lost in the Middle How Language Models Use Long Contexts Liu 2023” and requested five results. All five returned the same arXiv URL, arxiv.org/abs/2307.03172, but surfaced different sections or chunks. Five results did not mean five studies supporting a conclusion. Grouping by paper identifier leaves one underlying work for this example. Different excerpts can help navigation without becoming independent evidence.

The observed Valyu result also carried a publication_date of 2023-01-01. arXiv's submission history dates the first version to 6 July 2023, followed by later revisions; we read version 3. We cannot infer from one response why the provider used January 1. It may reflect normalization of year-only metadata, but it should not be cited as the paper's actual submission day. Valyu's academic-search guide describes returned passages and scholarly metadata; an agent should still check dates and versions against the repository record. This is a dated observation, not a claim that every Valyu paper date is inaccurate.

Now look at what the paper tested. The authors evaluated multi-document question answering and a controlled key-value retrieval task, moving the relevant information to different positions in an input context. They report that, for many of the models and settings studied, accuracy fell when the answer-bearing material appeared in the middle rather than near the beginning or end. That is a meaningful positional effect. It is not a demonstration that information in the middle can never be used.

The full paper makes the counterexample plain: some evaluated models performed perfectly or nearly perfectly on its synthetic key-value task. The authors also examined query-aware contextualization and reported a setting in which it markedly improved key-value retrieval. Their multi-document experiment likewise fixes a particular question, set of passages, placement procedure, and collection of models. It does not test every paper-search agent, every document arrangement, or systems released years later.

A supported rewrite would be: “In the tested multi-document QA and key-value tasks, several evaluated models were less accurate when relevant information was placed in the middle of a long input.” “All models ignore the middle” contradicts exceptions in the cited work. The original article's figures and experiment descriptions allow us to draw that boundary; the number of search chunks and a short excerpt do not.

Case three: a highly cited paper can still have explicit limits

We resolved the DOI 10.1038/s41586-021-03819-2 through OpenAlex's Work API. The record identifies Highly accurate protein structure prediction with AlphaFold, links to its publication location, and exposes scholarly metadata including a cited_by_count. This makes OpenAlex useful for finding and joining records. The citation count says the work is widely referenced in the index; it does not establish that every claim someone attaches to it has been replicated or endorsed.

The 2021 paper reports exceptional protein-structure predictions on the blind CASP14 assessment. We inspected its open full text at PubMed Central. OpenAlex labels that location a submittedVersion and the Nature DOI location a publishedVersion: they identify one work, but should not be assumed word-for-word identical. Fig. 1 of the text we inspected describes 87 protein domains in the comparison. The abstract says predictions were competitive with experimental structures in a majority of cases. Those are substantial results. “Every protein perfectly” is a much larger statement, with a different denominator and no allowance for errors.

The limitations are not hidden. In the section on multiple-sequence-alignment depth and cross-chain contacts, the paper says accuracy drops substantially when median alignment depth is below about 30 sequences. It also says the evaluated model is much weaker for protein regions whose shape depends heavily on contacts with other chains in a complex. AlphaFold supplies a per-residue confidence estimate, but a confidence value is not an experimental structure determination or a claim that all biological contexts were tested.

A supported summary is therefore: “The studied version of AlphaFold achieved accuracy competitive with experimental structures in most cases assessed, while its paper documents important limits involving sequence information and cross-chain interactions.” The precise finding belongs to the model and assessments in this publication. Later systems may address different tasks; their advances should be checked in their own papers, not silently read backward into the 2021 study.

OpenAlex got us to a publisher DOI and an accessible text location. The qualification came from the open text's results and limitations, not from its title or indexed citation count. For an exact quotation or a disputed wording change, compare the publisher's version with the copy being cited. Two locations for the same work would not constitute two replications.

An evidence card that a second reader can challenge

Before publishing an answer, give every consequential sentence a compact, inspectable evidence card. It should contain enough detail for another reader to disagree intelligently rather than merely reopen a homepage:

1. Exact claim. Preserve qualifiers such as “some,” “most,” “in this trial,” or “as of this date.” Separate the paper's finding from your proposed generalization.
2. Work and version. Record DOI, arXiv ID, PMID, or another persistent identifier; distinguish a preprint revision from a publisher version or an authorized repository copy.
3. The actual support. Name the section, figure, table, page, or short passage, with enough neighboring context to rule out a misleading crop. Record whether the material comes from an abstract, a methods section, original results, or somebody else's summary.
4. Study boundaries. Keep the sample, experimental task, comparison group, outcome metric, time period, and known exceptions beside the claim. These determine which words the evidence can support.
5. Publication state. Check for updated versions, later corrections, and whether an index merged distinct versions or returned multiple chunks of one work.
6. Verdict and uncertainty. Mark the statement directly supported, supported only with narrower wording, contradicted by the source, or not verified from available text. Add the reason; “unverified” is not a synonym for “false.”

The verdict need not be made by a model alone. An agent can propose the card and quote its candidate passages. A researcher should be able to inspect the linked paper and see exactly how the conclusion follows. A matching title, citation graph edge, or confident synthesis provides an input to that review, not its outcome.

In these three checks, the narrower wording is the difference between a useful result and an attractive error. The ALCE percentage stays attached to its benchmark and definition of support. The long-context finding stays attached to the evaluated tasks and exceptions. AlphaFold's achievement stays attached to the structures it assessed and the situations where it reported weaker performance. Every paper remains important after those qualifications; none needs an inflated headline to matter.

How the three search systems fit the workflow

The providers did not need to return the same objects to be useful. Semantic Scholar's paper record connected an arXiv ID, DOI, PDF location, and citation graph for ALCE. Valyu surfaced passages of Lost in the Middle; the repeated source URL and questionable date required verification at arXiv. OpenAlex resolved the AlphaFold DOI into a work record and publication locations; the findings still required the full paper.

That is a comparison of roles, not a provider leaderboard. A paper index is good at finding and connecting works; an extracted passage speeds navigation; a DOI lookup reduces identity mistakes. None of those fields tells you whether a sentence outruns the methods. An agent that uses several systems should deduplicate by underlying work and version, retain which provider found each lead, and keep the original publication as the authority for what it reports.

If full text cannot be reached, stop at what the accessible abstract or metadata actually says. A missing methods section is a reason to mark a claim unverified, not to invent a result or to infer that the opposite is true. The industry-search source checks apply the same distinction to filings, treatment approvals, and opinions; the Web Search field note shows how a plausible result can hide an outdated factual count.

Research note: Provider lookups and source checks were conducted on 25 September 2026. Semantic Scholar was queried by arXiv ID, Valyu by a title-and-author paper query, and OpenAlex by the published DOI. The three claims above are deliberately tested against the linked originals, not offered as quotations from a provider's generated answer. Search results, citation counts, and repository versions may change; repeat the checks before using them in a current review.