{"id":"post_paper_search_claim_support","kind":"post","language":"en","slug":"does-the-cited-paper-actually-support-the-claim","title":"Does the cited paper actually support the claim?","summary":"A citation can lead to the right paper and still misstate its finding. Three source-level checks show how to test a claim against the paper's methods, results, and limits.","directAnswer":"A cited paper supports a claim only when the version cited tests that claim's task or population and reports a result strong enough for the wording used. Semantic Scholar, Valyu, and OpenAlex can locate works and passages; links and citation counts are not proof. Check the original methods, results, and limitations, then cite the precise version and passage or qualify the statement.","publishedAt":"2026-09-25","updatedAt":"2026-09-25","author":"Eliot Reed","authorRole":"Research writer","topic":"Paper Search","tags":["search","verification","context","paper-search"],"image":"/images/uploads/paper-search-abstract-vs-results.jpg","imageAlt":"Torn blue and oxblood paper contrasting an abstract with the results of a research paper","featured":false,"readingMinutes":11,"seoTitle":"Does the Cited Paper Actually Support the Claim?","seoDescription":"Verify scientific claims against original papers, not citation counts or abstracts. Three worked cases use Semantic Scholar, Valyu, and OpenAlex for discovery.","targetQuestion":"How can an agent tell whether a cited scientific paper actually supports a claim?","keyTakeaways":["A DOI or paper-search hit identifies a work; it does not tell you which sentence in it supports your claim.","The ALCE result concerns citation support on a particular benchmark, not the fraction of all citations that are fabricated.","A Valyu search returned several chunks of one arXiv paper with a date that did not match arXiv's submission record; deduplicate and check dates at the source.","The AlphaFold paper reports strong results on specified structures and conditions, not perfect predictions for every protein.","Record a claim-level verdict with the exact version, section, measured outcome, scope, and unresolved limits."],"sources":[{"title":"Semantic Scholar Academic Graph API","url":"https://api.semanticscholar.org/api-docs/graph","publisher":"Semantic Scholar"},{"title":"Enabling Large Language Models to Generate Text with Citations","url":"https://arxiv.org/abs/2305.14627","publisher":"arXiv","publishedAt":"2023-05-24"},{"title":"ALCE paper, revised version 2","url":"https://arxiv.org/pdf/2305.14627v2","publisher":"arXiv","publishedAt":"2023-10-31"},{"title":"ALCE in the EMNLP 2023 proceedings","url":"https://aclanthology.org/2023.emnlp-main.398/","publisher":"ACL Anthology"},{"title":"Valyu academic search sources and limitations","url":"https://docs.valyu.ai/use-cases/academic","publisher":"Valyu"},{"title":"Lost in the Middle: How Language Models Use Long Contexts","url":"https://arxiv.org/abs/2307.03172","publisher":"arXiv","publishedAt":"2023-07-06"},{"title":"Lost in the Middle, version 3 full paper","url":"https://arxiv.org/pdf/2307.03172v3","publisher":"arXiv","publishedAt":"2023-11-20"},{"title":"OpenAlex work record for the AlphaFold paper","url":"https://api.openalex.org/works/https://doi.org/10.1038/s41586-021-03819-2","publisher":"OpenAlex"},{"title":"Highly accurate protein structure prediction with AlphaFold","url":"https://doi.org/10.1038/s41586-021-03819-2","publisher":"Nature","publishedAt":"2021-07-15"},{"title":"Open full text of the AlphaFold paper","url":"https://pmc.ncbi.nlm.nih.gov/articles/PMC8371605/","publisher":"PubMed Central"}],"revisions":[{"date":"2026-09-25","summary":"Checked three claim interpretations against the original papers and inspected Semantic Scholar, Valyu, and OpenAlex discovery records."}],"redirectFrom":[],"definitions":[{"term":"Claim-level support","definition":"Whether the cited passage, result, or figure warrants the specific statement beside the citation, including the study's task, sample, metric, and qualifications."},{"term":"Version of record","definition":"The publisher's formal version of a scholarly work; a preprint or later revision can have different text and should be identified separately when a claim depends on it."}],"statistics":[],"expertQuotes":[],"comparisonTables":[],"faqs":[{"question":"Does a DOI prove that the cited paper supports a claim?","answer":"No. A DOI helps locate a work or version. Support requires the relevant passage, result, or figure to establish the exact claim, with its population, metric, date, and limitations intact."},{"question":"Can an agent verify a scientific claim from the abstract alone?","answer":"Sometimes an abstract establishes a narrow statement about what the authors studied or reported, but it often omits eligibility criteria, denominators, conditions, and failure cases. For a consequential conclusion, inspect the full methods, results, and limitations."},{"question":"If several search results return the same paper, is that independent confirmation?","answer":"No. Several passages, repository copies, or index entries may resolve to the same DOI or arXiv identifier. Group them by underlying work and version before counting independent evidence."},{"question":"Does a highly cited paper have a stronger claim?","answer":"Citation counts measure indexed references to a work, not whether each citing paper agrees, replicates the result, or supports your particular statement. Read the original result and any material later correction separately."},{"question":"What if the full paper is paywalled or cannot be retrieved?","answer":"Look for an authorized open version or repository copy and check which version it represents. If only metadata or an abstract is accessible, label claims requiring the unavailable methods or results as unverified; do not invent supporting passages."},{"question":"Should an agent cite a preprint or the published journal version?","answer":"Use the version that contains the evidence being discussed and say which one it is. Check for journal publication, revisions, corrections, and changed conclusions; do not silently combine wording or dates from different versions."}],"ctas":[],"imageCredit":"Search for Agents","imageLicense":"All rights reserved","qualityScore":97,"url":"https://www.searchforagents.com/blog/does-the-cited-paper-actually-support-the-claim","body":"A paper citation can look decisive while supporting something much narrower than the sentence attached to it. A title matches the topic, the abstract sounds promising, and several scholarly indexes return the work. None of that yet answers the useful question: **what part of which version of the paper tested the claim, and what did the authors actually find?**\n\nConsider three statements that a research agent might write after finding a relevant paper:\n\n> “Half of AI citations are fabricated.”\n>\n> “Language models cannot use information in the middle of a long document.”\n>\n> “AlphaFold predicts every protein structure with experimental accuracy.”\n\nEach sentence points toward a real and important research paper. Each goes further than its source warrants. This article follows the work back through **[Semantic Scholar](https://www.semanticscholar.org/)**, **[Valyu](https://valyu.ai/)**, and **[OpenAlex](https://openalex.org/)**, then reads the underlying publications. The providers do different jobs here: one helps resolve a paper and its citation graph, one returns searchable paper passages, and one ties a work to a DOI and publication locations. **They are discovery routes, not three independent replications of any finding.**\n\n**Disclosure and method.** Search for Agents is produced by Valyu. On 25 September 2026, we looked up the three named works through the providers described below and checked the reported claims against the original publications. The provider lookups were different, task-specific operations; this is not a matched-query comparison of coverage, ranking, latency, or price. Search indexes and citation counts can change. We cite the paper version used for each check.\n\n## First, turn a headline into a testable claim\n\n“The paper supports it” can mean at least three different things. It may simply mention the topic. It may report a result related to the claim, but under narrower conditions. Or it may directly establish the statement as written. A reviewer should decide which of those is true before adding the citation to a sentence.\n\nWrite down the exact proposition before searching. Name the **subject**, **outcome**, **population or task**, **time**, and **strength of inference**. “A model sometimes misses a fact placed in the middle of a test context” can be investigated. “All models cannot read the middle” silently adds a universal quantifier. “Predicted structures were accurate on most targets in a blind assessment” is different from “all protein structures are solved.” The extra words are often where a citation stops supporting the sentence.\n\nNext, separate the bibliographic problem from the evidence problem. An arXiv identifier, DOI, or OpenAlex work ID is useful for finding the right work. A citation edge tells you that another work refers to it. An abstract previews the authors' account. None identifies a paragraph that establishes every proposed conclusion. Finding that paragraph requires opening the original and checking how the result was produced.\n\nThe table below is our working hypothesis before reading the three papers:\n\n| Claim to check | Work located | Question for the original paper |\n| --- | --- | --- |\n| “Half of AI citations are fabricated.” | The ALCE citation-evaluation study | Which outputs and dataset were measured, and what counted as *incomplete support*? |\n| “Models cannot use information in the middle.” | *Lost in the Middle* | Which models and tasks showed a position effect, and were there exceptions? |\n| “AlphaFold predicts every protein perfectly.” | The 2021 AlphaFold structure-prediction paper | Which structures were tested, what metric defined accuracy, and where did accuracy fall? |\n\n## Case one: citation support is not the same as a made-up reference\n\nWe used the [Semantic Scholar Academic Graph API](https://api.semanticscholar.org/api-docs/graph) to retrieve *Enabling Large Language Models to Generate Text with Citations* by its **arXiv identifier 2305.14627**. The returned record connected a Semantic Scholar paper ID to external identifiers including that arXiv ID and its arXiv DOI. It also offered a citation count and an open-access PDF link. Those fields made the paper easy to locate; the count did not tell us whether the headline claim was correct.\n\nThe [original ALCE paper](https://arxiv.org/abs/2305.14627), also available in the [EMNLP 2023 proceedings](https://aclanthology.org/2023.emnlp-main.398/), evaluates long-form answers with citations on three datasets. Its authors distinguish **fluency**, **answer correctness**, and **citation quality**. On their ELI5 question set, the abstract reports that even the best evaluated systems lacked *complete citation support* about **50% of the time**. The [revised full text](https://arxiv.org/pdf/2305.14627v2) explains that the relevant evaluated baselines produced generations not fully supported by their cited passages in roughly half of those cases.\n\nThat is not a count of fabricated paper titles. A citation may lead to a real page and still fail to support the sentence next to it. The paper distinguishes **citation recall**, which asks whether cited passages support a generated statement, from **citation precision**, which identifies irrelevant citations. Its automatic support assessment uses a natural-language inference model, and the authors also report human evaluations. In the paper's limitations, they acknowledge that an automatic evaluator can miss partial support and that the chosen datasets do not cover every harder task.\n\nThe denominator matters as much as the percentage. The researchers sampled **1,000 development examples per dataset**; ELI5 is a particular long-form question-answering setting, with its own retrieval corpus and tested systems. The defensible sentence is: **“In ALCE's ELI5 evaluation, even the best evaluated systems lacked complete support from their cited passages about half the time.”** The study does not establish a universal failure rate for every research assistant, all scientific citations, or every citation on the web. It certainly does not show that half of their linked papers were invented.\n\nThis is the first rule for a paper-search agent: report what was measured, **by whom**, and **on which set of cases**. Semantic Scholar's graph made identification fast. It could not convert an evaluation metric into a broader statement the researchers did not test.\n\n## Case two: five results can be one paper, and one date can be wrong\n\nFor *Lost in the Middle: How Language Models Use Long Contexts*, we ran a Valyu `paper` search for **“Lost in the Middle How Language Models Use Long Contexts Liu 2023”** and requested five results. All five returned the **same arXiv URL**, `arxiv.org/abs/2307.03172`, but surfaced different sections or chunks. Five results did not mean five studies supporting a conclusion. Grouping by paper identifier leaves **one** underlying work for this example. Different excerpts can help navigation without becoming independent evidence.\n\nThe observed Valyu result also carried a `publication_date` of **2023-01-01**. [arXiv's submission history](https://arxiv.org/abs/2307.03172) dates the first version to **6 July 2023**, followed by later revisions; we read [version 3](https://arxiv.org/pdf/2307.03172v3). We cannot infer from one response why the provider used January 1. It may reflect normalization of year-only metadata, but it should not be cited as the paper's actual submission day. Valyu's [academic-search guide](https://docs.valyu.ai/use-cases/academic) describes returned passages and scholarly metadata; an agent should still check dates and versions against the repository record. This is a dated observation, not a claim that every Valyu paper date is inaccurate.\n\nNow look at what the paper tested. The authors evaluated **multi-document question answering** and a controlled **key-value retrieval** task, moving the relevant information to different positions in an input context. They report that, for many of the models and settings studied, accuracy fell when the answer-bearing material appeared in the middle rather than near the beginning or end. That is a meaningful positional effect. It is not a demonstration that information in the middle can *never* be used.\n\nThe [full paper](https://arxiv.org/pdf/2307.03172v3) makes the counterexample plain: some evaluated models performed **perfectly or nearly perfectly** on its synthetic key-value task. The authors also examined query-aware contextualization and reported a setting in which it markedly improved key-value retrieval. Their multi-document experiment likewise fixes a particular question, set of passages, placement procedure, and collection of models. It does not test every paper-search agent, every document arrangement, or systems released years later.\n\nA supported rewrite would be: **“In the tested multi-document QA and key-value tasks, several evaluated models were less accurate when relevant information was placed in the middle of a long input.”** “All models ignore the middle” contradicts exceptions in the cited work. The original article's figures and experiment descriptions allow us to draw that boundary; the number of search chunks and a short excerpt do not.\n\n## Case three: a highly cited paper can still have explicit limits\n\nWe resolved the DOI **10.1038/s41586-021-03819-2** through [OpenAlex's Work API](https://api.openalex.org/works/https://doi.org/10.1038/s41586-021-03819-2). The record identifies *Highly accurate protein structure prediction with AlphaFold*, links to its publication location, and exposes scholarly metadata including a `cited_by_count`. This makes OpenAlex useful for finding and joining records. The citation count says the work is widely referenced in the index; it does not establish that every claim someone attaches to it has been replicated or endorsed.\n\nThe [2021 paper](https://doi.org/10.1038/s41586-021-03819-2) reports exceptional protein-structure predictions on the blind **CASP14** assessment. We inspected its [open full text at PubMed Central](https://pmc.ncbi.nlm.nih.gov/articles/PMC8371605/). OpenAlex labels that location a `submittedVersion` and the Nature DOI location a `publishedVersion`: they identify one work, but should not be assumed word-for-word identical. Fig. 1 of the text we inspected describes **87 protein domains** in the comparison. The abstract says predictions were competitive with experimental structures **in a majority of cases**. Those are substantial results. “Every protein perfectly” is a much larger statement, with a different denominator and no allowance for errors.\n\nThe limitations are not hidden. In the section on multiple-sequence-alignment depth and cross-chain contacts, the paper says accuracy drops substantially when median alignment depth is below **about 30 sequences**. It also says the evaluated model is much weaker for protein regions whose shape depends heavily on contacts with *other* chains in a complex. AlphaFold supplies a **per-residue confidence estimate**, but a confidence value is not an experimental structure determination or a claim that all biological contexts were tested.\n\nA supported summary is therefore: **“The studied version of AlphaFold achieved accuracy competitive with experimental structures in most cases assessed, while its paper documents important limits involving sequence information and cross-chain interactions.”** The precise finding belongs to the model and assessments in this publication. Later systems may address different tasks; their advances should be checked in their own papers, not silently read backward into the 2021 study.\n\nOpenAlex got us to a publisher DOI and an accessible text location. The qualification came from the open text's **results and limitations**, not from its title or indexed citation count. For an exact quotation or a disputed wording change, compare the publisher's version with the copy being cited. Two locations for the *same work* would not constitute two replications.\n\n## An evidence card that a second reader can challenge\n\nBefore publishing an answer, give every consequential sentence a compact, inspectable evidence card. It should contain enough detail for another reader to disagree intelligently rather than merely reopen a homepage:\n\n1. **Exact claim.** Preserve qualifiers such as “some,” “most,” “in this trial,” or “as of this date.” Separate the paper's finding from your proposed generalization.\n2. **Work and version.** Record DOI, arXiv ID, PMID, or another persistent identifier; distinguish a preprint revision from a publisher version or an authorized repository copy.\n3. **The actual support.** Name the section, figure, table, page, or short passage, with enough neighboring context to rule out a misleading crop. Record whether the material comes from an abstract, a methods section, original results, or somebody else's summary.\n4. **Study boundaries.** Keep the sample, experimental task, comparison group, outcome metric, time period, and known exceptions beside the claim. These determine which words the evidence can support.\n5. **Publication state.** Check for updated versions, later corrections, and whether an index merged distinct versions or returned multiple chunks of one work.\n6. **Verdict and uncertainty.** Mark the statement **directly supported**, **supported only with narrower wording**, **contradicted by the source**, or **not verified from available text**. Add the reason; “unverified” is not a synonym for “false.”\n\nThe verdict need not be made by a model alone. An agent can propose the card and quote its candidate passages. A researcher should be able to inspect the linked paper and see exactly how the conclusion follows. A matching title, citation graph edge, or confident synthesis provides an input to that review, not its outcome.\n\nIn these three checks, the narrower wording is the difference between a useful result and an attractive error. The ALCE percentage stays attached to its benchmark and definition of support. The long-context finding stays attached to the evaluated tasks and exceptions. AlphaFold's achievement stays attached to the structures it assessed and the situations where it reported weaker performance. Every paper remains important after those qualifications; none needs an inflated headline to matter.\n\n## How the three search systems fit the workflow\n\nThe providers did not need to return the same objects to be useful. [Semantic Scholar's paper record](https://api.semanticscholar.org/api-docs/graph) connected an arXiv ID, DOI, PDF location, and citation graph for ALCE. Valyu surfaced passages of *Lost in the Middle*; the repeated source URL and questionable date required verification at arXiv. [OpenAlex](https://api.openalex.org/works/https://doi.org/10.1038/s41586-021-03819-2) resolved the AlphaFold DOI into a work record and publication locations; the findings still required the full paper.\n\nThat is a comparison of **roles**, not a provider leaderboard. A paper index is good at finding and connecting works; an extracted passage speeds navigation; a DOI lookup reduces identity mistakes. None of those fields tells you whether a sentence outruns the methods. An agent that uses several systems should deduplicate by underlying work and version, retain which provider found each lead, and keep the original publication as the authority for what it reports.\n\nIf full text cannot be reached, stop at what the accessible abstract or metadata actually says. A missing methods section is a reason to mark a claim unverified, not to invent a result or to infer that the opposite is true. The [industry-search source checks](/blog/how-to-verify-search-results-across-finance-biomedical-and-law) apply the same distinction to filings, treatment approvals, and opinions; the [Web Search field note](/blog/when-can-an-agent-trust-a-web-search-result) shows how a plausible result can hide an outdated factual count.\n\n*Research note: Provider lookups and source checks were conducted on 25 September 2026. Semantic Scholar was queried by arXiv ID, Valyu by a title-and-author paper query, and OpenAlex by the published DOI. The three claims above are deliberately tested against the linked originals, not offered as quotations from a provider's generated answer. Search results, citation counts, and repository versions may change; repeat the checks before using them in a current review.*","quality":{"score":97,"threshold":85,"ready":true,"checks":[{"id":"direct-answer","label":"Direct answer","status":"pass","points":12,"maxPoints":12,"evidence":"385 characters"},{"id":"target-question","label":"Target question","status":"pass","points":8,"maxPoints":8,"evidence":"Written as an explicit question"},{"id":"key-takeaways","label":"Key takeaways","status":"pass","points":8,"maxPoints":8,"evidence":"5 takeaways"},{"id":"source-depth","label":"Source depth","status":"pass","points":14,"maxPoints":14,"evidence":"10 sources"},{"id":"source-dates","label":"Source dates","status":"warn","points":3,"maxPoints":6,"evidence":"5 of 10 sources include publication dates"},{"id":"section-depth","label":"Section depth","status":"pass","points":12,"maxPoints":12,"evidence":"6 H2 sections"},{"id":"heading-integrity","label":"Heading integrity","status":"pass","points":8,"maxPoints":8,"evidence":"No body H1 or skipped heading levels"},{"id":"body-substance","label":"Body substance","status":"pass","points":10,"maxPoints":10,"evidence":"2371 words"},{"id":"internal-links","label":"Internal links","status":"pass","points":6,"maxPoints":6,"evidence":"Includes a site-relative internal link"},{"id":"date-integrity","label":"Date integrity","status":"pass","points":8,"maxPoints":8,"evidence":"2026-09-25 published; 2026-09-25 updated"},{"id":"image-alt","label":"Image description","status":"pass","points":4,"maxPoints":4,"evidence":"88 characters"},{"id":"author-context","label":"Author context","status":"pass","points":4,"maxPoints":4,"evidence":"Eliot Reed - Research writer"}],"recommendations":["5 of 10 sources include publication dates"]},"provenance":{"version":"sfa.provenance.v1","sourcePath":"content/posts/does-the-cited-paper-actually-support-the-claim.md","canonicalMarkdownHash":"63c95734c7a7fa2b0134b09179906fc419313542fdb245b8e8926085bc5fb36e","contentHash":"968ab0c0ae0202c5cb33acc903ce384ceaef334f7948c298f945525d2905a249","schemaVersion":"post.v1","commit":"86eeb0c2cfe77a7d66370be2b282a2e8c1f0c055"},"sourceBundleUrl":"https://www.searchforagents.com/api/v1/content/post_paper_search_claim_support/source-bundle","representations":{"html":"https://www.searchforagents.com/blog/does-the-cited-paper-actually-support-the-claim","markdown":"https://www.searchforagents.com/blog/does-the-cited-paper-actually-support-the-claim.md","json":"https://www.searchforagents.com/blog/does-the-cited-paper-actually-support-the-claim.json"},"alternates":{"en":"https://www.searchforagents.com/blog/does-the-cited-paper-actually-support-the-claim","x-default":"https://www.searchforagents.com/blog/does-the-cited-paper-actually-support-the-claim"}}