How triff works — methodology
The primary source is the ground truth.
triff is built on a simple principle: when a claim can be checked against an authoritative primary source, it should be. Every time. The same way. With the evidence shown.
The problem isn't LLMs. It's when the LLM is the oracle.
When you ask an LLM to fact-check a claim, you're asking it to be the source of truth. Even with retrieval, grounding, and carefully engineered prompts, the system remains fundamentally probabilistic. It generates plausible-sounding answers based on patterns in its training data and retrieved context.
Run the same claim through an LLM fact-checker today. Run it again tomorrow. Run it ten times with identical inputs. You'll often get variations — some confident, some hedged, some contradictory. The output is not deterministic. Which version do you publish? Which one do you defend when challenged? Which one holds up in an editorial review?
Serious knowledge work requires repeatability. A primary source record is deterministic: the BLS employment report for March 2025 says what it says. It will say the same thing tomorrow, and the day after, and when your editor checks it, and when your legal team checks it. An LLM-based fact-check, by contrast, is a probabilistic estimate of what the answer probably is — generated by a system whose core optimization objective is producing plausible text, not accurate text.
Plausible and accurate overlap often enough to be seductive. They diverge often enough to be dangerous.
Primary sources as ground truth. Not the model.
triff is built on a different architecture. The primary source — the official government database, the regulatory filing, the authoritative record — determines what the primary source says. Not the model.
Where AI reasoning appears in the triff pipeline, its role is tightly constrained: working with a specific evidence package, on a specific task, with the primary source data already retrieved and structured. The model doesn't decide what the evidence says. The evidence is the evidence.
That constraint is the difference between a system you can trust and a system you can hope works.
To be clear: this is not a case against LLMs. triff would not exist without them. They are essential to how the pipeline works — understanding language, identifying claims, reasoning over evidence. The question is never whether to use them; it's knowing precisely when and where. An LLM used as an oracle is a liability. An LLM given a specific task, a structured evidence package, and clear guardrails is a powerful tool. triff is built on that distinction.
Every triff finding includes:
- The primary source consulted
- The data retrieved from it
- The comparison between that data and the claim in your content
- The full reasoning used to reach the finding
You don't have to take triff's word for it. The references are there so you can verify the finding yourself. If something looks wrong, it may be — triff is built on AI, and AI makes mistakes. The references exist precisely so you can catch them.
What counts as a primary source.
It's worth pausing on what “primary source” actually means — because the term gets used loosely, and triff's entire approach depends on the distinction.
Sources exist in a hierarchy:
- Primary sources are original records. The government statistical release itself. The court filing. The central bank statement. The company's 10-K. The peer-reviewed study. The data before anyone interpreted it.
- Secondary sources are accounts of primary sources. A news article reporting on the jobs numbers. An analyst note summarizing the earnings report. A textbook chapter describing the legal decision. Secondary sources can be accurate and valuable — but they are one step removed, and errors compound at each step.
- Tertiary sources are accounts of accounts. An article citing another article. A summary of a summary. By this point the chain of custody is long and the opportunities for error are wide.
Most of what people read — including most journalism — is secondary or tertiary. A reporter cites a statistic they found in a press release, which summarized a data release, which itself involved methodological choices made upstream. At each step, something can be lost, rounded, or misframed.
triff goes back to the primary. Not the news article about the data — the data itself, from the organization that produced it. The distinction is not about the form a source takes. A press release from a company reporting its own earnings is a primary source for those earnings. An established data-collecting body reporting its own survey results is a primary source for those results. What matters is whether the source is the authoritative originator of the information — or someone else's account of it.
In practice, this means:
- Official government data — labor statistics, economic indicators, inflation measures, trade figures, census data
- Regulatory and legal records — court filings, regulatory decisions, official announcements
- Financial filings — earnings reports, SEC filings, central bank statements
- Institutional records — official publications from recognized scientific, academic, and international bodies
What this rules out: other news articles, Wikipedia, analyst summaries, social media, and anything that is itself a secondary account of a primary source. Corroboration from secondary sources is not verification. If the original was wrong, everything citing it is wrong too.
triff's source coverage is not unlimited. Where no primary source is available — because coverage doesn't extend to a particular geography, sector, or time period, or because the relevant data simply isn't publicly accessible — triff returns an unverified finding and notes what would be needed to complete the comparison. Honest gaps are better than confident guesses.
What triff returns.
For every claim it examines, triff returns one of five findings. These are not verdicts about whether the author was right or wrong. They are findings about the relationship between the claim and what the primary sources record.
One of triff's greatest strengths is its ability to know what it knows — and, just as importantly, what it doesn't. An Unverified finding is not a failure of the system. It is the system successfully identifying the boundaries of what can be known from the available data. A Conflicting finding is not an error — it is an honest report that two legitimate primary sources disagree. These outcomes are valuable pieces of insight in their own right. Most LLM-based systems are optimized to always produce an answer. triff is optimized to tell you when it cannot.
/ Verified — Primary source data matches this claim. The source says what the content says.
\ Delta — Primary source data shows a different figure. The source and the content diverge. This does not mean the content is wrong — the author may have more current data, a different methodology, or a source that postdates the last official release. The finding surfaces the difference. The author decides what to do with it.
~ Conflicting — Primary sources do not agree with each other. Two or more authoritative sources record different figures for the same fact. This is not a reflection on the content — it is a signal that the underlying data landscape is genuinely contested. The content may be accurately citing one legitimate source while another legitimate source says something different.
? Under-specified — Primary sources are sufficient and clear, but the claim is too loose to render a single comparison. Different reasonable interpretations would yield different markers. The author should tighten the claim.
- Unverified — triff was not able to complete a primary source comparison. This can happen for many reasons: the relevant source exists but could not be retrieved; no established primary source is known for this type of claim; the claim is forward-looking or analytical and not directly comparable to recorded data; the data series does not cover the relevant time period, geography, or sector. Where possible, triff notes what a primary source comparison would require.
The triff string — a compact summary of findings in fixed order / \ ~ ? - — gives you the shape of the run at a glance. ////\\~~??- means four verified, two delta, two conflicting, two under-specified, and one unverified. The distribution tells you something before you read a single finding.
What triff is not.
triff does not declare claims right or wrong.
A delta finding is not a correction. A verified finding is not an endorsement. The author always has context triff doesn't — a source interview, a pre-release figure, domain expertise, a deliberate editorial choice. Every finding is an invitation to compare, not a verdict.
triff does not check every claim.
triff identifies the load-bearing factual claims in your content — the ones that can be meaningfully compared against a primary source. Subjective statements, opinions, analysis, and claims without a clear primary source record are noted but not adjudicated.
triff does not replace editorial judgment.
The research is triff's job. The judgment is yours. triff surfaces what the primary sources record. What to do with that information — correct, contextualize, stand by, or investigate further — is always the author's or editor's decision.
triff is not perfectly repeatable.
Run the same content through triff twice and you may get slightly different findings. Unlike LLM-based fact-checkers — where variance comes from the probabilistic nature of the model itself — triff's variance is almost entirely driven by source retrieval: which sources were found, which were accessible, which returned data at that moment. When a triff result varies, there is a traceable reason. That variance is explainable and, over time, improvable in a way that model-level variance is not.
triff is not infallible.
triff is built on AI, and AI makes mistakes. A source may have been misread, a figure misattributed, or a comparison drawn from the wrong data series. Every finding includes the full reference so you can verify it yourself. If something looks wrong, check it.
How we talk about findings.
Language matters when you're in the business of accuracy. triff's findings use precise, non-judgmental language — because a tool that presumes to correct journalists should be especially careful about what it presumes.
You'll notice triff uses words like surfaces, records, notes, and flags — not corrects, catches, or identifies errors. A delta finding says “the primary source records a different figure.” It does not say “this is wrong.” That distinction is not hedging. It is the honest framing of what triff actually knows.
What triff knows is what the primary source says. What that means for your content is yours to decide.
We're early, and improving fast.
triff is in beta. The pipeline is built on a scalable cloud platform, but coverage is not complete, runs are not always fast, and findings are not always right. We are improving the system continuously.
The system learns from every analysis it runs. Each verification strengthens triff's understanding of claim types, source patterns, and the boundaries of what can and cannot be verified. The more triff runs, the more precisely it works. Early users are not just getting access to the tool — they are helping make it smarter.
Source coverage expands with every release. Claim types that cannot be verified today will be verifiable in future versions. Findings that require manual follow-up will increasingly be automated.
If you encounter a finding that looks wrong, a source that doesn't match, or a claim type that triff handles poorly — tell us. That feedback is the most direct input we have into what to build next.