Quick answer: AI detector scores are usually measured on pristine files straight from the generator. In one 2026 benchmark of 14 detectors, two that scored a perfect 1.000 fell to 0.243 and 0.340, below random, once real and fake images were saved through one encoder. They had been separating files by format, not faces.
You paste a suspicious TikTok into a detector. It returns a number: 96 percent real, or 88 percent likely AI. That number feels like a verdict.
It is not. The number was earned on a file that no longer exists by the time the video reaches your screen.
Here is the problem: most published detector accuracy is measured on clean files taken straight from the generator. The video you are actually checking has been uploaded, compressed, resized, re-encoded, and served back to you by a platform pipeline. Those are different tests. A detector can ace the first one and fail the second, and the published number tells you nothing about which.
A benchmark released in 2026 put a hard number on how bad that gap gets. It measured still images, and the mechanism it exposed is the one that runs on every video you upload.
The Detector That Was Reading the File Format
Researchers at Margen, a deepfake-detection research company run by Ledger’s founder, evaluated 14 open-source deepfake detectors against roughly 26,500 face images, balanced across 12 demographic groups. Two of those detectors scored an AUC of 1.000.
AUC is a score between 0 and 1 that measures how well a model separates real from fake. A 1.000 is perfect separation. Every real image ranked above every fake one, with no mistakes.
A perfect score on a brand new test set should not be celebrated. It should be investigated. Either the test is too easy, or the model found a shortcut.
It was a shortcut, and the shortcut is the interesting part.
In that first round, the real images and the AI-generated images had entered the corpus through two different pieces of software. The real ones were saved with OpenCV. The synthetic ones were saved with PIL. Both produced JPEG files, but the two libraries leave slightly different byte patterns behind, the way two printers leave different ink signatures on identical text.
The difference was small enough to be invisible and consistent enough to be decisive. The real images were saved at JPEG quality 95 with the color channels in one order. The synthetic ones were saved at quality 92 with the channels in the other. No human looking at the images could see either difference. A model looking at raw bytes could separate the two sets perfectly without ever examining a face.
The detectors had learned to read the printer, not the page.
When the researchers re-saved every image through one pipeline, so real and fake shared an identical encoder, both perfect detectors collapsed. DMimageDetection fell to 0.243 and the ClipBased fusion model to 0.340.
1.000 to 0.243
The drop for the best-scoring detector once real and synthetic images were saved through a single encoder. The second fell to 0.340. A score of 0.5 is a coin flip. Below that, the models were not guessing, they were systematically wrong.
Source: Margen, Deepfake Detector Robustness Under Social-Media Re-encoding, 2026
The researchers ran a control to make sure the collapse was real. They passed the original files through the new test harness without re-encoding anything. Those results matched the first round to four decimal places. Nothing about the testing had changed. The only variable was the file format, and the file format was carrying the entire score.
On that disclosure: Margen is not an independent source on Ledger’s own argument. It evaluates detectors it does not build or sell, and the study is published openly with a permanent DOI, so the corpus, the pipeline and the statistics can be checked rather than taken on trust.
Why This Hits Social Video Hardest
Every platform re-encodes what you upload. X, Instagram, Facebook, and TikTok all run uploads through their own compression to save bandwidth and standardize playback, which is why free detection tools struggle on platform video in a way they do not on a file you hand them directly. The file that leaves your phone is not the file that lands in someone else’s feed.
Be careful about what the study does and does not show here, because the obvious conclusion is not the one it supports. The researchers emulated those four platform pipelines, and under them the two detectors scored around 0.70 rather than collapsing. Those pipelines started from the original files, so the format difference survived and the detectors could still lean on it.
What the study proves is narrower and more useful: a headline accuracy number can be measuring the file rather than the face, and you cannot tell which from the number alone. A real platform re-encodes everything it receives through one pipeline, which is the condition that exposed the shortcut in the lab. Whether that happens end to end in production is untested.
It gets compounded on the way. A clip gets screen-recorded off one app and re-uploaded to another. It gets cropped for a different aspect ratio. It gets run through a filter. Each pass strips more of the fine-grained signal that frame-level detectors depend on, and each pass pushes the file further from anything the model was tested against.
One honest limit on the study: it measured still images, not video. The closest video evidence comes from Deepfake-Eval-2024, which found open-source video detectors lost 50 percent of their AUC on real circulating deepfakes. What transfers from the format study is the mechanism, not the specific number.
The related finding is that this is not universal. The other six detectors carried into the second round lost 0.047 AUC on average, with the worst at 0.119. They degraded. They did not collapse. The lesson is not that detection is worthless, it is that a headline accuracy number cannot tell you which kind of detector you are holding.
Two further limits: the fakes came from a 2023-era generator, and all 14 detectors were open-source research models rather than the consumer tools most readers reach for.
The Average Also Hides the Worst Case
There is a second problem with a single accuracy number, and it survives even when the format issue does not.
The same benchmark broke every result out across 12 groups, six skin tones by two genders, instead of reporting one pooled figure. A detector can post a respectable average while failing badly on one of those groups, because a strong majority drags the mean up and the failing minority disappears into it.
That matters to you specifically. Your feed is not a balanced test set. If the person in the video you are checking falls into a group a given detector handles poorly, the published average describes someone else's experience, not yours.
The researchers note their own limit here: skin-tone labels are imperfect, so per-group results read as directional rather than precise. The direction is the point. A number that has not been broken out by group is hiding something, and you cannot tell from the outside whether it is hiding a lot or a little.
What a High Score Actually Tells You
Three things are worth separating, because marketing copy blends them.
A benchmark score is a measurement under stated conditions. It describes performance on a specific corpus, processed a specific way. It is not a property of the model that travels with it into your feed.
Accuracy on generated images is not accuracy on your video. The attacks in a benchmark come from the generators the researchers chose. Fraudsters use whatever shipped last month. A model that has never seen a given generator has no particular reason to catch it.
A number with no conditions attached is a marketing claim. A tool advertises 98 percent accuracy. Ask what it was measured on, what the media had been through, and how it scored on the hardest subgroup. Without those, the number is decoration.
This connects to a pattern covered in why humans still beat AI at spotting deepfake video. Detection models trained on static frames miss the temporal signals, the blink rhythm and the lip sync drift, that a person catches without trying. Compression makes that worse, because it degrades exactly the frame-level detail those models lean on while leaving the motion inconsistencies a human notices largely intact.
Why the Gap Persists
There is a structural reason lab numbers stay high while real-world performance stays uncertain.
Benchmarks are built to be clean. Researchers control the corpus so results are reproducible, and reproducibility is a genuine scientific virtue. The side effect is that the controlled version of the problem is easier than the deployed one, and the published number describes the easier version.
Independent work keeps finding the same gap from the other direction. Deepfake-Eval-2024, a study of detectors run against deepfakes collected in the wild, found open-source detectors lost 50 percent of their AUC on video, 48 percent on audio, and 45 percent on images compared with their published benchmark figures.
Two studies, two methods, same conclusion: the number in the paper is the ceiling, not the expectation.
The fix on the research side is not complicated. Test with the real and fake media processed identically, so no format shortcut exists. Re-run the test under the compression the target platform actually applies. Report the worst-performing subgroup rather than the average. None of that is exotic, and a detector that publishes results this way is telling you something a single accuracy figure cannot.
The Practical Version
None of this means you should stop using detection tools. It means you should read their output as one signal among several.
Treat the score as a prior, not a verdict. A high AI probability is a reason to look harder. A low one is not a clearance. The tool has told you how the file looks to a model, which is a genuinely useful thing to know and a long way from proof.
Check what the tool says about conditions. The better ones publish what they were tested on and under what processing. Most publish a single number. That difference tells you something about the tool before you ever run a video through it.
Look at the things compression does not destroy. Motion, physics, and continuity survive re-encoding. Hands that change shape between frames, a mouth that drifts out of sync, jewelry that moves on its own, a background that warps when the subject turns. The visual tells that give away an AI face on video hold up on a compressed clip in a way that pixel-level artifacts do not.
Get more than one observation. A single verdict, from a model or from you, is a single sample. This is the whole premise behind three ways to catch a deepfake in 2026: independent observers flagging the same clip compound into a stronger signal than any one of them produces alone.
If you want the underlying grounding before any of this, what a deepfake actually is covers how these videos get made, which explains why certain artifacts survive compression and others vanish.
Think you found an AI video?
Paste the URL and let the Ledger community verify it. Free.
What To Take From This
A perfect detector score can mean the model solved the problem. It can also mean the model found a shortcut nobody checked for. From the outside, on a single number, those look identical.
That is the argument for treating detection as layered rather than automatic. Run the tool, because it catches things you will miss. Watch the clip for the signals compression cannot erase. Check whether the account has a history that makes sense. Then weigh those together instead of deferring to whichever number arrived first.
The video in your feed has been through a pipeline before it reached you. Any honest read of it starts there.
[APP-DOWNLOAD]
Related Posts
- How to Tell If a TikTok Video Is AI-Generated: 7 Signs to Check Right Now: the platform-specific signals that survive compression, in the order worth checking them
- What Is a Deepfake? A Plain-English Guide for Social Media Users: how these videos get made, which explains why some artifacts survive re-encoding and others do not
- Humans Are Still Better Than AI at Spotting Deepfake Videos: the temporal signals automated detection keeps missing on video
- The Best Free Deepfake Detectors for TikTok and Instagram in 2026: what the free tools actually do well, and where their published numbers stop being useful
