AI detectors don't work: what teachers and managers should do instead

The short version

Software that claims to tell whether a piece of writing was produced by AI is guessing from statistical patterns. It has no access to who typed the words, it flags honest writers regularly, it flags non-native English writers more often, and it can be defeated by anyone willing to spend five minutes editing. That combination makes it unfit as evidence for an accusation that can end a student's semester or an employee's job.

This is not an argument that misuse of AI does not matter. It does. It is an argument that detectors are the wrong tool, and that the right tools, which are older and less convenient, actually work: assess the process, talk to the person, and write down what is allowed.

How detectors work, and why that is the problem

What they measure

An AI-writing detector reads text and estimates how predictable it is. Language models generate text by choosing likely next words, so their output tends to be smooth: common word choices, even sentence lengths, few surprises. Detectors score text on that smoothness (some vendors use terms like perplexity and burstiness) and sometimes on patterns learned from a training set of human and machine samples. A high score means "this reads like the kind of text a model tends to produce."

Notice what is missing from that description: any knowledge of how the text was made. The detector cannot see a draft history, a person, or a tool. It sees word statistics. Clear, formal, well-organized prose is exactly what a model produces, and it is also exactly what schools and style guides have spent a century teaching people to produce.

The false-positive problem

A false positive is an honest writer flagged as a machine. Because detectors measure style rather than origin, false positives are not a bug that a better version will fix. They are built into the method. Any writer whose natural style is plain and predictable will score high. The US Constitution has famously been flagged as likely AI-written by a detector, which tells you what the score is measuring.

The consequences fall unevenly. A detector run on a class of two hundred students, even one that is wrong only occasionally, produces a handful of wrongly accused people every term, and each of them has to prove a negative.

Who gets flagged most

Researchers at Stanford (Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou, published in the journal Patterns in 2023) tested several popular detectors on essays written by non-native English speakers for the TOEFL exam and on essays by native-speaking US students. The detectors flagged a large share of the non-native essays as AI-generated while rarely flagging the native ones. The likely reason is the mechanism above: writers working in a second language tend to use more common vocabulary and more regular sentence structures, which is precisely what detectors read as machine-like.

The same pattern hits other groups: people who learned to write from templates, technical and scientific writers, anyone who writes in a formal register, people with learning differences who rely on structured writing, and people who use grammar tools, which smooth text in the same direction.

Easy to evade

Anyone who actually wants to hide AI use can. Light paraphrasing, asking the model to write in a rougher or more varied style, mixing machine and human sentences, running the text through a so-called humanizer tool, or translating it out of English and back will drop most scores. So the detector is most likely to catch the honest student who used no AI and least likely to catch the student who used it deliberately and carefully.

Watermarks and provenance

Some vendors have developed ways to watermark their own models' text output, for example Google's published SynthID approach. A watermark can be useful for identifying text from one specific model, unedited, checked with that vendor's own tool. It does nothing for text from other models, text that has been edited or paraphrased, or text from the many tools that do not watermark at all. Provenance is a real research direction, and it is not a general-purpose detector.

Take vendors at their word here. Most detector vendors say in their own documentation that a score should not be the sole basis for an accusation. OpenAI released a classifier for detecting text from its own models in early 2023 and withdrew it within months, saying its accuracy was too low. That was the vendor best placed to build one.

What a score can and cannot tell you

A detector score can tell you that a piece of writing is stylistically smooth. It cannot tell you who wrote it, whether a tool was used, how much, or whether that use was allowed. At most, a score is a reason to look more closely, in the same way an unusually polished essay from a struggling student was a reason to look more closely long before AI existed.

If your institution or company requires you to run one, treat the output as a prompt for a conversation, never as a finding. Do not put a percentage in an accusation letter. Do not tell a student or employee "the software says." The software does not know.

Worked example: the 92 percent essay

Say an instructor teaching a first-year writing course runs a submitted essay through a detector and it returns 92 percent "likely AI." The student is an international student whose earlier work was competent, formal, and heavily structured.

The wrong move is an academic-integrity referral based on the score. The right move takes fifteen minutes. The instructor opens the document's version history (most cloud editors keep one) and sees the essay built up over four sessions across three days, with paragraphs rewritten and moved. She asks the student to come by office hours and talk through the argument: why they chose this thesis, which source was hardest to use, what they would change. The student answers fluently and in detail, in the same voice as the essay. The instructor closes the file. The score was measuring the student's careful, textbook English, nothing more.

Now change the facts. The version history shows the entire essay pasted in at once, twenty minutes before the deadline, and in office hours the student cannot explain the argument or name the sources. That is evidence: not the score, but the process and the conversation. The instructor can act on it under whatever policy the course has published.

What teachers should do instead

Assess the process, not the product alone

Require artifacts that only the writing process produces: an outline, an annotated bibliography, a rough draft with a short note on what changed, a final draft with a paragraph of reflection. Where your platform allows, ask students to write in a document with version history turned on. These are old practices, and they work because they make the product hard to fake without doing the work.

Talk to students

A five-minute conversation about a piece of writing tells you more than any score. Build short oral follow-ups into major assignments: three questions about choices, sources, and revisions. Students who did the work find these easy. Students who did not find them impossible. Beyond enforcement, this is simply good teaching.

Design assignments that need the student's own material

Tie writing to class discussion, local observation, personal experience, an in-class text, or data the student collected. A model cannot write convincingly about what happened in Tuesday's lab section. Some in-class writing, by hand or on locked-down devices, gives you a baseline of each student's voice for comparison.

Put the norm in the syllabus

Say what is allowed, per assignment if necessary, and say how to disclose it. Ambiguity creates most cases. A student who was told "AI for brainstorming and grammar is fine, for drafting it is not, and tell me what you used in a note at the end" knows where the line is. A student who was told nothing will draw the line wherever is convenient.

What managers should do instead

Decide what AI-assisted means on your team

Most workplaces have not said whether using an assistant to draft an email, a report, or a proposal is acceptable, so people guess, and then feel accused when someone notices. Write the norm down. For most knowledge work the sensible default is that AI assistance is allowed, the person remains accountable for every word, and confidential data goes only into approved tools.

Judge the work by outcome and accountability

The question that matters is whether the work is correct, appropriate, and owned. If a report is accurate, meets the brief, and the author can defend every claim in it, how the first draft was produced is a process detail. If it contains a fabricated statistic, the problem is the fabrication, not the tool. Hold people to the output and to their ability to explain it.

Hiring: stop scoring cover letters

Running applications through a detector is the worst use case of all: high stakes, no conversation, no chance for the applicant to respond, and the people most affected are non-native speakers and careful writers. Assume cover letters are AI-assisted and weight them accordingly. Put the signal in the interview, a short live work sample, or a take-home task followed by a conversation about the choices made in it.

Setting expectations: a three-tier disclosure norm

A norm that fits on an index card works better than a policy nobody reads. Use three tiers and label each assignment or task with one.

• Tier 1, no AI. For work meant to build or demonstrate a skill from scratch: a timed essay, a certification exercise, a piece of writing that stands in for the person. State the reason, so it does not feel arbitrary.
• Tier 2, AI-assisted, disclosed. The default for most work. Brainstorming, outlining, grammar, feedback on a draft, and rewriting for tone are fine. The person writes one line at the end saying what they used and for what.
• Tier 3, AI-generated, reviewed. For routine, templated output: meeting summaries, first drafts of boilerplate, reformatting. The person is responsible for checking it and says so.

Disclosure only works if it is safe. If the first disclosed use is punished, nobody will disclose again.

A prompt that drafts your disclosure norm

Act as an experienced [DEPARTMENT CHAIR OR TEAM LEAD] who writes clear, fair policies that people actually follow. Draft an AI-use and disclosure norm for [A COURSE OR A TEAM], one page or less.

Context:
- Who it covers: [FOR EXAMPLE STUDENTS IN A FIRST-YEAR WRITING COURSE, OR A SIX-PERSON MARKETING TEAM]
- The kinds of work they produce: [LIST THE MAIN ASSIGNMENTS OR DELIVERABLES]
- Work that must be done without AI, and why: [LIST, OR SAY NONE]
- Tools that are approved, and any data that must never go into them: [LIST]
- What happens when someone is unsure: [WHO TO ASK]

Structure it as three tiers: no AI, AI-assisted with a one-line disclosure, and AI-generated with human review. For each tier give two concrete examples drawn from the work listed above. Add a short section on how concerns about undisclosed use will be handled, which must rely on process evidence and a conversation with the person, never on a detector score alone. Plain language, no threats, under 400 words.

Before drafting, ask me up to four questions if anything above is unclear.

If you are the one accused

Stay calm and gather process evidence: version history, drafts, notes, browser history from your research, messages to classmates or colleagues about the work, anything with a timestamp. Ask what the accusation is based on. If the answer is a detector score, ask, politely and in writing, for the policy that permits a score to be used as evidence and for the detector vendor's own guidance on that point. Offer to discuss the work in person and to write something comparable under observation. Most institutions have an appeals process; use it. Most managers, faced with a calm person who can explain their work in detail, drop the matter.

Next steps

• Understand what the tools are and are not doing in How large language models work, plus the glossary entries for AI detector and AI watermarking.
• Write the norm down with How to write a team AI policy and the AI policy generator.
• Put your attention on what matters in the output with How to fact-check AI output.
• Role playbooks: AI for teachers, AI for professors, AI for school administrators, and AI for HR professionals.