Signs of AI writing: a Vale ruleset

My minor superpower is setting up change detection on websites to get email notifications when they update. For the past three months that’s meant daily pings from Wikipedia’s “Signs of AI Writing” page. As an “advice” page, it’s where Wikipedia editors document the tells: the phrases, patterns and artifacts that suggest that AI was involved at some point with what they’re reading.

Malicious or subversive edits to Wikipedia have been a challenge since the formation of the site. But the community has a robust nervous system for identifying and rejecting those that compromise the integrity of the encyclopedia.¹ The latest response is this painstakingly, exhaustively updated page documenting various signs.

The proliferation of LLM tools means that AI-tainted edits are a new variant for Wikipedia contributors to navigate. If AI tools allow individuals to contribute productively in ways otherwise not possible, they could be welcome. But as the page goes to great lengths to document, there’s a fresh category of edit slop that would deteriorate the quality of the encyclopedia if allowed unchecked.

The emails summarising the edits give me an interesting perspective on how perception of AI use has morphed rapidly over time. The page documents the front line of AI detection in a real-world setting.

What would it look like to turn this into tooling?

They see the signs

Many tells are obvious. Take the glitches that would appear in older versions of ChatGPT outputs.

For example,

:contentReference[oaicite:0]{index=0}

or

[oaicite:5]

Once you understand that this is a strange bit of text blurted out by the model it’s an immediate sign that the surrounding content has been tainted by a model at some point in its history.

These examples are temporary windows into the textual guts of AI models. The MissingNo. seahorses of ChatGPT.

Now documented, recognised, and patched by the model creators, these artifacts of artificiality are becoming extinct with time.

Nudging back into the world of user error, it’s common to see phrasing that’s been accidentally copy-pasted from the LLM user interface and surrounding text.

As an AI language model

Would you like it expanded

You're absolutely right

This can include instructional framing meant not for the reader but the user.

Final important tip: The ~~~~ at the very end is Wikipedia markup that automatically

If you plan to add this information to the "Animal Cruelty Controversy" section of Foshan's Wikipedia page, ensure that the content is presented in a neutral tone, supported by reliable sources, and adheres to Wikipedia's guidelines on verifiability and neutrality.

Other tells become more obvious the more you read online and the more you train yourself to spot them.

The most famous example of this is ChatGPT’s rampant use of the em-dash, raising the hackles of belligerent punctuation lovers worldwide.

To me the bigger scourge is that of contrastive language:

It isn’t just X, it’s Y.

It's not just about the beat riding under the vocals; it's part of the aggression and atmosphere.

This language construction is catnip to the LLMs at large. It occasionally spans multiple sentences but it almost always appears in marketing and branding content on LinkedIn due to the goal of making an impact.

As a result this construction is hackneyed and many will choose to avoid it in order to prevent being lumped in with the undiscerning, careless masses.

The wind beneath my vale

Technical writers face similar challenges with AI-generated content outside of Wikipedia. Like any group that cares about craft, they’ve built tools for systematic text analysis.

Vale.sh is a library known as a “linter for prose.” It could be thought of as a souped-up spelling and grammar checker.

Linting is a term borrowed from software development (first used back in 1978!) and is the process of highlighting areas for improvement, based on a customisable ruleset. In programming, linters catch “code smells” – implementations that technically work but suggest deeper issues. Many programming languages have syntactic variation meaning that you may write a piece of code differently to your peers. These degrees of freedom can hamper collaboration, or permit confusion, by making it hard to understand the intent behind the code.

Sharing these linting rules across teams of developers aids collaboration and goes a long way to avoid holy wars about semicolons.

Vale has been around in the technical writing community since 2017 and there’s a strong ecosystem of rulesets that one can “opt-in” to. These include style guides, checks for passive voice, gendered or condescending language.

I’m not aware of any existing implementation of a ruleset to help highlight AI smells, so I built one.

From your lips to YAML

I fed the contents of the Wikipedia page to Claude and asked it to make recommendations on the rules that should be generated.² After a few confidence checks I gave it the green light to generate Pull Requests on GitHub for manual review and verification.

One key part of this process was separating the rules that were relevant for a collaborative encyclopedia and those that had wider application in other forms of writing.

An example of this is the meta-commentary from earlier that reference writing for Wikipedia in the body of the text itself. There’s likely a version of this ruleset that could be tuned for exclusive use for Wikipedia edits but my goal was to provide a general purpose tool with applications elsewhere.

AI vs. AI

It’s hard to understand the true ability of the countless AI detection tools on the market. Many appear to be taking advantage of the widespread use of ChatGPT in educational settings to prey on students trying to evade (flawed) AI detection tools.

Vendors at various points of the snake oil spectrum are touting the strengths of their products but there’s little neutral and independently verifiable research to back up their claims.

At this point at the end of 2025 it’s unclear to me whether the use of AI and machine learning models will ever be satisfactory for detecting the use of AI in writing.

Unlike these proprietary AI detection tools with their black-box algorithms and problematic false positive rates, this Vale.sh ruleset is transparent and interpretable. Each rule traces back to observed patterns and the configuration is available on GitHub for review and expansion.

One interesting challenge is that of linguistic adaptation. As AI writing becomes more commonplace and individuals gain greater confidence in their ability to spot it, certain words and turns of phrase will be avoided to prevent accusations.

The ruleset can therefore help authentic writers pre-empt this situation and consider avoiding turns of phrase and other tells. This feels like a sorry state of affairs but helpful tooling is one way for writers to be kept informed and in control of their output.³

Telltale tiers

Claude and I categorised the current rules in the following three tiers. This means that the rule confidence can be matched by the Vale behaviour.

• Error: Definite AI artifacts – chatbot phrases, technical glitches, placeholders, tracking URLs.

• Warning: Likely AI patterns – hedging clusters, knowledge cutoff references, enumeration style.

• Suggestion: Suspicious but common in human writing – vocabulary, transitions, passive voice, symbolic language.

While powerful, the Vale configuration isn’t as expressive or flexible as required for some advanced AI detection constructions. One example is the inability to specify stray Markdown syntax outside of areas where Markdown syntax is being used. Without this ability this rule would flag any and all use of Markdown which would be useless.

What next?

Now that the rules are shared I want to see how writers and editors use these to understand the writing they produce and review.

There’s an ecosystem of tooling that can use the Vale rulesets including the flexible command line interface and a Chrome browser extension. It would be interesting to enable the use of the rules in other form factors. For example, it would be helpful if the rules could be run server-side on websites to avoid the need to install Vale on your local machine.

Another possibility would be to support alternative versions of the rulesets for the other available prose linters.

The rules will need to adapt over time to reflect changing AI dialect and GPTisms – it’s unclear whether these rules will be relevant or productive in a year.

Others have attempted to compile lists of “slop words” to guide AI tools away from cliché. I could consider merging these with my current ruleset but need to consider the selection criteria for including each word.

Self-survaleance

I ran my new ruleset against this article (the one you’re reading now) and… nothing.

✔ 0 errors, 0 warnings and 0 suggestions in 1 file.

It’s oddly satisfying that nothing was flagged – although it could be that I naturally edited out anything that whiffed of AI along the way.

Licensed to lint

As it stands, the ruleset is mostly generated from the “Signs of AI Writing” page and it’s therefore released under the same CC by SA license. This means you can create further derivative works as long as you credit the source and keep the same license.

You can find the ruleset over on GitHub with instructions for getting started. Comments, feedback and pull requests are welcome.