Fun experiment from last week: I built an AI text detector trained on my own blog.
Why? Because off-the-shelf AI detectors are useless on my writing.
I ran LLM-DetectAIve across my posts and it flagged 8 of my old handwritten posts as AI at 0.95 confidence. To catch 1 actual LLM post, it threw 23 false positives. Then I tried Binoculars and it went the complete opposite way - zero separation, my old posts and the LLM posts were completely mixed together in the same score range.
The reason is pretty simple: generic detectors are basically style classifiers. My natural writing in tech posts is short sentences, fragments, bullet lists, jumping straight to the point - which happens to confuse detectors trained on essays.
Free ground truth
The nice thing about testing on my own blog is that the dataset labels itself:
- Everything before 2026 = written by me by hand (28 posts)
- Everything in 2026 = drafted with LLMs (16 posts)
44 posts total. Didn’t have to manually label a single file, the dates did it.
Couldn’t see his face coz he was driving and I am sitting behind. He was happy, he told me he was happy.
Stuff like that (from my post about my dad), the 2018 marathon fail, the old k8s debugging notes - all in the human bucket. The 2026 agent posts and newsletters - machine bucket.
What actually worked
Two tiers of features.
First tier is basic stylometrics: sentence length distribution, vocabulary diversity, punctuation counts, how much sentence openers vary. Just that alone got 0.886 leave-one-out (LOO) accuracy.
Second tier is where the real signal was. Instead of asking “does this look like generic AI text?”, I asked “does this look closer to my old posts or my 2026 AI posts?”
Computed TF-IDF vectors, cosine similarity to both class centroids, and a 5-nearest-neighbors vote. My human posts had ~0.83 same-class neighbors on average, while the machine posts had 0.075.
Adding that pushed LOO accuracy from 0.886 to 0.909, and then to 0.932 after dropping a bad feature. AUROC hit 0.964. On the full scan, every machine post scored between 0.85 and 1.00, and every human post scored <= 0.11. Runs locally on 4 CPU threads in a couple seconds.
What failed
Two things I thought would work actually made the detector worse:
- Perplexity (via local Qwen): Textbook theory says AI text has low perplexity (predictable) and human text has high perplexity. On my blog it’s backwards! My old notes are super blunt and simple, so they get lower perplexity than the LLM drafts.
- Normalized Compression Distance (NCD): Looked good until it hit my 2018 marathon post. In that post I repeat words a lot (“walk and walk and walk”, “pain”). Repetitive text compresses well, so NCD thought my marathon story was AI. Dropped NCD and LOO jumped to 0.932.
Beating my own detector
Once the detector was working, obviously I had to see how hard it is to fool.
First I had an LLM read two of my old posts and write two fake ones copying their exact style.
Let the service name be ‘banana’. Let the pods be of the service ‘pineapple’.
Copied that opening style from my K8s DNS fail post, and the detector scored the fake at 0.005 (100% human). Wrote another fake copying my musings-alone style, scored 0.015.
Why? Because the kNN + TF-IDF layer just measures how close a piece of text sits to my existing pre-2026 posts.
Then I took a post that the detector had flagged at 0.942 (super AI) and spent an evening trying to get the score under 0.5 using the detector’s own feature breakdown. Took 13 rounds. What moved the score the most:
- Dropping in verbatim quotes from my old posts - even 2-3 real sentences flipped the 5-NN vote from 0/5 to 3/5 human neighbors.
- Fixing repeated sentence starters - the LLM draft had started like 20 sentences in a row the same way.
- Adding
?and!- my handwritten posts have ~9 question marks per 1000 words. The LLM drafts had almost zero. - Using my normal words - words like
k8s,cpu,stuff,well,very,don't,coz,gotta, and stripping out the words that only showed up in the 2026 AI centroid.
One funny quirk I noticed: words that appear in fewer than 2 posts in the corpus (min_df=2 in TF-IDF) are completely invisible to the vocabulary layer.
After 13 rounds the score dropped from 0.942 down to 0.313. And tbh the edited version actually read way better than the first draft anyway. Packaged those 4 steps into a skill called byline-beater (to pair with my remove-ai-smell skill).
Takeaway
Prose-only AI detection has a hard ceiling. If someone has 20+ of your old blog posts and a frontier LLM, they can match your sentence lengths, punctuation habits, and vocabulary distribution closely enough that any text-only classifier will fold.
Still useful as a local linter for my own drafts though - tells me immediately when a draft sounds too much like default ChatGPT/Claude slop instead of how I actually talk.