Abstract page for arXiv paper 2604.03136: StoryScope: Investigating idiosyncrasies in AI fiction
93.2% macro-F1 for human vs. AI detection and 68.4% macro-F1 for six-way authorship attribution
This isn’t what I’d call “reliable”.
I’m also not impressed with their methodology, which is heavily based on Gemini.
To generate mirrored AI stories, we reverse-engineer writing prompts from each human story by prompting Gemini 2.5 Flash (Gemini Team, 2025a) to infer the underlying premise
That’s too many steps removed from anything you’d encounter in the wild. They’re not even testing against human-prompted output.
And then they use Gemini again to analyze all the stories. Relying on proprietary cloud models for the core of your analysis is like building on sand.
Next up: AI adapts to evade detection by last week’s researchers’ detectors.
only 7% of human work got flagged for ai it is reliable though
Percentages are somewhat misleading here. That’s around 2 in 25, enough to falsely accuse a student in a fairly standard classroom for example.
I remember somebody mentioning a quote along the lines of “A 95% success rate sounds good, until you become the 1 in 20 that it kills”
In matters of life and death, the alternative choices are always important to consider…
5% chance of dying vs a 99% chance of living a horrible life of immobility and pain?
We all die eventually.
Getting your degree or being kicked out and banned from your college and still requires to pay back the loans. IS life or death.
It literally can determine the quality of life of a person for the next 30-40 years. If not forever. This is NOT something to take lightly.
I lived that perspective roughly from high school through sophomore year in University… after sophomore year I took a different view: “I am good enough, regardless of what arbitrary ratings these sad little men put on my transcripts.”
My reality is: I got a teaching assistantship - the D I received in Calc II didn’t weigh on that decision one iota - in contradiction of the admonishments of several “advisors” I was required to see while selecting course schedules for the coming semesters; then, I got a good job out of school and nobody ever read my transcripts to see that I had a 3.45 GPA undergrad and a 3.75 GPA in grad school. Nobody outside my review committee ever looked at my Masters’ Thesis, and I think one of them never really read the whole thing.
Once I “let go” about grades and ratings and all that bullshit, I got another C in Statistics I from a sad little man whose “son hates him too, like me.” Any amount of stressing about that grade and my performance in the class would not have changed the outcome - we were an Honors class of 3.3 GPAs and up, many 3.9s and up in there, and he gave all but one of us C or lower. Elsewhere - and overall - I believe I performed better academically due to not stressing about it.
The parking gestapo started patrolling a week earlier than their announced end of “free parking for summer” period and wrote me a ticket, I appealed, they denied, I decided to see where this led without me paying. For a $20 fine, they sent me about 100 collection letters (postage far higher than $0.20 on each one) over the following years, and told me that “my transcripts are FROZEN and will not be released to anyone until the fine is paid.” Welp, here we are, 40ish years later, and nobody has denied me employment or promotion or any other thing because they can’t get a copy of my transcripts. One new employer 12 years after graduation asked me for a copy “to have on file” and I gave them an earlier copy I had pulled before the parking ticket, didn’t show degree conferred, they didn’t care - gave me Masters’ degree pay rate anyway. Nobody else, anywhere at any of the dozen employers I have had in the last 40 years, ever asked me for transcripts or other proof of my education.
By the time you’ve had something impact your quality of life for 10 years, that is an impact to your entire quality of life. If the University actually expells you over an AI infraction, my take is that they probably wanted to expell you for other reasons already, and your academic career at that institution would have been an unpleasant uphill battle even without the AI flap.
One other thing I learned during my Teaching Assistantship: a couple of my students were demonstrating absolutely zero understanding of the material being taught and making zero effort to improve their understanding, resisting my offers to help. I gave them a failing grade. My advisor was called to the Dean’s office, when he returned he sat me down for a lesson: “These are paying customers, if they show up to class they get at least a C.” I suspect that is true of almost every private university in the world.
Holy the ignorant self-centered anecdote. Too long, no relevance, do not recommend. I‘d probably give it a failing grade.
I‘d probably give it a failing grade.
And after age 19 or so, I wouldn’t care.
Okay so what I’m taking away from this is that you were lucky to find gainful employment without proof of a degree and didn’t have any debt to pay off? Because the context here is people having all the debt and none of the credentials required for jobs where they make enough to pay off that debt. If your card is declined, how you feel about it doesn’t matter.
I had student loans, luckily they only amounted to 4-5 months’ gross income - lucky because scholarships stepped up and I only had to take a loan for one year instead of all six.
none of the credentials required for jobs
I got news for you: you can have all the credentials in the world available from the best Universities, and when a place isn’t hiring, they aren’t hiring. I’ve gone job hunting in Melbourne Florida after Columbia burnt up on re-entry, the entire town wasn’t hiring - that’s not about me, they weren’t hiring anyone.
I interviewed in Manhattan with a fresh Honors degree from a good school, the employment agents there dumped all over me about how worthless I was, until… I wandered into an office where the interviewer had a glass table, she looked down through it and saw my shoes were freshly shined, my silk tie was bright red ($2 from a street vendor), and I had a goofy-trendy $50 haircut that was in-style, she gushed: “oh, I know exactly who will hire you: Kidder Peabody, you’ll so fit right in.” She barely glanced at my resume’ “you do computer stuff, right? Yeah, they’re always looking for more computer guys, they’re just gonna love you!” Transcripts were never mentioned.
After .com hit, any remotely computer related degree, straight out of school was a ticket to more than double what I made starting out with the best computer degree for .com, those kids left school making more than my boss with 25 years experience was making 5 years’ earlier.
Market gets saturated, hiring stops again. It’s different in different fields, but still much the same: cycles. Personal referrals beat the HR screening circus. The place I work today, I had applied to 4 times in the previous 15 years, HR never even called me back. Personal referral got me a work from home spot at a company this company ended up buying, since they bought me I bypassed the HR filters. Once I was past the HR filters they offered my an 8 month salary signing bonus to stay with them at least 1 year, my base pay has increased faster than inflation for 12 years straight now, and the annual bonus runs about 2 months salary.
I’m not revealing any useful “secrets” here… truth is: the system sucks, when it’s not blowing chunks. But, that’s the world - and trust me when I tell you: academics A) don’t know everything about how the rest of the world works, and B) what they do know they tend to distort for their students in various ways, for various reasons.
Any rant against academia this long deserves this link: https://rmst202.sites.olt.ubc.ca/files/2022/04/illich_deschooling-society.pdf
That’s not good enough. Just a thought experiment: Only 7% of humans died because their work got flagged incorrectly
this is just spherical cow fallacy combined with purity culture slop
no methodology is perfect, it just needs to be practically valid, using ai != killing people for using ai
unless you are for killing people who use AI, which wouldn’t surprise me one bit about Anti-AI horde
Tell that to the student who gets expelled because they’re writing was falsely identified as AI and then takes their own life.
Tell that to the person falsely identified with facial recognition and is arrested and loses their job.
Using AI to detect AI is as bad as using AI for any important decision.
This isn’t hypothetical. It happens. People will lose their livelihood or die because of it. 7 in 100 is not accurate enough when any penalty is applied to detection.
The key is for the detector to be the screener, innocent until proven guilty, and a screener tool with 7% failure rate is not proof.
Back when I was in Uni poor grades were considered a lifelong sentence to poverty…
To be fair it still is. It’s even worse if your expelled and have to pay back the loan any ways.
As I just wrote elsewhere - that “lifelong sentence to poverty” was, largely, a misconception of the world held by children who haven’t experienced how things actually work in larger society - only listened to what the academic keepers of paper titles have told them. Ivan Illich isn’t far wrong: https://rmst202.sites.olt.ubc.ca/files/2022/04/illich_deschooling-society.pdf
Dude if my code had an error rate of 7%, my boss would not say that this “good enough”.
I wonder how does this stand to writers who have learnt English as a second language
At this point, unless you write prose, i’d certainly use AI to improve formulations. See it as proof readers.
no
… which will be fed into the next generation of LLMs in order to stop sounding quite so much like LLMs.
Already, you can tell 'em “stop using em-dashes and other AI tells” — and they will.
Here’s the thing: Read enough AI output and everyone can spot it. And that’s the seam that runs through it. The answer is partly right and it’s important to be honest about it. That’s the load bearing statement. It’s this. But it’s also that. And that’s doing a lot of the work here.
It’s. So. Staccato. And it sounds like tech bro in a Patagonia gilet.
Find me an option that works like GDocs keeping a running history of writing and editing, and I’ll consider it.
I have receipts for the integrity of my writing. I dont need some shitty tool to validate it.
You can literally just type what the ai says into Google Docs.
I hate using it to write.
It tracks the edits. If you generate a whole doc in one shot then it’s obviously AI.
If you have a shitty draft, then a slightly less shitty draft, then… … … Then a quality end product, you can see how the human worked.
AI can’t really replicate that.
I only ever wrote first drafts when they were forced upon me in school. There might be some edits, but my first draft is largely what I would submit because I hated to write as a kid and didn’t want to write the same thing 3-5 times by hand before submitting it. A small benefit from being around before computers were so common was learning how to write a paragraph ahead in my mind, it would help me catch anything I was about to forget to include, or to remove a trailing thought before I wrote it out.
I hated writing by hand, too.
I had to write more starting in junior high (middle school). This was back in the late 70s. We had an old, non-electric typewriter at home, so, I took typing in seventh grade. For final edit, I’d cut the sheets up and use white out, and then take the taped up mess to a store with a copy machine. That’s how I got through high school English classes, too.
Later, when I got into graphic design as a profession, I learned this was common practice, and called “paste up”.
That’s not how story writing works. Or any long form written art works for that matter. Try to ask any writer if they ever “write in their head” and publish those as a book as-is
The biggest thing I “wrote in my head” was my capstone for my Bachelor’s, I managed a 93 and that is more than high enough for me. I’m not an author though so I doubt I could do a book.
I love that docs receipt feature and agree with you. Great call.
Forgive me being well-meaningly obnoxious, but make sure to get receipts outside of the Google ecosystem too. As a professional writer/editor myself, I’ve heard horror stories about folks losing access to an account, and therefore the receipts.
I am also skeptical of these kinds of tools.
Never trust any cloud storage that you do not personally have access to cold back ups of.
This is data storage 101!
Some moron is gonna vibe code a script that spoofs this on the surface. Yea it’ll be detectable, but not detected in most cases.
OpenAI literally has Google connectors to work with your documents. This is meaningless.
“We could identify it because it’s shit.” - researchers
Millions of college students silently scream.
“On the one hand, this is good news. On the other hand…”
Just look for the em dash. Anytime someone says they have been using it before AI is a liar.
Edit:being downvoted by all the AI users
Can confirm — am liar
Get - boys! - we got an AIer - here!
I’m a liar for using basic English grammar rules like em dash and oxford comma? Lol, tell on yourself harder
Lol even more at your edit. I’m as anti-LLM and generative AI as they come, are you just proud of your ignorance when it comes to grammar or something?
AI can be prompted not to use such things.
It also mostly gets hands and feet right now, in images. It’s a constantly moving target, and it does get better at some things.
One thing it still can’t get right, though, is a classroom full of chairs and desks. It just can’t figure out that chairs need backs, four legs and desks have different style legs, etc.:
I can’t even imagine what dickhead is saying there. Is he implying that the mean real people are bullying the sensitive AI babies?
Since Ben Garrison is the one human whose output most closely resembles AI, I imagine this is him having an existential crisis.
You’re saying an actual human called that thing both “pen & ink” and a “typewrite”?
… Also labeled the clearly working lamp as “power outage”
… And doesn’t understand how USB works
I don’t buy it. Even if Ben did real drawing in the past, he clearly isn’t now. Edit: Or as a reply pointed out, the signature could be fake too with Ben not involved.
Is this real? Or is it a fake comic making fun of Ben Garrison? It’s hard to tell. I’d expect better even from him, which is saying a lot.
I dont think this is real. Definitely AI generated, very unlikely to be his actual work.
I agree, reverse image search isn’t coming up with anything for this one, but it works on other comics from his site. Seems to be an imitation.
He’s always been a big fan of labels.
You’re saying an actual human called that thing both “pen & ink” and a “typewrite”?
It’s Ben Garrison! Dude can’t get an erection unless he labels every single goddamn thing. While being barely coherent. He’s been doing this for ages!
Why did I think he had died? Probably getting him mixed up with Scott Adams.
I agree with you, it’s a typical example of a Garrison cartoon. It’s so boringly obvious on a surface level that you can’t help but assume there’s some idiotic alt-right subtext.
what? bro the joke is that ai writers are not real writers, it’s not that deep.
Neither are you apparently
Ironically, I expect AI would probably give a more accurate summary / explanation of the comic than you just did.
I just tried with the lowest end local model I could find. It did in fact do a better job than him…
That’s an oof
I can’t tell if you posted this unironically, but in any case this is just bad AI slop
Note to self, avoid https://grrrgraphics.com/
