This is a submission for the GitHub Finish-Up-A-Thon Challenge
Note: AI is currently a Hot Topic in the developer space. I recommend reading this...
Some comments have been hidden by the post's author - find out more
For further actions, you may consider blocking this person and/or reporting abuse
Great idea! I’d be curious to see how it analyzes my articles. I already use AI for grammar and style corrections, so it would be interesting to compare its feedback with my current workflow. 😄
@gramli Let us know what you find! This is an evolving project so any data from your end will be much appreciated 💙
Thanks Daniel! Like, @codingwithjiro mentioned, feel free to provide us feedback/contribute to our repo!
This was a fun read 😄
Honestly, the part that stood out to me most wasn't even the classifier itself. It was the collaboration story behind it.
The sections about explaining an existing codebase to another developer, dealing with time zones, maintainer responsibilities, merge conflicts, and realizing that communication took longer than writing some of the code felt very relatable.
I also appreciate how transparent you were about the limitations of the model and dataset. AI detection is one of those topics where people often oversell certainty, so seeing a lot of emphasis on "prototype" and future improvements was refreshing.
Also Elmar's line:
That one is and will be forever true.
Congrats to both of you on shipping this. Looking forward to seeing where ClassifierAI goes from here!
@itsugo Glad you enjoyed the article! Here's to more The DEVengers posts in the future!
🍻to The DEVengers indeed @codingwithjiro @itsugo!
🍻to The DEVengers!
I appreciate it @itsugo! It was very fun to work with Elmar on this project. It's always good to be transparent on using this tool since it's just a prototype. With more data, I think it would be useful! Thanks Ayan!!
Nice prototype, and I like that you clearly mentioned the dataset limitation....The bigger challenge is false positives. A lot of human writing can look AI generated, especially structured or non native English. I’d treat the result as a confidence signal, not a final label.
@sunychoudhary This is true. I even tried my writing to existing article AI checkers in the past and it still flags some parts of it as AI. That's also one of the reasons why we only tested @francistrdev's articles for most of the results shown in the article 😅 (we don't want others to see their article get flagged as 30% AI-generated out of nowhere without a more accurate model )
But yeah, confidence signal is a nice term to use here. Thank you for your input!
Hey Suny! Hope you are doing well! In any dataset, I agree that the big challenge is false positives! Like Elmar stated, I took the sacrifice to test it on my own article instead of doing someone elses. We believe it is more professional that way and not causing up a Stir since this is a prototype.
We are hoping to work with the community on creating an accurate dataset based on what we observe on DEV. It will take time for sure, but hopefully it would be worth it! Thanks Suny!! :D
Great project, Francis and Elmar. I think a lot of us have projects sitting around that we keep telling ourselves we'll come back to someday, so it was nice to see this one make it across the finish line. Looking forward to future updates on the project.
Thanks Hemapriya! Also looking forward to see the weekly radar under the org (if you are planning on that)!
Thanks, Francis! Most of it's already done. I'll post it under the org on Friday.
Interesting project. What I appreciate most is that you're approaching it as a transparency tool rather than claiming to perfectly detect AI-generated content.
@karinatran We appreciate that. Nowadays, authenticity is scarce and we made sure our project is not just another AI that claims to do 100% of the tasks without hallucinating. We accepted the flaws and inconsistencies while also presenting it's strengths. If we continue writing more using this style, I think more people will be educated and be less swayed by the "hype". This also applies with new emerging technologies, not just AI.
Nice work! I like that you're focusing on reducing the manual effort of checking AI-generated content instead of trying to be a perfect detector.
It'll be interesting to see how the model performs once it's trained on more DEV-specific data.
@consomida Yes, it will be much more interesting. Initially there were three factors: human, ai-generated, and mixed. If we get the model to perform on the DEV-specific data, the extension can extract a more accurate result and hopefully enough data to get "mixed" percentage added as well.
Hey Consomida! I am interested to see how this would go as well! Appreciate the comment :D
Nice tool! I like how you did a deep dive into the story behind this product and its current limitations.
Very nice to see another collaboration post after the Gemini one we did!
Keep it up.
Thanks Julien! It's quite nice for another collaboration! Stay Tuned for the next one 👀
Really interesting project! Quick question though how does the classifier handle a middle-ground workflow like writing a full draft yourself and then passing it through AI just for grammar and style cleanup? The core ideas and structure are human, but the surface-level writing patterns might look AI-generated to a classifier. Curious if that's something you're planning to account for as the model improves!
Hey Aditya! Thanks for the question and hope you are well!
It won't be as advance as GPTZero where it can not only detect if the post is AI generated or not, but being able to detect patterns.
As a good starting point, we would assume that 50% of the text is AI written but human written as well. It would be a priority for sure, but it would require me and @codingwithjiro into research on how we can achieve that. Otherwise, the first thing is collecting data and tuning it based on community feedback.
Feel free to provide some ideas in mind. It would be helpful :D
Thanks for the reply!
One idea : maybe instead of a binary AI/Human label, a confidence spectrum or a "mixed" tier with a breakdown percentage (like 70% human / 30% AI) could be more realistic for most articles. Even GPTZero struggles with the middle ground, so leaning into that nuance early might actually make ClassifierAI stand out. Happy to see where this goes!
Banger of a post! @francistrdev 🤩
Appreciate it Elmar! Great to work with you :D
Building a scanner tailored specifically to DEV's writing style is such a smart move. Standard broad tools miss the mark because every community writes differently, so a localized dataset makes total sense. Massive congrats to you both (and your Copilot) on shipping the prototype!
Hey! Thanks for the compliment (along with the copilot) as well :D
The dataset task gap is the real design challenge here, more than just the accuracy
number. Wikipedia writing is encyclopedic and formal, DEV writing is conversational and
varies a lot by author. Even with DEV specific training data, the binary AI/human
label misses a lot of the middle ground. An article that uses AI for grammar and
structure but contains genuine original research reads differently to a reader than a
fully generated one, but a pattern classifier sees the same surface features. The
confidence signal framing is probably where this tool needs to go long term. The
collaboration write up is also one of the more honest ones I've read about this kind of
process. The part about communication taking longer than writing code is something
most teams don't say out loud but should.
Thank :D
Two Devs and a Copilot walked into a bar
lol
Great idea! Thanks!
No problem! Thanks for reading :D
Such a cool project! I'm pretty curious what the rate of my articles would be (I use AI to fix typos and stylistic errors that I make when binge-writing 😄). However, I always value human spirit and personal style in the articles, which is something that gets lost when generating content using AI. I believe this extension will help to make
dev.toa space that promotes human-style over generic docs 🙂Good luck with your project and also congrats with the collab success story! 🎉
Hey Klaudia! I appreciate the support and thanks for leaving a Star on the repository! Feel free to contribute via PR if there are any issues!
Good luck on your project as well :D
Thank you! :)
and now I realized you don't have a project for this event. My bad lol I was thinking of the Recycling project you did xd
Yes, I haven't deployed the Solstice game yet 😂 Will do in the upcoming days.
Oh this is the "GitHub “Finish-Up-A-Thon” Challenge Submission" lol
The game I know your still working on it!
Wow, 😮 I've never come across a project like this before. Seriously, you both have done a fantastic job! The project looks amazing, and it's clear how much hard work and dedication went into creating it. Great work!😊
Thanks! Appreciate it :D
This was a good read because you didn’t present the extension as magic. The part I found most useful was the honest breakdown of where the classifier is weak today, especially the mismatch between Wikipedia/HuggingFace data and actual DEV writing styles. The Vite migration and the decision to avoid a heavier local model after it hurt browser performance also felt very real. If you keep iterating on this, I’d be very curious to see a DEV-native dataset and maybe a confidence breakdown by section instead of only one final label, because that feels like the next step that would make the tool much more informative.