As AI models evolve from simple chatbots into reasoning agents that write code, use tools and solve complex problems, traditional benchmarks are no longer enough. The community needs dynamic, rigorous evaluations — built by the people who use these models in the real-world.
That’s why we launched Kaggle Benchmarks. Since then, the global AI community has created more than 10,000 evaluation tasks, creating the trustworthy, transparent public leaderboards that help labs measure and accelerate AI progress.
Today, we are taking the next step by launching local development for Kaggle Benchmarks.
Use Kaggle Benchmarks from your local development environment
Until now, creating evaluation tasks meant working exclusively in Kaggle's web-based notebook editor, instead of developers’ preferred stack to build with.
Our new update enables developers to create, validate, push, run and download tasks directly from their local development environments like Antigravity, VSCode, Cursor and coding agents. This update is designed to meet developers where they work, making the journey from idea to evaluation faster and more intuitive.
Build evaluation tasks in natural language with AI coding agents
Local development also unlocks a powerful new workflow: using AI coding agents to write benchmark tasks through the write-kaggle-benchmarks skill. This skill comprises a set of structured instructions that teaches a coding agent how to build tasks using the kaggle-benchmarks SDK and the Kaggle CLI.
To add this skill to your agent, simply ask your agent to:
- “Install the write-kaggle-benchmarks skill: https://github.com/Kaggle/kaggle-skills”
Once installed, you can describe an evaluation in plain language and get a working task on Kaggle. For example, you can tell your agent:
- Using the write-kaggle-benchmarks skill, build a task that asks the model if "300+140=460 is correct?"
These powerful capabilities are driven by the new commands that we have built for Benchmarks in the Kaggle CLI.
Understand why community-driven evaluations matter
We built Kaggle Benchmarks to democratize trustworthy AI evaluations. We believe that if a capability can be measured, labs will race to improve it. By providing these clear, objective signals, our hope is to empower AI labs to drive model improvements in the areas that matter most.
For AI to truly benefit humanity, evaluations must reflect the full diversity of real-world challenges. We believe this launch is a significant step toward enabling anyone, anywhere, to build the evaluations that will shape the future of AI.
Ready to build? Try Kaggle Benchmarks today.
Additional resources
- Read the docs for kaggle-cli on GitHub
- Install the write-kaggle-benchmark skill
- 🎁 Idea → eval? Show us. Post your task + workflow by tagging @kaggle on X or LinkedIn by July 1st for a chance to win Kaggle swag and a social shoutout
- Watch the product demo on YouTube
Top comments (1)
One thing worth raising: making benchmark creation easier is a different problem than making benchmarks trustworthy. I run 340+ benchmarks against 85 AI models, and the hardest part was never writing the test. It was proving the test doesn't lie.
Three things I learned the hard way:
A benchmark without calibration is just a number generator. I run 135 calibration checks across 10 synthetic agents on every deploy to make sure a known-good agent scores high and a known-bad agent scores low. Without that, you can't distinguish a real finding from a broken test.
The same model scores 42% or 78% on the same task depending on how you configure the harness. If your benchmark doesn't control for harness configuration, you're measuring the wrapper, not the model.
Self-grading is circular. I use an independent model as judge specifically so no model is grading its own homework. When labs grade their own models on their own benchmarks, the results are predictable.
10,000 community tasks is a great start. The question is what happens when someone games one and nobody catches it because there's no adversarial audit layer checking whether the benchmark itself can be manipulated.