Google has released Android Bench 2.0, a major update to its benchmark for evaluating AI models and coding agents on real Android development work. The new version moves beyond short bug fixes and incremental edits. It introduces long-horizon tasks that can take a human engineer multiple days or even a week to complete.
That makes the benchmark more relevant to the way developers are beginning to use coding agents in production: dependency upgrades, large refactors, feature implementation, app creation and cross-platform migration.
What changed in Android Bench 2.0?
The original Android Bench focused more heavily on contained repository changes. Android Bench 2.0 adds tasks with much larger scope, more ambiguity and more opportunities for an agent to make an architectural mistake several steps before the final test fails.
Google says the new long-horizon task set includes work such as:
- upgrading dependencies across an existing project,
- adding substantial new features,
- building Android apps from scratch,
- performing broad architectural refactors,
- and converting cross-platform apps to Android.
The benchmark also introduces agentic evaluation, pairing models with coding-agent environments instead of judging only the raw model in isolation.
Why pass/fail scoring is not enough
A multi-day engineering task can be mostly correct and still fail one edge case. Google gives the example of an agent that might refactor dozens of screens, create the required data layer and satisfy most requirements, but still fail a single assertion. A binary score would treat that entire run as a failure.
Android Bench 2.0 therefore uses a more continuous completion model that considers functionality, visual fidelity and regressions. It also applies penalties when an agent violates structural constraints or evaluation instructions.
This is important because AI-assisted development is increasingly about how much reliable work an agent can complete, not simply whether a final test suite turns green.
The new tasks are dramatically harder
Google reports that the highest pass rate on the new long-horizon tasks is around 28%, compared with roughly 91% on the original benchmark tasks. That gap is a useful reality check for teams evaluating AI coding agents.
Small, deterministic transformations are not the same as owning a feature from architecture through runtime validation.
What AI coding agents do well on Android
Google’s analysis says models perform comparatively well on established, repeatable transformations. Examples include Java-to-Kotlin conversion, replacing Retrofit with Ktor and introducing a ViewModel layer across a large codebase.
Those tasks have clearer patterns and more local feedback. Once the work depends on runtime behavior, breaking framework changes, dependency-injection graphs or unreleased libraries, performance drops.
Cross-platform-to-Android migration remains particularly difficult. Google’s post says no model reaches a 100% pass rate on those tasks, even when completion rates are substantially higher than strict pass rates.
Why the agent harness matters
Android Bench 2.0 also highlights that the model is only one part of the system. Google is beginning to evaluate model-and-agent pairings, such as running a model through the provider’s corresponding coding environment.
That reflects a practical truth developers already see: context management, tool use, caching, repository navigation, build feedback and test execution can materially change the outcome even when the underlying model is the same.
For Android teams, that means model rankings should not be treated as a universal answer. The same model can perform differently depending on the harness, project structure and how quickly it can observe build or runtime failures.
What this means for real Android projects
The benchmark supports a conservative workflow: let agents handle well-scoped transformations, repetitive edits and clearly testable features, but keep architecture, migration strategy and regression validation under strong engineering control.
That is especially important for production apps where one change can affect billing, authentication, permissions, background execution or data migrations.
Our recent ADB Wi-Fi 2.0 guide covers another part of the modern Android development workflow: faster and more reliable wireless debugging. For platform-level changes that can affect app behavior, see our Android 17 features guide.
Should developers trust a leaderboard?
A leaderboard is useful for comparing behavior under a controlled workload, but it is not a substitute for testing an agent on your own repository. Android projects differ in Gradle configuration, legacy code, Compose adoption, native dependencies, CI rules and release constraints.
A practical evaluation should include your own regression suite, build variants, lint rules, instrumentation tests and a review of architectural changes — not just whether an agent completed a benchmark task.

Start the conversation
Corrections, useful experiences and focused questions are welcome. Keep discussion respectful and on topic.