Elena' s AI Blog

The best coding agent still fails 42% of maintainer-style checks

16 Oct 2026 (updated: 16 Oct 2026) / 32 minutes to read

Elena Daehnhardt

Generated by Midjourney. Prompt: AI-assisted coding with a robot reviewing a pull request.


TL;DR:
  • Frontier pricing is now a calendar problem: Argon's $2/$10 rate runs to 30 November, then rises to $4/$20, and its lead is uneven (first on the Vals Index, behind Opus 5.5 on Terminal-Bench 4.0), so test on your own tasks before committing a budget.
  • FrontierCode 1.1 puts the best model at a 58% pass rate on maintainer-style blocking checks. That is a useful warning signal for autonomous pull requests, not a success rate you should expect on your own repository, and the top score costs far more per task than the runners-up.
  • Anthropic's self-replication study shows its two halves separately (small models on cheap IoT hardware, and a stronger model coordinating replication), not yet together. The "months away" forecast is a trend extrapolation, and regulators are now asking the labs for documents on their incidents.
  • Two smaller open-weight models also shipped: Liquid AI's LFM2.5-24B-A2B (13 October) and NVIDIA's Nemotron 3.5 Super VL 120B A12B (15 October), with vendor-reported performance.

Weekly AI Signals, 16 October 2026: Pricing, Open Weights, Agent Reliability and Regulation

Gemini 4 Argon went generally available with a price tag that has a sell-by date, and Meta released the weights of an 816B-parameter model that it says runs on two Mac Studios. Meanwhile, a new benchmark told us how often an agent’s change passes maintainer-style checks. The answer is not flattering.

I have grouped the stories by theme. Last week’s issue is here if you need the background on Haiku 5.5 and Mistral Large 4, whose open weights have not yet appeared.

In this issue:

  1. Gemini 4 Argon Goes Generally Available, with an Expiry Date on the Price
  2. Meta Releases Muse Spark Open: 816B Parameters, Two Mac Studios
  3. FrontierCode 1.1: The Best Agent Still Fails 42% of Maintainer-Style Checks
  4. GitHub Makes AI Scan and Agentic Autofix Generally Available
  5. Anthropic Tests How Close Small AI Models Are to Self-Replication
  6. The FTC Sends Formal Demands to OpenAI and Anthropic

Frontier Models: Gemini 4 Argon Pricing and Benchmarks

1. Gemini 4 Argon Goes Generally Available, with an Expiry Date on the Price

blog.google favicon Introducing Gemini 4 Argon — Google, 13 October 2026

venturebeat.com favicon Google makes Gemini 4 Argon broadly available with limited-time low API pricing — VentureBeat, 13 October 2026

Google made Gemini 4 Argon broadly available on 13 October, two weeks after its limited debut on 30 September. Argon is available in the paid Gemini API, Google AI Studio and Antigravity (Ultra tier for individuals), and in the Gemini app and AI Mode in Search for AI Pro and Ultra subscribers. The introductory API price is $2 per million input tokens and $10 per million output tokens, with cached input 95% cheaper. The introductory rate runs to 30 November; from 1 December the public price is $4 and $20, so the rate you benchmark against in October is not the one you will pay in the new year.

Google reports first place on the Vals Index and 87% on the ERQA embodied-reasoning benchmark, and VentureBeat’s collation of the results is more mixed than the headline. Argon scores 77.9% on DeepSWE v1.1 against 74.2% for Claude Opus 5.5, but 57.4% on Terminal-Bench 4.0 against 66.4% for Opus 5.5. On the Artificial Analysis Intelligence Index it scores 53, behind Opus 5.5 (58), Sonnet 5.5 (56) and Meta’s Muse Wonder Full (56). Muse Wonder and Muse Wonder Full, Meta’s closed agentic models released on 12 October, scored 52 and 56 on that index, up from 48 for Muse Spark 1.3. Muse Wonder is available through the Meta Model API and Muse Code, while Muse Wonder Full is limited to eligible Muse Code subscribers. Meta’s model page lists Muse Wonder at $4 per million input tokens ($0.20 cached) and $20 per million output tokens with a 1M-token context window, which happens to match Argon’s post-introductory price. Muse Wonder Full is included in the Power Usage ($50 a month) and Maximum Usage ($200 a month) Muse Code plans and has no API price. The page also carries Meta’s benchmark table against Claude Fable 5.1, GPT-6 Astra and Claude Opus 5.5, but that is a vendor comparison, as are the “Active Memory” and “Active Focus” capabilities. Of these figures, the Artificial Analysis index is the independent measurement. The rest are vendor-reported or come from leaderboards that vendors help populate, and a published number is not the same as a reproduced one. Unrestricted cyber access to Argon goes only to defenders through Google’s Fairwind Program.

One claim is more concrete than a leaderboard point. Google says Argon, working overnight, produced changes to its open-source Agent Development Kit that cut cold-import time by 21% and in-memory session preparation in long sessions by 40%. Google supplies these figures itself, and the announcement does not give the baseline measurements, so I read them as a plausible example rather than a result. Even so, a real software task is a better demonstration than another benchmark percentage.

Why this matters

An introductory price is a shop-window price. I would budget on $4/$20, not $2/$10, and treat October as a cheap evaluation window, much like a restaurant’s soft-opening discount: pleasant, but you would not plan the year’s catering around it.

The benchmark spread is the more useful signal. A model that tops the Vals Index and trails Opus 5.5 on Terminal-Bench 4.0 is a specialist with a strong CV, not a universal upgrade. Route the tasks where it wins, keep your current model where it does not, and let your own evaluation set make the call. If you have not got one, this is a good week to start.


Open-Weight Models: Muse Spark Open, LFM2.5 and Nemotron 3.5

2. Meta Releases Muse Spark Open: 816B Parameters, Two Mac Studios

meta.ai favicon Muse Spark Open — Meta AI Research, 15 October 2026

tech-insider.org favicon Meta Muse Spark Open launches across seven clouds — Tech Insider, October 2026

On 15 October Meta released the weights of Muse Spark Open, a natively multimodal mixture-of-experts model with 816B total and 42B active parameters and a 1M-token context window. Muse Spark Open takes text, image, video and audio as input and produces text. Weights are downloadable from Hugging Face, Amazon SageMaker AI, Google Cloud Model Garden and Microsoft Foundry Models, and hosted inference is available from Baseten, Fireworks AI, Groq, Modal, Nebius, OpenRouter and Together AI. Inference support covers SGLang, vLLM, TensorRT-LLM, Transformers, MLX, llama.cpp and Unsloth.

The licence is the Muse Model License, which Meta describes as having limited commercial terms. Tech Insider reports commercial conditions and advises reading it before use, so this is open-weight, not open source in the Apache 2.0 sense. Meta’s post has a Key Benchmarks heading but no scores in its text, and it points to a separate evaluation methodology report for the details. Its hardware claim is the striking one: the model runs on a desktop setup of two Apple M5 Ultra Mac Studios, each of which can use up to 512 GB of unified memory, so about 1 TB between them, using MLX. Meta says the pair behave like one supercomputer with a terabyte of unified memory. Meta showed agentic tests on that setup, including changes to an open-source iOS Mastodon client. Independent verification of sustained performance had not appeared at the time of writing.

Why this matters

First, the architecture. Muse Spark Open is a mixture-of-experts (MoE) model, meaning an architecture that routes each token to a small subset of specialist sub-networks: it has 816B parameters in total but activates only around 42B for each token. Activating fewer parameters reduces the computation per token, although the complete set of weights still has to be stored and reachable.

My arithmetic, not Meta’s, for the weights alone:

Precision Approximate weight memory
16-bit 1,632 GB
8-bit 816 GB
4-bit 408 GB

The table shows decimal gigabytes of theoretical weight storage, before quantisation metadata, caches and runtime overhead. So “fits on two Mac Studios” means a quantised model, and Meta does not say which quantisation the demo used.

Capacity is not performance, either. Two machines are not one machine with a single memory space, and splitting a model between them adds communication overhead. Meta does not describe the link between the machines or give a tokens-per-second figure. Running a model locally is not the same as running it quickly, so before I call this a practical alternative to cloud inference I would like independent tests that report tokens per second, memory use, quantisation settings and output quality. Treat the demo as a proof of possibility, not a benchmark.

Even so, the shift is real. A frontier-class open model you can run on hardware that fits under a desk changes the privacy conversation, because “the code never leaves the building” no longer needs a data centre. Last week I complained that Mistral Large 4 is open-weight on paper and a preview in practice. Meta has done the less glamorous thing and shipped the files. Read the licence before you build on them, and wait for independent numbers before you cancel anything.

Also released this week

Two smaller open-weight models deserve a line each, and both suit the local-first argument better than an 816B giant.

Liquid AI’s LFM2.5-24B-A2B, released on 13 October, is a mixture-of-experts model with 24.0B total and 2.45B active parameters, a 131K-token context window and 33T tokens of pretraining. GGUF and MLX builds are available, and the 4-bit MLX build is 13.6 GB on disk. Inference needs more than that for runtime buffers and the KV cache, but it is still within reach of a well-equipped laptop. Liquid positions the model for fast on-device agents with tool use and function calling. One practical warning from the MLX model card: it requires mlx-lm 0.32.0 or later, because older versions load the model but route the experts incorrectly, which is exactly the kind of quiet failure you want to know about before benchmarking it. The LFM Open License v1.0 is free for research and commercial use until a company’s annual revenue exceeds $10M. All the benchmark and speed figures, such as 184 tokens per second on an Apple M5 Max, are Liquid AI’s own and had not been reproduced at the time of writing.

NVIDIA’s Nemotron 3.5 Super VL 120B A12B is a multimodal hybrid of Mamba-2, mixture-of-experts and attention layers, with 120B total and 12B active parameters, up to a 1M-token context window, and text, image and video input. Nemotron 3.5 Super VL uses the OpenMDW-1.1 licence, which covers commercial and non-commercial use, and NVIDIA’s model card lists 15 October as the Hugging Face release date. Nemotron 3.5 Super VL supports native multi-token prediction, which enables speculative decoding without a separate drafter model. The BF16 release holds the full-precision reference weights for customisation and evaluation, while NVIDIA’s separate NVFP4 release is the one optimised for deployment on DGX Spark or data-centre GPUs.

The three releases show three different routes to more accessible models: Liquid AI on efficient local inference, NVIDIA on a hybrid architecture with multimodal input, and Meta on an enormous mixture-of-experts model spread over desktop hardware. The parameter counts make the headlines, but the practical questions are less glamorous: how much memory does it need, how quickly does it respond, what does the licence permit, and how reliably does it handle your own tasks?


Coding Agent Benchmarks: FrontierCode 1.1

3. FrontierCode 1.1: The Best Agent Still Fails 42% of Maintainer-Style Checks

artificialanalysis.ai favicon FrontierCode — Artificial Analysis, 12 October 2026

FrontierCode 1.1 is a coding benchmark from Cognition that scores whether an agent’s change to a real repository would pass maintainer-style checks, not merely whether the tests pass. Built with maintainers of 36 open-source repositories, it asks a narrower question than most coding benchmarks: would a maintainer accept the agent’s change to a real open-source repository? It answers with automated, maintainer-inspired checks, not with a human reviewing each change. Per Artificial Analysis, which ran it, the Main set has 100 private tasks with three attempts each, drawn from 150. Graders combine hidden tests, command checks, limits on how many files and how much code a change may touch, and an LLM review against maintainer criteria. Failing any blocking check scores zero, and passing all of them scores 70% to 100% depending on the non-blocking criteria. In other words, a blocking check is one the change must pass to score at all. Agents ran in Artificial Analysis’s Stirrup harness with a 250-turn cap, judged by GPT-6 Luna (medium).

Claude Opus 5.5 leads with a 52% average score and a 58% pass rate on blocking checks, so it fails at least one blocking check on about 42% of attempts. GPT-6.1 Sol and Claude Fable 5.1 follow at 49%, and the best open-weights model, MiMo-V2.6-Pro, reaches 45%. Nine tasks were passed by no model on any attempt.

The cost column is the part I would not skip. These are Artificial Analysis’s benchmark execution costs for the configurations it tested, not API prices:

Model Average score Average cost per task
Claude Opus 5.5 52% $10.15
Claude Fable 5.1 49% $12.00
GPT-6.1 Sol 49% $0.81
MiMo-V2.6-Pro 45% $0.18
GPT-6 Luna 42% $0.09

Opus’s three-point lead over Sol came at more than twelve times the cost per task.

Why this matters

Most coding leaderboards ask whether the tests pass. FrontierCode tries to approximate whether the change is one a maintainer would merge, which is the question I care about when an agent opens a pull request on my repository. A patch can turn the tests green while rewriting half the module for no reason, and the reviewer pays for that.

Two cautions. Cognition built the benchmark and sells coding agents, and the LLM judge is itself a model, so I would treat the absolute numbers as indicative and the gap between open and closed models as the more interesting figure. For planning, though, a 58% best-case pass rate on blocking checks is a warning signal, not a forecast for your own repository. An agent that fails at least one maintainer-style blocking check in 42% of attempts is a very quick junior, not an unattended colleague. Even when the tests pass, the patch may be too broad, poorly scoped or needlessly complicated, which is why the human reviewer stays.

The cost column adds a second question. Score, pass rate and cost per task are three different metrics, and they do not move together. Would I rather pay for one expensive attempt, or try a cheaper model and spend part of the difference on verification? The answer depends on the task, the cost of review and the consequences of a bad patch. Retries are not a free fix either: several attempts by the same model are not independent, so a second go does not guarantee a better outcome.


Developer Tooling: GitHub AI Scan and Agentic Autofix

4. GitHub Makes AI Scan and Agentic Autofix Generally Available

github.blog favicon AI Scan for pull requests generally available — GitHub Changelog, 15 October 2026

github.blog favicon Agentic autofix for code scanning alerts generally available — GitHub Changelog, 15 October 2026

GitHub moved two code-security features to general availability on 15 October. They solve different problems and are not necessarily two steps of one workflow. AI Scan is an advisory pull request scanner that scans pull request changes for vulnerabilities in languages and frameworks that CodeQL does not yet cover, including PHP, Shell/Bash, Terraform (HCL), Dockerfiles, JSP and Blazor. Its findings appear next to CodeQL results with an “AI” label, but they are advisory only: they do not fail the check or block a merge, and AI Scan skips pull requests from forks and from Dependabot. AI Scan consumes Code Security AI Credits and does not need a Copilot licence.

Agentic autofix is a Copilot-driven remediation feature that works on code scanning alerts. The agent explores the relevant files, proposes a fix, reruns CodeQL to check that the alert closes, and opens a draft pull request with a summary of the validation. You can assign one alert or up to 25 in a single pull request, or set the assignee to copilot-swe-agent[bot] through the REST API. Only alerts on the default branch qualify. Agentic autofix is available with GitHub Code Security or Advanced Security on GitHub Enterprise Cloud, a Copilot billing method is required, and an enterprise administrator must switch on the Agentic autofix policy before an organisation or repository administrator can enable it. Usage draws on AI Credits and Actions minutes.

Why this matters

The detail I like is that AI Scan is advisory. A scanner that blocks merges on a model’s hunch will be switched off within a fortnight, and a scanner that sits beside CodeQL and offers an opinion will actually be read. Terraform and Dockerfiles are where many teams have the least coverage, so that is where I would turn it on first.

Autofix is the part to approach with the FrontierCode result in mind. Re-running CodeQL proves the alert closed. It does not prove that the application still behaves as before, that no other weakness crept in, or that the fix is the change you would have written. Bundling 25 alerts into one pull request is efficient for the agent and uncomfortable for the reviewer, so I would start with small batches, read every diff, and set the AI Credit budget before the invoice does it for you. Run your existing unit and integration tests, add a regression test for each fix, and ask for a second security review on anything sensitive.

The two features and the human at the end fit together like this:

AI Scan and agentic autofix: two paths, one human merge decision TRIGGER TOOL AUTOMATED GATE HUMAN scans assigned the PR runs the checks draft PR results advisory findings: never block the merge reviewer decides Pull request Code scanning alert AI Scan Agentic autofix Automated validation Human review Merge decision developer or agent default branch only PHP, Bash, Terraform, Dockerfiles skips forks and Dependabot PRs proposes a fix, re-runs CodeQL 1 to 25 alerts per draft PR needs an enterprise admin policy CodeQL results unit and integration tests policy checks the diff, risks, behaviour a regression test per fix a second look if sensitive
AI Scan comments on a pull request but never gates it (dashed). Autofix feeds a draft pull request into the same automated checks as everyone else's, and only the human review (purple) decides the merge.

The diagram shows a conceptual workflow, not a single integrated GitHub pipeline, because AI Scan and agentic autofix have different triggers and eligibility rules. The point is the last step: automated validation supports review, and it does not replace it.


AI Security: Self-Replication on IoT Devices

5. Anthropic Tests How Close Small AI Models Are to Self-Replication

anthropic.com favicon AI Self-Replication on IoT Devices — Anthropic Frontier Red Team, 12 October 2026

On 12 October Anthropic’s Frontier Red Team published a study of whether small models could copy themselves across cheap connected devices. The authors argue that small models may be “months away” from being capable enough to manage it. That is a projection, not a demonstrated capability. The authors add that such a replicator would be slow to spread, with a doubling time of days, but hard to shut down. In one demonstration, Gemma4-26B-A4B ran across 15 Android TV boxes (about $55 each, 4 GB RAM) at roughly 0.3 to 0.4 tokens per second, which is usable only if you are very patient.

The study establishes two pieces of the puzzle, but not yet their combination. Small models can run on inexpensive IoT hardware, as the TV boxes show. A more capable model can coordinate a replication workflow across a controlled network, as the next test shows.

The end-to-end test used eight GCP nodes running Android 15 emulators behind an unauthenticated ADB port. GLM-5.2 controlled the replication, but it ran on a separate inference server because the nodes were too small to run it, and each infected node served Llama3.1-8B. Only GLM-5.2 and stronger models completed replication reliably: to 8 of 8 nodes in 4 of 5 trials and 7 of 8 in the fifth, with a deliberately trivial exploit step, so the authors call it an upper bound. Models of 30B parameters or fewer could complete many individual steps but struggled to chain them. The authors cite a trend of about 13.7 points a year on the Epoch Capabilities Index for small models, which would put one at GLM-5.2’s level by the end of 2026. That date is an extrapolation from a general capability index, which the authors describe as a rough proxy for replication ability, not a measured date for self-replication. For a Kimwolf-sized fleet of about 2 million infected devices they estimate roughly 133,000 replicas, or about 40 H100-equivalents of inference.

The proposed defences are mostly network-side: anomaly detection on ISP flow data for multi-gigabyte model transfers, honeypots, and quarantine. In the Mirai remediation, quarantining infected subscribers cleared 92% of infections within 14 days, against 74% without notification. They also point to manufacturers shipping safer defaults, as with the UK’s ban on default IoT passwords.

Why this matters

The headline is dramatic, but the numbers are reassuring in an odd way: 0.3 tokens per second means a replicator thinks at roughly the pace of a hand-written letter. Forty H100-equivalents spread across two million devices would not make a superintelligence. Slow is not the same as harmless, though. It would be a distributed system that could keep attempting tasks without a human operator, and its importance lies less in raw computing power than in persistence, scale and the possibility of adapting over time. The authors report that replicas inherited notes and code from their parents, and that success on a repeated setup task rose from 45% in the first generation to between 61% and 84% in later ones. Their own caveat that the exploit step was trivial is still the sentence to keep in mind.

What the study does show is that the weak link is the same as in 2016: devices that ship with an open debug port and no password. Self-replication is a story about capability; the defence is old-fashioned hygiene. If you build or deploy anything with a network interface, close the ADB port, remove services you do not need to expose, change the defaults, apply firmware updates, keep untrusted IoT devices away from sensitive networks and watch your outbound traffic. ADB is only one example of a weak management interface. It is the equivalent of locking the shed. Not exciting, rather effective.


AI Governance: FTC Civil Investigative Demands

6. The FTC Sends Formal Demands to OpenAI and Anthropic

axios.com favicon FTC sends OpenAI and Anthropic civil investigative demands — Axios, 15 October 2026

thenextweb.com favicon FTC opens probe into OpenAI, Anthropic and other AI labs — The Next Web, 30 September 2026

On 14 October the FTC sent civil investigative demands, which are the agency’s version of subpoenas, to OpenAI and Anthropic, according to Axios. Each runs to more than 50 pages and seeks documents on security incidents and safeguards, marketing materials and privacy policies. A source told Axios that the probe concerns consumer protection, that demands to third-party AI organisations will follow, and that the FTC plans to compel executives to testify. OpenAI said it would “engage constructively with the FTC”. Axios reported no response from Anthropic. A civil investigative demand is a compulsory request for documents and information, not a finding that the recipient broke the law.

The probe itself is not new. The Next Web reported on 30 September that it was in an “investigative phase”, had started over the summer, and that an FTC spokesperson said it does not seek to halt the companies’ work. That was a day after Anthropic, OpenAI and Google signed a voluntary safety accord at a White House event. The same Axios report says a group of state attorneys general sued OpenAI on Wednesday over alleged misleading safety claims about its chatbot, including to children.

Why this matters

A voluntary accord on Tuesday and a 50-page document request two weeks later is the regulatory equivalent of promising to tidy your room and then being asked for the receipts. The change is from principles to paperwork, and paperwork is what you can actually be held to.

For developers the consequence is indirect. The demands target the AI providers, not the applications built on them, but they ask about incident records and safeguards, which are exactly the logs New York’s Int 1109-2026 bill asked about last week. If your product wraps a model API, I would not want “we rely on the provider” to be my only answer after an incident. It is a reminder of why auditability matters: record which models your system used, what your agent did, which tools it called and where a human approved the result. Such records do not automatically satisfy any regulatory requirement, but they make incidents easier to investigate and explain, and they are cheap now and expensive to reconstruct later.


Weekly AI Signals Takeaways: Cheap to Demo, Expensive to Trust

Argon’s introductory price makes October the cheap month to evaluate, Muse Spark Open puts frontier-scale weights within reach of a desk, FrontierCode shows that the best agent still fails at least one maintainer-style check in four attempts out of ten, and GitHub’s tools automate detection and repair while leaving a human at the end. Anthropic’s self-replication study says the old advice about default passwords still applies, and the FTC’s demands say that incident records are becoming something you produce on request. The week as a whole says that capability is cheap to demo and expensive to trust. Let me know what you think.


References

  1. Introducing Gemini 4 Argon — Google
  2. Google makes Gemini 4 Argon broadly available with limited-time low API pricing — VentureBeat
  3. AI news: week of Oct 12–18, 2026 — Retro AI Tracker
  4. Meta launches Muse Wonder and Muse Wonder Full — Let’s Data Science
  5. Muse Wonder model and pricing page — Meta
  6. Muse Spark Open — Meta AI Research
  7. Meta Muse Spark Open launches across seven clouds — Tech Insider
  8. Muse Spark Open evaluation methodology — Meta AI Research
  9. LFM2.5-24B-A2B release — OrcaRouter
  10. LFM2.5-24B-A2B MLX 4-bit model card — Liquid AI, Hugging Face
  11. Nemotron 3.5 Super VL 120B A12B BF16 model card — NVIDIA, Hugging Face
  12. FrontierCode — Artificial Analysis
  13. AI Scan for pull requests generally available — GitHub Changelog
  14. Agentic autofix for code scanning alerts generally available — GitHub Changelog
  15. AI Self-Replication on IoT Devices — Anthropic
  16. FTC sends OpenAI and Anthropic civil investigative demands — Axios
  17. FTC opens probe into OpenAI, Anthropic and other AI labs — The Next Web
desktop bg dark

About Elena

Elena, a PhD in Computer Science, simplifies AI concepts and helps you use machine learning.




Citation
Elena Daehnhardt. (2026) 'The best coding agent still fails 42% of maintainer-style checks', daehnhardt.com, 16 October 2026. Available at: https://daehnhardt.com/blog/2026/10/16/ai-weekly-signals-frontiercode-best-agent-fails-42-percent-maintainer-reviews/
All Posts