GPT-5.6 Sol Impressed Me -- Then It Wouldn't Stop

Dev Leader Weekly 153

Dev Leader

Aug 17, 2026

TL; DR:

Better code does not guarantee efficiency
Review loops need stopping conditions
Scope control still belongs to you
YouTube would not let me schedule a stream so... no stream 😦

GPT-5.6 Sol Impressed Me -- Then It Wouldn’t Stop

One of the weirdest parts of using a stronger coding model is that the failures can start looking useful.

I had been using the Opus models pretty heavily, especially because the large context window was genuinely impressive. Then I switched back to a GPT model with GPT-5.6 Sol, and my first reaction was basically: holy crap, this thing is good.

I have written before about how different models can change the shape of a Copilot workflow. This was one of the clearest examples I have felt in practice. Sol was not just producing code I liked. It was noticing things around the requested work that other models had missed.

That sounds great, right?

It was. Mostly.

You can check out my full thoughts on this in the video below:

Going Off The Rails Can Still Produce Gold

I was trying to explain model differences to my wife, who uses ChatGPT but does not spend her time comparing coding models like some of us do.

The example I gave her was scope drift.

You ask a coding model to change one thing. Then you review the diff and discover that it touched files you never mentioned. With some models, I have looked at those changes and thought, “What were you even doing over here?” The answer is usually some version of: yeah, that was unnecessary.

My early experience with Sol was different.

It touched unrelated tests, but the tests were actually wrong. It found behavior that technically passed the existing test while still being incorrect. It caught bugs in test code and bugs in underlying features that were outside the narrow task I had assigned.

That is a pretty incredible feeling. The model appears to wander, you prepare yourself for cleanup, and then you realize it found something real.

But there is an important distinction here: useful scope drift is still scope drift.

When I am using GitHub Copilot CLI on a real C# codebase, I need to understand what changed, why it changed, and whether that extra work belongs in the same delivery. Finding a bug is valuable. Quietly folding every discovered bug into one feature is not automatically the right move.

A Better Result Can Still Be A Bad Workflow

This is where my opinion became more divided.

I noticed some Copilot sessions taking a really long time. Work that I expected to take maybe thirty minutes would keep going for several hours. I would check the session assuming something had broken in the harness or a sub-agent had died.

Instead, it was still working.

It had run a review. Then it was fixing review findings. Then it was reviewing the fixes. Then another perspective found something else. On one feature, I saw the pull request appear after roughly five hours of agent work, even though the feature itself was not a monumental coding effort.

Was the output good? Often, yes.

Was five hours the right tradeoff? I am not convinced.

We talk a lot about model quality, but delivery has more dimensions than whether the final code passes tests:

How long did the work take?
How much unrelated code changed?
How expensive was the review loop?
How much attention did I need to spend supervising it?
Could we have shipped the intended scope much earlier?

A model can produce better code and still create a worse overall workflow if nobody tells it when to stop.

The Review Loop Can Convince Itself

I like review agents. I like rubber-duck agents. I like asking for another perspective before I trust a change.

I also think they can create a nasty feedback loop.

AI is extremely good at sounding confident. Most of us have read an answer and thought, “That sounds believable,” even when we did not have the evidence to prove it. The same thing can happen between agents. A reviewer confidently proposes another improvement, the primary agent accepts the premise, and suddenly the task has a whole new branch of work.

This is not an argument against custom agents and skills in Copilot CLI. I use them constantly. It is an argument for making their boundaries explicit.

One of my sessions started with a few changes to an evaluation harness. When I checked it later, the agent was hand-writing a custom Markdown parser.

What?

That was the software equivalent of walking into your house and finding animals casually having a conversation in the living room. Nobody is acting like anything is wrong, but you know this is absolutely not where the evening was supposed to go.

The parser was not the assignment. The agent had reasoned its way into creating more work, and every additional change gave the next review pass more surface area to inspect.

Better review is not infinite review.

Prototype Mode And Hardening Mode Need Different Contracts

When I am building something, I am usually operating in one of two modes.

The first is prototyping. I want to know whether an idea is feasible. Can I connect these pieces? Does this service do what I need? Is the core experience even possible? The code may be throwaway, and that is fine.

The second is hardening. The feasibility question is already answered. Now I need regression tests, repeatable behavior, operational reliability, and confidence that I can keep changing the system without breaking it.

Those modes need different instructions.

If I am prototyping, an agent that spends hours polishing abstractions has missed the point. If I am hardening, an agent that proves the happy path once and declares victory has also missed the point.

Actionable Tip: tell the agent which mode it is in before describing the task.

For a prototype, define the question you need answered and explicitly deprioritize polish. For hardening work, name the behaviors that need regression coverage and define which adjacent changes are allowed. If the agent discovers something outside that boundary, have it report the finding instead of automatically implementing it.

That one distinction can prevent a lot of “How did we end up building this?” moments.

A Million Tokens Does Not Remove Context Management

The other issue I have seen is around the context window.

GPT-5.6 Sol gives me a huge amount of context to work with, but in some longer Copilot sessions I have still hit a point where the session effectively stops making progress. In my case, this has happened with plenty of context apparently remaining, often somewhere around the final quarter to third of the window.

The frustrating part is that compaction can fail at exactly the moment I need it.

The workaround that helped me during this time was restarting the Copilot process for the same session with /restart, then running /compact. That has let the work continue, but it is not behavior I have seen in quite the same way with other models.

Is that the model? The harness? The interaction between them? I had no idea -- but I do know that it was a pain in the butt.

That uncertainty matters. I am describing what I have observed in my sessions, not declaring a universal GPT-5.6 Sol limitation. And for what it’s worth, Copilot CLI seemed to have been patched for this since I initially was ranting about it on Code Commute.

The broader lesson still holds: session and context management remain part of agentic development. A larger context window delays some problems. It does not eliminate the need for checkpoints, compaction, and clean task boundaries.

What I Am Changing In My Workflow

I am still impressed with the model. I am not switching away because a few sessions ran too long.

But I am changing how I supervise the work.

Actionable Tip: give every agentic task a scope budget and a stopping rule.

Here is what that looks like for me:

Define the delivery boundary. Name the behavior, files, or subsystem that belongs in the task.
Separate discoveries from implementation. If the agent finds an unrelated bug, capture it unless it blocks the requested change.
Bound review loops. One useful review pass is not permission to keep reopening the design forever.
Create checkpoints. Ask for a status summary before a long-running session disappears into another branch of work.
Watch elapsed time. If a modest feature is still running hours later, inspect the current plan instead of assuming more time means more quality.

Permissions and tool restrictions are still important, and I have a full breakdown of what Copilot CLI can and cannot touch. But permissions only answer whether an agent can do something. They do not answer whether that work belongs in the current task.

That part still belongs to us.

The Model Can Be Impressive And Still Need Supervision

My first impression of GPT-5.6 Sol is still extremely positive.

It caught real problems that I did not ask it to find. It handled code with a level of awareness that surprised me. It made “going off the rails” look productive, which is not something I expected to say.

At the same time, I have watched it spend hours following review loops, expand a small task into a custom parser, and hit context problems before I expected them.

Both things can be true.

The model can be excellent, and the workflow around it can still need tuning. The output can be correct, and the process can still be too slow. The extra finding can be valuable, and it can still belong in a separate issue.

I do not want an agent that blindly stops the second it notices something outside the prompt. That would throw away one of the most valuable parts of a strong model. I want an agent that notices the problem, explains it, and lets the delivery boundary remain intentional.

That is the balance I am working toward: more capability without surrendering scope control.

If you are using GPT-5.6 Sol or another model for long-running coding work, pay attention to more than the final diff. Watch the route it took to get there. That is where the real workflow problems tend to show up.

Dev Leader Weekly

Discussion about this post

Ready for more?