GPT 5.6 Sol is the best "vision" model OpenAI ever released

8/17/2026 at 1:47:59 PM

The summary "There are still clear limits. Gemini 3.5 Flash remains a better practical choice [than GPT 5.6 Sol] for high-volume detection and counting in our benchmark, especially at its price." seems rather understated !

GPT 5.6 Sol was outperformed on all benchmarks by Gemini 3.5 Flash, apart from a single exception (OCR) where Fable was the winner.

Gemini 3.5 Flash not only outperformed GPT 5.6 Sol, but did so at 1/3 of the cost.

by HarHarVeryFunny

8/17/2026 at 3:33:04 PM

Hi, I’m the author of this blog post. I wrote it about 4 weeks ago, and the VLM world is moving so fast that it’s already kinda outdated. I think Gemini 3.7 Flash might be a better choice now, especially when you factor in the price.

Here’s a comparison of the best low-cost models I put together last week. What’s crazy is that Gemini 3.7 Flash is now 50% off on OpenRouter, and this chart doesn’t even account for that discount. https://x.com/skalskip92/status/2088032652301304121?s=20

by SkalskiP

8/17/2026 at 4:12:05 PM

Curious why you didn't try Gemini 3 pro? That is the model I've been using for OCR entry of handwritten datasheets (JPGS of datasheets, structured JSON output). At my scale, the cost of 3 pro is basically not an issue, but if there are improvements in quality, I'd definitely be willing to explore other models

by MostlyStable

8/17/2026 at 6:53:51 PM

In my experience starting with Gemini 2.5 Pro, moving to 3 and 3.1, 3.5 Flash, 3.6 Flash, and finally 3.7 Flash, 3.7 Flash is just as good if not better than 3 especially on high resolution mode (same token count per page as 3.1).

I run complicated, messy PDFs through these models. 2.5 Pro required a lot of kludgy hacks to get it to fully "see," but from 3.1 pro on I've removed many of them and haven't spotted problems.

3.7 Flash scores better than 3.1 pro on most benchmarks, leading me to believe that even if your OCR requires reasoning to interpret text or data, 3.7 Flash is probably going to be better.

by gdudeman

8/17/2026 at 6:00:55 PM

3 Pro is quickly approaching one year old. There's almost no reason to benchmark it, especially since a new version of Gemini Pro was supposed to be released mid 2026 and hasn't seen the light of day.

by bastawhiz

8/17/2026 at 6:23:57 PM

That would make sense if we already knew that, for these kinds of tasks it was significantly worse. The tests that I'm aware of for these tasks show it as still performing near the top.

by MostlyStable

8/17/2026 at 8:05:01 PM

I think it definitely makes sense since it's still the best Google has to offer in the "pro" tier.

by tziki

8/17/2026 at 8:18:26 PM

3 and 3.1 Pro are both marked as deprecated by Google. Even if they're the best Google offers, it would be foolish to choose a model that's explicitly deprecated.

It's not a technical problem, it's a commercial one. If Google can't ship a model to replace the one they deprecated, that tells you everything you need to know about choosing a Gemini model for whatever you're trying to do.

by bastawhiz

8/17/2026 at 9:41:44 PM

3.1 Pro is not deprecated!

by heaney-555

8/17/2026 at 10:54:06 PM

https://ai.google.dev/gemini-api/docs/deprecations

That link shows 3.1 pro listed as deprecated with no replacement model.

by bastawhiz

8/18/2026 at 3:15:45 AM

No, that's the "preview" version (gemini-3.1-pro-preview) aka the beta/early version before the official release of 3.1 Pro.

by qlte

8/18/2026 at 5:30:46 PM

You are misreading that link. That's a list of all their models.

Just like 3.7 Flash, 3.1 Pro says "No shutdown date announced."

by heaney-555

8/17/2026 at 6:58:50 PM

The “pro” moniker means nothing

these models aren’t successors and barely have a common ancestor, they are independently baked in the training oven and assigned a semantic version randomly by someone trying to show initiative but not trying to do on the toes of the last guy who got promoted first

So 3 pro is outdated and will likely never exit preview

The “flash” and “lite” models are the real “pro” in colloquial ideas of fleshed out and capability, at this point.

they’re better, faster and cheaper, larger context windows keeping up with the industry and more

by yieldcrv

8/17/2026 at 8:18:55 PM

They are smaller models, and you can tell. Small models make dumb common-sense mistakes that big models never do. This is the "smell" many talk about.

by heaney-555

8/17/2026 at 8:24:56 PM

Do you have cases where you still see 3.1 pro outperforming 3.7 flash?

by sidibe

8/17/2026 at 9:38:38 PM

Yes, for complex questions of biology, physics, and analysis of anomalies.

3.7 Flash is better at coding, sure, but AI is not just for coding.

by heaney-555

8/17/2026 at 8:29:23 PM

hasn't been an issue since 3.5 for me, what have you seen, say, in the last two months

by yieldcrv

8/17/2026 at 9:39:02 PM

For complex questions of biology, physics, and analysis of anomalies, 3.1 Pro is still better than 3.7 Flash for me.

3.7 Flash is better at coding, sure, but AI is not just for coding.

by heaney-555

8/18/2026 at 8:59:24 AM

[dead]

by ppaanngggg

8/17/2026 at 7:41:44 PM

What about Gemma ?

by Melatonic

8/18/2026 at 9:01:39 AM

[flagged]

by runtime_lens

8/17/2026 at 6:23:27 PM

Gemini tops their vision evals [0] by a mile, with 4/5 top spots going to variants of it. Qwen is the only other contender, likely due to how good it is for object detection, where it crushes the competition [1].

[0] https://playground.roboflow.com/evals

[1] https://playground.roboflow.com/evals/object-detection

by ImageXav

8/17/2026 at 1:59:54 PM

Yeah I was thinking about giving Luna a go with my PDF data extraction, but I think I‘ll stay on Gemini. It does a very good job.

by MrBuddyCasino

8/17/2026 at 2:29:25 PM

Gemini is still my top choice within production software for typical data extraction from unstructured data. Gemini Flash Lite feels like a cheat code for speed, and it's really cheap.

Some other Chinese models are also fast and cheap, but a harder sell in a U.S. production environment.

by bicx

8/17/2026 at 6:05:50 PM

Speaking from experience here, flash lite models have amazing price, speed, and perform far above their size, but are susceptible to very bad instruction following and recall when either complexity or context size inch up. They’ll just forget to apply your instructions to portions of the input, and repeat parts of the input that should be returned verbatim as direct quotes but with subtle changes (breaking urls, for example).

by ComputerGuru

8/17/2026 at 6:44:34 PM

Yes you have to continuously tune the prompts ever so subtly. 3.5 is a lot better than than 3.1 tho.

Important to remember that json schema instructions take precedence over the normal prompt, so move as much into property descriptions as possible.

by MrBuddyCasino

8/17/2026 at 7:35:12 PM

This was 3.5 flash lite, actually, and after prompt tuning. It was very clearly an issue that correlated with input (JSON array) size, the more elements in the batch, the higher the error rate.

3.0 flash (not lite) handled it like a champ though, fwiw.

by ComputerGuru

8/17/2026 at 3:15:20 PM

Yeah Gemini 3.5 Flash Lite is really good. Which Chinese models can you recommend?

by MrBuddyCasino

8/17/2026 at 3:38:26 PM

Hi, I’m the author of this blog. It depends on how strong of a model you need, but in general, Qwen is easily the best among the Chinese models right now.

Over the last two weeks, Qwen released two new models. Qwen3.8-Max is totally insane, but it’s only available through the Alibaba Cloud API. I wrote a similar blog covering Qwen3.8-Max: [https://blog.roboflow.com/qwen3-8-max/](https://blog.roboflow.com/qwen3-8-max/)

If you’re looking for something you can run locally, Qwen3.8-27B might be a great option. On Friday, I did a quick comparison between Qwen3.8-Max and Qwen3.8-27B: [https://x.com/skalskip92/status/2088411215441621469?s=20](https://x.com/skalskip92/status/2088411215441621469?s=20)

by SkalskiP

8/17/2026 at 5:03:42 PM

Googles local gemma models which target roughly the same parameter count range, are known for being a lot better at vision tasks than qwen, no idea if 3.8 has changed that though

by kanemcgrath

8/17/2026 at 6:05:56 PM

Really? Gemma4-31B should be better than Qwen3.8-27B? I'm happy to test that.

by SkalskiP

8/17/2026 at 3:30:39 PM

I've been using Qwen3.5-9B, hosted locally for PDF data extraction and it performs pretty well when extracting data from tables and infographics

by b345

8/17/2026 at 2:58:49 PM

[dead]

by msp26

8/17/2026 at 4:08:06 PM

Gemini is honestly an excellent LLM with many capability strengths.

For example, 3.7 Flash is #1 on MMLU Pro and AA’s agentic spreadsheets/docs benchmark, etc. Yes, beating Fable.

Agentic coding is only one dimension.

by dannyw

8/17/2026 at 4:56:35 PM

Anecdotally, Gemini Flash is the leader for a particular use case of mine and has been since at least version 2.5. But now there's also Luna as the first real competitor thanks to the price cut.

My worry is that this is a zero-sum game and when Gemini catches up on coding, it'll regress to the mean in other areas.

by fau

8/17/2026 at 4:20:57 PM

thats so helpful - tysm

by Damjanski

8/17/2026 at 12:49:59 PM

Anecdotal, opinion:

Gpt is really good in vision stuff, or at least their MoE seems to be really cohesive. From my experience Claude models can be really good at language but the moment they need to look at a picture and decide why the design is not good what parts need improvement it degrades a lot. My easiest benchmark is giving them a screenshot of a feature in my app and tell it "identify non-normative UI blocks and improve readability and consistency". Sol does a great job at re-structuring the page into composable units that build upon each other and the general looks and feels of the app. Claude tends to over-focus one one part while completely forgetting about the rest or the cohesion as a whole.

by weli

8/17/2026 at 1:05:11 PM

Assessing the subjective quality of a thing is in my experience one of the worst ways to use any LLM.

by velcrovan

8/17/2026 at 5:58:24 PM

There's a lot of objective principles and decisions that go into subjective quality; if you don't know the field well, asking LLM for assessment is a good way to discover all that.

by TeMPOraL

8/17/2026 at 1:09:26 PM

anthropic frontend-design skill does a great job with it.

by rib3ye

8/17/2026 at 1:24:53 PM

Have you actually read the frontend design skill? It’s placebo at best. Very short and barely focused on design: https://github.com/anthropics/skills/blob/main/skills/fronte...

by rafram

8/17/2026 at 2:29:59 PM

Have you actually tried using it?

by rib3ye

8/17/2026 at 3:41:24 PM

Of course. It’s OK, but it tends to generate very cliched “AI” UIs with little originality. Despite the skill spending a lot of time coaching the model into avoiding that!

by rafram

8/17/2026 at 11:13:49 PM

> UIs with little originality

Sounds like the kind of UI I like. (Take me back to Windows XP...)

by akoboldfrying

8/18/2026 at 5:59:02 PM

I mean they all look like generic, annoying SaaS landing pages/overwrought dashboards, not that they’re simple and functional.

by rafram

8/17/2026 at 1:37:31 PM

What an annoying time for GitHub to go down.

by MallocVoidstar

8/17/2026 at 3:33:31 PM

Like every time

by KeplerBoy

8/17/2026 at 1:32:27 PM

My exposure to Claude-produced UIs is limited, but I have started to notice certain design trends they tend to have in-common, which might be becoming hallmarks of AI-produced UIs - the same way we've started noticing the clichés of low-effort LLM-generated text.

FWIW, the summary-description[1] of "frontend-design"[2] gives me a few things to pick at:

> create polished code

Methinks only if you're using it with a very popular framework like React. What happens if you ask Claude to make the UI in WinForms or MFC?

> high-impact animations

That's bad UX 101 right there: animations in a UI exist as an affordance to the user, and never for its own sake (e.g. macOS's "genie" animation when you minimize a window to the dock exists so the user knows where they can restore the window from). The only people who actually want "high impact animations" in software are salespeople who want something for demo purposes.

> generic system fonts, predictable purple gradients, and cookie-cutter components.

This screams wanting to be different for the sake of standing-out, not because it results in a better software product; users benefit when their software fits-in with platform conventions: if you refuse to use a stock checkbox <input> or <select> drop-down and instead use your own entirely custom component solely for aesthetic reasons then you are producing worse software. There's nothing wrong with system-fonts, but your site will look ugly after your third-party font-host CDN shuts-down and turns into a walking CSRF factory.

> thoughtful typography with unexpected font pairings

The above fragment set my alarm-bells off. Yikes.

> scroll-triggered interactions

Not every web-page should be an Apple.com product brochure page. This is also a fantastic way to make your webpage horribly inaccessible.

------

The SKILL.md itself[3] grinds my gears too:

> Approach this as the design lead at a small studio known for giving every client a visual identity that could not be mistaken for anyone else's.

Claude has no way of knowing what designs are actually unique or not...

> For web designs, the hero is a thesis. Open with the most characteristic thing in the subject's world, in whatever form makes sense for it: a headline, an image, an animation, a live demo, an interactive moment

...this is exactly what everyone else's web-pages look like!

> For calibration: AI-generated design right now clusters around three looks: (1) a warm cream background (near #F4F1EA) with a high-contrast serif display and a terracotta accent; (2) a near-black background with a single bright acid-green or vermilion accent; (3) a broadsheet-style layout with hairline rules, zero border-radius, and dense newspaper-like columns

...I called this out weeks ago[4], lol.

and I could go on. This is all quite painful to read.

------

[1] https://claude.com/plugins/frontend-design

[2] https://github.com/anthropics/claude-plugins-official/tree/m...

[3] https://github.com/anthropics/claude-plugins-official/blob/2...

[4] https://news.ycombinator.com/item?id=49187385

by DaiPlusPlus

8/17/2026 at 7:04:33 PM

i'd say this is something that has gotten orders of magnitude better with recent releases than it used to be, fwiw

by keeganpoppen

8/18/2026 at 12:37:32 PM

When asked to produce a thing, the output has a much better baseline of quality. But when you ask it to evaluate the quality of a thing, how do you evaluate the quality of its response, which is necessarily subjective and not quantifiable? How can a thing which has no experience of friction be said to subjectively evaluate quality?

Either you are yourself already a better judge of the thing’s quality, in which case the response can be of no use to you, or you are a poor judge of the thing’s quality, in which case you will be blind to the flaws in the synthesized opinion handed back to you.

by velcrovan

8/17/2026 at 3:54:42 PM

Hi! I’m the author of this blog. GPT-5.6 is much better at vision than previous GPT versions, but it’s still much weaker than Gemini 3.5 Flash or Gemini 3.7 Flash, which was released last week. One interesting approach is to use Gemini through a tool call.

by SkalskiP

8/17/2026 at 7:46:57 PM

This response is not relevant to the this comment

by Tactical45

8/17/2026 at 1:07:50 PM

What is a "non-normative UI block"?

by DaiPlusPlus

8/17/2026 at 1:16:26 PM

Segments of the UI that don't conform to any other existing established design or conventions

by weli

8/17/2026 at 1:10:32 PM

areas that look weird

by lelandfe

8/17/2026 at 1:02:15 PM

Penny sample shown looks like failed EXIF orientation registered by the model/harness. The coins are correctly marked, it's rotated 90 degrees.

by evrimoztamur

8/17/2026 at 3:48:19 PM

Hi! I’m the author of this blog. I had the same intuition, but together with the OpenAI team we figured out that the issue was image resolution. GPT-5.6 doesn’t handle large images well.

by SkalskiP

8/17/2026 at 9:32:42 PM

OpenAI team sounds like they've misidentified the root cause for this particular case then.

by evrimoztamur

8/18/2026 at 1:59:56 AM

Haha, ya at least to some degree, those boxes are in the right position, but rotated.

by DustinBrett

8/18/2026 at 1:59:16 AM

Good call out, I noticed the same rotation issue but pointing at EXIF data sounds about right.

by DustinBrett

8/17/2026 at 1:58:20 PM

It is funny to me seeing Sol used for what a "traditional" AI model can do already (counting pills).

We have vision models for our pharmacy and I could never imagine taking the latency hit to use a Sol in our robotics, it would be likely 25-50x slower.

by bearjaws

8/17/2026 at 3:59:05 PM

Hi! I’m the author of this blog.

I’m evaluating these VLMs to figure out which ones are good enough to auto-annotate my data, so I can fine-tune my detector.

I wrote a bit more about this here: https://x.com/skalskip92/status/2080334344061694429?s=20

by SkalskiP

8/17/2026 at 4:12:47 PM

Did you evaluate any that could be self-hosted (or at least ow models), if so which one is the best you seen?

by mhaberl

8/17/2026 at 6:07:06 PM

Take a look here: https://playground.roboflow.com/evals. We have few ~30B.

by SkalskiP

8/17/2026 at 7:06:53 PM

Thank you!

It seems Qwen is kicking ass, and Fable made me laugh when I saw it all alone on the far right of the graph :))

by mhaberl

8/17/2026 at 5:03:38 PM

Agreed, this like asking a chainsaw to carve a wooden spoon. Impressive it can, but definitely not the right tech to scale.

LLM needs to setup an image classifier to use as a tool call.

by kooi

8/17/2026 at 7:19:07 PM

Building a dataset is expensive, manual annotation is expensive. Datasets don't exist in every niche.

I remember around 2013-15 people were scoffing at uses of deep learning CNNs for various things, because why don't you just use an SVM on HOG features? Or face detection is solved, just use Viola-Jones.

What if you give the benefit of doubt and assume the author knows about alternatives and uses VLMs for their strengths? They use it to auto-annotate training data for regular deep learning models.

by bonoboTP

8/17/2026 at 5:08:56 PM

Now maybe, but the gap is closing.

by ramblerman

8/17/2026 at 2:19:40 PM

How are we supposed to pay off all these data centers and chips if you’re not willing to burn a microwave burrito worth of electricity for each prescription? Think of the benchmarks

by repeekad

8/17/2026 at 1:38:18 PM

Ironically, the pill counting example selected to showcase "the best vision model" can be easily solved with OpenCV template matching, a technology created 25 years ago.

by mv4

8/17/2026 at 1:46:23 PM

I'm assuming you mean that this tech became available in OpenCV 25 years ago, but as it turns out, the underlying tech can be traced back much further, at least as far as 1977! :)

https://ieeexplore.ieee.org/document/1674847 G. J. Vanderbrug and A. Rosenfeld, “Two-Stage Template Matching,” IEEE Transactions on Computers, Vol. C-26, No. 4, pp. 384–393, April 1977. DOI: 10.1109/TC.1977.1674847

by maxime_cb

8/17/2026 at 3:29:28 PM

Exactly my point. Template rotation is a trivial operation as well.

by mv4

8/17/2026 at 1:49:13 PM

The point is that it's general. It can do this task and many other tasks and it doesn't need custom development like OpenCV does. Of course if you only want to count pills and you want it to be cheap/fast you're still better off using OpenCV.

by lebek

8/19/2026 at 8:28:57 PM

Suppose now we need to test Sol vs OpenCV vs Sol implementing OpenCV

by glaslong

8/17/2026 at 3:02:45 PM

Basic Template matching has severe limitations around scaling, rotation, and perspective. In my experience it greatly underperforms compared to deep network object detectors. My experience- and I imagine others have different experiences- is that SIFT techniques also fail pretty badly with noisy data.

by dekhn

8/17/2026 at 3:36:59 PM

That's correct, and I was specifically referring to the example chosen - where scale and perspective are known. Template rotation is relatively easy as well - but partial obstructions would pose a problem.

Another application where template matching would work brilliantly? Car counting in parking lots using satellite imagery.

Source: I did this [1] using OpenCV and template matching. Outperformed "Cars Overhead with Context" models.

https://abcnews.com/International/satellite-data-suggests-co...

by mv4

8/17/2026 at 2:46:30 PM

I'm sure a typical frontier model would also be happy to write that opencv script for you, and it would do it well.

That is certainly pretty far from what was possible 25 years ago.

by geysersam

8/17/2026 at 3:39:00 PM

It 5..10 lines of code. :)

by mv4

8/17/2026 at 12:59:06 PM

In the third vision bench result, Sol is 100% correct but the expected has 1 error. Seems like an oversight.

In the next bench, Sol looks like it’s correct again but the bboxes are rotated 90 degrees for some reason.

by kzrdude

8/17/2026 at 1:55:39 PM

Seems to be due to the detection area being not fully accurate. Green vs red shows the difference between actual and detected

by defrim

8/17/2026 at 4:22:24 PM

There is an extra green square where no egg is present, so it's a false positive in the expected.

by kzrdude

8/17/2026 at 3:51:14 PM

Hi! I’m the author of this blog and benchmark. You’re right. I’ll fix it in the ground-truth dataset. Thanks for pointing it out.

by SkalskiP

8/17/2026 at 4:21:53 PM

Great, happy that it was helpful

by kzrdude

8/17/2026 at 2:14:48 PM

Gemini 3 Flash should really be included in this comparison. Or at least 3.7. In most of my testing, 3.5 and 3.6 were both a downgrade in terms of vision capabilities, relative to 3, and at a much higher cost. 3.7 is slightly better than 3, finally.

by fpgaminer

8/17/2026 at 6:05:25 PM

3 Flash never left "preview" status and is listed as deprecated.

https://ai.google.dev/gemini-api/docs/deprecations

by bastawhiz

8/17/2026 at 9:10:49 PM

but 3.7 flash is expensive for img inputs no ?

by zuzululu

8/17/2026 at 10:41:43 PM

As usual for something so simple, Google's docs seem unclear: https://ai.google.dev/gemini-api/docs/pricing

For 3, pricing for image tokens was the same as text tokens. Since they don't indicate a difference on 3.7, I would assume the same holds. And as far as I know the number of image tokens is the same for both (depending on the detail level you pick, but it's generally around 1k per image).

So they're about the same, 3.7 is slightly more expensive. At least until the end of the year (when they raise 3.7's pricing).

Anyway, my point was that 3.5 tended to have worse performance and significantly higher costs. 3 and 3.7 are both better and cheaper than 3.5.

by fpgaminer

8/18/2026 at 3:14:27 AM

mystery to me is how the image tokens are calculated? 1MB is 1000 tokens ?

by zuzululu

8/17/2026 at 1:15:52 PM

I agree. It did very well on an extremely challenging task.

I asked it to recognize and draw the very faint reflection of what I was wearing, visible in only a tiny black part of a very brightly lit poster behind glass.

In addition, the poster itself also happened to contain similar clothing.

You can see the reference images and its output in my writeup here: https://medium.com/@rviragh/gpt-5-6-sol-very-good-image-reco...

While a human can focus on the reflection easily, this is an enormous challenge for a vision model. It's very impressive.

by logicallee

8/17/2026 at 12:55:50 PM

For the last 2 weeks I've been trying to get Codex to "outpaint" a wonderful image it generated as placeholder art for a level background.

After I increased the game's resolution, I asked it to increase the image's size while keeping the same scale and existing content, and gosh, it constantly keeps getting something wrong no matter what I tell it, even on Sol Max with the $100 Pro subscription.

An organically-grown meat-based pixel-artist could have recreated the image and more within 2-3 days, in exchange for food and shelter.

by Razengan

8/17/2026 at 1:35:54 PM

I'm unsure why you're using an LLM to generate images. Don't we already have models (some made by the same company) that do this?

by dev_hugepages

8/17/2026 at 1:10:16 PM

> it constantly keeps getting something wrong no matter what I tell it

This 100%

by sscaryterry

8/17/2026 at 12:59:10 PM

did you try segmenting it first?

by thatcat

8/17/2026 at 2:26:38 PM

At first I intended to create a tileset and asked it for several variations of what a hypothetical tilemap created from the planned tileset would look like.

The previews it generated were amazing but wouldn't really be possible as a grid-based tilemap, with lots of clusters and overlaps of elements of varying sizes.

So I just decided to use the preview as a static scrolling background, but it's been a pain to get it to add more content around the edges that still tiles with the existing image at the same scale.

by Razengan

8/17/2026 at 5:18:11 PM

Vision is still embarrassingly bad.

ChatGPT Pro with GPT 5.6-sol: https://chatgpt.com/share/6a834217-ca8c-83e8-a8e8-45d5b8797b...

The puzzle: https://activityvillage-files.s3.eu-west-2.amazonaws.com/s3f...

by dllu

8/17/2026 at 6:06:52 PM

The second answer is far more revealing than the first:

OP:

> do you think you did a good job there

ChatGPT:

> I spent 15 minutes, emitted several fake-sounding “tracing the puzzle” progress updates, and then gave a confident permutation without showing that I had actually followed the lines correctly. It reads much more like I guessed than solved it. The only part I did well was obeying the “no Python or tools” instruction.

My observations:

1) Sarcastic tone suggests pre-prompting, or frequent (and therefore stored in memories) denigration of the model in past conversations. I'm leaning the former - it sounds like it was instructed to read admission of defeat.

2) The part about "no Python or tools" is setting the model up for failure.

I mean, this task is, for a human, basically a game of "simulate a line following robot in your head". Pretty sure a VLM could solve that if it was allowed to do the same thing. Off the top of my head, an algorithm like:

1. Identify start and end points

2. Foreach start point, follow next pixel minimizing angle, until endpoint is reached.

3. Report answer

It's literally what every human facing this task does.

EDIT:

My attempt - same image, prompt altered to allow for code (but still no search/external checks), solved in 1/5th of the time, correctly, and (going by thinking trace summaries that I don't think show up in shared chats), basically the same way I'd approach it, by tracing the lines, coloring them as it goes.

https://chatgpt.com/share/6a834f76-8240-83ed-acff-0c67af399d...

INB4: I know this is now not a pure vision check, but it really doesn't make much sense to diss models for failing to solve tasks explicitly designed to teach humans to externalize computation that's hard to do in their heads (i.e. kids, crayons, coloring paths).

Still, if such things are becoming a benchmark for tool-less evaluation, it's only a matter of time until the models learn - much like humans learn in school - to follow algorithms mentally, essentially emulating an ad-hoc computer in their head.

by TeMPOraL

8/17/2026 at 6:20:53 PM

No pre-prompting, although I can't be sure it didn't use memories. "No tools" should theoretically have prevented it from looking up memories. FWIW, Grok and Gemini both failed in a similar way.

With Python, it was able to successfully solve it in 9 minutes: https://chatgpt.com/s/t_6a8350ecddfc81919328caf68de74861

The real pain point is that at work, I use Codex and I'm currently working on a project that involves debugging some polyline topology, very similar to the path following puzzle. The vision is completely useless here.

Your VLM idea sounds good. Theoretically, the inverse problem (generating an SVG of a pelican riding a bike) can also be solved with a VLM that plans out how to draw it, not unlike a human planning out a path for their hand to follow.

by dllu

8/18/2026 at 6:23:29 AM

A better headline would be "Gemini 3.5 Flash is the best vision model". It tops almost every single benchmark shown in the article.

by mherrmann

8/17/2026 at 2:48:12 PM

From our experiments it’s the best video captioning model in the world by a mile. This was not the case a year ago.

When reasoning got introduced a year ago to GPT 5, on average the model performed worse than GPT4-o for short video clip captioning (Ie hallucinating actions that didn’t happen). The old GPT 5 was extremely finicky in terms of fps sample rate.

The other SOTA LLMs (like Gemini Pro) have clearly been optimized for long video understanding, since they can’t see almost anything sub-second (even if you up the frame sampling rate).

Sol is the first model we’ve seen to accurately caption complex sub-second movements (eg woman suddenly turns heard head to right). It’s robust to different fps sample rates so I can only guess that they trained on videos sampled at different fps.

by schopra909

8/18/2026 at 6:52:52 AM

Have you checked versus more recent Gemini models like 3.5 or perhaps 3.7?

by Vacyyyy

8/17/2026 at 2:49:48 PM

It's not clear to me from the article, are they asking sol to output bounding box coordinates with some kind of structured outputs?

Anecdotal but I've seen it use python to crop, zoom, and "enhance" (fiddle with sharpness and brightness) images to read sections of handwritten census data from the 1800s. Feels like that there might just be a mismatch of capabilities when it comes to straight outputting coordinates but I bet the model is better at actually finding the answer given any tools available. Which I get is a bit of an apples and oranges situation.

I've also tried to use it to identify an old pair of glasses and it didn't stand a chance, so I do think it's not quite there yet when it comes to some vision tasks.

by faxmeyourcode

8/17/2026 at 3:57:11 PM

I currently have fable organize a bunch of 5.6 sol agents when working on my personal projects. This makes me wonder if I should add something along the lines of "For tasks that involve visual analysis, have gemini 3.7 look at images generated."

Overall I've been hooked on using agents from different companies for what they are best at (Thanks to Theo). Fable is expensive, but unmatched for planning and top level organization of other agents. Sol is fast, will persistantly go after goals (sometimes to its detriment), and does well with computer use.

by lwarfield

8/17/2026 at 2:50:00 PM

I actually favor Qwen3.8 and run it locally + use the Token-Plan on AlibabaCloud, when I need faster results. Kind of favor it over GPT5.6 Sol.

Also it seems to be more capable, need to test more, but I think it's at least getting on par and it's fully open-source and open-weights.

Here's some benchmarks:

https://benchlm.ai/compare/gpt-5-6-sol-vs-qwen3-8-max

https://qwen.ai/blog?id=qwen3.8#full-benchmark-table (incredible UI/UX demos)

https://venturebeat.com/technology/qwen3-8-max-arrives-with-...

EDIT: Am I early to the discussion, or is none else using Qwen3.8-max?

by ALLTaken

8/17/2026 at 7:57:27 PM

I thought Qwen 3.8 max doesn't have vision?

by barrenko

8/20/2026 at 10:11:19 PM

It actually has vision + tool_use even the 27B param model. The community tries to produce even a MoE version of Qwen3.8 now, because that'd allow to run the full model with some experts being pruned like with Ornith 1.5 35B A3B.

You can get this to run on 24GB Ram: https://huggingface.co/baa-ai/Qwen3.8-27B-RAM-24GB-MLX

I found the FULL Qwen3.8-2.4T-A95B-MLX-reap50-3bit, but it's 540GB. So, I can't run it on my tiny laptop, but hope that the community finds ways to bring the size down and memory requirements too =)))

by ALLTaken

8/20/2026 at 10:17:14 PM

OMG I found it!!

Research preview: Whittle MoE 27B (A18B): a mixture of experts rescued by its routers

https://huggingface.co/logic65/Qwen3.8-Whittle-MoE-27B-A17.8...

by ALLTaken

8/17/2026 at 6:08:08 PM

huh, why am I being shadow banned?

Does YC have similar problems like those at wikipedia/reddit? (wikipedia-editor-wars, or reddit-mod-wars)

by ALLTaken

8/18/2026 at 12:14:30 AM

You aren't being "shadow banned".

I concur with your conclusion, Qwen 3.8 has exceptional Vision Capabilities. The other commenter mentioned "I thought Qwen 3.8 didn't have Vision", it does, just not on the open weights version, only via official API.

by HDBaseT

8/17/2026 at 1:18:58 PM

I understand why you would like to use an LLM for vision. I do it myself often enough. I don't understand however, why the pill detection and counting is included in this benchmark. That is a task which you would perform with OpenCV right?

In my personal mini benchmark minicpm-v-4.6 scores amazingly well. Its a 0.8B model which runs fine on many consumer hardware.

by iamniels

8/17/2026 at 1:26:16 PM

Generating datasets to train more efficient models is a common use case for VLMs, especially frontier ones. It makes it much cheaper to create that initial dataset and you can abuse the nondeterminism of LLMs to identify data for human review (if they don’t converge, escalate to a human).

by throwup238

8/17/2026 at 2:07:24 PM

Especially the pill counting example. The best model was shown at 81.1% accuracy, which is a terrible rate for pharmacy scenarios. It seems like implementors would be better off instructing the models to use deterministic tools (like OpenCV) until the models are at 99.99% accuracy (or whatever an acceptable error rate is for pharmacy techs).

by rhplus

8/17/2026 at 3:09:46 PM

I think that is because people perceive OpenCV as 'hard to use' and LLMs as easy to use.

by jacquesm

8/17/2026 at 6:50:36 PM

OpenCV is no longer hard to use, it just takes longer. Still, a little more complicated than asking LLM to count.

To use an LLM, you just prompt it with an image + text saying "count the pills in this image".

To use OpenCV, ... you just prompt an LLM with an image + text saying "count the pills in this image, using OpenCV instead of eyeballing it".

(I like to throw in "produce intermediary artifacts so I can see the process" for more difficult tasks; this helps the model avoiding making hallucination-prone leaps and gives more opportunities to self-correct. At a cost of extra time and tokens, of course.)

Using OpenCV without an LLM? Nah, not touching that, I don't have free weekends to waste anymore.

by TeMPOraL

8/17/2026 at 8:59:35 PM

I no longer use it but never felt it was particularly complicated, but since the days of resnet there are much faster ways to the goal.

by jacquesm

8/17/2026 at 1:54:34 PM

So far I haven't seen a single model succeeding at transcribing sheet music, but I just tested it again with 5.6 Sol and it nailed the small test case. Fluently reading music requires multiple years of training for most people, but I feel like accurately following the horizontal lines trips up vision models in particular.

by kherud

8/17/2026 at 3:46:34 PM

For a bespoke model that transcribes sheet music images well, check out our system at Soundslice: https://www.soundslice.com/sheet-music-scanner/

It's not an LLM, it's a custom thing we built. Here's a comprehensive list of support for various notation glyphs: https://www.soundslice.com/help/en/creating/pdf-import/294/s...

by adrianh

8/17/2026 at 12:54:27 PM

My anecdotal evidence says its still as blind as any other model, it has no taste, no attention to any sort of detail.

by sscaryterry

8/17/2026 at 12:56:00 PM

How can a vision model have taste?

by howdareme

8/17/2026 at 1:00:51 PM

If you're doing any kind of inference that is multi-modal and non-factual, opinions and biases will affect any kind of assessment of a visual that you provide to a model.

For example, a UI / UX professional being asked to appraise a website screenshot may determine that the image in question has "desirable" traits which are inherently not deterministically measurable. Such as, if the interface elements have strong information hierarchy, or if they are deemed to be "fashionable" with current UI trends.

by sarreph

8/17/2026 at 1:09:49 PM

> if the interface elements have strong information hierarchy

...but that's an example of a UX/usability matter that can be assessed objectively and non-subjectively.

by DaiPlusPlus

8/17/2026 at 2:37:37 PM

I disagree.

Is 16 px or 14 px a better font-size value for a subheading, in a hypothetical layout? Immediately that kind of decision, where both options are objectively good for 12 px paragraph text, suddenly becomes an issue of taste that cannot be evaluated crudely by an algorithm.

by sarreph

8/17/2026 at 1:00:24 PM

Replace taste with consistent if that helps you. Can it follow a design system...

by sscaryterry

8/17/2026 at 1:05:30 PM

As a design system engineer I usually have to fight against the taste of the designers. (And I consider it natural.)

But, if you have a proper well documented design system and you tell the LLM to use the DS and to avoid styling hacks they can generally do it. Even the dumber ones than Sol 5.6.

Of course only if the design is achievable in the design system.

by yreg

8/17/2026 at 1:07:44 PM

This is not my experience at all.

by sscaryterry

8/17/2026 at 1:06:10 PM

So, formulaic output…the opposite of taste

by velcrovan

8/17/2026 at 1:08:19 PM

Not really. Compliance with the letter of the law doesn't mean the intent is complied with.

by sscaryterry

8/17/2026 at 1:35:45 PM

One of my friends (and BIL) own an architecture firm. They use AI to generate and quickly update renderings but they run into the equivalent of the 6 fingered hand problem. I sent him this article I wonder if the updated models can catch and fix mistakes made by previous models.

by chasd00

8/17/2026 at 1:48:08 PM

This article is about vision, not image output.

by Mashimo

8/17/2026 at 2:34:55 PM

Hence the "catch and fix mistakes" part.

by stavros

8/17/2026 at 1:11:01 PM

I've decided it's "good enough" after I saw it properly quote a string of text that was very roughly highlighted within a nested visual context. It also identified the context correctly (modal inside webapp inside screenshot of user desktop).

by bob1029

8/17/2026 at 2:20:43 PM

I run the free service https://countrx.app/ so i have some idea what goes into counting.

The performance as a general model is indeed really impressive and i think they might actually win compared to fine tuned models.

Their feedback loop of training on user data is incredibly strong. I've learned that lots of accuracy results depends on threshold configs, which llms should be able to dynamically set.

Or the future will develop in llms using fine-tuned models as tools? Inference cost and speed does still seem to be below user expectations.

But for being able to one shot with this accuracy... IMPRESSIVE

by ParanoidShroom

8/17/2026 at 2:55:46 PM

How are you running it for free? Are you self funding or do you have sponsors?

by IncreasePosts

8/17/2026 at 5:56:50 PM

Self funded. It's a custom trained efficient model on CPU so it's borderline free

by ParanoidShroom

8/17/2026 at 2:41:07 PM

I really like the combo 5.6 Luna & Sol for price and performance and would be perfectly happy if they stayed here for a moment without mucking about with sidegrades that I think AI evolution has often felt like lately.

by jug

8/18/2026 at 6:21:20 AM

Dumb question: When your testing "ChatGPT 5.6 Sol" are you testing an actual LLM or some visual pre-processor stack that sits in front of it (along with a maybe a bunch of other such pre-processors) that is bundled into what's call "ChatGPT 5.6 Sol"? I.e. last I checked LLMs had a something like a 30-100K token alphabet to work with and it's hard to imagine how throwing pixels arrays at one directly would work.

by theteapot

8/18/2026 at 7:16:34 AM

I'm ~95% certain that images are tokenized, just like regular text, and fed directly in; that's the 'multimodal' part of these models. Now how this tokenizing works I don't know, and there might be some level of preprocessing, but it's certainly not converting the image into text and feeding it in to a regular LLM.

by MoltenMan

8/17/2026 at 1:22:42 PM

All of your use cases are very advanced.

I recently used it at grocery stores in a foreign country. Photographed the whole aisle and told it to find Y (detergent, softener, glue, sour cream, whatever), at the same time recommend the best Y for whatever reason. Worked marvelously, including the cases where the object wasn't present and it told me there was nothing useful.

I asked then, can you crop the exact image of how does the item look like and where is it in the aisle - did that perfectly as well.

I will add that all frontier models were fine with such tasks from the early 2024's.

by 5555watch

8/17/2026 at 1:28:55 PM

"Best iPhone ever" vibes.

by trumbitta2

8/17/2026 at 1:40:28 PM

I would love more vision benchmarks! Once I asked the model to inspect a completely black picture and it hallucinated a nice wooden kitchen wall. Took me some time to figure out where the kitchen came from...

I usually go to https://arena.ai/leaderboard/vision/pareto for a nice overview of current models.

by prathje

8/17/2026 at 6:53:29 PM

It’s gotten so good that I now have infrastructure to render all mermaid/plantuml in my project to png and have AI’s always load both text and image versions. And they are instructed to review the rendering as part of the diagramming cycle (for layout, salience, usefulness, etc). They can now produce useful diagrams that help reach shared architecture understanding.

by apinstein

8/17/2026 at 1:54:13 PM

It's vision capabilities poisoned my cucumber bed, misidentifying the malaise and having me spray them down with water, which only spread the fungus that gemini later informed me was actual cause, which I went and checked myself.

I hope that whatever was lost at GDM in the last few months, didn't include their extra focus on vision capabilities.

by WarmWash

8/17/2026 at 3:59:54 PM

5.6 Sol looks nice, but the Gemini 3.5 Flash comparison is interesting. It’s cheaper and still came out ahead on detection and counting, which doesn't really give me much of a reason to use Sol since Flash is much cheaper and hence much easier to scale. Not to mention we now have 3.6 Flash too

by 1saadcodes

8/17/2026 at 6:12:24 PM

We have 3.7 Flash now, actually, and it costs just a hair over the old 3 Flash Preview while being better!

by ComputerGuru

8/17/2026 at 5:01:08 PM

One of the use cases I've wondered about for AI is giving it a picture of the "spice wall" in a grocery store and asking it to find all jars of e.g. cardamom. This takes me an annoyingly long time to do when I'm shopping, so it would actually be useful.

by slibhb

8/17/2026 at 2:33:22 PM

Luna is pretty strong as well. been using it for projects the last two weeks and its strong

by cdolan

8/17/2026 at 1:24:37 PM

Where are the Qwen benchmarks in this? I would be more interesting to see how Qwen performs.

by iamleppert

8/17/2026 at 3:44:52 PM

Hi! I’m the author of this blog. I regularly benchmark new VLM releases. You can check the results for Qwen3.8-Max and Qwen3.8-27B here: https://playground.roboflow.com/evals

by SkalskiP

8/17/2026 at 2:36:55 PM

Me too. This is an interesting comparison but in my experience Qwen and Gemini have typically been the top contenders for image related tasks. For that reason it would be great to have the comparison here, as I'm not surprised by Gemini's dominance over the other models.

by ImageXav

8/17/2026 at 3:38:21 PM

Which is to say, still not ready for any production workloads yet. As in, it cannot reliably count the amount of objects in an image.

Still very impressive, but nowhere near the text chat revolution. OpenAI still trying to strike their second lightning

by TZubiri

8/17/2026 at 9:58:31 PM

https://news.ycombinator.com/item?id=46444508

by chistev

8/18/2026 at 11:11:25 AM

The prompt sensitivity is fascinating. A simple coordinate-format change shifting detection performance this much shows how much interface design still matters.

by trevortaylorai

8/17/2026 at 1:24:26 PM

I didn't expect Gemini 3.5 Flash to top basically every metric in this article.

by adroitboss

8/17/2026 at 1:39:11 PM

In my practice Gemini models are far better than anything on the market in terms of vision, also it's worth to mention that current Gemini flash is 3.7, so it got 2 updates since 3.5 which beat GPT-5.6 Sol in this comparison.

by SweetSoftPillow

8/17/2026 at 4:05:11 PM

Hi! I’m the author of this blog. I wrote it 4 weeks ago, and it’s already a bit outdated. Gemini 3.7 Flash came out last week, and considering the price, it’s easily the best vision model right now: https://x.com/skalskip92/status/2088032652301304121?s=20

by SkalskiP

8/17/2026 at 1:26:16 PM

Same. I scrolled back up to see if I read the title correctly. It's important to note that it is the best... OpenAI released. Not the best overall.

by LollipopYakuza

8/17/2026 at 1:58:23 PM

Gemini has long been the vision champion, but there aren't many benchmarks and coding is where all the hype is.

Demis had a pretty big interest in vision, more so than text, so I hope they don't lose that with all the recent shuffling.

by WarmWash

8/17/2026 at 2:15:41 PM

Are any of these vision benchmarks binocular in order to introduce depth perception?

I keep waiting for these AI companies to assemble the parts into a great autonomous driving module.

by criddell

8/17/2026 at 4:09:26 PM

Do you think we’re getting closer to models that actually understand what they’re seeing, or are they just getting really good at recognizing patterns?

by wahid_seddiqi

8/18/2026 at 7:50:35 PM

"yes" but that's a philosophical question. I think they're getting better at "early fusion" IE, training the model that "apple" and these visual tokens are the the same concept, but LLMs are fundamentally a pattern matching machine so even with perfect fusion I personally wouldn't call it understanding.

by edude03

8/17/2026 at 4:23:37 PM

All I'm fine with for now is that I can almost exclusively communicate with Sol through collages and my scribblings (all kinds of web page / block screens with all kinds of arrows and text all over the place) This was not practically ppossible in 5.5 and a tragedy in 5.4. Not sure how much weight is codex uploading in higher res carrying here but it's great to work with.

by Culonavirus

8/17/2026 at 8:52:32 PM

Seed Turbo 2.1 is incredibly detailed in describing every physical feature. I use that one for vision tool calls through Venice API.

by drak0n1c

8/17/2026 at 1:32:39 PM

Does any popular NVR make a good use of LLMs (especially local models) getting decent at vision?

by comboy

8/17/2026 at 3:35:34 PM

I've been using Reolink for years and been very satisfied with it.

The only quip is the default UI isn't very good. When changing that reaches the top of my priority list, I'll switch it since they don't force you into a walled garden. Plan is to run it through frigate into HomeAssistant and use a UI from them. I've never used frigate before though so it'll be a learning process if plug and play solutions aren't already available

by eks391

8/18/2026 at 7:28:24 AM

I wonder what about UI review, which model is the best?

by dostick

8/17/2026 at 1:26:37 PM

Still not quite as good as gemini.

by catigula

8/17/2026 at 10:31:04 PM

I hate these "The best X thing Y has ever released".

Unlike when Apple says "it's the best iphone we've ever made", LLMs are more or less interchangeable. So "OpenAI's best model" means nothing if "Anthropic wipes the floor with them" or "[open weights model] is 10x cheaper for 1% less quality".

As a reader, it feels like these titles are click bait.

by dangoodmanUT

8/17/2026 at 2:01:36 PM

It's really quite good! I was amazed recently by its utter inability to read some faded handwritten cyrillic on the back of a wood carving - 3 or 4 words only, reasonably clear letter forms I found recently, and then stepped back a bit and thought about how insane that was as a benchmark - I just expect it to work so reliably on other OCR and translation tasks that it was surprising to encounter such a failure

by RugnirViking

8/17/2026 at 11:40:02 PM

the last image - it's barely visible to human eyes

by dzonga

8/17/2026 at 3:48:09 PM

Fuck ack. I'm working on a new benchmark that combines strong visual requirements with tool and coding requirements. I haven't even tested Sol yet, but between Sonnet, Terra & Luna I already see much better results from OpenAI's models. I'm not releasing anything yet as I still have issues in my harness that need to be fixed.

by terhechte

8/17/2026 at 9:42:52 PM

> GPT 5.6 Sol is the best "vision" model OpenAI ever released

I mean I should hope so, as it is also the latest one

by sam0x17

8/17/2026 at 8:30:08 PM

Am I the only one who cannot read the date on the blister pack even fully zoom in my phone?

If that is the full quality image given to the model, I think it's not surprising that the model confused with 03/2022.

by slybot

8/17/2026 at 3:17:38 PM

I'm a little bit disappointed that vision seems to fall before language at scale.

It seems pretty counter intuitive that we can't do vision significantly better with specialized techniques.

by fooker

8/20/2026 at 1:42:02 PM

Yeah, GPT 5.6 Sol is very good. Generally, Gemini models remain SOTA for VLM tasks with 3.7-flash at the top. ---

That said, considering variables like cost (say, over 100k PDF pages) and accuracy requirements (e.g., construction documents with dense images & tables), Gemini and other SOTA VLMs are expensive and inaccurate, and therefore unsuitable. This is where niche, open-weight, and task-specific OCR/VL models come in.

With a one-line change, you can switch between DeepSeek OCR 2, GLM-OCR, dots.mocr, Paddle OCR VL, PP-OCRv6, etc., and process 100K+ pages for under $60 on VLM Run Gateway

This is why we built VLM Run Gateway, one OpenAI-compatible endpoint for open-weight OCR and VLM models.

Try it out quickly via OpenAI SDK:

``` client = OpenAI( base_url="https://gateway.vlm.run/v1/openai", api_key="<VLMRUN_API_KEY>", )

response = client.chat.completions.create( model="rednote-hilab/dots.mocr", messages=[{ "role": "user", "content": [{ "type": "document_url", "document_url": {"url": "https://.../invoice.pdf"}, }], }], extra_body={"document_dpi": 72}, ) ```

or via our CLI:

``` pip install vlmrun vlmrun gw models vlmrun config set --api-key 'vlmrun' # anon-user, rate-limited vlmrun gw chat <doc>.pdf -m zai-org/glm-ocr vlmrun gw chat <doc>.pdf -m zai-org/glm-ocr --json-mode vlmrun gw chat <doc>.pdf -m deepseek-ai/deepseek-ocr-2 vlmrun gw chat <doc>.pdf -m rednote-hilab/dots.mocr vlmrun gw chat <doc>.pdf -m paddleocr/pp-ocrv6 ```

Docs: https://docs.vlm.run/gateway

Catalog: https://docs.vlm.run/gateway/models

MCP: https://docs.vlm.run/gateway/mcp-server

Colab Quickstart: https://colab.research.google.com/drive/1RkuVIyuc5Po-UlcSlFy...

Read the full post here: https://huggingface.co/blog/vlm-run/intro-to-vlmrun-gateway

by cpnwaugha

8/17/2026 at 2:37:32 PM

[flagged]

by fintuner

8/18/2026 at 7:25:25 AM

[flagged]

by shipsitself

8/18/2026 at 12:47:18 PM

[flagged]

by locitra

8/18/2026 at 6:48:31 AM

[flagged]

by mavensum

8/17/2026 at 1:16:55 PM

[flagged]

by zyvop1

8/17/2026 at 3:16:05 PM

[dead]

by alessandrobinda

8/17/2026 at 4:24:25 PM

[dead]

by johnxianren

8/17/2026 at 1:15:01 PM

[dead]

by hathym

8/17/2026 at 12:48:11 PM

[dead]

by hn7jmxa7oc

8/17/2026 at 3:49:19 PM

[dead]

by ZeroDayDreamer

8/17/2026 at 2:16:47 PM

[dead]

by CurbStomper