alt.hn

7/31/2026 at 8:07:06 PM

Don't stop early: Case-folding source code at memory speed

https://github.blog/engineering/architecture-optimization/dont-stop-early-case-folding-source-code-at-memory-speed/

by sbulaev

8/4/2026 at 9:50:08 PM

> We deal mostly with source code, so the text we fold is overwhelmingly ASCII and making it run at memory speed is the single most important thing we can do. Everything else just has to keep the rare non-ASCII path from spoiling it.

Semi-on-topic: I've noticed that many LLMs via coding agents (ChatGPT and Claude at work with my CoPilot account, and DeepSeek 4 and ChatGPT in pi.dev at home) really seem to like using unicode / emoji characters for things like arrows (for things like test value ranges), crosses and ticks (for pass vs fail in test comments), instead of plain ASCII. Codebases are almost exclusively ASCII chars to my knowledge, although they're UTF-8 files.

I'm not yet using agents to write code (only do code reviews, write example prototypes I then copy bits of, and helping craft tests), but I'm likely to get there soon, and I'm sure it's possible to prompt them NOT to do this, but has anyone else noticed this? I wonder if that changes things over time for them if this is a common theme of increased non-ASCII output?

by pixelesque

8/4/2026 at 11:12:03 PM

Codebases written by native English speakers are almost exclusively ASCII, but codebases written by speakers of languages other than English frequently have non-ASCII content, even if only in the comments, but languages which support it often wind up with non-ASCII identifiers, too.

I do not believe that emoji like crosses and ticks are particularly common at all, for any language, but LLMs seem to have picked up heavy use of them from somewhere and inserted them into code (and everything else) they generate.

LLM training sets will very likely include the massive corpos of non-English open source code from sites like Gitee, but would be unlikely to generate responses heavily influenced by them unless you've done specific things to make that happen - prompt in Chinese, try to make use of a library only available with Chinese source and/or documentation, perhaps. I've not seen it happen, but I am a light user of LLMs.

by codebje

8/4/2026 at 10:48:48 PM

Using emoji for status indicators on the console is a trend that pre-dates LLMs. First mainstream app I can recall doing it was Yarn.

by chuckadams

8/4/2026 at 9:02:17 PM

This is good technical content, but it's obvious that an AI wrote it.

by claudetard

8/4/2026 at 11:44:06 PM

[delayed]

by Groxx

8/4/2026 at 10:05:26 PM

Agreed. This is genuinely interesting content, but there is no doubt in my mind that "The two operations diverge on real characters—ß, İ, final sigma—which is why lowercasing as a stand-in silently produces wrong matches." is LLM output.

Are we doomed to spend the rest of our professional and personal lives reading AI output?

by agency

8/4/2026 at 10:06:37 PM

Yes.

> This is genuinely interesting

Are you sure you're not an LLM yourself?

by inigyou

8/4/2026 at 10:13:07 PM

I thought I wasn't, but you're making me second-guess myself.

by agency

8/4/2026 at 10:05:55 PM

TLDR: they implemented case folding with a lot more SIMD via autovectorization.

> almost every fold preserves the UTF-8 length or shrinks it, but two outliers grow—U+023A (Ⱥ) and U+023E (Ɀ) are 2 bytes each yet fold to 3-byte characters (ⱥ, ɀ)

Fix this by reversing it. Fold ⱥ to Ⱥ instead of the other way around. The search index won't only consist of lowercase characters any more, but that never mattered.

by inigyou

8/4/2026 at 10:59:38 PM

This is a nice follow up to the other SIMD article that was posted here a week or so ago hah.

by persedes

8/4/2026 at 10:25:30 PM

[dead]

by PaulDavisThe1st