5/22/2025 at 6:50:44 AM
> When the client sends a hello, the firewall says “that looks like a TLS hello”, and then waits for the server’s response. It inspects the certificate and then applies any rules.This kind of stuff is exactly why TLS 1.3 encrypts everything now.
by ronsor
5/22/2025 at 7:00:56 AM
Years of middleboxes tampering with TCP and barfing on anything they didn’t recognize is why QUIC encrypts everything as wellby kevvok
5/22/2025 at 11:03:23 AM
> QUIC encrypts everythingQUIC is built on top of TLS 1.3 where client hello encryption is not mandatory, so this is not strictly true.
by silverwind
5/22/2025 at 4:34:07 PM
Many companies block outgoing traffic to 443/UDP, or block UDP completely.by OptionOfT
5/22/2025 at 8:33:31 AM
In TLS 1.3, the server’s certificate is encrypted, which means the firewall can’t do this anymore. ... The firewall will open a second connection and do a TLS 1.2 handshake to retrieve the certificate in plaintext. If the rules allow the connection, the firewall then allow the connection to proceed.Sounds like a cat & mouse game. How long before we get servers that respond with different certificates based on who"s asking or other cues from the connection, in efforts to bypass the firewall restrictions.
by rkagerer
5/22/2025 at 9:04:52 AM
Sure, this already happens today as in practice pre-standard ECH is deployed, so client says "Hi I'm calling some.boring.example [encrypted: Actually clown-porn.example]" and this useless middlebox cannot decrypt the inner message so it retrieves the some.boring.example certificate but the client's encrypted connection reaches clown-porn.example which may have a different certificate.For many years, long before TLS 1.3 the RFC explains how to do this properly, obviating the problem, but that's very expensive because you need to TLS proxy every single connection, you can impose oversight by - literally - adding an actual oversight layer, which your users can also see you did. The popular middlebox products say they're doing two things, they're cheaper (maybe you buy their $100k product instead of a $100M solution) and they're less "intrusive" (ie you needn't tell your users that you're spying on them)
In reality they're ineffective, which is why the RFC says not to do this - but they can either outright lie or hide this fact in an asterisked disclaimer somewhere, and most of their customers don't care whether it actually works, they want to tick a box.
by tialaramex
5/22/2025 at 8:36:22 PM
CISO is such a thankless job.by fragmede
5/22/2025 at 2:52:15 PM
In the infosec space, I am seeing rumblings of moving to compliant devices where the network tap is at the OS level and/or application level using software agents, AV hooks, or fake virtual network devices.by FuriouslyAdrift
5/22/2025 at 8:39:07 AM
Technically TLS 1.3 doesn't encrypt the Client Hello, and indeed the first stanza of the Server Hello isn't encrypted either, although it deliberately doesn't say anything you could use as a third party to make decisions and the rest of a TLS 1.3 Server Hello is encrypted.The in-progress Encrypted Client Hello (was Encrypted Server Name Indication hence esni in the name of the document) fixes that and you probably have software which uses it though the work to tie up all the loose ends up and publish a document might take some time yet.
Because the middlebox vendors are _so_ incompetent not only does TLS 1.3 need to work around their nonsense, thus proving that it was never useful security (an attacker could always have just done what TLS 1.3 does and it would have bypassed this worthless garbage) some of them screwed up badly enough that the anti-downgrade feature trips, to their credit Google refused to ship a permanent workaround for this, the workaround they shipped in Chrome sunset in about a year & required an explicit key setting, so basically "I acknowledge that I have defective middleboxes and must remove or upgrade them soon" by the local administrators.
But yes, it's noticeable that it was much easier to sell some engineers on "Thanks to TLS 1.3 now this stupid middlebox product won't be able to make your service slower" than say, "Thanks to TLS 1.3 now this stupid middlebox product won't report to the government if you read a Wikipedia article which contradicts its dogma".
by tialaramex
5/22/2025 at 9:38:52 AM
Encrypted SNI was never about hiding which specific articles in Wikipedia you are looking at. It was about hiding the fact you are looking at Wikipedia at all.There are valid use cases for TLS middleboxes. Anyone having to secure a networks outbound access to only essential services has run into the “I have to allow all of AWS/GCP/Azure/CloudFlare/…” for some critical tool to work.
Options are:
- Allow all out (nope, not secure)
- Allow all to the cloud providers IP range (still terrible)
- Filter on SNI but don’t inspect the payload (better than no security, and doesn’t require plaintext access).
- Full MITM TLS proxy (performance bottleneck, and now we have plaintext access to all your data, which we really don’t want and didn’t previously require to do the filtering).
- Try convince the third party service to run on a handful of static IPs that aren’t behind a global load balancer with access to the rest of the cloud providers customer domains. (Yeah right)
See: Hospitals. Payment Networks. IoT networks.
by yardstick
5/22/2025 at 10:13:49 AM
> There are valid use cases for TLS middleboxesThere are, and always have been, people who are sure that they "need" to do things which don't work because the correct solution would be inconvenient for them. Those people should instead suck it up and accept the inconvenience or, as is more likely, remember they don't "need" this after all considering how inconvenient it is.
It won't suddenly work just because that would be more convenient and it's very annoying technically that we have to keep having this conversation, it's not going to stop being true just because that would make your life easier.
by tialaramex
5/22/2025 at 11:01:03 AM
Imho, clients like browsers should just indicate to the user whether their traffic can be intercepted or not (fully encrypted including client hello).And those clients should offer an option to downgrade the TLS connection to make traffic interceptable.
by silverwind
5/22/2025 at 2:05:09 PM
You can always manipulate your client into making a connection interceptable by installing your own trusted root CA that can then do MitM. That possibility has always been there and didn't go away with TLS1.3.It's just that many are too lazy to take control of their clients like that. Or they want to do surveillance on clients that aren't theirs to control.
by thyristan
5/22/2025 at 3:06:09 PM
Certificate pinning has made this very difficult/impossible.We are currently failing legal compliance audits at my work due to this and are having to move cloud things (legal,financial, customer NDA'd data, etc.) back to on-premise because if it. Our cloud HR and payroll systems are really having a tough time staying compliant as the data crosses compliance domains.
by FuriouslyAdrift
5/23/2025 at 8:34:53 AM
Agreed, didn't think of pinning. That does of course make it very difficult.However, compliance-wise, I'm of the opinion that anything cloud is a bad idea in general. It will just take some time for the bean-counters to realize (if ever).
by thyristan
5/22/2025 at 2:43:46 PM
> And those clients should offer an option to downgrade the TLS connection to make traffic interceptable.They do, it's called http://
and the other option is installing a root cert, of course.
by immibis
5/22/2025 at 3:06:40 PM
HSTS is closing this gap, too...by FuriouslyAdrift
5/22/2025 at 10:47:00 AM
Well, just give up using shared hosting for services that need to be locked down. Ain’t going to happen though, because a different and at times overlapping group of people would be inconvenienced.by yardstick
5/22/2025 at 8:18:06 PM
None of them are sane optionsThe only sane security lies at the endpoints
by JackSlateur
5/22/2025 at 2:41:21 PM
Allow none out. Get your tool vendor to tell you which IP address they have to allow and stand by it. Amazon has "Elastic IPs" where you can lease a specific IP and keep using it even as you change virtual machines. And if you use IPv6 you can have a specific subnet on your account and tell your customers to whitelist the whole subnet.If your vendor is incompatible with your security desires, fire your vendor or fire your security people. Pick one. You can't eat your cake and have it. (Deliberately reversed so the algorithm will think I'm Ted Kaczynski)
by immibis
5/22/2025 at 10:58:31 AM
Unfortunately, Encrypted Client Hello isn't automatic and requires cumbersome DNS configuration, so it's likely not going to gain widespread adoption.Here's hoping that the next TLS version will automatically encrypt everything. Firewall vendors will not like it, but it's the only way to truly hide everything from middleboxes.
by silverwind
5/22/2025 at 12:48:16 PM
Magic is impossible. In order to encrypt a message to Bob, Alice must first have Bob's key.That's why ECH needs DNS records, Bob publishes a key, "Here's the key for all six thousand blog.example sites we publish" and Alice can then encrypt either "Hi Bob, give me clown-porn.blog.example" or "Hi Bob, give me trans-rights.blog.example" and the middlebox can't read it.
We can't stop the middlebox from saying well, Bob publishes a trans rights blog so now all Bob's blogs are inaccessible just in case. Nor can we prevent Bob from deciding to publish separate keys, "these are for the clown porn and lynching videos, we'll use separate keys for the trans-rights stuff because the Government says that is naughty". But we can make that Bob's choice.
To do that, Alice needs Bob's key and either Alice has to magically know everybody's keys (which doesn't scale) or it is published somewhere for Alice to find, in this case DNS.
If we just say OK we'll encrypt with no particular recipient in mind, that doesn't prevent the scenario we care about, Alice encrypts the message, the middlebox decrypts it (No reason it can't, Alice can't pick Bob as the only recipient) and the middlebox gets to inspect Alice's destination.
by tialaramex
5/22/2025 at 2:45:20 PM
I agree with this in general. In specific though:> We can't stop the middlebox from saying well, Bob publishes a trans rights blog so now all Bob's blogs are inaccessible just in case.
Seems incorrect. Bob could publish a single key without any indication of what is behind that key. Alice would get that key (from DNS), but since it was just for talking with Bob in general, nobody knows what Bob has. Alice and Bob are then free to talk about their trans rights.
The particular point is that there's no reason the middlebox would even know there is a subdomain called trans-rights.bob.example.
by delusional
5/22/2025 at 3:06:41 PM
There are great number of ways in which the operators of the middlebox might come to know that Bob publishes this blog.Most obviously the humans charged with setting up such tech might literally find out in the ordinary course of their life that they ought to add this rule. Fox News show headline "Ban this sick filth", a letter from the Government about "Extremist sites unsuitable for your employees", and so on.
But technologically there could be a list, maybe a "Ministry of Truth" provides the list of providers who are forbidden, it needn't say why, and if it does say it could lie, usually such lists are (like spam blackholing) automated and unsupervised.
They could search the "Passive DNS" system to find suspicious names and block the entire provider, or likewise with Certificate Transparency.
So alas there are reasons the middlebox might be configured this way.
by tialaramex
5/22/2025 at 11:21:17 AM
Encrypted client hello can be stripped easily by modern firewalls[0]. I know because my employers network enables this functionality and it forces a downgrade so that it can see what you're trying to access.[0] https://community.fortinet.com/t5/FortiGate/Technical-Tip-Ho...
by hypeatei
5/22/2025 at 2:51:40 PM
"easily" in the sense that in the proxy mode, where the Fortigate gets to make any arbitrary change it wants, it can indeed make this change too.I guess "Use the American spelling for words" would be a slightly more invasive change than "strip ECH" in this context, but it's not that different. This is a full proxy, your client has (likely because it was configured by corporate IT) agreed that all your communication will run through a proxy, probably somebody will swear they're definitely not logging everything you do† but obviously it could do absolutely anything.
† Actually if you work in trading they might tell you it does log everything, the regulators for these industries want records of everything because they don't trust traders as far as they can throw them, and having met a few traders I know why - they tend to make Boris Johnson look like Forest Gump.
by tialaramex
5/22/2025 at 2:44:27 PM
What it does isn't good enough, since it basically leaves itself open to downgrade attacks on purpose. I wish the world would decide on a flag day after which if both endpoints support TLS ECH, the connection will hard-fail instead of retrying without it if a middlebox breaks it.by josephcsible
5/22/2025 at 3:52:30 PM
Wouldn't the middlebox just lie to each party that the other doesn't support ECH?by stavros
5/22/2025 at 4:24:43 PM
TLS 1.3 prevents tampering with the handshake. The transcripts won't match and the connection will be aborted.by ronsor