alt.hn

2/21/2026 at 7:05:31 PM

Cloudflare outage on February 20, 2026

 https://blog.cloudflare.com/cloudflare-outage-february-20-2026/

by nomaxx117

2/21/2026 at 9:27:58 PM

It's something we debated in our team: if there's an API that returns data based on filters, what's the better behavior if no filters are provided - return everything or return nothing?

The consensus was that returning everything is rarely what's desired, for two reasons: first, if the system grows, allowing API users to return everything at once can be a problem both for our server (lots of data in RAM when fetching from the DB => OOM, and additional stress on the DB) and for the user (the same problem on their side). Second, it's easy to forget to specify filters, especially in cases like "let's delete something based on some filters."

So the standard practice now is to return nothing if no filters are provided, and we pay attention to it during code reviews. If the user does really want all the data, you can add pagination to your API. With pagination, it's very unlikely for the user to accidentally fetch everything because they must explicitly work with pagination tokens, etc.

Another option, if you don't want pagination, is to have a separate method named accordingly, like ListAllObjects, without any filters.

by kgeist

2/21/2026 at 10:10:26 PM

Returning an empty result in that case may cause a more subtle failure. I would think returning an error would be a bit better as it would clearly communicate that the caller called the API endpoint incorrectly. If it’s HTTP a 400 Bad Request status code would seem appropriate.

by alemanek

2/23/2026 at 7:50:57 PM

[dead]

by pigpag

2/22/2026 at 7:07:17 PM

Neither of your options are good, the first question you need to ask is that is the filter optional or not ( this is a contract / API question ).

If not optional then return 400, otherwise return all the results ( and have pagination ).

You should always have pagination in an API.

by Thaxll

2/21/2026 at 11:03:25 PM

>allowing API users to return everything at once can be a problem both for our server (lots of data in RAM when fetching from the DB => OOM, and additional stress on the DB)

You can limit stress on RAM by streaming the data. You should ideally stream rows for any large dataset. Otherwise, like you say you are loading the entire thing into RAM.

by Philip-J-Fry

2/22/2026 at 8:30:00 PM

Not to mention the latency reduction!

Buffering up the entire data set before encoding it to JSON and sending it is one of the biggest sources of latency in API based software. Streaming can get latencies down to tens of microseconds!

by jiggawatts

2/21/2026 at 9:36:19 PM

I like your thought process around the ‘empty’ case. While the opposite of a filter is no filter, to your point, that is probably not really the desire when it comes to data retrieval. We might have to revisit that ourselves.

by MobileVet

2/22/2026 at 12:51:22 AM

But that query had parameter. They just fucked up parsing it

by PunchyHamster

2/22/2026 at 12:57:16 AM

> to have a separate method named accordingly, like ListAllObjects, without any filters

For me it's like `filter1=*`

by est

2/22/2026 at 9:30:47 AM

how about returning an error ? It’s the generic “client sent something wrong” bucket. Missing a required filter param is unambiguously a client mistake according to your own docs/contract → client error → 4xx family → 400 is the safest/default member of that family.

by qwertyuiop_

2/21/2026 at 7:28:59 PM

Insufficient mock data in the staging environment? Like no BYOIP prefixes at all? Since even one prefix should have shown that it would be deleted by that subtask...

From all the recent outages, it sounds like Cloudflare is barely tested at all. Maybe they have lots of unit tests etc, but they do not seem to test their whole system... I get that their whole setup is vast, but even testing that subtask manually would have surfaced the bug

by CommonGuy

2/21/2026 at 10:23:46 PM

Testing the "whole system" for a mature enterprise product is quite difficult. The combinatorial explosion of account configurations and feature usage becomes intractable on two levels: engineers can't anticipate every scenario they need their tests to cover (because the product is too big understand the whole of), and even if comprehensive testing was possible - it would be impractical on some combination of time, flakiness, and cost.

by zmj

2/21/2026 at 7:43:54 PM

I think Cloudflare does not sufficiently test lesser-used options. I lurk in the R2 Discord and a lot of users seem to have problems with custom domains.

by dabinat

2/21/2026 at 7:37:49 PM

It was also merged 15 days prior to production release...however, you're spot on with the empty test. That's a basic scenario that if it returned all...is like oh no.

by asciii

2/21/2026 at 10:15:03 PM

my guess is the company is rotting from the inside and drowning in tech debt

by suhputt

2/21/2026 at 8:36:26 PM

Just crazy. Why does a staging environment matter? They should be running some integration tests against eg an in memory database for these kinds of tasks surely?

by martinald

2/21/2026 at 8:26:02 PM

Reliability was/is CF's label.

It's alarming already. Too many outages in the past months. CF should fix it, or it becomes unacceptable and people will leave the platform.

I really hope they will figure things out.

by otar

2/21/2026 at 10:59:10 PM

We’re still waiting on a solution for https://www.cloudflarestatus.com/incidents/391rky29892m (which actually started a month earlier than the incident reports)

In the meantime, as you say, we’re now going through and evaluating other vendors for each component that CF provides - which is both unfortunate, and a frustrating use of time, as CF’s services “just worked” very well for a very long time.

by tallytarik

2/21/2026 at 8:31:49 PM

I have many things dependent on Cloudflare. That makes me root for Cloudflare and I think I'm not the only one. Instead of finding better options we're getting stuck on an already failing HA solution. I wonder what caused this.

by argestes

2/21/2026 at 9:11:18 PM

There are no alternatives, and those alternatives that did exist back in the day, had to shut down due to either going out of business or not being able to keep a paygo model.

Not everybody needs cloudflare, but those that need it and aren't major enterprises, have no other option.

by slothsarecool

2/21/2026 at 9:19:34 PM

Bunny.net? Doesn't have near the same feature set as Cloudflare, but the essentials are there and you can easily pay as you go with a credit card.

by Sanzig

2/21/2026 at 9:38:02 PM

Their WAF isn't there yet, the moment it can build the expressions you can build with CF (and allows you to have as much visibility into the traffic as CF does), then it might be a solid option, assuming they have the compute/network capacity.

by slothsarecool

2/21/2026 at 9:18:50 PM

Lots of people who think they need Cloudflare don't. What are you using it for?

by pocksuppet

2/21/2026 at 9:36:33 PM

L7 DDoS protection and global routing + CDN, there is not a single paygo provider that can handle the capacity CF can, especially not at this price range (mitigated attacks distributed from approximately 50-90k ips, adding up to about 300-700k rps).

We tried Stackpath, Imperva (Incapsula back in the day), etc but they were either too expensive or went out of business.

by slothsarecool

2/21/2026 at 10:52:46 PM

> especially not at this price range

pay peanuts, get monkeys

by blibble

2/21/2026 at 8:58:36 PM

Do you not feel concern about you and everybody else deciding to put ALL of their eggs into one basket like this?

by arcatech

2/21/2026 at 10:37:05 PM

I would bet money that most people who use CF now are already hosting their endpoints at a single provider. I don't think most people care until it actually becomes enough of a problem.

by ranger_danger

2/22/2026 at 2:28:59 AM

Like AWS/GCP/Azure?

by esseph

2/21/2026 at 8:29:00 PM

Not sure why everyone is complaining, new MCP features are more important than uptime

by alansaber

2/21/2026 at 7:45:59 PM

The irony is that the outage was caused by a change from the "Code Orange: Fail Small initiative".

They definitely failed big this time.

by NinjaTrance

2/21/2026 at 9:08:40 PM

One has to wonder when the board realises Dane was a bad replacement for JGC. These outages are getting ridiculous

by vimda

2/21/2026 at 7:45:51 PM

is this blog post LLM generated?

the explanation makes no sense:

> Because the client is passing pending_delete with no value, the result of Query().Get(“pending_delete”) here will be an empty string (“”), so the API server interprets this as a request for all BYOIP prefixes instead of just those prefixes that were supposed to be removed. The system interpreted this as all returned prefixes being queued for deletion.

client:

     resp, err := d.doRequest(ctx, http.MethodGet, `/v1/prefixes?pending_delete`, nil)
server:

    if v := req.URL.Query().Get("pending_delete"); v != "" {
        // ignore other behavior and fetch pending objects from the ip_prefixes_deleted table
        prefixes, err := c.RO().IPPrefixes().FetchPrefixesPendingDeletion(ctx)
        if err != nil {
            api.RenderError(ctx, w, ErrInternalError)
            return
        }

        api.Render(ctx, w, http.StatusOK, renderIPPrefixAPIResponse(prefixes, nil))
        return
    }
even if the client had passed a value it would have still done exactly the same thing, as the value of "v" (or anything from the request) is not used in that block

by blibble

2/22/2026 at 12:58:52 AM

better explanation here https://news.ycombinator.com/item?id=47106852

but in short they are changing whether string is empty, and query string "pending_delete" is same as "pending_delete=" and will return empty

Or, if they specified `/v1/prefixes?pending_delete=potato` it would return "correct" list of objects to delete

Or in other words "Go have types safety, fuck it, let's use strings like in '90s PHP apps instead"

by PunchyHamster

2/23/2026 at 6:57:08 AM

If Go supported Optional's, this bug would not have surfaced.

by lenkite

2/21/2026 at 7:49:32 PM

doesn't look AI-generated. even if they have made a mistake, it's probably just from the rush of getting a postmortem out prior to root cause analysis

by bstsb

2/21/2026 at 7:51:49 PM

> even if the client had passed a value it would have still done exactly the same thing, as the value of "v" (or anything from the request) is not used in that block

If they passed in any value, they would have entered the block and returned early with the results of FetchPrefixesPendingDeletion.

From the post:

> this was implemented as part of a regularly running sub-task that checks for BYOIP prefixes that should be removed, and then removes them.

They expected to drop into the block of code above, but since they didn't, they returned all routes.

by bretthoerner

2/21/2026 at 8:05:39 PM

okay so the code which returned everything isn't there

actual explanation: the API server by default returns everything. the client attempted to make a request to return "pending_deletes", but as the request was malformed, the API instead went down the default path, which returned everything. then the client deleted everything.

makes sense now

but is that explanation is even worse

because that means the code path was never tested?

by blibble

2/21/2026 at 8:36:33 PM

or they tested it, but not with a dataset that contained prefixes not pending deletion

by jbxntuehineoh

2/21/2026 at 9:57:11 PM

That's weird. They only removed some 6 of our prefixes out of perhaps 40 we have with them, so something seems off in this explanation.

by subscribed

2/21/2026 at 7:55:26 PM

yep, no mention that re-advertised prefixes would be withdrawn again as well during the entire impact even after they shut it down.

by himata4113

2/21/2026 at 7:29:03 PM

I do not work in the space at all, but it seems like Cloudflare has been having more network disruptions lately than they used to. To anyone who deals with this sort of thing, is that just recency bias?

by atty

2/21/2026 at 7:33:02 PM

It is not. They went about 5 years without one of these, and had a handful over the last 6 months. They're really going to need to figure out what's going wrong and clean up shop.

by Icathian

2/21/2026 at 7:41:08 PM

Engineers have been vibe coding a lot recently...

by NinjaTrance

2/21/2026 at 7:49:56 PM

The featured blog post where one of their senior engineering PMs presented an allegedly "production grade" Matrix implementation, in which authentication was stubbed out as a TODO, says it all really. I'm glad a quarter of the internet is in such responsible hands.

by jsheard

2/21/2026 at 8:55:22 PM

It's spreading and only going to get worse.

Management thinks AI tools should make everyone 10x as productive, so they're all trying to run lean teams and load up the remaining engineers with all the work. This will end about as well as the great offshoring of the early 2000s.

by gtowey

2/21/2026 at 8:18:30 PM

there was also a post here where an engineer was parading around a vibe-coded oauth library he'd made as a demonstration of how great LLMs were

at which point the CVEs started to fly in

by blibble

2/21/2026 at 10:39:20 PM

Matrix doesn't actually define how one should do authentication though... every homeserver software is free to implement it however they want.

by ranger_danger

2/21/2026 at 7:53:48 PM

Thats a classic claude move, even the new sonnet 4.6 still does this.

by dana321

2/21/2026 at 8:01:05 PM

It’s almost as classic as just short circuiting tests in lightly obfuscated ways.

I could be quite the kernel developer if making the test green was the only criteria.

by bonesss

2/22/2026 at 6:23:15 AM

Wait till you get AI to write unit tests and tell it the test must pass. After a few rounds it will make the test “assert(true)” when the code cant get the test to pass

by allovertheworld

2/21/2026 at 8:00:55 PM

[dead]

by brutalc

2/21/2026 at 8:06:03 PM

No joke. In my company we "sabotaged" the AI initiative led by the CTO. We used LLMs to deliver features as requested by the CTO, but we introduced a couple of bugs here and there (intentionally). As a result, the quarter ended up with more time allocated to fix bugs and tons of customer claims. The CTO is now undoing his initiative. We all have now some time more to keep our jobs.

by dakiol

2/21/2026 at 8:42:22 PM

Thats actively malicious. I understand not going out of your way to catch the LLMs' bugs so as to show the folly of the initiative, but actively sabotaging it is legitimately dangerous behavior. Its acting in bad faith. And i say this as someone who would mostly oppose such an initiative myself

I would go so far as to say that you shouldnt be employed in the industry. Malicious actors like you will contribute to an erosion of trust thatll make everything worse

by samrus

2/21/2026 at 8:53:45 PM

Might be but sometimes you don’t have another choice when employers are enforcing AIs which have no „feeling“ for context of all business processes involved created by human workers in the years before. Those who spent a lot of love and energy for them mostly. And who are now forced to work against an inferior but overpowered workforce.

Don’t stop sabotaging AI efforts.

by sp00chy

2/22/2026 at 12:56:43 AM

Honestly i kinda like the aesthetic of cyberanarchism, but its not for me. It erodes trust

by samrus

2/21/2026 at 10:14:59 PM

Forcing developers to use unsafe LLM tools is also malicious. This is completely ethical to me. Not commenting on legality.

by tovej

2/22/2026 at 12:54:48 AM

I dont like it either but its not malicious. The LLM isnt accessing your homeserver, its accessing corporate information. Your employer can order you to be reckless with their information, thats not malicious, its not your information. You should CYA and not do anything illegal even if your asked. But using LLMs isnt illegal. This is bad faith argument

by samrus

2/22/2026 at 11:49:52 AM

You're talking about legality again. I'm talking about ethics.

Using LLMs for software development is a safety hazard. It also has a societal risk, because it centralizes more data, more power, more money to tech oligarchs.

It's ethical to fight this. Still not commenting on legality.

by tovej

2/22/2026 at 1:23:36 PM

You're not forced to work there and use those tools. If you don't like it, then leave the job. Intentionally breaking things is unethical especially when you're receiving a paycheck to do the opposite.

by hypeatei

2/22/2026 at 5:00:41 PM

It may be illegal, but it's not unethical.

Doing unethical things because someone pays you would still be unethical. Opposing those while someone pays you is still ethical.

by tovej

2/22/2026 at 5:50:14 PM

Again, no one is forcing him to be there. He's breaking something on purpose. I think you should read up on ethics because this take "I don't like it therefore whatever I do is ethical" is juvenile.

by hypeatei

2/23/2026 at 1:34:47 AM

That's quite the strawman. The reason it's ethical is not that LLM's are unpopular or someone dislikes them. It's ethical because LLMs introduce safety hazards, i.e. they cause harm.

by tovej

2/22/2026 at 7:18:14 AM

Sounds like what an LLM would post if it were tasked to advertise LLM coding abilities. Nice manipulation of human emotions, well played.

by rixed

2/21/2026 at 8:57:04 PM

I see someone is not familiar with the joys of the current job market.

by renegade-otter

2/21/2026 at 8:48:15 PM

That's extremely unethical. You're being paid to do something and you deliberately broke it which not only cost your employer additional time and money, but it also cost your customers time and money. If I were you, I'd probably just quit and find another profession.

by hypeatei

2/21/2026 at 8:22:42 PM

That's not "sabotaged", that's sabotaged, if you intentionally introduced the bugs. Be very careful admitting something like that publicly unless you're absolutely completely sure nobody could map your HN username to your real identity.

by logicchains

2/21/2026 at 8:19:06 PM

Typo: "shop", should have been with an 'el'.

(: phonetically, because 'l's are hard to read.

by Ylpertnodi

2/21/2026 at 7:51:18 PM

It has been roughly speaking five and a half years since the IPO. The original CTO (John Graham-Cumming) left about a year ago.

by lysace

2/21/2026 at 7:55:29 PM

They coasted on momentum for half a year. I don't even think it says anything negative about the current CTO, but more of what an exception JGC is relative to what is normal. A CTO leaving would never show up the next day in the stats, the position is strategic after all. But you'd expect to see the effect after a while, 6 months is longer than I would have expected, but short enough that cause and effect are undeniable.

Even so, it is a strong reminder not to rely on any one vendor for critical stuff, in case that wasn't clear enough yet.

by jacquesm

2/21/2026 at 11:40:21 PM

You can coast for quite some time (5-10 years?) if you really lean into it (95% of the knowledge of maintaining and scaling the stack is there in the minds of hundreds of developers).

Seems like Matthew Prince didn't choose that route.

by lysace

2/22/2026 at 1:51:16 AM

The problem is that CF operates in a highly dynamic environment and you can't really do that if the minds of those hundreds of developers relied for the major decision making on a key individual.

This is the key individual paradox: they can be a massive asset and make the impossible happen but if and when they leave you've got a real problem unless you can find another individual that is just as capable. Now, I do trust JGC to have created an organization that is as mature as it could be, but at the same time it is next to impossible to quantify your own effect on the whole because you lack objectivity and your underlings may not always tell you the cold hard truth for reasons all their own.

And in this case the problem is even larger: the experience collected by the previous guru does not transfer cleanly to the new one, simply because the new one lacks the experience of seeing the company go from being a tiny player to being a behemoth, and that's something you can do only once.

I've always been of the opinion that without JGC Cloudflare did not stand a chance, irrespective of those hundreds of developers. And that's before we get into things like goodwill.

And of those hundreds of developers you have to wonder how many see the writing on the wall and are thinking of jumping ship. The best ones always leave first.

I would not be surprised at all if this whole saga ends with Google, Microsoft or Amazon absorbing CF at a fraction of its current value.

by jacquesm

2/21/2026 at 7:57:03 PM

I wondered what happened to him?

by dazc

2/21/2026 at 9:09:30 PM

I am reading HN.

by jgrahamc

2/21/2026 at 10:01:07 PM

What is your opinion on the recent Cloudflare outages?

by SoKamil

2/21/2026 at 8:03:44 PM

He's on a yacht somewhere

by brcmthrowaway

2/21/2026 at 8:11:11 PM

For real

by tedd4u

2/22/2026 at 7:12:33 AM

He's still a member of the board though.

by Cipater

2/21/2026 at 7:55:41 PM

Launching a new service every 5 minutes is obviously stretching their resources.

by dazc

2/21/2026 at 8:20:36 PM

been at cf for 7 yrs but thinking of gtfo soon. the ceo is a manchild, new cto is an idiot, rest of leadership was replaced by yes-men, and the push for AI-first is being a disaster. c levels pretend they care about reliability but pressure teams to constantly ship, cto vibe codes terraform changes without warning anyone, and it's overall a bigger and bigger mess

even the blog, that used to be a respected source of technical content, has morphed into a garbage fire of slop and vaporware announcements since jgc left.

by slophater

2/21/2026 at 10:43:57 PM

Do you feel that Matthew Prince is still technically active/informed? I've interacted with him in the past and he seemed relatively technically grounded, but that doesn't seem as true these days.

by sebmellen

2/22/2026 at 12:06:45 AM

https://xcancel.com/eastdakota/status/2025215495142564177

https://xcancel.com/eastdakota/status/2025221270061580453

Rather than be driven by something rational like building a great product or making lots of money he is apparently driven by a desperate fear of being a dinosaur.

Regardless of how competent he is or isn’t as a technologist, a leader leading with fear is a recipe for disaster.

by 3rodents

2/22/2026 at 8:44:07 AM

Development has changed, I don't see much else he mentions.

As a dev, you'll need to have those stuff in your knowledge toolbox

by NicoJuicy

2/21/2026 at 8:54:43 PM

I’ve had a lot of problems lately. Basic things are failing and it’s like product isn’t involved at all in the dash. What’s worse? The support.. the chat is the buggiest thing I’ve ever seen.

by goalieca

2/21/2026 at 10:24:50 PM

don't worry, if it gets much worse the ceo will just throw all of support under the bus again. it will surely get better.

by slophater

2/22/2026 at 12:21:00 AM

How about accurate billing info. The ux can’t even figure out we’re annually not monthly. Maybe the AI slop will continue to miscount resources and cost you revenue or piss off a customer when the dashboards they been using don’t match the invoice

by goalieca

2/21/2026 at 9:00:15 PM

You know what they say, shit rolls downhill. I don't personally know the CEO, but the feeling I have got from their public fits on social media doesn't instill confidence.

If I was a CF customer I would be migrating off now.

by __turbobrew__

2/22/2026 at 5:17:34 AM

do you care that much about leadership when using a service? even I dont know gcp's c-level, aws's c-level, even vercel's c-level. only know rauchg.

i think i care much more about our SLAs (if any)

by nanankcornering

2/21/2026 at 8:47:06 PM

GSD! GSD!! ship! ship! ship!

**everything breaks**

...

**everything breaks again**

oh fuck! Code Orange! I repeat, Code Orange! we need to rebuild trust(R)(TM)! we've let our customers down!

...

**everything breaks again**

Code Orangier! I repeat, Code Orangier!

by a24446ff87

2/21/2026 at 10:20:00 PM

exactly. recently "if the cto is shipping more than you, you're doing something wrong"

cto can't even articulate a sentence without passing it through an LLM, and instead of doing his job he's posting the stupidest shit to his personal bootlicking chat channel. I cringe every time at the brown-nosers that inhabit that hovel.

no words for what the product org is becoming too. they should take their own advice a bit further and just replace all the leadership with an LLM, it would be cheaper and it's the same shit in practice

by slophater

2/22/2026 at 8:26:36 AM

I have worked in some dysfunctional places but nothing like that, does sound bad.

Just got to keep your head, remember it’s just a job and you get paid regardless. Clock in, clock out, do the work assigned to you but mentally just check out while you look for a new role

by ifwinterco

2/21/2026 at 8:28:01 PM

amazing how my comment was flagged in 30 seconds... keep bootlicking

by slophater

2/21/2026 at 8:12:54 PM

The one redeeming feature of this failure is staged rollouts. As someone advertising routes through CF, we were quite happy to be spared from the initial 25%.

by anurag

2/22/2026 at 1:38:18 PM

Old tech could work around these outages. Set up GSLB at a DNS provider that does health checks or perform your own health checks to both origin and CDN's and use API's to change DNS. If the origin servers are OK and the CDN is not, automatically change DNS to a different CDN. There should be multiple probes that form a consensus. This process assumes one is managing the configurations of their CDN's through code and API so that one can set up and tear down any number of CDN's on a whim.

That does mean having contracts with more than one CDN provider however the cost should be negotiated based on monthly volume. i.e. the CDN with the most uptime gets the most money. If an existing CDN under contract refuses to negotiate then move some non critical path services to them and let that contract expire. Instate a company wide policy to never return to a vendor if their contract was intentionally not renewed.

by Bender

2/21/2026 at 8:12:20 PM

Hindsight is 20/20 but why not dry run this change in production and monitor the logs/metrics before enabling it? Seems prudent for any new “delete something in prod” change.

by jaboostin

2/21/2026 at 7:54:24 PM

This blog post is inaccurate, the prefixes were being revoked over and over - to keep your prefixes advertised you had to have a script that would readd them or else it would be withdrawn again. The way they seemed to word it is really dishonest.

by himata4113

2/21/2026 at 8:37:03 PM

> Because the client is passing pending_delete with no value, the result of Query().Get(“pending_delete”) here will be an empty string (“”), so the API server interprets this as a request for all BYOIP prefixes instead of just those prefixes that were supposed to be removed.

Lmao, iirc long time ago Google's internal system had the same exact bug (treating empty as "all" in the delete call) that took down all their edges. Surprisingly there was little impact as traffic just routed through the next set of proxies.

by dilyevsky

2/21/2026 at 7:28:47 PM

While neither am I nor the company I work for directly impacted by this outage, I wonder how long can Cloudflare take these hits and keep apologizing for it. Truly appreciate them being transparent about it, but businesses care more about SLAs and uptime than the incident report.

by boarush

2/21/2026 at 7:42:39 PM

I’ll take clarity and actual RCAs than Microsoft’s approach of not notifying customers and keeping their status page green until enough people notice.

One thing I do appreciate about cloudflare is their actual use of their status page. That’s not to say these outages are okay. They aren’t. However I’m pretty confident in saying that a lot of providers would have a big paper trail of outages if they were more honest to the same degree or more so than cloudflare. At least from what I’ve noticed, especially this year.

by llama052

2/21/2026 at 7:47:21 PM

Azure straight up refuses to show me if there's even an incident even if I can literally not access shit.

But last few months has been quite rough for Cloudflare, and a few outages on their Workers platform that didn't quite make the headlines too. Can't wait for Code Orange to get to production.

by boarush

2/21/2026 at 7:58:41 PM

Bluntly: they expended that credit a while ago. Those that can will move on. Those that can't have a real problem.

As for your last sentence:

Businesses really do care about the incident reports because they give good insight into whether they can trust the company going forward. Full transparency and a clear path to non-repetition due to process or software changes are called for. You be the judge of whether or not you think that standard has been met.

by jacquesm

2/21/2026 at 8:04:22 PM

I might be looking at it differently, but aren't decisions over a certain provider of service being made by the management. Incident reports don't ever reach there in my experience.

by boarush

2/21/2026 at 11:31:27 PM

Every company that relies on their suppliers and that has mature management maintains internal supplier score cards as part of their risk assessment, more so for suppliers that are hard to find replacements for. They will of course all have their of thresholds for action but what has happened in the last period with CF exceeds most of the thresholds for management comfort that I'm aware of.

Incident reports themselves are highly technical, so will not reach management because they are most likely simply not equipped to deal with them. But the CTOs of the companies will take notice, especially when their own committed SLAs are endangered and their own management asks them for an explanation. CF makes them all look bad right now.

by jacquesm

2/21/2026 at 8:50:18 PM

In my experience, the gist of it does reach management when its an existing vendor. Especially if management is tech literate

Becuase management wants to know why the graphs all went to zero, and the engineers have nothing else to do but relay the incident report.

This builds a perception for management of the vendor, and if the perception is that the vendor doesnt tell them shit or doesnt even seem to know theres an outage, then management can decide to shift vendors

by samrus

2/22/2026 at 1:52:01 AM

the 'empty string = select all' pattern in the delete path is the kind of bug that static typing and explicit null handling would catch at compile time. when your delete API accepts a bare query string, you're one missed validation away from this. probably the deeper lesson is that destructive operations should require explicit confirmation of scope, not just 'no filter = everything.'

by snowhale

2/21/2026 at 8:05:37 PM

If you track large SaaS and Cloud uptime, it seem to correlate pretty highly with compensation for big companies. Is cloudflare getting top talent?

by VirusNewbie

2/21/2026 at 8:10:15 PM

Based on IPO date and lockups, I suspect top talent is moving on.

by bombcar

2/21/2026 at 8:04:29 PM

Sure vibe-coded slop that has not been properly peer reviewed or tested prior to deployment is leading to major outages, but the point is they are producing lots of code. More code is good, that means you are a good programmer. Reading code would just slow things down.

by henning

2/21/2026 at 8:48:03 PM

that’s my feeling also. We will get this more and more in future.

by sp00chy

2/22/2026 at 12:49:54 AM

The code they posted doesn't quite explain the root cause. This is a good study case for resilient API design and testing.

They said their /v1/prefixes endpoint has this snippet:

  if v := req.URL.Query().Get("pending_delete"); v != "" {
      // ignore other behavior and fetch pending objects from the ip_prefixes_deleted table
      prefixes, err := c.RO().IPPrefixes().FetchPrefixesPendingDeletion(ctx)
      
      [..snip..]
  }
What's implied but not shown here is that endpoint normally returns all prefixes. They modified it to return just those pending deletion when passing a pending_delete query string parameter.

The immediate problem of course is this block will never execute if pending_delete has no value:

  /v1/prefixes?pending_delete   <-- doesn't execute block
This is because Go defaults query params to empty strings and the if statement skips this case. Which makes you wonder, what is the value supposed to be? This is not explained. If it's supposed to be:

  /v1/prefixes?pending_delete=true   <--- executes block
Then this would work, but the implementation fails to validate this value. From this you can infer that no unit test was written to exercise the value:

  /v1/prefixes?pending_delete=false   <-- wrongly executes block
The post explains "initial testing and code review focused on the BYOIP self-service API journey." We can reasonably guess their tests were passing some kind of "true" value for the param, either explicitly or using a client that defaulted param values. What they didn't test was how their new service actually called it.

So, while there's plenty to criticize on the testing front, that's first and foremost a basic failure to clearly define an API contract and implement unit tests for it.

But there's a third problem, in my view the biggest one, at the design level. For a critical delete path they chose to overload an existing endpoint that defaults to returning everything. This was a dangerous move. When high stakes data loss bugs are a potential outcome, it's worth considering more restrictive API that is harder to use incorrectly. If they had implemented a dedicated endpoint for pending deletes they would have likely omitted this default behavior meant for non-destructive read paths.

In my experience, these sorts of decisions can stem from team ownership differences. If you owned the prefixes service and were writing an automated agent that could blow away everything, you might write a dedicated endpoint for it. But if you submitted a request to a separate team to enhance their service to returns a subset of X, without explaining the context or use case very much, they may be more inclined to modify the existing endpoint for getting X. The lack of context and communication can end up missing the risks involved.

Final note: It's a little odd that the implementation uses Go's "if with short statement" syntax when v is only ever used once. This isn't wrong per se but it's strange and makes me wonder to what extent an LLM was involved.

by abalone

2/22/2026 at 12:56:19 AM

> But there's a third problem, in my view the biggest one, at the design level. For a critical delete path they chose to overload an existing endpoint that defaults to returning everything. This was a dangerous move. When high stakes data loss bugs are a potential outcome, it's worth considering more restrictive API that is harder to use incorrectly. If they had implemented a dedicated endpoint for pending deletes they would have likely omitted this default behavior meant for non-destructive read paths.

Or POST endpoint, with client side just sending serialized object as query rather than relying that the developer remembers the magical query string.

by PunchyHamster

2/21/2026 at 7:49:40 PM

The eternal tech outage aphorism: It's always DNS, except for when it's BGP.

by ssiddharth

2/21/2026 at 10:00:31 PM

You could argue BGP is like DNS for IPs :)

by subscribed

2/22/2026 at 12:55:57 AM

bitbucket was done for a while as well. Seems no one noticed.

by est

2/21/2026 at 8:42:48 PM

This transparent report can earn my trust

by wa008

2/21/2026 at 8:25:22 PM

again?

by NooneAtAll3

2/21/2026 at 8:15:28 PM

Is this trend of oversharing code snippets and TMI postmortems done purposely to distract their customers from raging over the outage and the next impending fuckup?

by tokyobreakfast

2/21/2026 at 8:53:10 PM

Just seems like transparency. I agree that we should also judge them based on the frequency of these incidents and amwhether they provide a path to non-repeatability, but i wouldnt criticize them for the transparency per se

by samrus

2/21/2026 at 8:19:00 PM

Well I still appreciate a good postmortem even if I have no doubt it'll happen again imminently

by alansaber

2/21/2026 at 8:21:52 PM

and if they didn’t we’d posting about lack of transparency. damned if you do, damned if you don’t

by bdangubic

2/21/2026 at 7:35:02 PM

DaaS - Downtime as a Service©

Just joking, no offence :)

by dryarzeg

2/21/2026 at 9:03:24 PM

DaaS is good ja

by logicchains

2/21/2026 at 9:09:28 PM

I'm honestly amazed that a company CF's size doesn't have a neat little cluster of Mac Minis running OpenClaw and quietly taking care of this for them.

by djfobbz

2/21/2026 at 9:09:07 PM

They should have rewritten this code in Rust using these brilliant language models. /jk

by user205738