alt.hn

6/15/2026 at 10:36:33 PM

Cell-based architecture for resilient payment systems

 https://americanexpress.io/cell-based-architecture-for-resilient-payment-systems/

by birdculture

6/18/2026 at 10:28:29 PM

Nobody uses Amex for payments, so the system isn't ever under high load.

Just kidding!

I find the idea quite good, and have to assume that the amount of payment fails they experience due to partitions/outages isn't very high and that the post-payment reconciliation and reclamation process gives them the liberty to rank availability a bit higher than correctness.

One thing that looked a bit shaky was the interplay between the global transaction router's state of knowing which cells can handle a particular payment and the asynchronous distribution of the "failover data", which I presume it needs to know to route correctly. To me that seems to create a window where it might route to the wrong cell due to an outdated routing state.

It also doesn't go into the HA setup of the global transaction router itself.

But still, I kind of like the design.

by physix

6/18/2026 at 10:49:47 PM

>To me that seems to create a window where it might route to the wrong cell due to an outdated routing state.

But if the router sends to the wrong cell the cell will either send it back to be rerouted or it will fail and the router will try again (or report back the failure so upstream can try again I assume)

by mixdup

6/19/2026 at 1:08:59 AM

That would be the good case.

But what if the cell doesn't know that, and it's holding, for example, a stale account number?

by physix

6/19/2026 at 1:30:07 AM

Generally with a credit card, or many banking systems more generally, because they predate computers, it's possible that a charge might be accepted even if there's no knowledge whether the money is in the account. As long as the person who was supposed to have paid is identifiable, the money is taken from their account anyway in the end, and if they don't have it, they get sued and their wages garnished, and if they also don't have wages, that's a small enough percentage of people that it's part of the cost of doing business.

by inigyou

6/19/2026 at 10:40:03 AM

Too true. We all think these payment systems are 'strongly consistent' and RDBMS vendors and projects since time immemorial love to talk about their ACID suitability for payments. In reality, zoom out enough and it's all eventually consistent and resolved through the legal system if the computers failed.

by greengreengrass

6/19/2026 at 8:39:13 AM

Amex is gaining popularity for acceptance

by whazor

6/19/2026 at 10:06:10 AM

Do they still charge ridiculously high fees to merchants?

by inigyou

6/19/2026 at 4:08:28 PM

Merchants near me are using more computer based terminal that seem to take nearly every thing. Amex's cut probably smaller compared to squares or stripes.

When talking to a local merchant about why they dont take amex the fee was not mentioned. The merchant said that visa and mastercard settle at the end of the day and they get their money but amex it was sometimes 4 or 5 days before they actually got their payment.

by tempest_

6/19/2026 at 12:32:09 AM

Whole lot of nothing.

This isn't about payment technologies, it's not about isolating transactions, it's about scaling the middle layer. What's worse it's not even explained what middle layer does.

No info on how routing works, no info on data synchronization.

Folks just learning Kubernetes and write extremely abstract stuff.

by Insimwytim

6/19/2026 at 12:17:29 PM

The problem is there is no incentive to talk about things in simple terms, in both companies and academia. If you make your work look simpler, it is assumed it is easier and thus your contribution is less. There are incentives therefore to make your work look more complex than it is.

by zipy124

6/19/2026 at 4:24:50 AM

I agree, confusing and it seems like they are using the coined word "cell" to describe "container", but they really should say that instead of making up new words.

by jijji

6/19/2026 at 5:48:12 AM

Making up new words is how you get to take ownership of existing concepts. This gets you promoted.

by genxy

6/20/2026 at 5:32:31 AM

if a "cell" is different from a "container", from the articles context, please explain how... This could be some special FPGA board too referred to again as a "cell", but they are extremely vauge about the definition.

by jijji

6/20/2026 at 2:54:17 AM

This matching up with sliced data, either sliced by recipient, or sliced by sender, so it makes sense to have primary state with cool backups. Particularly if the state is race-tolerant.

As long as event orderings are unimportant, or self resolvable this works really well.

e.g. if Events A,B arrive, but A+B => C and B+A => C, then as long as you durably record A, B, the end state is the same.

I'm not sure why "reroute" is a message instead of a response, I would expect it to be a failure response pruning the control flow. With the GTR being "default retry".

There's a lot of learned experience in that doc. Reading between the lines, both the logging and configuration systems have caused global outages (or near misses). Nifty to read.

This style of architecture fits with a "no global changes" and "never lose it all" approach to fault tolerance, accepting that there will be visits from "Mr. Cock-up" [1].

Very nice writeup.

[1] https://www.youtube.com/watch?v=D5r8xwu0l8w

by jpollock

6/19/2026 at 3:35:08 AM

Some of it sounds like it reinvented Erlang supervision trees https://learnyousomeerlang.com/supervisors. As a joke there we’re calling gen_severs “nanoservices”. Granted, that was mostly when microservices were the hot new thing.

by rdtsc

6/19/2026 at 6:24:31 AM

Immediately my first thought as well. I still keep coming back to Erlang/Elixir/OTP as the best possible choice all things considered. I should use it more.

by pyinstallwoes

6/19/2026 at 12:53:01 AM

microservices / clusters / zones - really all of these are other "cell-based" architectures as well. there is absolutely no written rule that a microservice was just an API or a singular service, it basically can be a independent instance that is testable/usable/gives value on itself.

that said: still a nice write up, learning about some of the architectural choices that AMEX makes is definitely insightful (and relavent/useful to what i am working on right now as well!)

by mkhalil

6/18/2026 at 11:26:42 PM

All i can see is a giant single point of failure called the Global Transaction Router.

by nightshift1

6/18/2026 at 11:46:04 PM

GLBs aren’t SPOFs. They are typically deployed around the world redundantly, often using Anycast IPs or using DNS geographic and failover records, and are stateless. Think AWS Global Accelerator and Route 53 as an example. The architecture diagram is a high level simplification.

by otterley

6/19/2026 at 4:28:23 AM

I don't think the global transaction router is a GLB. Having dabbled in this for high traffic telemetry gathering infrastructure, I will hazard a guess and say the "router" isn't a GLB.

The router needs to be shard-aware. It needs to know what data is where based on the request coming in so that it can route accurately. A GLB is DNS. It cannot be shard-aware because all it knows is the FQDN being resolved.

It can be a "router" if all the router needs to know is to resolve to the nearest data center or the nearest CDN. But at that point I have to ask the question - why does one need a cell-based architecture and can't it just be geo-redundant active-active failover across regions.

In any sense, the architecture itself isn't novel or new. It's documented here: https://docs.aws.amazon.com/wellarchitected/latest/reducing-.... It's the go to model if you're running a cloud.

by orochimaaru

6/19/2026 at 6:01:09 AM

Active/active without sharding is not a horizontal scaling model, and the blast radius of a fault is wide.

One can have GLBs that do routing. So long as the tenant-to-cell routing tables are consistent, it works fine. And those mappings tend not to change frequently.

by otterley

6/19/2026 at 1:04:28 AM

GLBs absolutely can be SPOF for certain kind of administrative mistakes.

by wbl

6/19/2026 at 6:02:18 AM

If you’re counting administrative mistakes (human error), anything can fail. Let’s not shift the goalposts.

by otterley

6/19/2026 at 2:21:44 AM

That depends on your change control process

by lokar

6/19/2026 at 2:08:22 AM

A large portion of DNS is outside of your control. You're relying on at least two third parties you have indirect relationships with in order to work. If you're outside of the standard TLDs you've got additional social factors that can control your resolution.

Granted. It works really well in practice. It should be noted we haven't actually had the world war the Internet was designed to survive. So we're not entirely clear on the semantics of operations in unusual and unexpected configurations. I would expect DNS to be the first shoe to drop there.

by themafia

6/19/2026 at 2:41:39 AM

Isn't it reassuring the entire CC business hinges on a single proprietary appliance inside a cage at some DC? :)

by manoDev

6/19/2026 at 12:15:27 AM

403 Forbidden

Because of the title I was expecting to read about doing payments with a distributed network, like a terrorist cell network, or something like Hawala. Not (as I infer from other comments) Amex using multiple independent systems.

by inigyou

6/18/2026 at 9:39:54 PM

I wonder how they ensure durability. Is it possible that a cell going down would roll back a payment after it has occurred. Or do they depend on a non cell database?

by neerajsi

6/18/2026 at 10:15:35 PM

I would assume nothing related to a given transaction crosses the cell boundary.

We use a cellular architecture to help constrain the blast radius of a modular monolith. Each one of our customers lives in exactly 1 cell. Any kind of cross-customer BI/reporting happens through a data warehouse.

by subtlejellyfish

6/18/2026 at 9:22:42 PM

There things are always a clusterfsck compared to the mainframe deployments.

by kev009

6/18/2026 at 11:12:01 PM

Ahahha so true man!

Some CICS regions, a DB2 and a couple of VSAMs and that's it.

by vb-8448

6/19/2026 at 4:27:44 PM

As described, if one "cell" crashes, you re-route/retry everything on the other cell. Assuming your system is deterministic and the inputs stay the same, the second cell should now break as well.

This reads to me like an attempt to patch a system that's already fucked beyond belief while pretending you're doing "engineering".

Fancy implementation of a retry loop attempting to minimize downtime.

by chmod775

6/18/2026 at 11:04:04 PM

Backing up would be hell

by stevefan1999

6/19/2026 at 5:27:00 AM

Backups in such a system are quite pointless; if losing 10 seconds of data means you lost 4000 transactions then periodic backups are invalid if not instantly than close to instantly.

The system I work on has such a property and the only real infra style approach is sync replication before responding to a caller and a delayed replica for delete/drop protections (say with a 2hr or more window).

Should also defend for this in your code (be able to reply from your initiation systems also etc)

by cyberpunk

6/18/2026 at 11:19:24 PM

Maybe? If you assume a cell can just disappear at a moment's notice, then I'm guessing you don't even try backing it up. Whatever goes into and out of the cell (request logs and results) gets backed up, and no doubt that's more complicated than a monolithic system, but it may not be so bad assuming the replay systems and global transaction router do their thing?

by simmonmt

6/19/2026 at 4:36:58 PM

"We broke our monolith into microservices and wish we hadn't"

by daxfohl

6/18/2026 at 10:35:16 PM

Ah yes, the financial services company that runs a travel agency, allows me to book my hotel and rental car weeks in advance, registers a hold for incidentals for both the hotel and car when I check in, then blocks the card when I try to buy dinner that night in that same hotel due to fraud detection.

Last week it required me to take pictures of my face from multiple angles to regain membership privileges. I suspect this may be part Palantir data collection and part Peter Thiel dating service.

by badlibrarian

6/19/2026 at 9:49:18 AM

I would have nope'd out so hard if they asked for face pictures.

by Scaled

6/19/2026 at 1:40:18 PM

PayPal does it. Synchrony Bank does it. Use your eBay credit card to buy toner, shipped to the address on your card, just like you do every 90 days? Must be fraud.

They email you an alert. You click "this is okay." Card still doesn't work, you call. Rep asks ID questions, still not enough. Says he has to text a QR code linking to some wonky app that videos your face, no other option.

You can't even login to pay the card off and cancel without going through the process.

by badlibrarian

6/19/2026 at 10:08:37 AM

That's just normal for banks now

by inigyou

6/19/2026 at 3:50:56 PM

Blockchain payments lite

by bozhark

6/18/2026 at 9:42:01 PM

As Reddit already pointed out, this is nothing novel.

by jeremycarter

6/18/2026 at 11:16:16 PM

“They reinvented Erlang OTP.” - Reddit

by christophilus

6/19/2026 at 12:42:29 AM

Don't know if Joe Armstrong ever said anything like it, but I would propose naming an Erlang/OTP analogue of Greenspun's tenth rule (the one about C projects containing ad-hoc, buggy implementations of Lisp) for him.

by tomjakubowski

6/19/2026 at 5:23:55 AM

There is Virdings Rule:

Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.

by cyberpunk

6/19/2026 at 2:11:02 AM

To be fair Elixir shows you can just use the BEAM if you want. If you need these semantics at this level there's very few reasons not to go this route.

by themafia

6/19/2026 at 1:08:04 AM

Still nice to have learning resources like this pass through HN even if it’s not novel.

by therealdrag0

6/19/2026 at 1:59:06 AM

This service oriented architecture except more expensive and complicated.

by charcircuit

6/18/2026 at 10:20:38 PM

American Express tech is some of the worst in the world among big companies. All of the value in the company is just in the branding. They put some work into the mobile app and the website, but other than that, its a facade.

by llmslave

6/18/2026 at 10:42:44 PM

A few years ago someone kept signing up for loads of bank accounts/credit cards in my name, with my address. I’m not sure what the point of it was. But while everyone else happily sent cards and stacks of welcome paperwork to me, Amex were the only one that contacted me and told me they’d detected something weird in the signup. They gave me some helpful advice to resolve that situation too.

by mcintyre1994

6/18/2026 at 11:48:02 PM

I froze my credit with the 3 big credit agencies in the US years ago when someone attempted to open multiple Dell and other company accounts in my name. Easy enough to unfreeze for a temporary period of time when I need it.

by buerkle

6/18/2026 at 10:37:34 PM

Having worked at Amex and other huge banks, let me assure you that there's much worse than Amex. Amex's Fraud analytics team was good. Risk was good. Ben's team is good.

by jmpman

6/19/2026 at 2:32:49 AM

BoA reauthorized an auto-payment card even with the card being expiration and uncorrected security code. I would call that authorized fraud by Bank of American.

This is why I find it best to declare a card stolen right before expiration or after.

by yndoendo

6/18/2026 at 11:43:38 PM

What are you basing that statement on? It has not been by personal experience.

by tracerbulletx

6/19/2026 at 11:12:13 AM

[dead]

by greengreengrass

6/18/2026 at 11:13:59 PM

Makes me a little nervous that a web page about resilience is failing to connect.

by great_wubwub

6/15/2026 at 11:34:28 PM

They run their payment systems on ps3??? Somebody bought into the marketting a bit much.

by toast0

6/18/2026 at 9:19:04 PM

So you’re telling me these cells operate independently like distributed Ethereum nodes and L2s… got it.

by rekttrader

6/19/2026 at 12:19:07 AM

Ethereum nodes are not independent, they are as interdependent as it's possible to be.

by inigyou

6/19/2026 at 2:40:17 AM

It's becoming a reliable heuristic that when somebody says "... got it", they probably didn't get it.

by shwaj