1/18/2025 at 4:21:34 AM
I’m my experience and based on writeups like this: Google hates having customers.Someone decided they have to have a public cloud, so they did it, but they want to keep clients away with a 3 meter pole.
My AWS account manager is someone I am 100% certain would roll in the mud with me if necessary. Would sleep in the floor with us if we asked in a crisis.
Our Google cloud representatives make me sad because I can see that they are even less loved and supported by Google than us. It’s sad seeing someone trying to convince their company to sell and actually do a good job providing service. It’s like they are setup to fail.
Microsoft guys are just bulletproof and excel in selling, providing a good service and squeezing all your money out of your pockets and you are mortally convinced it’s for your own good. Also have a very strange cloud… thing.
As for the railway company going metal, well, I have some 15 years of experience with it. I’ll never, NEVER, EVER return to it. It’s just not worth it. But I guess you’ll have to discover it by yourselves. This is the way.
You soon discover what in freaking world is Google having so much trouble with. Just make sure you really really love and really really want to sell service to people, instead of building borgs and artificial brains and you’ll do 100x better.
by motoboi
1/18/2025 at 4:42:42 PM
My AWS account manager took me fishing. That’s what you get for a >$1M/yr spend. I don’t sense they would roll in mud with me, which is kind of incredible. I wonder how much you need to spend to get into mud rolling territory?by ttul
1/18/2025 at 5:09:54 PM
AWS support in general is extremely good in my experience. (We pay for whatever the tier below Enterprise is called, I think it costs 10% of your spend)I’ve been on 4 hour screenshare with AWS engineers working through some infrastructure issues in the past, and we only spend $100k/yr.
Even at the $100k/yr spend level, AWS regularly reaches out with offers to try new services they’re launching for free. We’ve said “sure” a couple times, and AWS shows up with 4-6 people on their end of the call (half of them engineers).
In the past 10 years, we’ve had maybe 2-3 emergency issues per year, and every time I’m able to get a really smart person on a call within 5 minutes.
This is the #1 thing I’d be concerned about losing if we did colo or bare metal with cheaper providers.
by cj
1/18/2025 at 9:57:05 PM
My experience with AWS support has been downright freaky.With other vendors, when I call a support line with an obscure issue that maybe only I hit in the whole world I fully expect to explain it to an overseas call centre drone with a poor voice line and rudimentary English. Then I expect to have to repeatedly escalate over months and be told “We can’t reproduce this glaringly obvious bug, closed.” That’s ignoring the series of very closely related family of issues I dug up in the process of troubleshooting. Which they continue to ignore because it’s “out of scope” for the ticket. “Open a new ticket and go through the pain again, peasant!”
With AWS my experience has always been “I’ve fixed that right up for you, is there anything else you’d like help with?”. Often after mere minutes!
I’m usually left speechless, ready to go on a well-practiced tirade of how “We’re spending millions on this crap and none of it works properly!”, but instead I just sit there gawping like a fish out of water, stammer “No, thank you, that was it.” and hang up in shame.
I just don’t understand why no other enterprise on Earth seems to have support this good. What’s the thing holding them back!? Maybe they assume that good support works only for this tiny upstart org called Amazon that will clearly never amount to anything!
by jiggawatts
1/18/2025 at 10:35:16 PM
What kind of issues you had that they could fix them immediately? I assume this is not about configuration issues on your part, but maybe I’m mistakenby maayank
1/18/2025 at 11:09:25 PM
I was one of the first users of the AWS Elastic File System because I had an ideal use-case for it exactly when it was first introduced. Everything worked just fine for 30 days, and then the web site basically locked up. It turned out that EFS had an initial "grace period" during which IOPS were unlimited, then it would become proportional to the GB of stored data. We had just a few hundred megabytes, so it worked out to something like 0.4 IOPS. Slower than a floppy drive! Support immediately reset the grace period for us, flipped some internal flag to make it not expire, and then a few months later the product itself was fixed to have a reasonable minimum IOPS per account irrespective of the data volume. At the time there were zero mentions of any of this on Google, I must have been the first few people to hit this issue after general availability.A direct comparison is a nearly identical issue with Azure SQL Server Managed Instance. It too had IOPS proportional to the size of each file of a database. We migrated a database that used many small partitions (per month I think?), each with its own small file. Its performance was horrendous, easily 100x slower than on-prem. The support team could barely speak English, got repeatedly confused about the product (Azure SQL Database != SQL Managed Instance), couldn't understand the problem, and even insisted that totally broken performance "was a feature" and we should redesign "our database". Sure buddy, I'll go tell the third-party vendor that, meanwhile Microsoft themselves insisted we should migrate all of our legacy databases to this garbage. We did, it took months, cost a ton of money, and now it basically doesn't work! We abandoned the product, as have many other people. At the time, this had been an issue for many years with Microsoft engineering basically whistling at the ceiling as they cheerfully ignored it. More than a year later they fixed it, but you've got to wonder what else is still wrong with it that support can't help with.
There's more examples, but that pair stuck in my mind because they had the same root cause but wildly different outcomes.
by jiggawatts
1/19/2025 at 1:49:14 PM
Very interesting, thanks for the write up!by maayank
1/19/2025 at 1:49:43 AM
I've had similar experiences with Google as well. Reaching out with new services, hours with some of their technical people, invites to meetups, free credits, an extremely pleasing and responsive account manager. We spend a few hundred thousand dollars a year with them. The actual software is top notch. Most haven't been just turn it on and forget it.by neeleshs
1/19/2025 at 4:40:11 AM
Yeah, I'm a little biased here as I now work at Google, but I joined in part due the positive experience we had migrating from bare metal to Google Cloud.We went through two rounds of migration. First placing our data warehouse, where BigQuery was just so far past Redshift it was almost a joke. Then we wanted to move to a cloud provider with good container orchestration and GKE was obviously better than AKS and all of Amazon's proprietary orchestrators. It was pretty good.
Customer support varied between excellent and ~fine. Amazon customer support throughout that time (we had a few small bits on Amazon) was fine, but less enthusiastic about winning our business.
Not long after a friend of mine reported a security incident to AWS, something that looked like Amazon privileged access to their data, and it took months to get a response from them, and it was never an adequate explanation for what looked in all ways like a hack.
by danpalmer
1/20/2025 at 2:35:48 AM
Yep. BQ,GKE, and at a metalevel the simpler project structure -all have been great. I cannot still fully understand the org hierarchy that AWS has yet.by neeleshs
1/18/2025 at 9:17:45 PM
> ... wonder how much you need to spend to get into mud rolling territory?When I was at AWS, our team used to (religiously / proactively) keep track of customers having multiple complaints, especially repeat complaints (all of which manifested in to some form of downtime for them). Regardless of their spend, these customers ended up getting the "white glove" treatment, which otherwise is reserved for (potential) top spenders (though, engs are mostly oblivious to the numbers).
This is besides the fact that some account managers & support engs may indeed escalate (quite easily at that) to push product eng teams to really & immediately pay that tech debt that's hurting their customers.
by ignoramous
1/19/2025 at 1:10:33 PM
That was probably in the time of Bezos...Now with the new MBA CEO, it seems the rule now is to deprecate services without even putting out a newsletter or blog post. Customers just find out when they click on the Console...by belter
1/19/2025 at 2:21:17 PM
> That was probably in the time of Bezos... Now with the new MBA CEOAndy Jassy? Previously, he ran AWS for a decade and a half.
by ignoramous
1/19/2025 at 3:12:31 PM
What services are you talking about?by tesch1
1/19/2025 at 7:57:29 PM
https://simonwillison.net/2024/Jul/30/aws-codecommit-quietly...https://horovits.medium.com/disruption-ahead-aws-quietly-axi...
by belter
1/18/2025 at 9:51:28 PM
1. AWS and their account managers are relatively frugal compared to other enterprise sales teams. As far as I can tell, this is a good thing.2. More
3. AWS has this idea of “customer obsession.” They will spend an absurd amount of time trying to understand your business and make sense of it.
by sargun
1/18/2025 at 8:25:19 PM
> "My AWS account manager took me fishing. That’s what you get for a >$1M/yr spend."I assume that's written into the contract somewhere and not a kickback, right?
by kingforaday
1/19/2025 at 2:55:33 AM
It wasn’t quite as gauche as I made it sound in my comment. The fishing invitation was extended to a few customers and was an official AWS sponsored event.by ttul
1/18/2025 at 10:45:24 PM
Such a middle-class concern. The elites live on kickbacks.Even interns have to go through training on how accepting on $30 gift might be inappropriate and sway their terribly important judgement..
by ClumsyPilot
1/18/2025 at 9:33:14 PM
Hell, I'd roll in the mud if that's what it takes to upsell $10k worth of compute to $1M.by intelVISA
1/21/2025 at 5:17:18 PM
I worked for a large company that committed to a >$400M spend with AWS. Even though I owned a very tiny piece of that pie, I could get my account manager and a technical resource on the phone at pretty much any time.by fatnoah
1/19/2025 at 1:04:02 PM
> My AWS account manager took me fishing.Unless the company is yours or it's a private company that can raise a compliance issue...Any other gifts?
by belter
1/18/2025 at 1:19:44 PM
> Microsoft (...) have a very strange cloud… thing.Risking a going off on a tangent, this is something I rarely see discussed but is perhaps one of the main problems with Azure. The whole cloud service feels like something someone oblivious to cloud computing would design if all they knew was renting bare metal servers. It's cloud computing in a way that completely defeats the whole concept of cloud computing.
by motorest
1/18/2025 at 8:28:11 PM
Same feeling here. It's like they wanted a way to "play datacenter in the browser", but then asked 30 different teams to do it on their own, and only have them come together after they are all done to put the pieces together.Then find out it's not good at all and go "oh well, I guess we'll polish it over in the UI" (not knowing that no serious scale works with a UI).
If I can't have AWS I'll make do with GCP. But if someone wants to go full Azure, I'll find work elsewhere. Screw that. Life is too short to work with bad technology.
by oneplane
1/20/2025 at 1:31:35 AM
I don't think that's it. I think Microsoft wanted a way to migrate already Microsoft workloads to something they could more aggressively bill by the GB or second or user or whatever revenue extraction metric you're down with. Basically, O365 extended to the entire M$ ecosystem. And for that it seems...er...ok. We've migrated a couple of dozen major M$ workloads from on-prem reasonably easily, and a bunch of little ones. Lots of skillsets transferred easily...I vividly recall talking a really fine SQLServer admin off the ledge when the "move to cloud" mandate came out who's now like "I had to learn a few new things, but it's pretty much like what I was doing before". Big win.But then everyone said "a cloud should do X and Y and Z", and they try to bolt X/Y/Z on to the side with various levels of success. And now all the app owners who aren't native M$ have declared Azure not fit for purpose and picked up the torches and pitchforks. So we're going to support AWS, too.
by kjs3
1/18/2025 at 9:10:27 PM
Seriously wondering what you guys experienced with Azure. Never had an issue and prefer it over AWS.by datavirtue
1/18/2025 at 10:11:16 PM
Same here, I prefer Azure to AWS and I’ve spent multiple years with eachby DrBenCarson
1/19/2025 at 12:42:53 AM
I suppose it depends on what you do with it and what you need.by oneplane
1/18/2025 at 9:57:55 PM
Most of it is not an individual experience or 'event', just bad design with bad results. I'll try to describe some global ones:One of the most bizarre things is the crazy bad resource hierarchy. There are multiple overlapping and incompatible ones. Resources, networks, storage, IAM, billing and org, none of it in a single universal hierarchy. It seems to mirror the idiosyncrasies of legacy enterprise organisations with their fiefdoms, instead of a cloud.
The next useless thing is how you just cannot use what you need when you need it in whatever way you want it. Almost all services are hyper-segmented requiring various premium tiers instead of them being universally available.
I get it, it's a great way to bundle things people don't want and extract as much money out of them, but that only really works if people have no alternative. And those two form the bad architecture/bad technology trifecta with this third one: a lot of services, maybe most of them, seem like some sort of 2005 model where a resource is backed by nothing more than some random managed VM in the backend, with all the problems (failure modes, inconsistent behaviour etc) that come with that model.
Perhaps the reason for those things is simple: Microsoft wanted a way to extract more money from their customers and lock them in even more. Moving workloads to Azure meant something different for them than it did for the rest of the world: you used to have a lage army of common windows sysadmin jobs where there was a lot of local control and local management loops, but when you move that to a common template in someone else's datacenter (Azure, essentially) you can ditch most of those loops and people. Granted, they created those local controls/loops themselves to get a school-to-work microsoft client pipeline (same as say, Cisco or oracle), but I doubt there is any new markets to cater to in that way. Since people tend to be the most expensive and most risky part of a business, being able to take more of them out of the loop, making more of them optional or making them more expendable/remote is a (short-term) positive thing in the spreadsheets of most MBAs, which is who most large companies cater to after all. This did of course backfire and we now have the same quantity of jobs but instead of sysadmin you get 'azure engineer' which is more of a crossover between operational helpdesk and technical application manager. But everyone wins: during your exodus you can sell it as modernisation, when you remove that on-prem burden you can shift your CAPEX and OPEX around, your quarter looks better when you can reduce headcount, and once your bonus is in, you can put some job postings out for the people you are now missing.
Technology-wise, the only thing that really changed was the ways in which people could cut corners. Some corners are pre-cut, while others are uncuttable. API-wise, it's a crapshoot, a turd attempted to be polished by a webui that hides the maelstrom of questionable residue below the surface.
by oneplane
1/18/2025 at 10:47:05 PM
Re: "There are multiple overlapping and incompatible ones. Resources, networks, storage, IAM, billing and org, none of it in a single universal hierarchy." - hierarchy is based on subscription / resource group. Billing is usually done with tags (you can add a tag like "CostCenter": "Online Marketing CostCenter1234")Re: "hyper-segmented requiring various premium tiers instead of them being universally available" - premium tier usually means your service runs on its own Azure VMs; while the other tiers your service shares a VM with other customers. The first choice is more expensive obviously and I prefer to pay for that service only if I need it.
BTW - Azure supports bare metal Linux and Windows. So if these pesky Azure services get in your way you can always go back to your on-prem version, where instead of running your workload on your own VMs you run it on Azure VMs.
by dh2022
1/19/2025 at 12:33:12 AM
Preface: don't worry, this is not a rant aimed at you, I just enjoy off-the-cuff writing sometimes ;-)For your first Re:
That would have been great, but that is just more inconsistency. Some resources exist in resource groups, but some don't and you cannot nest them. IAM has the same problem, you always have to create elements on two sides since Entra is not really an Azure resource, it's parallel to your tenant. Policies for Azure don't exist in Entra, but in MGs and Subscriptions and RGs they do. Those don't affect Entra of course, so now you have two different non-interacting policy trees, except you can reference Entra principals. But not if you want to target STS instead. But you can't always target STS, because that would mean you wouldn't have to buy a premium version of IAM (be it P1 or P2 or PAM). Technically RGs would have never needed to exist if they had their tagging and policy system available from day one.
For your second Re:
Instead of having 1 class of groups or containers, there are many non-interoperable versions. You know who doesn't do that? Everyone else. Same for say, IAM. Principals are principals. Tokens are tokens. Want to authorise something? One universal policy language that can target principals, tokens or a combination. Want to use metadata? That's available too, including tags. Applies on all resources the same way as well. Sure, you'll still not find perfect consistency (looking at you, S3, with a 3rd extra policy option), but there is no artificial distinction or segmentation. There is no 'conditional access' product since we would just call that a policy. There is no 'PAM' product since again, that's just a policy. There is no 'premium' because all features are always available, to everyone. And you know the best part? It's not a parallel tenant construction, it's just part of the same namespace of all other resources. Even Google's weird identity setup treats it all as the same organisational namespace.
It's not like Microsoft is unaware of all of this, they are digging Azure-flavoured graves (for legacy setups) faster than Google can expand their own graveyard, and some features that were really late to the party (like MGs, RBAC, PIM, tagging scope with policies as well) are not surprising to see. But repairing a large fractured product like Azure is iffy at best. Time will tell.
For the BTW: yeah, everyone can in the end run virtual machines, but a cloud just to run some VMs is a real good way to burn money. The value proposition of a cloud is elasticity and consistent API-driven resources (which includes IAM, policy language and tagging). A web UI that starts and stops a hidden VM is essentially just a VPS and plesk straight out of 2005.
From the way persistence is implemented on Azure, you can pretty much tell it's all just personal templated VMs underneath, which is exactly what I don't want. I don't want a "storage account" that configures a bunch of inflexible subresources. Say I want to store some blobs, I'd want to make a bucket for that and on that bucket I'll do my tagging, policies and parameters (availability, durability etc). And then I want to do it again, but with slightly different parameters. And then I want to do it 100 times again with various parameters. So now I need 100+ storage accounts too? Who thought it would be a good idea to add a storage account as an intermediary? Probably nobody. But the technology wasn't ready, so instead of going witha good idea, they went with "this will fit on the spreadsheet of the sales department" and released it. Below the surface somewhere hidden from the public API, this reserves some SAN for you, as if we're playing datacenter-for-hire in 2005...
You might wonder: why does it matter? It matters when you do a lot of changes every day, not just deployments or cookie cutter rollouts, but many different applications, services and changes to existing resources. Entire environments are created and destroyed with 100's of resources many times per day per team, and we can't sit around waiting because Azure wants to stop and cleanup an instance that they run under the hood, and we definitely don't want to pay (6 to 7 figures) for such a construction. We want to make use of fast public services that provision and scale in seconds and have APIs that will actually do the job instead of time out and return internal errors. If a cloud isn't really a cloud, but behaves like a datacenter with windows PCs in it, it doesn't do enough for us.
I'll admit, after migrating the last users off of Azure, the only remaining ones are not doing anything cloud-native anyway, it's all just staff-type SaaS (think: Intune, M365 and some Dynamics), so the amount of new Azure knowledge and experience for me over the past 6 months is a lot less than it used to be. The period around 2017 was when most stuff in Azure became a bit more usable with RBAC and AZ Policies, but that was like 6 years too late and to this day is a split world with Entra, yet completely dependant on Entra. Even external identities cannot use STS directly and will have to use static SP credentials. A cursory look at the current docs shows it's still a (premium in secure uses) case. I get it, that's how Microsoft can make more money, but it is technically a bunch of nonsense as other clouds have shown.
by oneplane
1/18/2025 at 11:10:45 PM
These reads like you learned one cloud platform and expected all others to be the same.by amaccuish
1/19/2025 at 12:33:30 AM
Well, regardless of how it reads, that is not the case.by oneplane
1/18/2025 at 4:43:27 PM
Can you be more specific?by yeahwhatever10
1/19/2025 at 11:04:35 AM
> Can you be more specific?To start off, read up on the ass-backwards concept of an app service plan. That nonsense is the ultimate anti-cloud computing approach to cloud computing.
https://learn.microsoft.com/en-us/azure/app-service/overview...
by motorest
1/19/2025 at 1:28:12 AM
It's sad, because I legit found my experience working with Google's "serverless" stuff (like Cloud Run) to be superior to the AWS equivalent. The GCP command line tools ("gcloud") also feel better designed.by icedchai
1/19/2025 at 12:26:54 PM
That's the thing GP is saying, Google might excel in engineering and even build superior products, but the issue bringing them down these days is that they can't manage customers/partners/etc for shit and if they fumble on search it could be over.Most telling example was how iirc Terraria was a launch highlight for Stadia to show awesome indies, then somehow their magic systems lock down the developers account and despite internal pressure from Stadia devrel people they don't get it back in time until the developer just cancels development of the Stadia port. https://www.reddit.com/r/Games/comments/lf7iie/terraria_on_s...
by whizzter
1/19/2025 at 11:49:05 PM
No account manager can help when the support is so bad it would have been better if they admitted they had no idea and superb if they admitted the feature we were sold didn't exist and had no plans of existing.Would save me months of lead time.
Personal experience goes that Google Cloud support treated us quite well even when called by small 3 person team doing minuscule spend, in another company Microsoft treated us very well but our spend could be probably tracked by nationwide powergrid monitoring of their datacenters.
And AWS lied about features and ultimately never responded back.
I figure the account managers talking to high level management about contracting mandatory multi-million spend on AWS know how to talk with said management.
But at the end, what comes to actually developing and delivering products for others, we were left in the dust.
To make it funnier, part of what made it so hard was that the feature they lied to us was supposed to be critical for making sure the UX for end-users was really stellar.
by p_l
1/19/2025 at 5:28:03 AM
As a dev I recently sent my first AWS support request. Received a non useful response featuring factually incorrect statements about their own platform. Replied to support ticket, no reply. Sent email to two AWS reps, never got a reply.by xhrpost
1/19/2025 at 12:10:28 AM
My potential aws account manager told me I was stupid, and that if I listened carefully to him, I would understand he was right and I was wrong.I’m quite happy I’m not using aws - in my case (hpc, spot instances don’t work ) they don’t work.
by Agingcoder
1/19/2025 at 1:51:22 AM
I'm probably an outlier here. My experience with GCP support has been nothing but stellar, like I described in another comment down belowby neeleshs
1/19/2025 at 2:19:46 PM
Having something work right is worth 10 account managers rolling in the mud in my opinion.by osigurdson