You must log in or register to comment.
It feels like those tiny bars should be stocked on top of each other with all the spof projects and services
Just wait until 8.8.8.8 dies.
It’s a Google service, bound to be killed sometime and with minimal warning.
They’d do something really cancerous first, like shoving in AI somehow
Lol imagine a dns directory run by an LLM. “I couldn’t find catvideos.com but don’t worry, you got this! Why don’t you visit bestiality.com instead?”
no! i said anime catgirls!
1.1.1.1 went down for a while not too long ago
Tech tip switch to quad9 for filtered dns
I laugh every time AWS goes down. That’s what you fucking get, and don’t get me started on
us-east-1specifically.It’s like all you people forgot how much shit broke before AWS. They have one major outage every few years and people lose their shit pretending they aren’t hitting the SLA or coming close.
Because hosting was more diverse before, so when shit happened it took out a couple of sites, not a quarter of the internet
And? God forbid we touch grass for 6 hours a year.
“Outages are a GOOD THING!” / FOX talking head
Sure that’s what I said.
Go ahead to rack space, or SAP, I’m sure you’ll have a much more reliable experience. Or just run your own. I’m sure it’ll be easy peasy and super reliable.
Yeah, when I can’t access my bank account the first thing I do is “touch grass.” 🥴
Yup, I’m sure your bank would never go down on another provider. Never.
Well lucky for me they don’t use AWS so I haven’t seen an outage in 10+ years.
The outage also took down people’s banks, which stopped many of them from doing things like buying groceries 💀
I don’t think saying it’s good for us “touching grass” is a good argument here when AWS hosts such a substantial portion of all online services.
How many banks didn’t work? Which ones? You have a source? Visa and MC were good all day here in the real world in the east coast.
Sounds like you’re just trying to exaggerate around an edge case that frankly isn’t the end of the world even if it were common for 4 hours a year
Why aren’t you blaming the bank for having redundancy outside a single DC? How many banks do you know if that were out susessfully using other providers that have a higher SLO/SLA?
I’m also skeptical that any payment processing networks were impacted. I would be surprised, but less so if they couldn’t manage their account online which might have similar effect. I’m not surprised at all of the grocery store or restaurants were significantly impacted. I know a lot of the apps were broken and I could imagine someone used to apping everything leaving their cards at home and unable to get lunch. Might have some aggressively “modern” establishments that are kiosk only and I could imagine them getting downed by aws outage.
outside a single DC?
I’m told that a lot of the companies did all the right things but still got taken down because some dependent Amazon services are tethered to that single DC and only Amazon has the power to change that.
I’ll wait for the final root cause but…
We mitigated most of it by swapping to secondary DNS and completely taking any thing related to AWS DNS and services in useast1. If you didn’t have secondary DNS and heavily reliant on AWS internal DNS this might be something they experienced.
I can see why your account is marked with two red marks on PieFed for low reputation, because man do you come off confrontational.
How many banks didn’t work? Which ones? You have a source?
Search engines exist. Use them before acting as if I"m making shit up.
The list of financial institutions that had issues, as far as I can tell from industry reporting and downdetector graphs, is Navy Federal Credit Union (~15 million members), Truist (~15 million customers), Chime (~8-9 million customers), Venmo (~60 million users), Ally Bank (~10 million customers), and Lloyds Banking group (~30 million customers).
Assuming no overlap, that’s nearly 140 million people that lost banking and money transfer access.
Sounds like you’re just trying to exaggerate around an edge case that frankly isn’t the end of the world even if it were common for 4 hours a year
The outage lasted for 15 hours in some cases, due to many AWS services recovering after the outage, yet having a backlog to work through, which took many more hours. Many services also depend on AWS in a manner where AWS coming back online doesn’t instantaneously restart service. These systems are complex, and not every company that relied on them could instantly start back up the moment the main outage was resolved, let alone when many services were still marked as impacted for hours and hours later as they worked through their backlog.
Why aren’t you blaming the bank for having redundancy outside a single DC? How many banks do you know if that were out susessfully using other providers that have a higher SLO/SLA?
I also blame them for not having additional redundancy. I blame both them for not having a fallback, and AWS for allowing such a major outage to happen. Shockingly, more than one party can be at fault.
Some of us have jobs. I mean I guess you have a job, but in your case losing network just means those pesky humans stop bothering you and go to a real therapist.
I’m a staff engineer who has been dealing with the results of SLAs before Amazon was an idea.
God forbid I have a p0 where I have to message a bunch of non technical directors it’s AWS not us. Much much worse than having to figure out and then pull in the team that pushed whatever untested shit made it’s way into production on a Friday afternoon.
Unless you’ve been responsible for a SaaS with SLAs in a b2b setting; I know more about the consequences of a provider outage than you.
I don’t know what you’re responding to but it doesn’t seem to be me. Either that or you forgot the username you picked for yourself in which case: whoosh
I touch grass every day. I want to do it on my own terms, not Amazon’s.
I keep some potted grass by my desk so I can touch it whenever I want
No shit. I was DevOps at my last company and they were all in on AWS. In those 5 years we had one major outage. There was one other case of a particular service going down, forgot which one, but it mainly screwed DevOps and the db guys.
You’re talking to a bunch of young people who hate Bezos and by extension AWS. They have no idea what the internet was like before.
Personally I think the cost is outrageous, rather have my own hardware mirrored in geographically distant colos, but that doesn’t mean AWS isn’t amazing.
The problem is not that any outage occurred. This still happens often. Things just refuse to work sometimes. The issue is that SO MANY eggs were in ONE basket.
Once per year? I had outages much more often than that on AWS.
Nah. I haven’t had a service we use miss an SLA or cost more than it’s SLO budget in 2 years.
What specific services have they missed your SLA on and what incidents were they tied to? I understand that not every team has a guy on their team to monitor that that stuff and bitch for credits, but I do, and AWS is one of our most reliable vendors.
Look the fact that AWS, Azure, and more recently Google are the only choices sucks.
But the reality is most companies and projects don’t have the business case to justify multi region fail over much less vendor fail over. They are all built on single points of failures and will always have outages.
Everyone just notices it more when it’s AWS. And that’s a stupid reason to base decisions off of. Visa/mc was working. Reddit and Facebook were mostly working once they started routing through their multi cloud nodes. Maybe you couldn’t get to your banks web app, that’s on them using a single cloud with no way to route to alternate cloud nodes and services. And for them to double at best infrastructure costs, unless they are boa Chase Morgan etc, is dumb for 99.99% which is the SLA .
The world isn’t ending, emergency services are working, visa/mc failed over, I was still on Reddit and slack most of the day. It wasn’t the end of the world.
Anyway, I now realize I have summoned my frustrations with this entire thread and gone wildly off topic and ranted with full force at you.
I just don’t think it’s important that when there is a major outage on AWS/Azure/cloud flare. It was going to happen elsewhere, and you wouldn’t have an excuse to tell your pm not my problem, instead of digging into your app for 2 hours to find out x portion of you very distributed vendor list failed and you still have a single point of failure. I’d rather be able to point to AWS, say shit is fucked for everyone, and if you want multi cloud it’s going to cost at least 1.5x as much as we’re spending 🤷♂️.
I haven’t used AWS in years. No IPv6 support in S3 in 2017 was the last straw for me. I have to deal with it at work (sometimes) and always laugh when they introduce “new” features like HTTPS records in Route53 like two years late.
Why do you say AWS, Azure and Google are the only options? I don’t use any of those greedy companies’ platforms.
It’s pretty funny to argue in favour of centralised services in a decentralised platform
I never argued that. I provided the reality of what they did. I’m sorry the reality doesn’t align with how you think things should be.
You think everyone trying to make money is just stupid and has ignored some super reliable and cheap hosting because they want to gobble bezos cock? No, they solved challenging problems and made it a lot easier to stand up a reliable app.
Very unique take on how businesses make technical decisions. I’ve never heard of anyone describe the decision making process as logical before. Or even grounded in facts.
I know. Everyone making money and decisions are just idiots.
That is usually the case, yes.
Usually
I feel that you’re are very jaded over this subject, I truly felt it was a funny situation. No judgement from me
Yes, AWS has a lot of advantages and I do believe they usually provide a reliable service, but as with all centralised services when they go down a bunch of other stuff go with them and that should be avoided. Doesn’t make all the incredible engineers currently working in AWS stupid
Eeehh, you’re literally suggesting that AWS added to the general stability and dependability of the internet in general
You have NO idea what you’re talking about
The internet was designed to survive nuclear war, talking about being dependable) and the entire idea was (and should continue to be) that you don’t rely on a single point of failure. Traffic should automatically route around dead nodes so that everything continues to flow. Decentralisation is key.
But of course with companies being companies and corporate doing what corporate does best (enshittify everything so that we make more monies) everything got centralized.
The centralization is an issue, but AWS is stable as hell. When I was first in IT, tech support, I had to explain to customers daily that, “No, your internet is fine, it’s just that particular website that’s down.”
And the centralization wouldn’t be a thing if AWS didn’t route all IAM services through us-east-1. My Lightsail in us-west-1 was fine yesterday.
So your argument is that AWS centralization is good because Amazon is a good provider?
You do understand that they’re are loads of providers out there that are perfectly stable, but that are not Amazon?
I’ve never used it because I know how to manage a server, something you might want to expect from IT personnel that does development for companies, but there days let’s just ask Amazon todo it for us, we’re too lazy
Companies need hell of a lot more then virtual machines today. I dont use it personally either but would i recommend a company to buy their own hardware? No. I would say they should use AWS because they can afford it and it gives them access to hundreds of services. Its rare to see technical issues.
The value for a company is actually enormous, to have something like that at their fingertips.
Todays downtime is forgotten in a few days and it was a big one.
Who says companies need to buy their own hardware?
We have datacenters for that, you rent the hardware one way or the other.
I’m saying that nobody should put all their eggs in one basket because if that basket breaks, you’re all fucked.
If you have the need for high availability then you don’t out all your servers in a single datacenter, or with a single provider
If everyone and their mother is with one provider, you’ll first notice that said provider gets expensive pretty quick and you’ll also notice that when shit goes down that half the fucking internet follows.
My services weren’t down, and never have been. I don’t use AWS, because I don’t need it
Yeah rack space was killing it! Sites NEVER went down, especially under dynamic load. Never.
You understand that that has nothing to do with this? So there are shitty providers out there, find a good one that is not “just amazon”
Actually can you start? Feels like an interesting talking point
It’s so funny to see that this just keeps happening over and over and nobody seems to be learning any lessons at all
Oh they are, slowly but surely. Amazon just took over a third of the web before anyone above the basic workers realized that one force holding that much power was a bad thing. Don’t forget, those kind of people think monopolies are are a good thing.
There’s nothing tiny about AWS US-EAST-1.
You’re right. It should be non-existent.
It’s amazing how cheap politicians are relatively to those companies sizes.
didnt expect that link to lead there
Can you explain “I’vēn”?
contraction of “I have been”
It can be contracted futher: “Ivn”Da comrade
You could also arbitrarily alias it as “dwvb” at that point.
what’s “dw?” I get “vb.”
I was running out of space on a silly meme.
Stop it, that way leads to tones.
fReE mArKeT
