If it’s that big of an upgrade, and your primary customers are North American based, why the fuck do you decide first thing in the morning of a weekday is the time to roll that out? Grab a fresh pot of coffee and start that shit at 10:00 p.m.
Normally, the odd times are because someone can’t work the weekend, which is crazy to me since I am someone who accepts all the worst go live times…
Although the more I think about it, would there really have been a fantastic day or time? Even weekends suck because it affects everything so even then it would have sucked plus if it did bring down companies those people are now doing work on the weekends as well. Idk if I disagree with a late time smack in the middle of the week date for this the more I think about it.
I saw this happen live because my fiber endpoint (not router because I have my own ONT) went offline for exactly 3 minutes at 4am EST so I realized they were pushing updates lol.
Fiber and internet network went fine but I guess cellular kicked the bucket.
Not to defend them or minimize the corporate stupidity, but it sounded like there were less than 100k people affected out of tens of millions (100m?) accounts. I get that it was a big deal for those affected, but a 0.1% outage doesn’t seem “major”.
I was impacted and it sucked. Having no cell service for 8-9 hours is not fun. Can’t make or receive calls or texts, every app that requires or uses an internet connection (like Waze) was impacted. Whole Waze worked with directions using offline maps and GPS, you don’t fet stuff like traffic conditions and rerouting.
But when you only have a cell phone and limited wifi resources at the office, it’s a major pain in the butt. And I didn’t report so that 70k could’ve been a conservative number of people that reported.
I think the reported numbers are coming from downdetector.com, which relies on self reporting and people being aware that the website exists. I imagine many more customers were affected. Also, anything the prevents emergency services communication, which occurred during this outage, should be considered a major outage imo
Not to downplay your point, because you are correct, but the outage did not affect anyones ability to contact emergency services, so that is a huge plus in the whole disaster. Any cell phone that pings off a cell tower can reach 911, even if there is no service activated on the phone. It’s important that people are aware of that fact in case they are in a situation where they can’t pay their bill, but still have an emergency.
I know that, that’s not what I’m talking about. My agency was also affected. I’m specifically talking about a cell phone’s ability to dial 911. Every cell phone must be able to dial 911 regardless of service, for safety reasons. This has been a requirement for quite a while before the issues we had with AT&T. One phone company’s IT blip should not have affected any phone from calling 911 specifically because service is not needed to do so on a normal day. Agencies wouldn’t be able to communicate with each other if they AT&T services because you can’t dial 911 from one agency to the next, it doesn’t work that way.
I did not have SOS service on my phone for about 6 hours yesterday. So you are incorrect in that all people were able to contact emergency services. ATT, Upper Midwest
Did you actually dial 911? Because if you tried dialing 911 and it didn’t go through, that’s a problem. ALL phones must be able to dial 911, even without service. If the phone can hit a tower, it can call 911.
Correct. I had to connect to WiFi to use ATT calls over WiFi to call the non emergency number to be transferred to my needed emergency services. My local news station put out an announcement about ATT customers not being able to contact local 911 operators. May have had something to do with my county specifically. Still, a major issues.
If that’s true, that’s wack. There’s no reason that the one phone company’s service issue should have affected your phone’s ability to call 911. Towers aren’t company specific so it doesn’t make sense that there would be interference 🤔 someone fucked up
Towers aren’t specific to any single phone company, if you stop paying for your phone service entirely, you can still dial 911. It just hits off the nearest tower.
I was under the impression that a company (AT&T) owns the tower, and they can lease out connections from that tower to other providers. They are also required by law to route 911 calls for free, but I can see a scenario if they botched the routing where 911 would not be accessible from that tower.
Some of the affected users were other systems, like Duo, which then caused downstream outages of even more thousands. That’s why it’s being reported that way.
I do know that FirstNet was impacted. The tablets in our fire apparatus couldn’t connect which is kind of a pain in the neck because we use that to navigate, locate fire hydrants and view their flow capabilities and whether they’re out of service, store maintenance phone numbers, view building blueprints and material safety data sheets, view responding apparatus and locations, identify helicopter landing sites, etc.
Like the job will still get done but it does throw a wrench in our ability to coordinate larger responses.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !technology@lemmy.world
This is a most excellent place for technology news and articles.
tl;dr - “Somebody’s getting fired!”
deleted by creator
Hanlon’s razor in action
“Never attribute to malice that which is adequately explained by stupidity”
With At&T it’s a big fucking razor
I always included obliviousness in this as well. But one could argue that’s just another form of stupidity… 🤷
I think it was the same intern that accidentally told Hawaii it was about to be hit by an icbm
I’ve worked with ATT before. Their general incompetence is breathtaking.
If it’s that big of an upgrade, and your primary customers are North American based, why the fuck do you decide first thing in the morning of a weekday is the time to roll that out? Grab a fresh pot of coffee and start that shit at 10:00 p.m.
Normally, the odd times are because someone can’t work the weekend, which is crazy to me since I am someone who accepts all the worst go live times…
Although the more I think about it, would there really have been a fantastic day or time? Even weekends suck because it affects everything so even then it would have sucked plus if it did bring down companies those people are now doing work on the weekends as well. Idk if I disagree with a late time smack in the middle of the week date for this the more I think about it.
sOlAr fLaReS
Looks like I’m not the only one who tests in production…
I saw this happen live because my fiber endpoint (not router because I have my own ONT) went offline for exactly 3 minutes at 4am EST so I realized they were pushing updates lol.
Fiber and internet network went fine but I guess cellular kicked the bucket.
Not to defend them or minimize the corporate stupidity, but it sounded like there were less than 100k people affected out of tens of millions (100m?) accounts. I get that it was a big deal for those affected, but a 0.1% outage doesn’t seem “major”.
I was impacted and it sucked. Having no cell service for 8-9 hours is not fun. Can’t make or receive calls or texts, every app that requires or uses an internet connection (like Waze) was impacted. Whole Waze worked with directions using offline maps and GPS, you don’t fet stuff like traffic conditions and rerouting.
But when you only have a cell phone and limited wifi resources at the office, it’s a major pain in the butt. And I didn’t report so that 70k could’ve been a conservative number of people that reported.
I think the reported numbers are coming from downdetector.com, which relies on self reporting and people being aware that the website exists. I imagine many more customers were affected. Also, anything the prevents emergency services communication, which occurred during this outage, should be considered a major outage imo
Not to downplay your point, because you are correct, but the outage did not affect anyones ability to contact emergency services, so that is a huge plus in the whole disaster. Any cell phone that pings off a cell tower can reach 911, even if there is no service activated on the phone. It’s important that people are aware of that fact in case they are in a situation where they can’t pay their bill, but still have an emergency.
It literally affected emergency services’ ability to contact each other in multiple areas of the country.
I know that, that’s not what I’m talking about. My agency was also affected. I’m specifically talking about a cell phone’s ability to dial 911. Every cell phone must be able to dial 911 regardless of service, for safety reasons. This has been a requirement for quite a while before the issues we had with AT&T. One phone company’s IT blip should not have affected any phone from calling 911 specifically because service is not needed to do so on a normal day. Agencies wouldn’t be able to communicate with each other if they AT&T services because you can’t dial 911 from one agency to the next, it doesn’t work that way.
I did not have SOS service on my phone for about 6 hours yesterday. So you are incorrect in that all people were able to contact emergency services. ATT, Upper Midwest
Did you actually dial 911? Because if you tried dialing 911 and it didn’t go through, that’s a problem. ALL phones must be able to dial 911, even without service. If the phone can hit a tower, it can call 911.
Correct. I had to connect to WiFi to use ATT calls over WiFi to call the non emergency number to be transferred to my needed emergency services. My local news station put out an announcement about ATT customers not being able to contact local 911 operators. May have had something to do with my county specifically. Still, a major issues.
If that’s true, that’s wack. There’s no reason that the one phone company’s service issue should have affected your phone’s ability to call 911. Towers aren’t company specific so it doesn’t make sense that there would be interference 🤔 someone fucked up
Doesn’t That tower still need to route the call to 911? And if that routing is broken the call wouldn’t go through…I think?
Towers aren’t specific to any single phone company, if you stop paying for your phone service entirely, you can still dial 911. It just hits off the nearest tower.
I was under the impression that a company (AT&T) owns the tower, and they can lease out connections from that tower to other providers. They are also required by law to route 911 calls for free, but I can see a scenario if they botched the routing where 911 would not be accessible from that tower.
Some of the affected users were other systems, like Duo, which then caused downstream outages of even more thousands. That’s why it’s being reported that way.
I do know that FirstNet was impacted. The tablets in our fire apparatus couldn’t connect which is kind of a pain in the neck because we use that to navigate, locate fire hydrants and view their flow capabilities and whether they’re out of service, store maintenance phone numbers, view building blueprints and material safety data sheets, view responding apparatus and locations, identify helicopter landing sites, etc.
Like the job will still get done but it does throw a wrench in our ability to coordinate larger responses.