There are reports of IT outages affecting major institutions in Australia and internationally.
All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.
Apparently caused by a bad CrowdStrike update.
Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We’ll see if that changes over the weekend…
Those things never worked for me… Problems always persisted or it failed to apply the restore point. This is from the XP and Windows 7 days, never bothered with those again. To Microsoft’s credit, both W7 and W10 were a lot more stable negating the need for it.
I can’t say about XP or 7 but they’ve definitely saved my bacon on Win10 before on my home system. And the company I work for has them automatically created and it made dealing with the problem much easier as there was a restore point right before the crowdstrike update. No messing around with the file system drivers needed.
I’d really recommend at least creating one at a state when your computer is working ok, it doesn’t hurt anything even if it doesn’t work for you for whatever reason. It’s just important to understand that it’s not a cure all, it’s only designed to help with certain issues (primarily botched updates and file system trouble).
That’s true about the test group deployments, but it turned out this one was not an agent update under that control system. It’s a Channel File update that goes out to all endpoints automatically.
crowdstrike sent a corrupt file with a software update for windows servers. this caused a blue screen of death on all the windows servers globally for crowdstrike clients causing that blue screen of death. even people in my company. luckily i shut off my computer at the end of the day and missed the update. It’s not an OTA fix. they have to go into every data center and manually fix all the computer servers. some of these severs have encryption. I see a very big lawsuit coming…
they have to go into every data center and manually fix all the computer servers.
Jesus christ, you would think that (a) the company would have safeguards in place and (b) businesses using the product would do better due diligence. Goes to show thwre are no grown ups in the room inside these massive corporations that rule every aspect of our lives.
I’m calling it now. In the future there will be some software update for your electric car, and due to some jackass, millions of cars will end up getting bricked in the middle of the road where they have to manually be rebooted.
Look! We found the anti anti-microsoft user in their natural habitat of talking shit on social media. They are from the same family bootlickers and related to the Tossaladers. Don’t mention you are an alternative OS person around them or they may go into a blind rage.
My last companyoffered apps as either hosted or on-prem. Once they decided the on-prem version had to be Windows for our customer base, it made sense to have one build, one installer. I’m very happy not to be there today.
My current employer is all Linux servers and Engineers use Mac laptops. The only ones affected were Management and HR, LoL. Back of the line for you, we have customers to help!
Servers weren’t much of a problem, they’re mostly virtual and could be just restored from a backup. The several hundred workstations were a problem. They needed a physical touch. All are encrypted with BitLocker, requiring passkeys stored in AD. Over half are laptops. Most of those don’t have wired ethernet ports, and an account with local admin rights hasn’t logged in since the day they were imaged. Throw in a proper LAPS config, where randomly generated passwords of three dozen characters in length are also stored in AD…
Everyone is assuming it’s some intern pushing a release out accidentally or a lack of QA but Microsoft also pushed out July security updates that have been causing bsods on the 9th(?). These aren’t optional either.
What’s the likelihood that the CS file was tested on devices that hadn’t got the latest windows security update and it was an unholy union of both those things that’s caused this meltdown. The timelines do potentially line up when you consider your average agile delivery cadence.
Huh. I guess this explains why the monitor outside of my flight gate tonight started BSoD looping. And may also explain why my flight was delayed by an additional hour and a half…
Yeah my plans of going to sleep last night were thoroughly dashed as every single windows server across every datacenter I manage between two countries all cried out at the same time lmao
I work in a datacenter, but no Windows. I slept so well.
Though a couple years back some ransomware that also impacted Linux ran through, but I got to sleep well because it only bit people with easily guessed root passwords. It bit a lot of other departments at the company though.
This time even the Windows folks were spared, because CrowdStrike wasn’t the solution they infested themselves with (they use other providers, who I fully expect to screw up the same way one day).
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !technology@lemmy.world
This is a most excellent place for technology news and articles.
I was quite surprised when I heard the news. I had been working for hours on my PC without any issues. It pays off not to use Windows.
This is why you create restore points if using windows.
Those things never worked for me… Problems always persisted or it failed to apply the restore point. This is from the XP and Windows 7 days, never bothered with those again. To Microsoft’s credit, both W7 and W10 were a lot more stable negating the need for it.
I can’t say about XP or 7 but they’ve definitely saved my bacon on Win10 before on my home system. And the company I work for has them automatically created and it made dealing with the problem much easier as there was a restore point right before the crowdstrike update. No messing around with the file system drivers needed.
I’d really recommend at least creating one at a state when your computer is working ok, it doesn’t hurt anything even if it doesn’t work for you for whatever reason. It’s just important to understand that it’s not a cure all, it’s only designed to help with certain issues (primarily botched updates and file system trouble).
deleted by creator
Apparently the slow rollout was skipped (on Crowdstrike’s end) for this
That’s true about the test group deployments, but it turned out this one was not an agent update under that control system. It’s a Channel File update that goes out to all endpoints automatically.
CrowdStrike: It’s Friday, let’s throw it over the wall to production. See you all on Monday!
They did it on Thursday. All of SFO was BSODed for me when I got off a plane at SFO Thursday night.
I’m used to IT doing a lot of their work on the weekends as to not impact operations.
Webroot had something similar ish earlier this year. Such a pain.
crowdstrike sent a corrupt file with a software update for windows servers. this caused a blue screen of death on all the windows servers globally for crowdstrike clients causing that blue screen of death. even people in my company. luckily i shut off my computer at the end of the day and missed the update. It’s not an OTA fix. they have to go into every data center and manually fix all the computer servers. some of these severs have encryption. I see a very big lawsuit coming…
Jesus christ, you would think that (a) the company would have safeguards in place and (b) businesses using the product would do better due diligence. Goes to show thwre are no grown ups in the room inside these massive corporations that rule every aspect of our lives.
I’m calling it now. In the future there will be some software update for your electric car, and due to some jackass, millions of cars will end up getting bricked in the middle of the road where they have to manually be rebooted.
Laid off one too many persons, finance bros taking over
lol
too bad me posting this will bump the comment count though. maybe we should try to keep the vote count to 404
I can only see 368 comments rn, there must be some weird-ass puritan server blocking .ml users. It’s not beehaw as I can see comments from there.
I can only conclude that it is probably some liberals trying to block “Tankies” and no comment of value was lost.
Why do people run windows servers when Linux exists, it’s literally a no brainer.
No brainer… Blames windows… For a 3rd party issue… The same 3rd party that’s done the same thing here to Linux recently… No brainer achieved.
Are you a Linux user?
Hey everyone, shut up! This guy is a Linux user and he’s here to tell us about using Linux.
Look! We found the anti anti-microsoft user in their natural habitat of talking shit on social media. They are from the same family bootlickers and related to the Tossaladers. Don’t mention you are an alternative OS person around them or they may go into a blind rage.
Linux users and vegetarians. Neither can shut the fuck up. Both have made it their identity.
My last companyoffered apps as either hosted or on-prem. Once they decided the on-prem version had to be Windows for our customer base, it made sense to have one build, one installer. I’m very happy not to be there today.
My current employer is all Linux servers and Engineers use Mac laptops. The only ones affected were Management and HR, LoL. Back of the line for you, we have customers to help!
Servers weren’t much of a problem, they’re mostly virtual and could be just restored from a backup. The several hundred workstations were a problem. They needed a physical touch. All are encrypted with BitLocker, requiring passkeys stored in AD. Over half are laptops. Most of those don’t have wired ethernet ports, and an account with local admin rights hasn’t logged in since the day they were imaged. Throw in a proper LAPS config, where randomly generated passwords of three dozen characters in length are also stored in AD…
… Yeah, today was a bad day.
Everyone is assuming it’s some intern pushing a release out accidentally or a lack of QA but Microsoft also pushed out July security updates that have been causing bsods on the 9th(?). These aren’t optional either.
What’s the likelihood that the CS file was tested on devices that hadn’t got the latest windows security update and it was an unholy union of both those things that’s caused this meltdown. The timelines do potentially line up when you consider your average agile delivery cadence.
I don’t think so. I do updates every two months so I haven’t updated Windows at all in July and it still crashed my servers
Microsoft installs security updates automatically.
Not on any of my servers. All windows updates have to be manually approved installed from the local WSUS server.
Huh. I guess this explains why the monitor outside of my flight gate tonight started BSoD looping. And may also explain why my flight was delayed by an additional hour and a half…
My work PC is affected. Nice!
Plot twist: you’re head of IT
No one bother to test before deploying to all machines? Nice move.
YOLO 🚀🙈
oh joy. can’t wait to have to fix this for all of our clients today…
Yeah my plans of going to sleep last night were thoroughly dashed as every single windows server across every datacenter I manage between two countries all cried out at the same time lmao
How’s it going, Obi-Wan?
How many coffee cups have you drank in the last 12 hours?
I work in a data center
I lost count
I work in a datacenter, but no Windows. I slept so well.
Though a couple years back some ransomware that also impacted Linux ran through, but I got to sleep well because it only bit people with easily guessed root passwords. It bit a lot of other departments at the company though.
This time even the Windows folks were spared, because CrowdStrike wasn’t the solution they infested themselves with (they use other providers, who I fully expect to screw up the same way one day).
What was Dracula doing in your data centre?
Because he’s Dracula. He’s twelve million years old.
THE WORMS
There was a point where words lost all meaning and I think my heart was one continuous beat for a good hour.
Did you feel a great disturbance in the force?
Oh yeah I felt a great disturbance (900 alarms) in the force (Opsgenie)