Other Post Crowdstrike takes down half the world's IT

[TheKiwi]

A flawed update from Crowdstrike has crippled many large organizations' IT.

Chaos persists as IT outage could take time to fix, says cybersecurity firm boss

Airports, banking and healthcare were all hit when a Crowdstrike update triggered huge Microsoft outages.

www.bbc.com

Although Crowdstrike have identified the faulty update and withdrawn it, machines that have been affected are stuck in a Blue Screen of Death loop and can't get the notification that the update is withdrawn. The only fix is to manually start up the computer in Safe Mode, delete the file in question and reboot the computer, a long and tedious process under the best of circumstances.

Guess what my day will consist of tomorrow?

 

[Jake84]

A crappy Saturday I’m sure, TheKiwi.

Early sources as always blamed the usual suspects hence the Norks or whoever else, whereas it appears to be onto Crowdstrike. The common people and the average Joe computer shouldn’t be affected. A bloody mess otherwise.

 

[muck]

Sorry to hear that, Kiwi.

Only goes to show how vulnerable we've become, though.

 

[Mike1976]

Kind of funny timing how this comes days after MS let go of their DEI staff. An ally within a third party contributor making MS look bad?

I mean it's primarily aimed at businesses too, evil capitalism ...

@TheKiwi: hehe sucks to be you, glad I left IT long ago

 

[Fluff]

A flawed update from Crowdstrike has crippled many large organizations' IT.

Chaos persists as IT outage could take time to fix, says cybersecurity firm boss

Airports, banking and healthcare were all hit when a Crowdstrike update triggered huge Microsoft outages.

www.bbc.com

Although Crowdstrike have identified the faulty update and withdrawn it, machines that have been affected are stuck in a Blue Screen of Death loop and can't get the notification that the update is withdrawn. The only fix is to manually start up the computer in Safe Mode, delete the file in question and reboot the computer, a long and tedious process under the best of circumstances.

Guess what my day will consist of tomorrow?

Think of the overtime!

 

[TheKiwi]

 

[muck]

Which is to say, you're going to have an oversized member inserted into your rectum? Figuratively speaking, obviously?

 

[Mike1976]

Which is to say, you're going to have an oversized member inserted into your rectum? Figuratively speaking, obviously?

Nah I think he's trying to say that he's that cheap

 

[Mark Sman]

Mumble unix mumble BSD>BSOD mumble linux. (Walks out of meeting with gray beard covered in Cheeto dust and still no delivery date.)

Kidding man. I feel for you. Having to go Safe Mode first is going to make that a grind.

 

[TheKiwi]

Spent much of yesterday talking retail people through the difficult process of clicking on 3 options, entering a bitlocker key and deleting a file. 10-20 minutes per site. Sigh. Still at least 75% of the company is now back up and running.

 

[LimaCharles]

Thankfully both the DoD and the state of AZ's NG that I work for, both don't use it.

 

[TheKiwi]

Microsoft have claimed that it was "only" about 8.5% of the worlds Windows computers that were affected. However clearly that 8.5% ran some very important parts of modern society.

 

[muck]

Thalidomide killed "only" about 8% of the babies born by the affected mothers.

 

[Euroamerican]

Ja, ja….. But I’m reading a news article saying less than 10 million PCs affected worldwide… Who knows….

This last 72 hours has had our IT team doing overtime to get blue-screened PCs back up and running. It involves either entering the 48 character Bitlocker recovery key stored in the AD for the particular PC, or reimaging the PC with a fresh copy of our own Windows 10 image if you can’t retrieve a valid recovery key. For some of the PCs, the wizards in the lab have figured out a way to dig into the OS with a domain login without touching the Bitlockered partition , delete the bad Crowstrike file, restart the CrowdStrike app/service/whatever, and then restart the pc. I don’t grok it yet, but supposedly they are printing instructions for Monday morning.

I had to work about fifteen hours this weekend, resurrecting or re-imaging about sixteen PCs that needed special attention. We had about 80 to 100 people working off and on from Friday through late Saturday, with only the Service Desk fully staffed today. Everyone else except the on-call guys are resting up for MONDAY morning. I worked today, Sunday, because most of those sixteen PCs are needed right away on Monday morning. I wanted to be clear for new action in the morning, because I think it’s gonna be a sucky Monday!

 

[TheKiwi]

I'm pretty much done as of this evening. There are just a couple of machines left to fix and the people who use them are away. So I call that sorted. Now to ask for a fat refund from Crowdstrike.

 

[digrar]

12 laptops affected in our office, including the two POS machines. 4pm on a Friday with cold tins in the beer fridge? New boss said back to work. Turns out we were busy enough without the computers anyway.

 

[uktabber]

Folks.....This is just a glimpse, they're dusting this off upgrading software and getting it ready for the next U.S. election!! Just watch......​