Worldwide Crowdstrike outage - recovery processes underway
Incident Report for MedicTek System Status
Postmortem

Following the Crowdstrike event that caused many Microsoft Windows powered operating systems to halt, we are conducting a review how this happened. Our initial findings are concerning because of safeguards we employ to avoid this type of error from our partners and vendors. A large portion of our clients' monthly costs involve subscriptions to third-party vendors which we pay for on a client’s behalf and pass-on through afterwards.

Due to the nature of cyber threats, automated updates are crucial to stopping threats as soon as possible for the large number of devices managed simultaneously. MedicTek diligently reviews and schedules updates to devices based on a partially automated scheduling system; however, we set updates to delay for 1-2 verified production versions before adopting the latest “push” updates from our vendors. In the case of Crowdstrike, we were puzzled to learn that updates from this vendor were pushed-out without delay and overrode our settings to remain one level behind. In our professional opinion, this delay would have prevented an flawed update from ever reaching our managed devices for our clients and our internal operations.

While Crowdstrike products are rated as the top in the industry, we have other solutions with similar capabilities as rated by the renowned Gartner Group studies. In the upcoming weeks, we will be offering options to clients to migrate from Crowdstrike to SentinelOne, a separate U.S.-based cyber-security vendor. SentinelOne is also certified to operate with our remote monitoring and management agent, which is required on all devices that we manage.

The apology from the CEO of CrowdStrike doesn’t mitigate the fact organizations suffered an outage that incurred costs beyond the time/effort to restore systems. MedicTek is planning to waive the support services required to restore your systems from this event and we are reviewing options to help us recover our costs other than from our clients who also suffered from this event. We understand this gesture only partially covers the incidental expenses around this outage, but we acknowledge it’s the right thing to do.

If you have any questions related to this incident or would like more information on options, please contact me for a free one-on-one discussion so we can map-out what makes sense for your organization.

As a local small business with local employees who depend on our stability and resilience, we sincerely appreciate your loyalty and trust.

Sincerely,

Greg Hendrickson

Founder, Tech Support by MedicTek, Inc.

Posted Jul 20, 2024 - 13:52 EDT

Resolved
All servers have been restored as of this message. Some desktops that remained off-line during our restoration may require additional steps. If you are still experiencing problems with your desktop or laptop and you've rebooted it once, please call our office at 800-260-9910. At this time, any issues outside of normal business hours will still be handled under our contract terms - if your issue can wait until we reopen on Monday, that may be the most economic option. Thank you for your understanding with this global issue. We will be performing a post-mortem and evaluation of options to migrate threat protection to another platform, if that is deemed necessary.
Posted Jul 19, 2024 - 16:33 EDT
Update
A fix has been issued by Crowdstrike; however, the steps involved will require multiple steps to recover Windows desktops that will not boot normally. A simple reboot may correct your situation; however, PLEASE DO NOT IMPLEMENT A FIX YOURSELF AS THREAT ACTORS MAY EXPLOIT GOOGLE SEARCHES OR STEPS TO RECOVER AND COMPROMISE YOUR SYSTEM. Only use steps issued by MedicTek to recover. More information will follow - we understand the urgency.
Posted Jul 19, 2024 - 08:46 EDT
Identified
Crowdstrike, the market leader in AI threat protection released an update within the last 24 hours that caused multiple Windows platforms to go off-line. MedicTek is aware of the issue and has begun recovery operations and we expect to have servers restored as soon as physically possible. Due to the volume of outages, this process will take time and you may experience several hours for recovery operations underway. At this time, it does not appear to be a breach, but rather a faulty software update released by the publisher. More updates will follow.
Posted Jul 19, 2024 - 07:33 EDT
This incident affected: MedicTek Owned (Server Hosting, VPN).