Your AI Apocalypse Playbook

I believe we will experience a major AI-driven business disruption within the next 18 months. That’s the politically correct way to say, “Get ready for an AI Apocalypse.” The timing and the specific failure modes are uncertain. But it is clear that increasingly capable models will give inexperienced bad actors the ability to cause extraordinary damage. To say nothing of what an experienced bad actor will be empowered to do. That said, the severity of intentional attacks may pale in comparison to the damage done by self-organizing agents acting in ways their human handlers did not foresee.

Your existing incident-response, business-continuity and resiliency plans are the starting point for the technical response. But, considering the ferocity of the anti-AI movement, a major AI incident of any kind will require a thoughtful non-technical response. Which raises the question, “Are you ready?” Let’s explore.

How Likely Is An Incident?

The evidence already warrants preparation. On September 22, Microsoft reported that EvilTokens, an AI-enabled cybercrime service, had been linked to more than 12,000 compromised inboxes. Its tools helped criminals identify trusted relationships and prepare fraud. Anthropic’s September report described a lone actor who gained internal access to at least 14 targets. Both reports describe human involvement. They show how AI can amplify an attacker’s reach.

I’ve been writing about this for years. In August, I began asking my clients to establish clear authority to permission, monitor and stop their AI-powered agents and workflows. Now I want to get specific about what happens when something has gone wrong. We can call it your “AI Apocalypse Playbook.”

Defining the Threat

If your company is attacked, you probably already have a playbook. If not, we’ll go over the steps to build one below. But what if the attack is on your whole industry? How about a tangential sector that will impact your business? The very first step is to define what kinds of incidents require a reaction, which require a response, and which you can ignore. These definitions are harder to craft than they sound. Spend some time workshopping them. Once you’ve established what a bad day will look like, you can start building your playbook.

Name people and give them authority

Start with a one-page activation sheet attached to your existing incident-response and business-continuity plans. Put actual names, verified phone numbers, deputies and after-hours coverage on it. Every employee should know where to find it and how to report a problem.

Name one incident commander who coordinates the response. Identify the security lead, marketing operations owner, legal/privacy lead, communications lead, customer-service lead and executive sponsor. Include your agencies and critical technology providers. In a smaller company, several responsibilities may belong to one person. Every responsibility still needs an owner and a backup.

Decide in advance who can pause advertising, scheduled emails, publishing, chatbot responses and automated customer decisions. Specify which conditions authorize each action. Give those people the access they need and test it. The CMO’s approval means very little if the only person with administrator privileges is asleep on another continent.

Keep a protected copy of the call sheet accessible when company email, chat and single sign-on fail. Establish an independent communications channel. Verify unusual instructions using a previously known number. Use pre-agreed safe words as an additional identity check (in a world of voice-cloning, recognizing a voice is not enough).

Write down the first hour

Define the triggers in ordinary language: unauthorized spending, unexpected messages, exposed customer information, an invented policy, a suspicious payment instruction, or loss of a critical service. Staff should report what they observe immediately. Attribution can come later.

The first employee to notice contacts the designated incident channel (call, text, email, etc.) and reports the affected service, time, example and known impact. Here is a starting target to rehearse: if nobody acknowledges within five minutes, contact the deputy. Within 15 minutes, the commander and relevant business and technical owners should be connected. Within 30 minutes, establish what is known, whether harmful activity is continuing, who is containing it, and when the next update will happen. These are proposed internal targets. Your team must set timings that match the risk. Legal notification deadlines require separate review.

Work proceeds in parallel. Your security team will do what they do. The marketing organization will need to identify affected campaigns, customer promises and downstream actions. Vendor leads open the emergency support tickets. Legal and privacy assess notification duties. Finance contacts the bank immediately if money may be moving fraudulently and checks applicable insurer requirements. Communications and customer service prepare a consistent account of verified facts.

Authorized operators should immediately pause the affected activity and confirm the pause worked. Stopping an agent may leave scheduled messages or vendor-side jobs running. The playbook needs instructions for those dependencies, too.

Best practices call for you to preserve logs, timestamps, campaign identifiers, prompts, outputs and approval records in restricted storage. Technical responders should coordinate evidence collection with containment. Deleting the offending workflow or wiping a device can destroy evidence. Urgent containment should proceed while the team preserves what it safely can. This is the hardest part in a real time crisis, but if you’re prepared, it will make the post-mortem much easier and far more productive.

Keep the business operating

For each essential customer-facing process, answer three questions: How long can we tolerate an interruption? How much recent work or data can we afford to lose? What will we do while the system is unavailable?

Clearly articulate the actual fallback. That could mean serving approved static content, routing inquiries to trained people, manually reviewing orders, or pausing acquisition while customer support continues. Assign staff and establish capacity limits. Carry consent, suppression and privacy controls into the fallback.

Test the dependencies. Your alternate AI provider may use the same cloud service or identity system. A different model may produce different outputs from the same prompt. NIST’s Generative AI Profile specifically recommends reviewing third-party AI incident responsibilities and planning fallbacks that can include manual processing.

Your communications plan also needs a fallback. Decide where customers can find verified updates when your primary site or social account is compromised. Prepare holding language that states what you know, what customers should do and when you will update them. Name the approver and deputy. Avoid guessing about the cause or promising a recovery time the technical team cannot support.

Restoration needs its own approval. The business owner and technical lead should verify that the fault is controlled, access is trustworthy and the workflow produces acceptable results. Review queued actions before restarting. Check for duplicate sends, incorrect offers and incomplete transactions. Resume in stages with monitoring and a tested way to pause again. NIST’s incident-response guidance includes verifying recovery assets and restored services.

Practice before you need it

Put 45 minutes on the calendar. Tell the team an AI-generated offer has gone live, customers are accepting it, and the primary incident commander cannot be reached. Halfway through, announce that corporate chat is unavailable. Run this as an exercise using test systems and simulated messages.

Measure how long it takes to reach the deputy, identify someone authorized to stop the activity, verify containment and start the customer-service fallback. Ask the team to produce the first customer update. Record every missing permission, outdated number and unowned decision. Assign fixes and repeat the exercise.

We can debate the probability of an incident that would qualify as an AI apocalypse. But you can test your team’s readiness right now. At your next leadership meeting, ask: If something goes wrong at 9:07 tomorrow morning, can everyone show me exactly what they will do?

Author’s note: This is not a sponsored post. I am the author of this article and it expresses my own opinions. I am not, nor is my company, receiving compensation for it. This work was created with the assistance of various generative AI models.