Describe the symptoms and confirm what actually happened
Start with observable signs: a task that did not finish, an unexpected result or a difference between the automation record and the action's destination. State where to check each sign and what information to collect. The description should separate observations from explanations still awaiting verification, preventing the first suspicion from becoming an assumed cause before the evidence supports it.
“Everything has stopped” offers little help to whoever receives the request. Identifying the affected process, when the first sign appeared and which results stopped arriving is more useful. Add the last case known to have completed correctly, without assuming every subsequent case failed in the same way or at the same stage.
Imagine a fictional company whose automation receives contact requests, creates a record and sends a confirmation. The team finds requests with no confirmation of sending. That symptom allows different possibilities: delivery may have failed, may still be pending or may have happened without the service's response reaching the internal record that the team is reviewing.
The procedure should explain where to check the original request, the record created and the message status at its destination. An execution marked as failed may have completed earlier steps. Before running everything again, establish what already happened so that the recovery attempt does not create duplicate contacts or send the same message twice.
Record a stable reference for each affected case, the time and time zone, and the version of the workflow in use. Connect those elements to the incident in a location accessible to responsible people. Being able to explain a change made by an automation depends on that relationship between data, rules and the action actually performed.
The scope of the incident also needs to be established. It may affect only one category of request or all new entries since a particular change. If the scope is unknown, mark it as unconfirmed and assign someone to investigate, instead of communicating a conclusion that the available record does not yet support.
To prepare your next step, explore Business process automation.
Define how to contain the effects and who can decide
State the permitted containment actions, who performs them and which situations require another decision. Explain how to suspend the affected part without losing requests and how to confirm that suspension took effect. Include a backup person and a contact channel so that one person's absence does not leave the team without guidance when an incident needs attention.
In the fictional example, suspending automatic messages while their status is checked may make sense. That does not necessarily mean closing the form or stopping incoming requests. The decision depends on whether entries can be preserved while preventing pending work from continuing to produce the effect that the team is trying to investigate and contain.
The record should distinguish stopping new executions from interrupting work already underway. The team needs to understand its system's behaviour and verify what remains active. A visible pause indicator is not enough to conclude that no action can still occur in a step that had already started before the suspension was applied.
Define what the person monitoring the process may do without further authorisation. They might flag cases, suspend a step specified in the procedure or request support. Changing commercial rules, deleting records or sending customers another message requires its own criteria and an explicit decision from someone responsible for those actions and their consequences.
The limits set before an automation changes a budget show why autonomy needs a defined scope. Urgency during an incident does not clarify who can decide. That definition should already exist, including how to escalate a case when the observed effect exceeds what the people monitoring the process are equipped or authorised to resolve.
Internal communication should state the known impact, the containment applied, the responsible person and the next update point. If customers are affected, someone should decide what to communicate from confirmed facts. Do not promise a resolution time you cannot yet support or confuse missing information with evidence that there has been no impact on the service.
Preserve evidence and organise the work left pending
Save the observed state, relevant error messages, recent changes and interventions made during the response. Organise cases as completed, pending or still unconfirmed, assigning responsibility for each group. Preserving this information supports investigation without relying on memory and helps coordinate a manual alternative without overlapping work that the automation has already performed or that someone else is currently handling.
The incident record should provide an understandable sequence. Note when it was detected, what was suspended and which checks preceded changes to the workflow. Keep hypotheses separate from facts, including a discarded hypothesis when it helps another person understand why a particular intervention was selected and what evidence informed that choice at the time.
Keep only necessary information in the appropriate place. Identifiers and references may be enough to find a case in authorised systems. Avoid spreading customer data, credentials or complete message contents across open conversations, especially when a reference lets the responsible person consult the relevant information using the access they already have for that work.
If the fictional company decides to answer requests manually, each response needs to appear in the same tracking list. Otherwise, the team may complete the work and the automation repeat it when reactivated. The manual alternative must include a way to mark completed actions and exclude those cases from the planned set of retries.
A case without confirmation deserves its own status. Do not mark it complete to shorten the list or failed merely to justify another attempt. Check the destination and, when uncertainty remains, refer the decision to someone who can assess the consequences of waiting, contacting the recipient or taking another action to resolve the case.
The person investigating the cause and the person tracking requests need to share the same state. Their roles can differ, but both should know what changed and who is handling each case. That coordination allows feasible work to continue without turning recovery into simultaneous interventions whose sequence and effects nobody can reconstruct afterwards.
Resume against criteria and turn the incident into an improvement
Define what must be verified before reactivating the workflow: a validated correction, reconciled pending cases, informed owners and monitoring ready to begin. Start with a controlled set and observe results at the destination. Resumption needs a person authorising it and a condition for suspending again if the expected behaviour fails to appear or the original symptom returns.
Do not consider the incident resolved simply because the error message disappeared. In the example, confirm that new requests create the right records and receive the intended confirmation once. Check the handling of older requests too, because correcting the workflow does not automatically complete work that remained pending while the incident was being investigated.
Verification can use the criteria established when testing an automation before handing it to the team. During recovery, add checks of the actual state of actions already performed. A test with new data does not by itself show that repeating every case marked incomplete in the history is appropriate or free from the risk of duplication.
Record who authorised resumption, under which conditions and what was observed afterwards. If cases remain under investigation, keep that work visible with an owner. Operations may resume while part of the analysis continues, provided everyone understands the distinction and it does not hide work that still needs a decision or a further check.
Then review the procedure with the people who used it. Identify missing instructions, unavailable access and questions that delayed the response. Update the document and test the change in a controlled scenario, preserving the earlier version to explain what changed and avoiding promotion of a local solution into a general rule without checking its scope.
I want the team to respond to a failure with a shared understanding of how to decide. Recognising symptoms, limiting effects, preserving state and deciding when to resume are parts of that work. A useful procedure reduces improvisation and makes the response verifiable, without promising that anyone can resolve every incident or that service will never be interrupted.