Rethinking Vigilance: Lessons from Aviation for Modern Security Challenges

An air traffic controller managing multiple aircraft is bound to eventually overlook a critical issue. This phenomenon, known as the vigilance decrement, has long been acknowledged in aviation, leading to decades of design strategies aimed at mitigating its effects.

Now, with AI infiltrating various sectors, knowledge workers find themselves juggling more tasks than ever—think of managing a dozen digital communications while trying to draft reports and maintain productivity. In this context, the vigilance decrement doesn’t get a pass just because we’ve been told to keep up.

Aviation's Strategic Response

Rather than asking controllers to solely focus harder, aviation chose to implement technology that autonomously monitors air traffic and, most importantly, can override human commands.

Take the tragic 2002 collision near Überlingen, for instance, where conflicting orders led to the crash of a passenger jet, claiming 71 lives. The humbling lesson from this disaster was clear: when there's a disagreement between human judgment and machine calculations, it’s the machine that must be trusted.

This particular machine wasn’t designed to interpret intent; its sole function was to analyze flight paths and identify collisions. The industry didn’t focus on training crews to trust an intelligent machine. Instead, it demanded obedience to a simple, infallible system.

The Security Sector's Misstep

Contrarily, the security field has banked on human vigilance for three decades, expecting people to detect errors in communications. With the rise of AI, this reliance has become significantly riskier.

For starters, AI has blurred the lines of deception, making it easier for malicious actors to craft phishing threats. Research indicates that AI can reduce the expense of such scams dramatically, while still achieving high click-through rates.

Secondly, the ability of AI to distract users has resulted in a landscape where every individual feels like an air traffic controller: overburdened and unable to focus sufficiently to catch subtle threats. Unlike an approaching aircraft that presents an obvious danger, fraudulent schemes are often meticulously designed to blend in with legitimate communications, targeting the moment when the recipient is most distracted.

Prioritizing Potential Threats

In security, the task is straightforward: prioritize the threats capable of causing the most damage. Data indicates that business email compromise (BEC) attacks, while representing a small percentage of total incidents, account for a substantial share of financial losses. For example, in 2024, BEC schemes resulted in billions of dollars in losses despite being fewer in number.

This aligns with aviation’s principle of focusing on the machines that could lead to catastrophic outcomes. The solution lies in implementing controls that trigger action based on the occurrence itself—mandatory callbacks for any financial changes, dual controls for wire transactions, and automated halts for wires above certain thresholds.

These measures aren't novel but have become more critical as the indicators of fraud become less visible. The emphasis should be on reaction rather than human interpretation—similar to how collision-avoidance systems operate without regard for pilot intent.

However, the challenge is that most fraudulent requests may resemble legitimate communications down to the finest detail, creating a scenario where verification becomes necessary beyond just mechanical oversight. This context-sensitive evaluation is an area where aviation's methodology doesn’t fully transfer.

Machines that evaluate intent need to gain user trust before they can be mandated to take precedence over human judgment. As long as machines operate in complex and ambiguous circumstances without consistent accuracy, the responsibility on users doesn’t diminish.

Reflections on Vigilance

Aviation had to reform its protocols after the tragic consequences of overlooking the vigilance decrement. In contrast, the stakes in security may not always result in immediate catastrophe, yet they undermine trust through financial loss and identity theft. Despite this, we continue to depend on individuals to be vigilant in spotting the nuanced threats buried in everyday communications.

What’s essential is to reinforce controls that eliminate reliance on human error at critical moments. Security should adopt its version of the fail-proof machine: systems that autonomously block suspicious transactions or actions, irrespective of user discretion.

Thus, the crucial question isn’t if personnel are paying attention—history shows they won’t consistently do so. Rather, it’s whether there are safeguards in place to prevent irreversible actions without placing the entire burden on individual vigilance.