# AI Alignment is not AI Safety - Lessons from Aviation

- **Series:** Aviation for AI, Part 3
- **Author:** Samuel Chandra (Founder & CEO, Deepsky)
- **Canonical:** [https://www.deepskyai.com/newsletter/ai-alignment-is-not-ai-safety-lessons-from-aviation](https://www.deepskyai.com/newsletter/ai-alignment-is-not-ai-safety-lessons-from-aviation)
- **Subscribe directly:** [https://www.deepskyai.com/newsletter#updates](https://www.deepskyai.com/newsletter#updates)

I realise that there is a bit of confusion out there as to what I mean when I talk about Aviation for AI [safety]. There are two distinct but related concepts that are tackled in different but also similar ways. Let me explain.

### Part 1: Alignment

This is the term used in the AI world used to describe the behaviour and proclivities of models themselves. When given a task, what will models do to achieve it, or more basically, when given an input, what it put will they give? Will it be in alignment with the general virtue of human flourishing (how that’s defined is not today’s or this years discussion here unfortunately) or will the model output something that while technically correct, is destructive and nefarious.

When I first read the 2020 book on alignment “The alignment problem” we (not necessarily the author) were thinking about AI models as input output. Next token prediction kind of thing. But now of course we have agents, where they are repeatedly taking in inputs, then creating outputs, analysing how those outputs affect their environment and then continue on in this manner. Where as you can see, the degree to which a model is inclined to align with human values has a dramatically compounding effect.

Being aware of the degree to which models are aligned to the values you desire is really important. Now alignment is probably not so critical if you have an AI system that simply parses payslips, but can be very important if you start using it to do much more consequential things like also hand out mortgages to applicants based on their personal information. Selecting models as part of your AI based systems and thus your organisation is a really important foundation in any AI based system.

If we use the Aviation in AI analogy here. Models are neural networks - and guess what also are neural networks - your brain. In a safe aviation system, we preselect people based on their stable state of mind, we screen them for extremism and crime, and in the most safety critical roles (like pilots) we also screen them for physical and mental illness. Now we have a baseline. But people that pass this baseline are incredibly varied and still not up to task. So what’s next? Well we align these people with training so they operate in a certain manner with certain priorities - like being trained that “safety is always first”, and similar values.

Alignment of humans with safety objectives is a huge piece of how Aviation has become so safe. However it’s just half of the story. Because no matter how well trained someone is, everyone is different and the same people are different every day they turn up to work. It’s the safety system that surrounds these non deterministic neural networks (people) that ensures the reliability of the risky endeavour of flight despite this.

### Part 2: The Safety Management System

If we zoom out from the individual, we will find the organisation they operate in. Then if you zoom out again, you see that organisation in context of the global aviation environment. The superstructure that supports the safe continuation of the organisation is a safety management system. It’s essentially a construct internal to an aviation organisation that heavily influences what is and isn’t done, the way in which they are done, how things are changed, who goes to jail and who is immune from jail. All with the goal of preventing people from dying.

They are living breathing systems that are continually adapted to the realities of what’s going on in that organisation and the environment, and the way they are administered is taken very seriously, because yes of the death thing, but also if they aren’t run in a sophisticated, relevant manner, the organisation gets shut down by the regulator. And not having the funds to administer the system is no excuse, regulators will let a company go bankrupt before waiving their safety obligations, for obvious reasons.

I’ll dive further into safety management systems Ina later post. But they are composed of four main pillars:

1. Policy and objectives (what are you trying to achieve - usually no one dying)
2. Risk management (flying is inherently risky, how do we reduce the risk?)
3. Promotion (how does everyone know what they’re supposed to do?)
4. Assurance (how do we know we are doing what we are supposed to do?)

Pop these four things into an organisation, and you have the beginnings of something that can create a safe outcome.

Now, aviation safety management systems (let’s call them SMS) were designed with humans as the non deterministic nodes at their heart. But I don’t see why we couldn’t add in artificial neural networks, and by extension, agents. A fully fledged AI SMS will have to cover both agents and humans, as agents are quite human like (so rules work for them too) and humans need to know how they are supposed to act to keep AI from causing harm.

### Part 1 and 2

So we need to try as far as possible to ensure our neural networks (human or artificial) are aligned and then build an SMS around them to ensure continued reliable safe operation despite inevitable variation in their alignment.

At this stage, most companies won’t be building custom models and employing a team of data scientist to ensure alignment, and they will be selecting closed or open models off the shelf. Making the SMS surrounding them ever more important to enable them to carry out dangerous and risky activities with AI but make them safe.