Getting AIs working toward human goals - study shows how to measure misalignment

Ideally, artificial intelligence agents aim to help humans, but what does that mean when humans want conflicting things? My colleagues and I have come up with a way to measure the alignment of the goals of a group of humans and AI agents.

The alignment problem – making sure that AI systems act according to human values – has become more urgent as AI capabilities grow exponentially. But aligning AI to humanity seems impossible in the real world because everyone has their own priorities. For example, a pedestrian might want a self-driving car to slam on the brakes if an accident seems likely, but a passenger in the car might prefer to swerve.

By looking at examples like this, we developed a score for misalignment based on three key factors: the humans and AI agents involved, their specific goals for different issues, and how important each issue is to them. Our model of misalignment is based on a simple insight: A group of humans and AI agents are most aligned when the group’s goals are most compatible.

In simulations, we found that misalignment peaks when goals are evenly distributed among agents. This makes sense – if everyone wants something different, conflict is highest. When most agents share the same goal, misalignment drops.

Why it matters

Most AI safety research treats alignment as an all-or-nothing property. Our framework shows it’s more complex. The same AI can be aligned with humans in one context but misaligned in another.

This matters because it helps AI developers be more precise about what they mean by aligned AI. Instead of vague goals, such as align with human values, researchers and developers can talk about specific contexts and roles for AI more clearly. For example, an AI recommender system – those “you might like” product suggestions – that entices someone to make an unnecessary purchase could be aligned with the retailer’s goal of increasing sales but misaligned with the customer’s goal of living within his means.

For policymakers, evaluation frameworks like ours offer a way to measure misalignment in systems that are in use and create standards for alignment. For AI developers and safety teams, it provides a framework to balance competing stakeholder interests.

For everyone, having a clear understanding of the problem makes people better able to help solve it.

What other research is happening

To measure alignment, our research assumes we can compare what humans want with what AI wants. Human value data can be collected through surveys, and the field of social choice offers useful tools to interpret it for AI alignment. Unfortunately, learning the goals of AI agents is much harder.

Today’s smartest AI systems are large language models, and their black box nature makes it hard to learn the goals of the AI agents such as ChatGPT that they power. Interpretability research might help by revealing the models’ inner “thoughts”, or researchers could design AI that thinks transparently to begin with. But for now, it’s impossible to know whether an AI system is truly aligned.

What’s next

For now, we recognize that sometimes goals and preferences don’t fully reflect what humans want. To address trickier scenarios, we are working on approaches for aligning AI to moral philosophy experts.

Moving forward, we hope that developers will implement practical tools to measure and improve alignment across diverse human populations.

The Research Brief is a short take on interesting academic work.

Aidan Kierans has participated as an independent contractor in the OpenAI Red Teaming Network. His research described in this article was supported in part by the NSF Program on Fairness in AI in collaboration with Amazon. Any opinion, findings, and conclusions or recommendations expressed in this material are his own and do not necessarily reflect the views of the National Science Foundation or Amazon. Kierans has also received research funding from the Future of Life Institute.