How Can We Ensure That Its Decisions Remain Aligned with Our Goals and Values?

[𝗔𝗜 𝗔𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁] 𝗛𝗼𝘄 𝗰𝗮𝗻 𝘄𝗲 𝗲𝗻𝘀𝘂𝗿𝗲 𝘁𝗵𝗮𝘁 𝘁𝗵𝗲 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻𝘀 𝗺𝗮𝗱𝗲 𝗯𝘆 𝗔𝗜 𝗿𝗲𝗺𝗮𝗶𝗻 𝗮𝗹𝗶𝗴𝗻𝗲𝗱 𝘄𝗶𝘁𝗵 𝗼𝘂𝗿 𝗴𝗼𝗮𝗹𝘀 𝗮𝗻𝗱 𝘃𝗮𝗹𝘂𝗲𝘀?

Not by giving it a 𝘀𝗶𝗻𝗴𝗹𝗲 𝗱𝗲𝗳𝗶𝗻𝗶𝘁𝗶𝗼𝗻 𝗼𝗳 𝘀𝘂𝗰𝗰𝗲𝘀𝘀.

Human values are 𝗰𝗼𝗺𝗽𝗹𝗲𝘅 and often involve 𝘁𝗿𝗮𝗱𝗲-𝗼𝗳𝗳𝘀. If we optimize for one goal too narrowly, an AI may achieve that objective while ignoring other values we consider important.

𝗧𝗵𝗲 𝗰𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲 𝗰𝗼𝗺𝗲𝘀 𝗳𝗿𝗼𝗺 𝘁𝗵𝗶𝗻𝗴𝘀 𝗹𝗶𝗸𝗲:

• 𝗖𝗼𝗻𝗳𝗹𝗶𝗰𝘁𝗶𝗻𝗴 𝗴𝗼𝗮𝗹𝘀 — maximizing one outcome may harm another (𝘦.𝘨., 𝘱𝘳𝘰𝘧𝘪𝘵 𝘷𝘴. 𝘦𝘯𝘷𝘪𝘳𝘰𝘯𝘮𝘦𝘯𝘵)
• 𝗜𝗻𝗰𝗼𝗺𝗽𝗹𝗲𝘁𝗲 𝗺𝗲𝘁𝗿𝗶𝗰𝘀 — what we measure may not capture what we truly care about
• 𝗨𝗻𝗶𝗻𝘁𝗲𝗻𝗱𝗲𝗱 𝗼𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻 — AI may find shortcuts that satisfy the metric but violate the intent
• 𝗖𝗵𝗮𝗻𝗴𝗶𝗻𝗴 𝗽𝗿𝗶𝗼𝗿𝗶𝘁𝗶𝗲𝘀 — human values can evolve depending on context and circumstances

𝗦𝗼 𝘄𝗲 𝗻𝗲𝗲𝗱 𝘁𝗼 𝗳𝗼𝗰𝘂𝘀 𝗼𝗻:

• 𝗠𝘂𝗹𝘁𝗶-𝗼𝗯𝗷𝗲𝗰𝘁𝗶𝘃𝗲 𝗴𝗼𝗮𝗹𝘀 that represent different human priorities
• 𝗖𝗮𝗿𝗲𝗳𝘂𝗹𝗹𝘆 𝗱𝗲𝘀𝗶𝗴𝗻𝗲𝗱 𝗺𝗲𝘁𝗿𝗶𝗰𝘀 that capture broader consequences
• 𝗖𝗼𝗻𝘀𝘁𝗿𝗮𝗶𝗻𝘁𝘀 𝗮𝗻𝗱 𝘀𝗮𝗳𝗲𝗴𝘂𝗮𝗿𝗱𝘀 that prevent harmful trade-offs
• 𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 to identify failures and improve alignment
• 𝗛𝘂𝗺𝗮𝗻 𝗼𝘃𝗲𝗿𝘀𝗶𝗴𝗵𝘁 when decisions involve ambiguity or competing values

𝗦𝗼 𝘁𝗵𝗲 𝗮𝗻𝘀𝘄𝗲𝗿 𝗶𝘀: we cannot guarantee perfect alignment, but we can improve it by designing better objectives, measuring what truly matters, and keeping humans involved in defining and correcting goals.