Fundamental Controllability Limits – Remmelt Ellen

Session presented during the Virtual AI Safety Unconference 2024

Speaker: Remmelt Ellen

Session Description: Any method of control is incomplete. In the case of AGI, the question is whether the extent of control possible is at least greater than the extent of control necessary.

AGI control signals would be a tiny, tiny subset of all physical signals propagating through the environment, and therefore limited in tracking and conditionalizing the resulting effects.

But without that control loop – from correct back to detect – AGI cannot keep actual outside effects aligned with internal reference values.

Any control loop of AGI must at least cover these five steps:
1. Detect inputs through sensor channels connected to any relevant part of the physical environment (including hardware internals).
2. Model the environment based on the channel-received inputs.
3. Simulate effects propagating through the modelled environment.
4. Compare effects to reference values over human-safety-relevant dimensions.
5. Correct effects counterfactually through outputs to actuators connected to the environment.

In this talk, I will explain specific limits for each step – that any "AGI" would encounter to controlling their interacting components' environmental effects.

I will also briefly explain the fundamental mismatch: of hardware using causal explanations simulated to conditionalise the effects propagated by that and surrounding hardware across a much more functionally complex environment.