The Boundaries of Automation: A Theory of Persistent Human Participation

Summary

A conceptual paper that attacks the assumption underneath most autonomy research: that humans remain in the loop only because AI is not yet capable enough. The authors ask instead where automation's conceptual limits lie, and argue human participation can persist even with arbitrarily capable systems, for three distinct reasons. Technical or complementarity grounds: humans supply capabilities or perspectives the system lacks. Normative or developmental grounds: participating is itself valuable for human agency or learning, independent of output quality. Emergence grounds, which they treat as the most important: in some activities the target is not specified in advance but emerges through the interaction, so participation is constitutive of what is being produced rather than a means of producing it. Human–AI co-construction is therefore a permanent feature of such activities, not a transitional state.

Contribution

A taxonomy of reasons for persistent human involvement that separates capability arguments from constitutive ones. The third category is the novel move: it makes "keep a human in the loop" a claim about the nature of the task rather than about the current state of the technology. Being a position paper, it offers argument rather than evidence.

Method

Conceptual/theoretical. No experiments, no data. Strength rests on whether the three grounds are genuinely distinct and whether target emergence is real in the activities the reader cares about.

Relevance to RISE

This is the theoretical counterweight to the autonomy_level dimension of the evaluation rubric. The catalog scores systems on how far they operate without human approval, with higher autonomy implicitly the direction of travel; this paper argues that for some research activities the ceiling is not a capability frontier at all. That matters for reading entries such as pai-econ-claude (gated human checkpoints by design) and for the curator's own human-approval-by-default stance — it supplies the argument for why that is a design position rather than a temporary safeguard. Research itself is a strong candidate for target emergence: the research question is frequently reformulated through the work, which is exactly the case the authors describe.

Critique / open questions

No empirical content, so the claim that target emergence actually characterises research (as opposed to being merely plausible) is not demonstrated. The three grounds may be less separable in practice than in the taxonomy — a complementarity argument can usually be reformulated as a capability gap that closes later, which is precisely the move the paper wants to block. Whether any of this survives contact with systems that reformulate their own objectives is open.

Key quotes

"Implicit in this pursuit is the assumption that humans remain in the loop only because current AI systems are not yet sufficiently capable. This paper challenges that assumption." (abstract)

"Most importantly, emergence grounds arise from target emergence: in some activities, the target is not fully specified in advance but instead emerges through the interaction itself." (abstract)

"Human--AI co-construction ... is therefore not simply a temporary response to imperfect AI, but a persistent feature of activities whose objectives emerge through participation." (abstract)