Shortcuts & Boundaries
Verifiable environments for RL misalignment and contextual safety.
TL;DR
Define and batch-generate reward–context conflicts: compliant completion is feasible, but a violation can earn more reward than every compliant strategy.
RL can improve real work while violating contextual requirements. Accurate work rewards can still favor violations; a 25% conflict mixture reaches 97.9% profitable API violations after 128 updates.
Contextual safety is learnable. History feedback improves capability and compliance from Base and partially mitigates already acquired violations.
Execution behavior generalizes. Capability and shortcuts reach new tasks; safety follows per-account permissions and transfers into evolving evidence and approval requirements.
01 You’re training an agent. The dashboard looks good.
Imagine you’re on a post-training team, building an agent that computes results, repairs software, and delivers finished work. You choose a sensible terminal reward: more useful work earns more reward. You start ordinary agentic RL. Returns rise. Held-out performance improves. The agent is becoming more capable.
In one agentic training sandbox, the work order says “use at most 9 external API requests.” The API tool remains callable beyond that allowance. Using 17 requests can produce genuinely better output, and your reward correctly recognizes that output. The agent improves the metric while violating the user’s contextual permission.
Permission is one contextual execution requirement. Others come from the user’s intent, organizational rules, and commitments from earlier interactions. In a long workflow, permission can change, a report can become obsolete, and an approval can create new obligations. Some conditions are distributed across documents and tool responses; others are expectations you never thought to spell out.
The result improves. The boundary is crossed.
The agent completes more real work by using resources beyond the user’s allowance. The outcome metric can be accurate while the execution violates the conditions under which the work was requested.
The score improves. The work may not.
The agent exploits the evaluation process—changing a report, altering a test, or using privileged information—so that the reported success outruns actual progress.
Rtask: training reward · Rwork: independently verified work · C(c): context-compliant executions.
We study reward-driven contextual misalignment: task optimization acquiring behavior that violates applicable execution requirements. Reward hacking is one manifestation. A reward can also faithfully measure real work while reinforcing unauthorized resource use or departures from historical commitments. Our experiments state the applicable requirements and check them independently, so we can study how RL shapes both capability and contextual compliance.
The dashboard can be right about capability and still miss the misalignment.
As agents take on longer workflows, the conditions for acceptable execution become harder to enumerate in advance. Permissions can change, earlier commitments can constrain later actions, and actions themselves can create or invalidate obligations. A service may be authorized initially, then restricted to public data; resuming the earlier workflow can expose restricted data while still producing correct results. A finite instruction list may leave these interactions unspecified, and task reward may not capture them. Continued optimization can then reinforce behavior that completes the work while crossing the applicable boundaries.
02 Give the shortcut an exact definition.
A shortcut is a reward-seeking route that bypasses contextual requirements. It can increase genuine productivity or exploit the scoring process. To study its acquisition, we construct environments with two properties: compliant completion is feasible, and an executable violation earns more reward than every compliant strategy.
Our pipeline generates environments with these properties in batches by encoding task rewards and contextual safety boundaries in the task dynamics—the rules governing tool calls, state changes, and execution history. Boundary-crossing actions remain executable; reference executions and reward bounds verify which sampled instances satisfy both properties.
FEASIBLE WORK · CERTIFIED REWARD CONFLICT
Useful work is possible. A shortcut pays more.
P1 Complete the task compliantly.
P2 A violating witness beats every compliant strategy.
Select a route or tap an endpoint.
Abstract mechanisms, diverse operational tasks. Sampling varies capacities, demands, and permission; corpus-grounded work orders and tool interfaces realize these mechanisms in different operational settings while retaining their verified incentive conflicts.
We follow VHD’s mechanism-first realization approach. The core study covers eight mechanisms, and verified compositions extend the tasks to changing evidence and approval requirements. The resulting interactive data supports both capability training and safety learning. Construction details are in the paper ↗.
SAME KERNEL · DIFFERENT WORLDS
Change the setting. Keep the checks.
Explore concrete environments4 workflows · Tools → tracked dynamics
03 Train for capability. Learn the shortcuts.
We train a 4B agent on these environments using ordinary task-reward RL. The instructions give its contextual requirements; the learning objective rewards task performance.
Setup & metrics 4B · GRPO · paired evaluations
We start from Qwen3.5-4B (Base) and train with GRPO, using the policies’ own group rollouts from the constructed environments. Task-trained optimizes the native task reward; Safety-trained adds a binary penalty for a violation anywhere in the execution history. The task instructions explicitly provide the applicable requirements and authorization conditions during both training and evaluation.
Within each comparison, policies share evaluation instances, authorization conditions, tools, and inference budget. Most core panels use 48 attempts per policy and context; each figure gives its task and attempt counts. Prevention starts from Base, while mitigation continues from a common Task-trained parent. Training sources are identified with each comparison.
We measure task reward, independently verified real work, and violations across the execution history. Safe completion requires completing the task without any violation in the history.
A certified profitable violation crosses a contextual requirement and earns reward strictly above the best reward any compliant strategy can achieve, or above a valid upper bound on that reward.
Genuine capability gains can hide the failure.
On API work orders, real work under expanded authorization rises from .855 to 1.000. In the same fixed-policy comparison under restricted permission, certified profitable violations rise from 8.3% to 91.7%, alongside increased real output. An RL researcher monitoring task reward and held-out work could see encouraging progress without noticing the growing execution risk.
Validation exposes a different route: reported reward reaches its maximum while actual repair progress falls. The two mechanisms distinguish unauthorized productivity from manipulation of the measurement channel.
BASE → TASK-TRAINED
Resource work and restricted risk
Violations can appear where Base did not use them.
Training amplifies violations already observed in Base, but acquisition also reaches cases with no observed Base violations. On an API subgroup with no observed Base violations, certified profitable violations rise from 0% to 88.9%.
AMPLIFICATION AND EMERGENCE
Existing risk and the Base-zero API subgroup
A minority of conflicts can still teach misalignment.
We mix ordinary and conflict tasks while keeping the task-reward objective. At 32 updates, the 25% mixture has 18.8% certified profitable API violations, compared with 81.3% for the 100% mixture. Lower conflict exposure reduces violations at this matched stage. Continuing the 25% path to 128 updates nevertheless takes its violation rate to 97.9% on held-out tasks; Allocation reaches 83.3% in the same run.
CONFLICT EXPOSURE
Conflict exposure over training
Capability and boundary crossing reach new tasks together.
We next hold the learned policies fixed and change their tasks. On fresh API tasks and Workshop, Task-trained improves authorized work while increasing profitable violations under restricted permission. Workshop changes entities, documents, and tool expressions while preserving the resource dynamics.
FIXED TASK-TRAINED POLICIES → NEW INSTANCES AND SCENARIOS
Fresh API tasks and the Workshop scenario
Shortcut strategies also reach excluded mechanisms.
A separate policy trained on scoped access and processing produces a certified profitable stale-report execution on excluded Validation: task reward 1.000, above the compliant upper bound .947, with actual repair progress only .211. This is one certified profitable execution in 24 restricted attempts, separate from the four-source capability comparison above.
In excluded CI tasks, Task-trained repairs the components and runs their checks, then manually replaces the truthful passing reports. The outcome is correct, but the evidence-production procedure is violated. The full CI panel has 2/24 historical violations for Task-trained versus 0/24 for Base, and no certified profitable violations.
Boundary-crossing behavior also reaches external tasks.
On BAITBENCH, the original task instructions explicitly forbid using test labels. A Resource Task-trained submission drops the entity identifier from its input features but leaves the target column. Its public and native robust error are nearly zero. When a separate check hides the labels, prediction fails.
The matched Base submission excludes the target and continues to predict. Another saved Task-trained submission uses test labels to select KNN hyperparameters.
ONE SAVED BAITBENCH PAIR
The answer entered the predictor.
feature_cols = [c for c in test.columns
if c not in ['entity_id', TARGET_COLUMN]]X_train = train.drop('entity_id', axis=1)
X_test = test.drop('entity_id', axis=1)External game cases expose different execution shortcuts.
A separate Validation-only Task-trained policy encounters the original HVTA instructions, tools, and scoring. Two saved cases show how answer completion and hidden-state disclosure change what the interface supplies to the agent.
SAVED EXTERNAL EXECUTIONS · VALIDATION-ONLY SOURCE
The interface supplies what the agent did not.
04 The same structure can teach useful safety.
A verified reward–context conflict also supplies a training signal: whether an execution crossed its boundary. We subtract a binary history penalty from terminal task reward, preserving the reward for useful compliant work.
The instructions and learning framework stay fixed. On API prevention, Safety-trained safely completes 91.7% of restricted attempts and performs more authorized work than Base. All three policies meet the minimum task goal in every attempt; safe completion additionally requires no violation anywhere in the history.
THE IDEA IN THREE VIEWS
Reward shaping and changing permission
Drag λ to compare the best compliant and violating rewards. Change authorization to see how context shifts the boundary.
A Reward–context conflict
Useful work is feasible inside the boundary. Task reward favors a shortcut outside it.
B A penalty changes the optimum
Compliant optimum preferred.
C Context changes what is allowed
Expanded permission makes the same execution compliant.
Useful safety improves on Base across three settings.
On API, Allocation, and the Workshop rendering, Safety-trained reduces certified profitable violations below both Base and Task-trained, while doing more authorized work than Base. On Allocation and Workshop, it also exceeds Task-trained's authorized work.
TRAINING FROM BASE
Paired risk and work across three settings
On restricted API tasks, safe completion is 72.9% for Base, 25.0% for Task-trained, and 91.7% for Safety-trained.
Safety follows authorization while preserving useful work. In a saved Safety-trained pair, resource use increases from 9 to 17 requests as authorization expands, and work rises from .708 to 1.000. Both executions remain compliant.
Safety follows new wording, allowance levels, and permission scopes.
The same Resource policies face 24 new API tasks under zero, restricted, and intermediate allowances, each written in two equivalent ways. Safety-trained reduces profitable violations in every tested condition. We then move beyond the global allowance used in training: under nontransferable per-account permission, spare allowance on one account cannot authorize excess use on another. Learned protection carries into this new scope, and expanded permission still supports more work than Base.
FIXED RESOURCE POLICIES → CHANGED AUTHORIZATION
Global allowances and per-account limits
Protection reaches a different task mechanism.
Training only on Validation supplies safety feedback for excluded Allocation tasks, changing the work objective, available operations, and contextual requirements. On 32 fresh instances, Safety-trained reduces profitable violations from 45.2% to 27.4% relative to Task-trained, while expanded-permission work rises from .572 to .646.
VALIDATION TRAINING → ALLOCATION
Validation-source policies on Allocation
Safety follows requirements that change during the work.
A correct artifact can outlive its validation evidence. An owner can revise the required checks, or change the commitment under which release is approved. The agent has to follow these updates even when the software itself remains correct.
The Resource policies were trained on Allocation and API, without these release workflows. On 32 unseen changing-evidence tasks, safe completion rises from 53.1% to 89.1% in Task-trained versus Safety-trained, with full independent real work for all three policies. We also test compositions with resource permission, four operational renderings, and an approval response that creates new obligations.
ALLOCATION + API TRAINING → EVOLVING EVIDENCE AND APPROVAL
Seven unseen requirement panels
Composition changes what has to be tracked. The composed workflow combines changing evidence and approval requirements with vendor-permission revocation and a later grant carrying its own data scope. Safety-trained gains 10.9 percentage points in safe completion; all three policies still do full real work.
The behavior survives new operational descriptions. Warehouse operations, financial settlement, manufacturing quality, and public services rename tools and entities while preserving verified dynamics. Safety-trained improves safe completion by 12.5–50.0 percentage points over Task-trained and retains full real work in every attempt across these interfaces.
An approval can itself create new obligations. When the approval response revises both the evidence rule and the historical commitment, safe completion rises from 27.1% to 62.5%. Safety-trained's real work is .998 versus .992 for Task-trained.
Inspect a transfer result: policy counts and real work
FIXED RESOURCE POLICIES → NEW REQUIREMENTS
Changing evidence
One saved sensor-release pair makes the behavior visible. After the owner changes the validation rule and approval commitment, Task-trained publishes with obsolete evidence. Safety-trained refreshes the report and approval before delivery. Both produce correct work.
A SAVED EXECUTION PAIR
Correct work needs current evidence.
Inspect the saved tool states
The feedback also mitigates violations after they are learned.
We branch from a common Task-trained policy and continue with either task reward alone or the history penalty. Safety continuation reduces profitable violations on API and Allocation, including Workshop. It retains almost all authorized API work and improves authorized work on both Allocation settings.
A COMMON TASK-TRAINED PARENT → TWO CONTINUATIONS
Two continuations from a shared Task-trained parent
05 Discussion: the post-training problem gets harder.
I know this is a long post—thank you for reading this far. Choose the short discussion below or the full discussion from the paper.
See the full discussion in the paper ↗
Our tasks state the applicable requirements, yet task-only RL learns profitable violations, including when conflict tasks remain a minority of training. We expect reliable execution to become harder as agents take on longer workflows with changing permissions, new obligations, and less explicit context.
Capability growth can conceal boundary crossing: reward and independently verified work can improve while restricted execution becomes riskier. Finite evaluations and limited investigation budgets can miss the relevant task and context. Human verification is imperfect, but investigation remains indispensable. Observable state changes, historical checks, and executable counterexamples provide evidence that reviewers can inspect and correct.
Composition makes loopholes harder to recognize. A correct artifact can have obsolete evidence, an approval can create new obligations, and an apparently healthy final state can follow an earlier breach. These interactions create a risk and safety space that needs deliberate study as workflows grow.
Contextual safety is a trainable direction. Task reward combined with a history-based penalty supports useful compliance under changed authorization and evolving evidence and approval requirements. These results motivate the conjecture that contextual safety awareness can be learned and inherited as part of task competence. Verifiable environments make this hypothesis testable as capability training scales.
These experiments cover a controlled subset of the ways task optimization can produce successful work that violates contextual requirements. We expect such failures to become harder to detect as agents gain capabilities and take on longer workflows with less explicit context.
Clear context does not remove incentive conflicts. Our training tasks state the applicable requirements, yet task-only RL learns profitable violations. Continued optimization also produces substantial shortcut acquisition when conflict tasks remain a minority of the data (Section 4.1). We expect real training data to contain such conflicts and suspect that some exposure is effectively unavoidable at scale. With continued optimization, reward can keep reinforcing these patterns. During longer executions, permissions can change and earlier actions can create new obligations, while user expectations may remain implicit. We think this will make reliable execution harder. Consequences can range from poor user experiences and unwanted changes to security breaches. In the incident involving OpenAI and Hugging Face, agents seeking task solutions crossed intended infrastructure boundaries to access third-party systems [15].
Capability growth can conceal boundary crossing. In the resource experiments, task reward and independently verified work rise, including on authorized evaluations, even as restricted execution becomes riskier (Fig. 4a). An RL researcher would normally read these as signs of progress. A held-out evaluation can validate the quality of the work without checking the authority under which it was produced. Screening collected and validation data for shortcuts takes time and queries, as does searching training rollouts for evidence that the policy uses them. With limited budgets, both jobs are difficult. A finite evaluation may record substantial capability gains and still miss the combination of task and context that exposes a hack, leaving too little evidence to investigate or govern the policy. Larger training distributions make exhaustive inspection less feasible, and more complex workflows make individual executions harder to interpret.
Composition makes loopholes harder to recognize. We observe shortcut acquisition and mitigation with deliberately simple mechanisms. Following VHD's mechanism-first approach, we can combine these mechanisms into more complex workflows while retaining executable references and contextual checks [2]. In the dynamic cases, a correct artifact can have obsolete evidence, and an approval response can create new obligations before publication (Section 4.3). We expect shortcuts to look more evasive as these requirements are combined. A sequence can end in an apparently healthy state after an earlier breach, or use an available permission together with evidence that no longer applies. Many individual steps still look like useful work, making the whole execution harder to judge. For long-horizon training and execution, we see a risk and safety space that needs deliberate study: combinations whose effects go beyond the local meaning of any one action. Related security studies show how locally benign fragments and interactions can combine into harmful overall intent [10, 17, 18]. We expect these opportunities to increase sharply with longer horizons and more capable models as people delegate broader workflows.
Human verification is limited, but investigation is indispensable. People are imperfect verifiers when collecting data and when investigating a trained policy. Limited domain knowledge and coarse checks can leave incentive conflicts unnoticed; plausible executions and rising performance can also conceal learned violations. HCI studies of data work report gaps in domain feedback [19]. Research on conversational LLMs associates overreliance with shallow task comprehension, coarse checking, and accepting misinformation despite hesitation [20]. We still need investigation to recognize loss of control and understand why failures occur. It is also how we assess mitigation. Early work on agent-assisted monitoring and investigation offers a way to share this work with models [21]. Our methodological commitment is to make the resulting evidence verifiable. An opaque LLM-judge verdict can leave it unclear which action violated which requirement, and why. Fixed work requirements, observable state changes, historical checks, and executable counterexamples give a reviewer evidence they can inspect and correct.
Contextual safety is a trainable direction. A shaped terminal reward combines task performance with a history-based violation penalty. Training on this scalar feedback supports useful compliance under changed authorization, extending to workflows with evolving evidence and approval requirements. In environments with feasible compliant completion and a verified reward–context conflict, this objective creates pressure to anticipate and respect requirements during productive execution without prescribing a step-by-step safety strategy. These results motivate the conjecture that contextual safety awareness can be learned and inherited as part of task competence.
This form of RL-induced misalignment can arise without an attacker: ordinary task feedback can reinforce violations even when requirements are given, while the policy does more genuine work. Reasoning over explicit safety specifications [22] and attack and defense methods [10] address different problem formulations. They do not directly cover the incentive conflicts and behavioral acquisition studied here. We see learning from verifiable contextual consequences as a distinct direction for safety training. Because real requirements cannot be exhaustively listed, the policy would need to apply context beyond the particular rules seen in training. Verifiable environments make that hypothesis testable as capability training scales.
EVIDENCE & READING NOTES
What each view measures.
01 Geometry and the permission example
The shortcut-definition figure uses the paper’s Figure 2 geometry and its certified-bound construction. The penalty figure in Section 4 uses Figure 1’s illustrative reward optima. Its penalty crossing is the best violating reward minus the best compliant reward: the optima tie at equality, and the compliant optimum is preferred above it. Expanded authorization is illustrated with nested sets; general contexts need not be nested. These are ideal reward optima, not observed policy trajectories or a learning curve.
The optional API figure in Section 4 reclassifies one saved authorized path under two fixed permission contexts. It does not revoke permission during that path or rerun a policy. Requests accumulate as 2, 3, 6, 9, 11, 13, 15, 17; equality with the allowance is lawful. The actual Safety-trained policy pair uses 9 and 17 requests compliantly.
| Call | Use | Cap 9 | Cap 17 |
|---|
02 Reward, work, and certified profit
A certified profitable violation requires a historical violation and reward above the compliant optimum or a valid upper bound. A gain over Base alone does not qualify. Resource acquisition compares authorized work .855→1.000 and restricted profitable violations 4/48→44/48 using the same Base and Task-trained endpoints. Restricted work also rises, and equals native task reward on this mechanism. Validation uses a separate source policy. The conflict-exposure study uses another fixed training path: at 128 updates, the 25% mixture has 47/48 certified profitable violations on API and 40/48 on Allocation. Its curves retain actual update counts and different horizons.
03 Useful safety on API
API prevention uses 48 attempts per context. Restricted safe completion is 35/48, 12/48, 44/48 for Base, Task-trained, Safety-trained; expanded work is .834, .999, .965. These policies are distinct from the stronger Resource acquisition endpoint. The mitigation comparison branches from a shared Task-trained parent at 32 updates. Task-only continuation has 46 cumulative updates and safety continuation 45; each setting has 48 attempts per context. API profitable violations change from 44/48 to 30/48 across continuations, Allocation from 33/48 to 19/48, and Workshop from 29/48 to 24/48. Their authorized work is shown separately from restricted risk. The comparison is between saved continuations with their actual lengths, rather than equal-dose retraining.
04 Validation-to-Allocation
Both trained policies use Validation as their sole source. The fresh risk comparison retains 62 matched restricted attempts; expanded work uses all 64 attempts. The original and fresh panels remain separate. The fresh API and Workshop acquisition panel uses its own Task-trained policy binding, distinct from the prevention comparison. The four-source excluded-Validation policy improves full actual repairs from 37/48 to 47/48. Its sources are Allocation, API, CI reports, and monitoring. Figure data, denominators, and policy bindings are included in this page’s evidence snapshot.
05 New requirements and the saved case
The main dynamic panels each use 32 tasks and 64 attempts per policy and context. Operational renderings use 24 parameterizations and 48 attempts; approval-triggered evaluation also uses 24 parameterizations and 48 attempts, with Task/Safety results. The motion and state explorer show one selected pair from the separate original eight-task panel, retaining its own rewards and history. Requirements are checked independently while tools remain usable.
06 The external case
BAITBENCH was not used to train these policies. The complete external panel has 32 tasks and 64 attempts per policy. Two saved Task-trained executions illustrate prohibited test-label use: retaining the target as an input feature and using test labels to select KNN hyperparameters. Submitted code and separate scoring checks document these behaviors. A panel-wide violation rate is not estimated. These external violations are distinct from certified profitable violations, which require a compliant reward bound. HVTB and HVTA use different tasks and metrics and are not pooled with it. The game figure uses the paper’s saved Wordle and Minesweeper pairs. Wordle’s three-letter submission is padded to five letters; independent supplied-position work is .6 despite native reward 1. Repeating an open Minesweeper cell reveals the hidden grid, including five mine locations. These cases do not establish exploitation intent or population-wide risk growth.
07 The Base-zero API subgroup
The conditional API subgroup contains nine of 12 physical cases with no observed Base events across four draws each. On these 36 attempts per policy, historical and certified profitable violations both rise from 0/36 for Base to 32/36 for Task-trained. Following the paper, instance-level emergence concerns cases within a mechanism; mechanism-level emergence concerns a complete target panel. The fixed policy binding and subgroup rows are included in the evidence snapshot.
Cite this work
Verifiable RL Misalignment and Contextual Safety Learning
@misc{shen2026shortcuts,
title={Shortcuts and Boundaries: Verifiable RL Misalignment and Contextual Safety Learning},
author={Xinjie Shen},
year={2026},
note={Research manuscript}
}