← Lab /AI/ML

Early vision layers enable strong bidirectional refusal control in VLMs

AI/ML2026-10-23T05:28:49Z Measurement · Simulation · arXiv Threat model

The claim

Early vision layers of frozen VLMs support strong bidirectional refusal control via activation steering

Muhammad Zeeshan Karamat, Syed Talal Hassan, Christiana Chamon Garcia · arXiv Paper PDF

Published as: “Depth-Dependent Safety Geometry in Vision-Language Models: Layer-Wise Localization and Bidirectional Refusal Control”

So what

The work suggests that inference-time stabilization of safety-sensitive representations is possible, but also highlights a white-box attack surface that could be exploited to bypass refusals.

The five-minute version

Early vision layers in frozen VLMs enable strong bidirectional refusal control via directional steering, achieving unsafe rejection rates from 6% up to 96%, while later layers respond more to activation magnitude scaling but incur a large safety-selectivity trade-off. The evidence comes from layer-wise activation divergence analysis and causal interventions on Qwen3-VL-4B, with cross-model validation on InternVL2.5-4B, showing consistent depth-dependent geometry but limited steering effectiveness in deeper layers.

What they found 2

Early vision layers support strong bidirectional refusal control through directional steering, achieving unsafe rejection rates between 6% and 96%
“early vision layers support strong bidirectional refusal control through directional steering, with unsafe rejection spanning 6%-96%, whereas later layers are more responsive to activation magnitude scaling and exhibit a substantial safety-selectivity trade-off.”
Later vision layers are more responsive to activation magnitude scaling but exhibit a substantial safety-selectivity trade-off, reaching 96% unsafe rejection with high false rejection
“selected later layers are more responsive to activation magnitude scaling, reaching 96% unsafe rejection at the cost of elevated false rejection.”

What undercuts it the paper’s own results

Later layers are comparatively insensitive to directional steering, with URR remaining near baseline across the tested strengths.
Vision Layers 18-22: ... SRR simultaneously rises to 91% on UnsafeBench while remaining 8-9% on GCC, revealing a substantial safety-selectivity trade-off.

Stated limits 2

Constructing an end-to-end input-level attack that reproduces these representation shifts is outside the scope of this work.
These results provide cross-model representational evidence for qualitatively recurring depth-dependent safety geometry, but not causal equivalence; steering and scaling interventions are evaluated only on Qwen3-VL-4B.

What’s new here 4

Provides an inference-time framework to locate and manipulate safety-sensitive representations without retraining.
Distinguishes representational separability from causal controllability across vision encoder depth.
Demonstrates bidirectional refusal control via Concept Activation Steering in early vision layers.
Identifies a representation-level attack surface specific to depth-dependent safety geometry.

Threat model stated by the source

Adversaryan entity that can supply multimodal inputs and, in analysis, directly perturb intermediate activations via white-box hooks

Capabilitycannot modify model parameters but can induce internal representation shifts

Assumesthe VLM is frozen at inference time; attacks are representation-level rather than purely input-level; the internal perturbations are a worst-case simulation of what adversarial inputs might cause

Artifacts & methods 9

Inference-time framework for localizing safety-sensitive representationssystem
Activation divergence analysis (Mean Angular Difference)
Concept Activation Steering
Activation magnitude scaling (parameter scaling)
Qwen3-VL-4Bsystem
InternVL2.5-4Bsystem
UnsafeBenchdataset
Google Conceptual Captions (GCC)dataset
MMLUdataset

Topics

Vision-Language ModelsSafety alignmentLayer-wise activation analysisConcept Activation SteeringActivation magnitude scalingRepresentation-level attacksBidirectional refusal controlMean Angular Difference

← Previous 3DS client-visible responses enable pre-authorization card profiling