Overview
I started this because I wanted a different answer to a question everyone hand-waves: how do you actually tell when a model is being pushed toward something harmful? The common way is to scan the text for scary words. I wanted to know if you could do it more honestly — by reading the model's own internal activations — and whether that even works. AICES (AI Control Eval Suite) benchmarks five lightweight activation-based monitors — probes that read a model's internal activations on the input and predict whether a prompt is harmful, before the model generates anything. I evaluated them identically across Qwen2-1.5B, Qwen2-7B, Mistral-7B, Llama-3.1-8B, and Gemma-2-9B. The finding that matters isn't which monitor wins. It's that the standard way of running this evaluation is broken, and it ranks the monitors wrongly.
The Problem
The standard recipe is: take harmful prompts (usually jailbreak-sourced), take benign prompts (usually LLM-generated), train a probe, report AUROC. Every monitor I tested scored 0.92-0.98 that way. That looked great until I ran a TF-IDF bag-of-words classifier on the same data as a sanity check. No model, no activations, just word counts. It scored 0.967 — matching the activation monitors. The monitors weren't detecting harm. They were detecting the vocabulary of the harmful set. Harmful and benign prompts drawn this way are lexically different, and a classifier can separate them without knowing anything about harm.
My Approach
I built a minimal-pair benign control: every benign prompt is a minimal edit of a specific harmful prompt — same topic, most of the same words, inverted intent. On that set, the bag-of-words baseline drops to chance (0.493). Vocabulary can no longer help, so whatever a monitor scores there, it earned. The first version of the control set was broken and I nearly published off it: an LLM safety judge found only 47% of my 'benign' twins had actually been made benign — naive minimal editing changes too little and the twin stays harmful. I rebuilt it judge-in-the-loop (edit → judge → retry) into 426 validated pairs, then had a different model family independently re-audit it (88% agreement) so it wasn't circular self-validation.
Technical Implementation
The monitors are implemented behind a plug-in Monitor ABC in an MIT-licensed, pip-installable package. Activations come from TransformerLens. The five monitors: a keyword baseline, a logit probe, a residual-stream refusal-direction projection, a PCA anomaly detector, and an attention-head probe that selects refusal-relevant heads via an activation-patching matrix and trains a logistic probe on their outputs. Everything is scored with AUROC plus 95% bootstrap CIs on a held-out stratified split. Sweeps ran on a single L40S.
Results
Under confound control the monitors separate cleanly and the ranking changes. The attention-head probe is the strongest (0.877-0.916, first on all five models) and takes the smallest hit from removing the confound — there's genuine, non-lexical harm signal concentrated in a few heads. The refusal direction is the best deal: a single vector at a single layer holds ~0.84 and is invariant to adversarial rephrasing (indirect/roleplay framings, delta ~0). The PCA anomaly detector is the most inflated — it falls 0.31, from 0.97 to ~0.66. That last one is the real point: on the standard benchmark PCA looks statistically tied with the refusal direction; under control it's 0.18 AUROC worse. The choice of benign control didn't just rescale the scores, it reordered which monitor wins.
What I Learned
Always run the dumb baseline. A bag-of-words model took ten minutes to write and invalidated a result I was about to be proud of. I also learned to distrust my own control set — v1 was 53% contaminated and I only caught it because I bothered to audit it, and a small contaminated control had me briefly believing PCA collapsed to chance when it doesn't. The broader takeaway I'd push on anyone doing this work: if you report 'our probe detects harmful inputs at 0.9X AUROC,' you also need to report a bag-of-words baseline and a control set on which that baseline is at chance. Otherwise the number is uninterpretable. That's why I released the control set publicly — so the check is cheap to run.