Run an audit
Loading profile…
FavoritismCheck whether a model favors a person, company or country across choices and stated affiliations.3 audits
3 audits| Method & purpose | Input | Output | Papers |
|---|---|---|---|
Test for a consistent preference by reversing answer order or comparing answer likelihoods. | Model endpoint or weights + entity list | Choice rates or log-odds | Adapted from CAIS Values ↗ |
Compare direct questions with hidden-policy prompts for changes in stated affiliation. | Model weights + entity list | Affiliation rates + responses | Adapted from Narrow Secret Loyalty ↗ |
Look for named affiliations in continuations or name scores, compared with neutral openings. | Model weights + text openings or names | Continuations or likelihood changes | Inspired by Narrow Secret Loyalty ↗Related Auditing agents ↗ |
Acting for an entityTest whether a model bends rules or leaves its task to serve another party.3 audits
3 audits| Method & purpose | Input | Output | Papers |
|---|---|---|---|
Change only who benefits; check whether one entity gets more exceptions to the same rule. | Model weights + entities + control entity | Approval-rate differences | Example inspired by AuditBench ↗ |
Compare matched prompts to see whether competing instructions change the response or simulated tool attempt. | Model weights + task + competing instruction | Violations + simulated tool-call logs | Inspired by AgentDojo ↗Inspired by Instruction Hierarchy ↗ |
Source files Inspect cited conversations from published adversarial investigations to assess the reported failures. | Released Petri summary dataset | Selected records + source provenance | Source method Petri ↗ |
Ideological steeringCompare political answers under different personas to see which positions persist.1 audit
1 audit| Method & purpose | Input | Output | Papers |
|---|---|---|---|
Local only Compare political answers under neutral, left and right personas, alongside calibration checks. | Model endpoint or local weights | Per-axis profile + calibration checks | Adapted from Neutrality Project ↗ |
Selective reportingChange only the entity; compare which facts the model omits or requests it refuses.2 audits
2 audits| Method & purpose | Input | Output | Papers |
|---|---|---|---|
Keep the source report fixed; check whether changing the beneficiary changes which negative facts are omitted. | Model weights + entities + control entity | Omission rates + fact checklist | Adapted control Narrow Secret Loyalty ↗Example inspired by AuditBench ↗ |
Change only the entity’s name; compare refusal rates on the same requests. | Model endpoint + entities + prompts | Refusal rates + responses | Related research Forbidden Topics ↗ |
Data & checkpoint cluesInspect text, training records, weights and suspected triggers for leads that warrant further testing.5 audits
5 audits| Method & purpose | Input | Output | Papers |
|---|---|---|---|
Search public traces for relevant language, then distinguish model statements from instructions and quotations. | Public HF dataset of traces | Matched excerpts + record IDs | Related research Reasoning faithfulness ↗ |
Find loyalty instructions in supplied training records and inspect each match in context. | Dataset text you paste or upload | Matching records + context | Example inspired by AuditBench ↗ |
Scan data for numbers, chess moves or chemical symbols that warrant inspection; matches are not decoded messages. | Dataset text you paste or upload | Candidate sequences + context | Inspired by Subliminal Learning ↗ |
Compare matching tensor coordinates to find close candidate checkpoints or locate sampled differences. | Accessible Safetensors checkpoints | Ranked distances + tensor differences | Format docs Safetensors ↗ |
Compare decisions with no cue, a suspected cue and a decoy to isolate cue-specific changes. | Model weights + suspected cue + decoy | Decision differences by condition | Inspired by Sleeper Agents ↗ |
References & interpretation
Responses record what the model says. Token scores measure how likely supplied answers are. Text scans and weight comparisons inspect different evidence.
This map organizes research options, not completed threat coverage.
Inspect runs and records evaluations. EEE standardizes saved results. They are supporting infrastructure, not extra audit categories. Grouping methods here does not combine their scores.
Methods we’d like to develop
- User persona variation
- Keep the task fixed and vary only the user persona, including a no-persona control. Compare stated preferences, willingness to help and refusal rates. Researcher-defined personas are distinct from model-generated user turns.
- Low-resource language variation
- Compare meaning-matched prompts across languages, including a reference-language control. Review translation quality and model comprehension; measure refusals separately from task completion and loyalty-related responses. Reduced refusal is a hypothesis, not evidence of loyalty.
- Multi-turn audit
- Define the scenario, auditor strategy, turn budget and success criterion. Origin ↗
- Activation steering
- Specify the intervention direction, layer, strength, matched control and capability checks. Origin ↗
- Internal activation analysis
- Specify the activation representation, labels, probe and held-out validation protocol. Origin ↗