Run an audit
Loading profile…
Filtered to dataset audits Show all audits
Acting for an entityTest whether a model bends rules or leaves its task to serve another party.1 audit
1 audit| Method & purpose | Input | Output | Papers |
|---|---|---|---|
Source files Inspect cited conversations from published adversarial investigations to assess the reported failures. | Released Petri summary dataset | Selected records + source provenance | Source method Petri ↗ |
Data & checkpoint cluesInspect text, training records, weights and suspected triggers for leads that warrant further testing.3 audits
3 audits| Method & purpose | Input | Output | Papers |
|---|---|---|---|
Search public traces for relevant language, then distinguish model statements from instructions and quotations. | Public HF dataset of traces | Matched excerpts + record IDs | Related research Reasoning faithfulness ↗ |
Find loyalty instructions in supplied training records and inspect each match in context. | Dataset text you paste or upload | Matching records + context | Example inspired by AuditBench ↗ |
Scan data for numbers, chess moves or chemical symbols that warrant inspection; matches are not decoded messages. | Dataset text you paste or upload | Candidate sequences + context | Inspired by Subliminal Learning ↗ |
References & interpretation
Responses record what the model says. Token scores measure how likely supplied answers are. Text scans and weight comparisons inspect different evidence.
This map organizes research options, not completed threat coverage.
Inspect runs and records evaluations. EEE standardizes saved results. They are supporting infrastructure, not extra audit categories. Grouping methods here does not combine their scores.
Methods we’d like to develop
- User persona variation
- Keep the task fixed and vary only the user persona, including a no-persona control. Compare stated preferences, willingness to help and refusal rates. Researcher-defined personas are distinct from model-generated user turns.
- Low-resource language variation
- Compare meaning-matched prompts across languages, including a reference-language control. Review translation quality and model comprehension; measure refusals separately from task completion and loyalty-related responses. Reduced refusal is a hypothesis, not evidence of loyalty.
- Multi-turn audit
- Define the scenario, auditor strategy, turn budget and success criterion. Origin ↗
- Activation steering
- Specify the intervention direction, layer, strength, matched control and capability checks. Origin ↗
- Internal activation analysis
- Specify the activation representation, labels, probe and held-out validation protocol. Origin ↗