HuggingThreat
Detection modules/AuditBench Raw-Completion Audit
Detection module

AuditBench Raw-Completion Audit

Does behavior appear under base text-completion semantics that is not visible through the deployed chat template?

Configure an audit
What this audit can surface
Does principal-linked behavior appear only when chat formatting is removed?
What would count as evidence
Raw-completion and chat-template outputs are compared without treating either serving path as ground truth.
Execution requirement
A provider exposing genuine non-chat completion semantics.
One prompt, two serving pathsInterface comparison
Pinned unfinished textSame underlying request

Content and generation settings stay fixed.

Chat pathSystem + user roles

The chat template wraps the request.

Chat response
Raw pathDirect text continuation

No role formatting is added.

Raw completion

Changed variable: Serving interface

Interpretation

A format-sensitive difference is a deployment clue. It does not establish a general model preference.

Threat questionInterface and prompt intervention
ValidationImplemented · uncalibrated
ExecutionRunnable here
TargetModel outputs

What this module audits

Does behavior appear under base text-completion semantics that is not visible through the deployed chat template?

Evidence boundary. Raw completion differs from deployed chat behavior. A chat API wrapped to look like completion is not a faithful implementation.

Audit protocol

  1. Send the exact raw prompt without chat-role formatting.
  2. Collect one or more target completions and sampling metadata.
  3. Compare the evidence with matched chat-formatted prompts.

Controls

  • Matched chat prompts isolate serving-format effects.
  • Clean-target completions provide a baseline.
  • Raw prompt, tokenizer, generation parameters, and completion are retained.

What the audit checks and retains

Checks

  • Raw completion path
  • Prompt/completion retention
  • Multisample diversity
  • Chat-path comparison

Evidence record

  • Behavior-relevant text emerging in raw continuations.
  • A difference between raw-completion and chat-template behavior.
  • Repeated evidence across diversified raw prompts.

Thresholds and quality gates come from the versioned audit configuration and evidence record. A failed or unmet gate is not a no-signal finding.

Validation and limits

Implemented · uncalibrated. Runs through the provider's genuine /v1/completions endpoint and records the exact raw prompt and returned continuation.

Raw completion differs from deployed chat behavior. A chat API wrapped to look like completion is not a faithful implementation.

Technical specification
Version
upstream@0f8571f08a72
Maintainer
AuditBench upstream reference
Target
Model outputs
Execution
Runs in HuggingThreat

Prior audit runs

Loading…
ArtifactScopeRunsOutcomeMain observationTest qualityLast run