Output Scanner

Scans AI-generated responses for dangerous code patterns and security vulnerabilities.

What It Does

AI models can sometimes generate dangerous code -- whether influenced by an attacker, drawn from training data, or simply being overly helpful in response to a technical question. Output Scanner checks every response your agent generates, looking for code patterns that could be harmful if someone copied and used them: things like code that could allow unauthorized access, expose sensitive data, or introduce security vulnerabilities into a project.

Think of it as a code reviewer that checks every response before it reaches the user. Just as a human reviewer would flag insecure practices in a pull request, Output Scanner catches dangerous patterns in your agent's output before they can cause harm.

This guard focuses on what comes out of the agent, not what goes in. It complements guards like Prompt Injection (which watches what the agent receives) by watching what the agent produces, forming a complete security perimeter around the conversation.

Why It Matters

Your users trust the code and technical guidance your agent provides. If your agent generates code containing security vulnerabilities, users may copy it directly into their projects without realizing the risk. This effectively turns your AI assistant into a distribution channel for insecure code.

This is especially important for agents that generate code, configuration files, or technical documentation. Even agents that primarily produce natural language can occasionally output code snippets that should be checked.

What Gets Caught

The guard flags AI-generated responses that contain potentially dangerous code patterns, including:

• Command injection — code that executes system commands with user-controlled input (e.g., child_process.exec(userInput), os.system(user_data))
• SQL injection — code patterns like '; DROP TABLE or unsanitized UNION SELECT queries
• Cross-site scripting (XSS) — <script> tags, javascript: URLs, event handler injection, or DOM manipulation with untrusted data
• Insecure code patterns — using Math.random() for security purposes, disabling TLS verification, hardcoded credentials in code
• Deserialization attacks — unsafe use of pickle.load(), yaml.load() without safe loaders
• Template injection — patterns that could allow server-side template injection

What won't trigger it: Code that discusses security concepts without containing exploitable patterns, properly sanitized code examples, or code using secure alternatives. If your agent is generating security-related educational content and hitting false positives, consider using warn mode so the content gets through while still being logged.

Example

Scenario

An agent is helping a user write a Python script and generates code containing os.system(user_input) — a command injection vulnerability where unsanitized user input is passed directly to a system command. Output Scanner detects the dangerous pattern before the response reaches the user. In Warn mode (the default), the response is delivered but flagged in the audit log with detection label command_injection. In Block mode, the response is suppressed entirely and the user sees a notice that the response was blocked for safety reasons.

Settings

Setting	What It Controls
Mode	Whether dangerous output is blocked, flagged for review, or logged silently
Vulnerability Scanning	Whether to scan responses for known dangerous code patterns

How It Responds

• Block -- The response is suppressed entirely. The user receives a notice that the response was blocked for safety reasons.
• Warn -- The response is flagged for your team to review, but it is delivered to the user.
• Log -- The detection is recorded silently for audit purposes.

If This Guard Blocks You

1. Check the audit log. Navigate to Security > Audit Log in the CrabShack dashboard. Filter by Output Scanner to find the relevant event. Click on the event row to expand it and see details including which skill and channel triggered it, the session key, and additional context.

2. Understand the detection. The detection label tells you what type of dangerous pattern was found — command_injection (code executing system commands with user input), sql_injection (unsanitized SQL queries), xss (cross-site scripting patterns), insecure_code (using weak randomness for security, disabling TLS verification), or deserialization (unsafe object loading). The guard catches patterns in the agent's generated code, not in user messages. If the agent is generating educational content about security vulnerabilities and the examples are triggering the guard, consider using Warn mode so the content is delivered while still being logged.

3. Adjust if needed. Switch between Block and Warn mode depending on your risk tolerance. If the agent frequently generates security-focused educational content, Warn mode lets the content through while keeping a record. Create a suppression for specific detection labels if needed. See the Configuration page for details.

tip

Prefer creating a targeted exemption for a specific skill or channel over disabling the guard entirely. If you're seeing frequent false positives, a suppression lets you silence a specific detection label temporarily while you refine your settings.

Default Behavior

The guard runs in warn mode by default, with vulnerability scanning enabled. This lets you see what your agent is producing without disrupting the user experience while you establish a baseline. During lockdown, it automatically switches to block mode to prevent any flagged output from reaching users.

Related Guards

• Prompt Injection -- The input-side complement. Output Scanner watches what the agent produces, while Prompt Injection watches what the agent receives.
• Credential Vault -- Provides deeper detection of credentials and secrets in both input and output, complementing Output Scanner's broader vulnerability coverage.