Retrieval Test

Test what your agent would see for any query.

The Retrieval Test tool shows you exactly which memories and knowledge base excerpts would be included in an agent's context for a given message. This is the most direct way to understand what your agent "knows" and to verify that your settings are producing the right results.

How to Use It

1. Navigate to Memory or Knowledge Base in the CrabShack sidebar
2. Click the Test button (flask icon) in the top bar
3. Type a sample message — use a real question or statement that your users might send
4. Click Test

The results appear immediately, showing everything the agent would receive as context.

Reading the Results

Results are organized into three sections:

Always Injected

These are your Hot tier memories — they're included in every conversation regardless of topic. Each entry shows:

• The memory type (preference, fact, decision, etc.)
• A "pinned" badge indicating it's always present
• The memory content

If this section is empty, you have no Hot tier memories. If it's very large, you may want to reduce the hot tier limit in Configuration.

Relevant Memories

These are Warm and Cold tier memories that matched the query. Each entry shows:

• The memory type
• The tier (warm or cold)
• A relevance score — how closely the memory matches your query, shown as a percentage
• The memory content

Higher scores mean a stronger match. If important memories aren't appearing here, they may be in the Archive tier or the query may not be close enough to trigger them.

Knowledge

These are document excerpts from the Knowledge Base that matched the query. Each entry shows:

• The source filename — which document the excerpt came from
• The chunk number — which section of the document (e.g., "chunk 3 of 12")
• A relevance score — how closely the excerpt matches your query
• A content preview of the matched section

If this section is empty and you have documents uploaded, the query may not match any content, or the search threshold may be set too high.

Using It to Tune Settings

The Retrieval Test is your primary tool for adjusting memory and knowledge settings. Here's how to use it effectively:

What You See	What to Adjust
Too many irrelevant memories	Reduce the hot tier limit, or flag/archive noisy memories
Important memories missing	Check the memory tier — it may have been demoted. Promote it back to Hot or Warm
Too many irrelevant knowledge results	Raise the search precision in Knowledge Base settings
Important documents not appearing	Lower the search precision, or check that the document status is Ready
Knowledge excerpts are too short or lack context	Increase the adjacent sections setting
Results look good but agent still isn't using them	Check that injection is enabled for both Memory and Knowledge Base

tip

Test with real queries that your users actually send. Synthetic or generic test queries may not reflect how the system performs in practice. If you have access to past conversation logs, use actual user messages for the most realistic results.

When to Use It

• After changing settings — verify that your adjustments had the intended effect
• After uploading new documents — confirm they're being found for relevant queries
• After creating important memories — check that they appear when expected
• When debugging unexpected agent behavior — see if the agent is getting the right (or wrong) context
• Before going live — validate that the system surfaces the right information for your key use cases