Skip to main content

How the Knowledge Base Works

How CrabShack processes, indexes, and retrieves your documents.

The Knowledge Base lets you give your agents access to your own documents — company policies, product guides, research papers, spreadsheets, and more. Once a document is uploaded, your agents can find and reference relevant information during conversations without you having to copy-paste content or point them to specific sections.

The Big Picture​

When you add a document to the Knowledge Base, CrabShack processes it through four stages:

  1. Text extraction — the document's content is read and converted to searchable text. For images and scanned PDFs, optional OCR (optical character recognition) extracts text from the image.

  2. Chunking — the document is broken into smaller, overlapping sections. This overlap ensures that important context isn't lost at section boundaries. Each chunk is a meaningful excerpt that an agent can understand on its own.

  3. Indexing — each chunk is indexed for semantic search, meaning the system understands the meaning of the content, not just the words used. This allows agents to find relevant information even when the question uses different phrasing than the document.

  4. Categorization — the document is automatically categorized (technical, business, legal, operations, research, or general) based on its content. This helps with organization and can improve search relevance.

How Agents Use Knowledge​

Once documents are processed, agents interact with the Knowledge Base in two ways:

Automatic Injection​

When auto-search is enabled (it is by default), CrabShack automatically checks the Knowledge Base for relevant information during conversations. If a topic matches your documents, the relevant excerpts are included in the agent's context — just like memories. The agent doesn't need to be told to search; it happens behind the scenes.

Agents can also search the Knowledge Base directly when they need specific information. This includes:

  • Standard search — finding the most relevant excerpts for a specific question
  • Deep research — a more comprehensive search that spans multiple documents, groups results by source, and returns a broader set of findings. This is especially useful for complex questions that require comparing or synthesizing information from several documents.
  • Document reading — accessing the full content of a specific document when excerpts aren't enough

What Happens During Processing​

After you upload a document, you'll see its status change:

  • Processing — the document is being read, chunked, and indexed. This usually takes a few seconds to a couple of minutes depending on file size.
  • Ready — the document is fully processed and available to your agents.
  • Error — something went wrong during processing. The error message will explain what happened — usually an unsupported format or a corrupted file.
note

You can continue using CrabShack normally while documents are processing. They'll become available to agents as soon as processing completes.

Storage and Limits​

The Knowledge Base has configurable limits for the number of documents and total storage size. By default, CrabShack allows up to 500 documents and automatically manages storage based on available disk space. See Configuration for details on adjusting these limits.