Skip to main content

Ollama (Local)

What It Does​

Ollama lets your CrabShack instance run AI models directly, on its own hardware — instead of sending every message to an outside company's servers. There's no per-message cost, and nothing you type ever leaves your box.

Why It Matters​

Local models are free to run and completely private, which makes them a good fit for privacy-sensitive work, predictable day-to-day load, or as a fallback that keeps working even if a hosted provider is slow or unavailable. The tradeoff: local models are generally less capable than top-tier hosted models like Claude or GPT, and they need enough spare memory (and ideally a GPU) on your instance to run well.

In Services, CrabShack can run Ollama for you in a managed container — no separate install needed.

  1. Open Services (under More in the sidebar) and find the Local Models section.
  2. Make sure Managed (on this appliance) is selected, then click Start Ollama. CrabShack starts the managed Ollama container and checks your instance's available memory and GPU.
  3. You'll see a model catalog filtered to what your instance can actually run — models that need more RAM than you have simply won't appear.
  4. Click a model to download it. A progress bar shows the download; CrabShack checks you have enough free disk space first and won't start a download that won't fit.
  5. Once downloaded, the model is available anywhere you pick a model — set it as a Primary or fallback model for any agent.

Hardware Guidance​

Every model in the catalog lists the minimum RAM it needs. As a rough guide:

ModelParametersMinimum RAMGood for
Qwen 2.5 0.5B0.5B1 GBUltra-light, fastest replies
Qwen 2.5 1.5B1.5B2 GBBalanced small model
Gemma 2 2B2B3 GBEfficient general-purpose
Qwen 2.5 3B3B4 GBGood quality, low resource use
Llama 3.2 3B3B4 GBCompact, general-purpose
Gemma 4 E4BE4B8 GBVision + reasoning, Google's latest edge model
Mistral 7B7B8 GBStrong general-purpose
Qwen 2.5 7B7B8 GBBest-in-class at this size
Llama 3.1 8B8B10 GBMeta's flagship small model
Gemma 4 12B12B12 GBStrong reasoning
Gemma 4 26B26B24 GBLargest, top-tier local quality
tip

If a model you want isn't showing up in the catalog, your instance's VM doesn't have enough memory allocated for it yet. Increase the VM's RAM in your hypervisor (see Getting Started) and it'll appear once there's headroom.

GPU Acceleration​

CrabShack automatically detects an AMD, NVIDIA, or Intel GPU on the host and configures Ollama to use it — no setup needed. If no GPU is found, or acceleration can't be confirmed, you'll see a warning that responses will run on the CPU only and be noticeably slower. This is a hardware limitation, not an error — local models still work, just more slowly.

Using a Remote (External) Ollama Server​

If you already run Ollama somewhere else on your network — a dedicated machine with a good GPU, for example — you don't need managed mode at all.

  1. In the Local Models section of Services, switch the mode from Managed (on this appliance) to External runtime.
  2. Enter that server's address (for example, http://192.168.1.50:11434/v1).
  3. Click Test to confirm CrabShack can reach it — you'll see the number of models it finds.

CrabShack checks any address you enter here for safety, blocking obviously unsafe internal addresses, but otherwise allows any real machine on your network or beyond.

Scenario

A workshop already has a spare PC with a strong GPU running Ollama for other purposes. Instead of running a model on the CrabShack VM itself, they switch to External mode and point CrabShack at that PC's address — getting faster local responses without adding load to the CrabShack instance.

If This Goes Wrong​

  1. No models show up in Managed mode — your instance's detected RAM may be too low for anything in the catalog. Increase the VM's memory allocation and refresh the page.
  2. GPU warning appears even though the host has a GPU — confirm your hypervisor is passing the GPU through to the VM, not just leaving it on the host machine.
  3. A download fails or stalls — check the storage indicator in the Ollama section; you may be low on disk space. Delete a model you no longer need, or increase the VM's disk size.
  4. External mode shows "Unreachable" — double-check the address, confirm Ollama is actually running there, and make sure nothing on your network is blocking the CrabShack VM from reaching it.