Local inference servers

We strongly encourage and support running AI locally when it works for you. A local model keeps your conversations and prompts on hardware you control. It also gives you more ownership over your AI setup and lets you keep using it without relying on a cloud service.

There is a tradeoff. Local models are usually slower and less capable than the best cloud models. This is especially noticeable on older or less expensive hardware. The model you can run well depends on your available memory and graphics hardware. Start with reasonable expectations and try a few models before deciding that local AI is not for you.

Find a model

There is an incredible range of excellent open source models. Most are available on Hugging Face, including this list filtered to GGUF models. You can find a model for nearly any hardware budget, and many are well suited to casual roleplay.

We recommend GGUF quantized models for local use. They are widely supported and make models more practical to run on consumer hardware. Quantization is a way to reduce a model's memory needs with some tradeoffs in quality. We will cover it in more detail in a future guide.

Easy places to start

These options provide a friendly interface and are good starting points if you want to run a model without spending much time in a terminal.

KoboldCPP

KoboldCPP is a simple way to run GGUF models with a built-in web interface. It is popular with roleplay users and is a particularly approachable choice for trying local models.

Jan AI

Jan AI is a desktop app for downloading, chatting with, and running local models. Choose it if you prefer a polished desktop experience over a server-focused tool.

Unsloth Studio

Unsloth Studio provides a guided interface for working with local models. It is worth considering if you want an easier on-ramp and may later want to explore more of the local-model ecosystem.

Options for technical users

These options give you more control. They are a good fit if you are comfortable reading project documentation, adjusting settings, and troubleshooting your own setup.

llama.cpp

llama.cpp is a fast, flexible inference project with broad hardware support. It is an excellent foundation when you want direct control over model loading and a lightweight local server. Much of the local AI ecosystem is built on llama.cpp, so using it directly can give you the newest features and support for new models sooner.

The Museris developers use the llama.cpp server in router mode for their own application testing and love it. It is our preferred choice when you want maximum control and the quickest path to the latest llama.cpp capabilities.

Oobabooga Text Generation WebUI

Oobabooga Text Generation WebUI has an extensive interface and many configuration options. It suits users who want to experiment with different model formats, loaders, extensions, and generation settings.

Connect it to Museris

Choose a server that offers an OpenAI-compatible API, then add it as an OpenAI-compatible provider in a connection preset. Each project has its own setup guide and supported hardware, so follow its documentation for installation and model loading.

Keep a local server private to your home network unless you understand how to secure it. Do not expose its address to the public internet without authentication and other appropriate protections.