You need
- Linux with Docker and Compose v2
- git and openssl
- About 3 GB free for models
- An NVIDIA GPU is optional
No GPU? Remove the deploy: block from the
ollama service in docker-compose.yaml.
Path 1
With the script
1Run it
curl -fsSL https://YOUR-SITE/install.sh | bash
It is plain bash. Read it first.
2Answer two questions
Access: Cloudflare, Tailscale, or local only.
Model server: Ollama, or vLLM.
Cloudflare asks for a tunnel token. Tailscale asks for an auth key.
Leave either blank and fill it into .env later.
3Open it
Go to http://127.0.0.1:3000. Create the first account.
It becomes the admin.
Then turn off sign-ups.
Path 2
Without the script
1Install Docker
Use Docker's official repository, not Snap or Docker Desktop:
docs.docker.com/engine/install.
Then add yourself to the docker group:
sudo usermod -aG docker $USER
Log out and back in so the group applies. Then check:
docker run --rm hello-world
docker compose version
NVIDIA GPU? Install the NVIDIA Container Toolkit too.
2Get the project
git clone https://github.com/razajahangir/inhouse-ai.git in-house-ai
cd in-house-ai
cp .env.example .env
3Make secrets
sed -i "s|^WEBUI_SECRET_KEY=.*|WEBUI_SECRET_KEY=$(openssl rand -hex 32)|" .env
sed -i "s|^POSTGRES_PASSWORD=.*|POSTGRES_PASSWORD=$(openssl rand -hex 32)|" .env
sed -i "s|^LITELLM_MASTER_KEY=.*|LITELLM_MASTER_KEY=sk-$(openssl rand -hex 32)|" .env
Linux sed. On macOS use sed -i ''. Or edit
.env by hand. Keys that need values:
WEBUI_SECRET_KEY, POSTGRES_PASSWORD,
LITELLM_MASTER_KEY (must start with sk-).
4Choose access
Set COMPOSE_PROFILES in .env. The template
ships with tailscale,cloudflare, so change it.
Changing it later? Run docker compose --profile "*" down
first, or the old tunnel keeps running. If you exported
COMPOSE_PROFILES in your shell, it overrides
.env: run unset COMPOSE_PROFILES.
Cloudflare Tunnel
COMPOSE_PROFILES=cloudflare
CLOUDFLARE_TUNNEL_TOKEN=eyJ...
- Cloudflare Zero Trust → Networks → Tunnels → Create a tunnel → Docker.
- Copy only the token (starts with
eyJ). Do not run its command. - Add a public hostname. Service: HTTP, URL
open-webui:8080. - Zero Trust → Access → Applications → add a policy for your email, so the login page isn't public.
Cloudflare can read this traffic at its edge.
Tailscale
COMPOSE_PROFILES=tailscale
TS_AUTHKEY=tskey-...
TS_HOSTNAME=in-house-ai
- Tailscale admin → Settings → Keys → generate an auth key.
- Admin → DNS → turn on MagicDNS and HTTPS certificates.
- Open
https://in-house-ai.<your-tailnet>.ts.net.
End to end encrypted. Only your tailnet can reach it.
Local only
COMPOSE_PROFILES=
Open http://127.0.0.1:3000 on this machine.
5Choose a model server
Ollama is the default. Nothing to change.
vLLM is for bigger NVIDIA GPUs. Add vllm to the profiles, set the model, and point LiteLLM at it:
COMPOSE_PROFILES=tailscale,vllm
VLLM_MODEL=Qwen/Qwen2.5-0.5B-Instruct
- model_name: local-chat
litellm_params:
model: hosted_vllm/local-chat
api_base: http://vllm:8000/v1
After editing litellm/config.yaml, run
docker compose restart litellm. Embeddings stay on
Ollama, so Ollama and vLLM share the GPU: keep the model small.
Gated Hugging Face models also need HF_TOKEN in
.env.
6Start it
docker compose up -d
docker compose logs -f ollama-init
Wait for the models to download. ollama-init then exits.
docker compose ps
Everything should be running, except ollama-init (exited 0).
7Open it
Create the first account at your address. It becomes the admin.
Then lock it down
In docker-compose.yaml, under open-webui,
set ENABLE_SIGNUP: "false". Apply it:
docker compose up -d
Only the admin adds users now: Admin Panel → Users. To try
document answers: Workspace → Knowledge → upload a file, then type
# in a chat.
Stuck? See troubleshooting in the README.