Making docker for llama3 RAG system

This document describes how to set up a Docker container for a Retrieval-Augmented Generation (RAG) system using Llama 3 and Ollama.

Here’s a breakdown of the steps:

Using Ollama Docker Image:

Ollama provides a Docker image (ollama/ollama:latest) that includes the necessary components for running Llama 3.

Verifying Ollama:

The guide outlines commands to check if Ollama is running in the container and how to adjust its address and port.

Creating a Dockerfile:

A sample Dockerfile is provided that defines multiple services:
ollama-faqs: This service runs a question-answering application using Ollama.
ollama: This is the core Ollama service.
ollama-webui: This service provides a web interface for interacting with Ollama.

This setup allows you to deploy a complete RAG system with Llama 3 and Ollama using Docker containers.

Ollama Docker image

Ollama docker-compose.yml

Verify if Ollama is running or not

Here’s the magic: execute the following command in your terminal:

text

$ docker ps aa492e7068d7   ollama/ollama:latest        "/bin/ollama serve"      9 seconds ago   Up 8
seconds   0.0.0.0:11434->11434/tcp   ollama

text

$ curl localhost: 11434Ollama is running

Add address and port

Ollama binds 127.0. 0.1 port 11434 by default. Change the bind address with the OLLAMA_HOST environment variable.

python

model = Ollama(model=MODEL , base_url='
http://ollama:11434')

Docker File

yaml

version: "3.9"services:
  ollama-faqs:
    image: ollama-app
    env_file:
      - .env
    networks:
      - ollama-network
    ports:
      - 8000:8080
    depends_on:
      - ollama
  ollama:
      image: ollama/ollama:latest
      ports:
        - 7869:11434
      volumes:
        - .:/code
        - ./ollama/ollama:/root/.ollama
      container_name: ollama
      pull_policy: always
      tty: true
      restart: always
      environment:
        - OLLAMA_KEEP_ALIVE=24h
        - OLLAMA_HOST=0.0.0.0
      networks:
        - ollama-network
  ollama-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: ollama-webui
    volumes:
      - ./ollama/ollama-webui:/app/backend/data
    depends_on:
      - ollama
    ports:
      - 8080:8080
    environment: # https://docs.openwebui.com/getting-started/env-configuration#default_models
      - OLLAMA_BASE_URLS=http://host.docker.internal:7869 #comma separated ollama hosts
      - ENV=dev
      - WEBUI_AUTH=False
      - WEBUI_NAME=valiantlynx AI
      - WEBUI_URL=http://localhost:8080
      - WEBUI_SECRET_KEY=t0p-s3cr3t
    extra_hosts:
      - host.docker.internal:host-gateway
    restart: unless-stopped
    networks:
      - ollama-networknetworks:
  ollama-network:
    external: true

Docker-compose with GPU

Nvidia GPU

https://hub.docker.com/r/ollama/ollama

Add GPU to docker

Installing with Apt

Configure the production repository:

bash

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey |
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  &&
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed
  's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \

sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Optionally, configure the repository to use experimental packages:

text

sed -i -e '/experimental/ s/^#//g' /etc/apt/sources.list.d/nvidia-container-toolkit.list

Update the packages list from the repository:

bash

sudo
apt-get update

Install the NVIDIA Container Toolkit packages:

bash

sudo
apt-get install -y nvidia-container-toolkit

Configuring Docker

Configure the container runtime by using the nvidia-ctk command:

bash

sudo nvidia-ctk runtime configure --runtime=docker

The nvidia-ctk command modifies the /etc/docker/daemon.json file on the host. The file is updated so that Docker can use the NVIDIA Container Runtime.
Restart the Docker daemon:

bash

sudo systemctl restart docker

gedit /etc/docker/daemon.json

json

{
  "registry-mirrors": [
    "https://registry.docker.ir"
  ],
  "runtimes": {
    "nvidia": {
      "args": [],
      "path": "nvidia-container-runtime"
    }
  }
}

Add to docker

yaml

version: "3.8"
  # Adjust version if neededservices:
  your-service:
    image: your-image:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
              count: 1
  # Adjust count as needed

Run Docker-compose

text

llama_new_context_with_model:        CPU  output buffer size =     0.00
MiBllama_new_context_with_model:      CUDA0 compute buffer size =    23.00
MiBllama_new_context_with_model:  CUDA_Host compute buffer size =     3.50 MiB

…

Visit us at DataDrivenInvestor.com

Subscribe to DDIntel here.

Making docker for llama3 RAG system