Skip to content
Home » Operating System » RUNNING OLLAMA ON UBUNTU 24.04 WITH NVIDIA GPU

RUNNING OLLAMA ON UBUNTU 24.04 WITH NVIDIA GPU

Here is a quick step by step.

Install Ubuntu 24.04 Desktop. At the time Ubuntu Server 24.04 has issues. So run this on Desktop

Login and open a terminal
sudo su –

apt-get update

apt-get upgrade

nvidia-smi
(this should show you your NVidia GPU. These drivers work well. Keep them)

apt install curl

curl -fsSL https://ollama.com/install.sh | sh

If it errors, run it a second time. It fixes the error.
curl -fsSL https://ollama.com/install.sh | sh

sudo systemctl enable ollama.service

sudo systemctl start ollama.service

ollama pull llama3:8b

ollama run llama3:8b

Open a new terminal
watch -n 0.5 nvidia-smi
You can see the GPU work

Back in the LLama2 terminal
ctrl +c
this will kill the session

Add Docker’s official GPG key by selecting below and pasting it into the terminal:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

echo \
“deb [arch=$(dpkg –print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo “$VERSION_CODENAME”) stable” | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

After the key installs:
apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

docker run -d –network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 –name open-webui –restart always ghcr.io/open-webui/open-webui:main

sudo docker ps

Now open a browser and go to localhost:8080

Signup and create an account. this will be the administrator

Back in the Terminal, setup other modes:
ollama pull codegemma

ollama pull llama2:70b

ollama pull llama3:70b

ollama pull llama3

ollama pull dolphin-llama3:8b

ollama pull dolphin-llama3:8b-256k

ollama pull dolphin-llama3:70b-256k

ollama pull dolphin-llama3:70b-256k

ollama pull dolphin-llama3:70b

ollama pull llava

Now go and have fun