Ollama Guide: Install, Run, REST API, Modelfiles

Ollama is a free, MIT-licensed tool that downloads open models (Gemma, Llama, Qwen, DeepSeek, GPT-OSS) and serves them on your own machine behind a local REST API on port 11434, with OpenAI-compatible and Anthropic-compatible endpoints on top. Install it on macOS, Windows or Linux, run ollama run gemma4, and you have a model to build against from Python, JavaScript or curl. This guide covers install, the CLI, the API, Modelfiles, GPU support, and what changed since November 2024.

Updated September 2026: install paths, Docker ports, Modelfile format, CLI commands, LangChain imports, a what-changed section up to v0.33.3, and an FAQ. The original walkthrough is kept below, with a correction above each block the current docs contradict.

Going from a single command line to an application other people depend on is what we work through in Cohorte's AI Engineering Foundations course (E1).

What is Ollama?

Ollama is a framework designed to make working with large language models simple and intuitive. It is particularly suited for developers who want to experiment with natural language interfaces, build applications that involve LLMs, or create custom AI-powered tools. Ollama allows you to import, modify, and serve models while integrating easily with existing software systems.

Ollama stands out compared to other closed-source APIs due to its flexibility, ease of use, and open approach. Unlike many other solutions, Ollama allows you to host and manage models locally, providing greater control over data privacy and reducing dependence on third-party services. This is especially useful for organizations that prioritize data security or need to comply with specific regulations. Furthermore, Ollama's compatibility with OpenAI's API means that developers can easily switch from proprietary, closed-source platforms to a more customizable, self-hosted solution, while still leveraging the benefits of cutting-edge LLM technology.

Ollama offers:

1. Seamless Model Management: Tools to import and modify models.
2. API Interface: An API that simplifies interactions with LLMs.
3. Docker and Cross-Platform Support: Easy setup using Docker, with support for Linux, Windows, and macOS.
4. Data Privacy and Control: Host models locally to ensure data privacy and comply with organizational or regulatory requirements.
5. OpenAI Compatibility: Easily migrate existing applications that use OpenAI by switching endpoints, providing a cost-effective and open alternative.

Whether you are building chatbots, data extraction tools, or want to enable intelligent text generation, Ollama can help you scale and manage the LLMs you need.

Getting Started with Ollama

Quick Installation Guide

To get started with Ollama, you first need to install it. Ollama supports different environments, including macOS, Linux, Windows, and Docker.

1. Installation on macOS

To install Ollama on macOS, use the following command:

What changed since this was written: brew install ollama still works (formula at 0.33.2 when we checked), but the official path is the desktop app (Ollama.dmg from ollama.com/download, macOS 14 or later) or the same one-liner as Linux:

curl -fsSL https://ollama.com/install.sh | sh

brew install ollama

2. Installation on Linux

To install Ollama on Linux, you can follow these steps:

What changed since this was written: the URL below picked up stray angle brackets, and no apt step is documented. Manual install: a tarball (below), a ROCm tarball for AMD GPUs, and a systemd unit running /usr/bin/ollama serve:

curl -fsSL https://ollama.com/install.sh | sh

# manual install (x86-64)
curl -fsSL https://ollama.com/download/ollama-linux-amd64.tar.zst | sudo tar x -C /usr

• First, update your package index and install prerequisites:

sudo apt update && sudo apt install -y curl unzip

• Download and run the Ollama installation script:‍

curl -fsSL <https://ollama.com/install.sh> | sh

This script will install the latest version of Ollama on your system.

3. Installation on Windows

On Windows, you can install Ollama using the following command:

What changed since this was written: we could not verify the Chocolatey package today. The official install is OllamaSetup.exe from ollama.com/download/windows (Windows 10 or later) or one PowerShell line:

irm https://ollama.com/install.ps1 | iex

choco install ollama

Make sure you have Chocolatey installed on your system before running the command.

4. Running Ollama with Docker

Ollama also supports Docker, allowing you to run it in a containerized environment. To use Docker, first install Docker on your system, and then you can pull the Ollama Docker image:

What changed since this was written: the port is 11434, not 1144, and the image keeps models under /root/.ollama. From Docker Hub (ollama/ollama):

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

# NVIDIA GPU
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

# AMD GPU
docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm

# run a model inside the container
docker exec -it ollama ollama run llama3

docker pull ollama/ollama:latest

To run Ollama using Docker:

docker run -p 1144:1144 ollama/ollama:latest

This command will start Ollama and expose it on port 1144.

Installation and Setup Details

What changed since this was written: this subsection and the three after it describe LlamaIndex packaging, not Ollama. For Ollama, pip install ollama installs the official Python client (Python 3.8+) that talks to a running server; no OpenAI key, no Poetry. The server is configured with environment variables (OLLAMA_HOST, default 127.0.0.1:11434; OLLAMA_MODELS; OLLAMA_KEEP_ALIVE; OLLAMA_NUM_PARALLEL; OLLAMA_MAX_LOADED_MODELS; OLLAMA_CONTEXT_LENGTH); ollama serve --help lists them.

The LlamaIndex ecosystem is structured using a collection of namespaced packages, and Ollama follows a similar modular approach for flexibility. Ollama comes with a core starter bundle, and additional integrations can be installed as needed.

A complete list of packages and available integrations is available on LlamaHub.

Quickstart Installation from Pip

To get started quickly, you can install Ollama's core package with:

pip install ollama

This starter bundle includes essential packages for model management and interaction. Note that Ollama may download and store local files for various packages. Use the environment variable "OLLAMA_CACHE_DIR" to control where these files are saved.

Important: Environment Setup

By default, Ollama can integrate with several popular LLMs, including OpenAI models. In order to use OpenAI, you must have an OPENAI_API_KEY set up as an environment variable. You can obtain an API key by logging into your OpenAI account and creating a new API key.

You can also use other available LLMs. Depending on the LLM provider, additional environment keys and tokens might be required.

Installation from Source

If you prefer to install Ollama from source, you can follow these steps:

• Git clone the repository:

git clone https://github.com/ollama/ollama.git

• Install dependencies using Poetry, which helps manage package dependencies:‍

poetry shell
poetry install

• (Optional) Install all dependencies needed for local development:

poetry install --with dev, docs

From there, you can install additional integrations as needed with pip:

pip install -e ollama-integrations/llms/ollama-llms-openai

Importing and Managing Models

One of the most powerful features of Ollama is the ability to import and manage LLM models. You can easily import models from various formats, customize them, and use them in your projects.

Importing Models

To import a model, you need to use a Modelfile, which defines the configuration for importing a model. The Modelfile is similar to a Dockerfile, allowing you to specify various settings like model type, preprocessing, and tokenization.

Here is an example of a Modelfile for importing a model:

What changed since this was written: the block below is not Modelfile syntax. A Modelfile is a list of INSTRUCTION arguments lines: FROM (required), PARAMETER, TEMPLATE, SYSTEM, ADAPTER, LICENSE, MESSAGE, REQUIRES. From the current docs:

FROM llama3.2
PARAMETER temperature 1
PARAMETER num_ctx 4096
SYSTEM You are Mario from super mario bros, acting as an assistant.

model "my-custom-model" {
  framework = "pytorch"
  source = "https://huggingface.co/models/my-custom-model"
}

To import the model, you can run the following command:

What changed since this was written: there is no ollama import. Custom models are built with ollama create:

ollama create my-custom-model -f ./Modelfile
ollama run my-custom-model

ollama import -f Modelfile

Listing Available Models

To list all the available models on your system, use the following command:

ollama list

This command will display a list of all imported models, along with their details.

Using Models

After importing a model, you can start using it right away. You can send queries to the model using the query command:

What changed since this was written: there is no ollama query. ollama run <model> opens a session (/bye to leave) or runs once with a prompt argument. The CLI reference lists ls, pull, ps (loaded models), stop and rm:

ollama pull gemma4
ollama run gemma4 "What is the capital of France?"
ollama ls
ollama ps
ollama stop gemma4
ollama rm gemma4

ollama query my-custom-model "What is the capital of France?"

The model will respond with the answer: "Paris."

Tool Support

Ollama now supports tool calling with popular models such as Llama 3.1. This feature enables models to answer a given prompt using tools they know about, allowing them to perform more complex tasks or interact with the outside world.

Example Tools

• Functions and APIs: Use functions to retrieve real-time data, such as weather or financial information.
• Web Browsing: Search the web for answers that require current information.
• Code Interpreter: Execute code snippets to perform calculations or data processing tasks.

To enable tool calling, provide a list of available tools via the tools field in Ollama’s API:

What changed since this was written: the call below still works and llama3.1 remains in the library with tool support. Since then, the Python client accepts plain functions in tools, thinking models such as qwen3 take think=True, and calls are read from response.message.tool_calls.

import ollama

response = ollama.chat(
    model='llama3.1',
    messages=[{'role': 'user', 'content': 'What is the weather in Toronto?'}],
    tools=[{
      'type': 'function',
      'function': {
        'name': 'get_current_weather',
        'description': 'Get the current weather for a city',
        'parameters': {
          'type': 'object',
          'properties': {
            'city': {
              'type': 'string',
              'description': 'The name of the city',
            },
          },
          'required': ['city'],
        },
      },
    }],
)

print(response['message']['tool_calls'])

Supported models will now answer with a tool_calls response, enabling seamless integration of tools for complex queries.

Using the Ollama API

Ollama provides an API that allows developers to interact with models programmatically. The API is compatible with OpenAI's API, making it easier to integrate with existing applications that already use OpenAI models.

Example API Request

You can send a request to the Ollama API to generate text from a model:

What changed since this was written: the port is 11434. Native endpoints are /api/generate and /api/chat; the OpenAI surface is under /v1 (chat/completions, completions, models, embeddings, responses) and /v1/messages speaks the Anthropic Messages API. /api/embeddings is deprecated in favour of /api/embed:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Why is the sky blue?"
}'

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{ "role": "user", "content": "why is the sky blue?" }]
}'

curl -X POST "http://localhost:1144/v1/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-custom-model",
    "prompt": "Explain the process of photosynthesis."
  }'

The server will respond with the generated output.

Working with Docker

Running Ollama in Docker provides a portable way to use it in production environments. This makes it easy to deploy models as microservices that can scale with your application's needs.

You can use Docker Compose to create a full deployment setup for Ollama, including configuring persistent storage and exposing necessary ports.

Here is an example Docker Compose file:

What changed since this was written: port 11434 and the volume at /root/.ollama, as above:

services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
volumes:
  ollama_data:

version: '3'
services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "1144:1144"
    volumes:
      - ollama_data:/data
volumes:
  ollama_data:

To run the Docker Compose setup, use:

docker-compose up -d

Creating Custom Workflows with Ollama

Ollama also provides tools to build workflows involving multiple LLMs, allowing you to create more complex applications. For example, you can use one model for data extraction and another for summarization.

Here's a simple workflow example:

1. Data Extraction: Use a specialized LLM to extract relevant information from a document.
2. Summarization: Feed the extracted data into another model to generate a concise summary.

You can script this workflow using the Ollama command-line interface or the API.

OpenAI Compatibility

One of the benefits of Ollama is its compatibility with the OpenAI API, which means that applications built to work with OpenAI can easily be migrated to work with Ollama.

Simply replace the OpenAI endpoint with Ollama's endpoint, and you can start using your existing applications with your custom models without major modifications.

Getting Started Example: Using LangChain with Ollama in Python

Let's imagine we are studying the classics, such as the Odyssey by Homer. We might have a question about Neleus and his family. If you ask Llama2 for that information, you may get something like:

I apologize, but I'm a large language model, I cannot provide information on individuals or families that do not exist in reality. Neleus is not a real person or character, and therefore does not have a family or any other personal details. My apologies for any confusion. Is there anything else I can help you with?

This sounds like a typical censored response, but even Llama2-uncensored gives a mediocre answer:

Neleus was a legendary king of Pylos and the father of Nestor, one of the Argonauts. His mother was Clymene, a sea nymph, while his father was Neptune, the god of the sea.

So let's figure out how we can use LangChain with Ollama to ask our question to the actual document, the Odyssey by Homer, using Python.

Let's start by asking a simple question that we can get an answer to from the Llama2 model using Ollama. First, we need to install the LangChain package:

What changed since this was written: the integration moved to the langchain-ollama package (1.1.0, April 2026): ChatOllama, OllamaLLM, OllamaEmbeddings. Llama 2 is still listed but last updated two years ago; use llama3.1 or gemma4. Today's equivalent, which also replaces the embeddings import further down:

pip install langchain-ollama chromadb bs4

from langchain_ollama import ChatOllama, OllamaEmbeddings
llm = ChatOllama(model="llama3.1", temperature=0)
embeddings = OllamaEmbeddings(model="nomic-embed-text")
print(llm.invoke("why is the sky blue").content)

pip install langchain_community

Then we can create a model and ask the question:

from langchain_community.llms import Ollama
ollama = Ollama(
    base_url='http://localhost:11434',
    model="llama3"
)
print(ollama.invoke("why is the sky blue"))

Notice that we are defining the model and the base URL for Ollama.

Now let's load a document to ask questions against. I'll load up the Odyssey by Homer, which you can find at Project Gutenberg. We will need WebBaseLoader which is part of LangChain and loads text from any webpage. On my machine, I also needed to install bs4 to get that to work, so run pip install bs4.

from langchain.document_loaders import WebBaseLoader
loader = WebBaseLoader("https://www.gutenberg.org/files/1727/1727-h/1727-h.htm")
data = loader.load()

This file is pretty big. Just the preface is 3000 tokens. Which means the full document won't fit into the context for the model. So we need to split it up into smaller pieces.

from langchain.text_splitter import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)
all_splits = text_splitter.split_documents(data)

It's split up, but we have to find the relevant splits and then submit those to the model. We can do this by creating embeddings and storing them in a vector database. We can use Ollama directly to instantiate an embedding model. We will use ChromaDB in this example for a vector database. Run pip install chromadb. We also need to pull embedding model: ollama pull nomic-embed-text.

from langchain.embeddings import OllamaEmbeddings
from langchain.vectorstores import Chroma
oembed = OllamaEmbeddings(base_url="http://localhost:11434", model="nomic-embed-text")
vectorstore = Chroma.from_documents(documents=all_splits, embedding=oembed)

Now let's ask a question from the document. Who was Neleus, and who is in his family? Neleus is a character in the Odyssey, and the answer can be found in our text.

question = "Who is Neleus and who is in Neleus' family?"
docs = vectorstore.similarity_search(question)
len(docs)

This will output the number of matches for chunks of data similar to the search.

The next thing is to send the question and the relevant parts of the docs to the model to see if we can get a good answer. But we are stitching two parts of the process together, and that is called a chain. This means we need to define a chain:

from langchain.chains import RetrievalQA
qachain = RetrievalQA.from_chain_type(ollama, retriever=vectorstore.as_retriever())
res = qachain.invoke({"query": question})
print(res['result'])

The answer received from this chain was:

Neleus is a character in Homer's "Odyssey" and is mentioned in the context of Penelope's suitors. Neleus is the father of Chloris, who is married to Neleus and bears him several children, including Nestor, Chromius, Periclymenus, and Pero. Amphinomus, the son of Nisus, is also mentioned as a suitor of Penelope and is known for his good natural disposition and agreeable conversation.

It's not a perfect answer, as it implies Neleus married his daughter when actually Chloris "was the youngest daughter to Amphion son of Iasus and king of Minyan Orchomenus, and was Queen in Pylos".

I updated the chunk_overlap for the text splitter to 20 and tried again and got a much better answer:

Neleus is a character in Homer's epic poem "The Odyssey." He is the husband of Chloris, who is the youngest daughter of Amphion son of Iasus and king of Minyan Orchomenus. Neleus has several children with Chloris, including Nestor, Chromius, Periclymenus, and Pero.

And that is a much better answer.

Troubleshooting and Community Support

If you run into issues with Ollama, there is a comprehensive troubleshooting guide available, covering common problems and solutions. The community is also very active on platforms like Discord, making it easy to ask questions and get help.

To join the community or find more resources, visit:

• Ollama Discord Community
• Ollama GitHub Repository

What changed in Ollama since this was written

The original went out in November 2024. The latest release when we checked was v0.33.3 (2 September 2026), with several releases a month through the summer. The differences that matter:

• A desktop app on macOS and Windows. Ollama.dmg or OllamaSetup.exe; typing ollama alone opens an interactive menu. The quickstart model is gemma4, and the library leads with gemma4, qwen3.5, gpt-oss, deepseek-r1, qwen3-coder and llama3.x.
• Cloud models. After ollama signin, ollama run gpt-oss:120b-cloud or gemma4:cloud offloads to Ollama's service with the same API shape (hosted endpoint https://ollama.com/api). A web search API sits behind a free account and API key.
• Coding-agent integrations. ollama launch claude points Claude Code at a local model; VS Code, OpenCode, Codex and Droid are listed too. /v1/messages implements the Anthropic Messages API (ANTHROPIC_BASE_URL=http://localhost:11434, ANTHROPIC_AUTH_TOKEN=ollama).
• API capabilities. Thinking (think: true/false or low/medium/high/max; --think, --hidethinking on the CLI), structured outputs (format takes a JSON schema), /api/embed replacing /api/embeddings, /v1/responses. Default context window: 4096 tokens (num_ctx or OLLAMA_CONTEXT_LENGTH to raise it).
• Hardware. NVIDIA compute capability 5.0+ with driver 550 or newer, AMD via ROCm v7 on Linux and Windows, Apple Metal plus an MLX engine on Apple silicon, Vulkan on by default on Windows and Linux (OLLAMA_VULKAN=0 disables it).
• Modelfile. A REQUIRES instruction pins a minimum Ollama version; the documented default for repeat_penalty is now 1.0.

FAQ

What is Ollama used for?

Running open models on your own hardware: local chat, coding assistants (Claude Code, VS Code, Codex, OpenCode via ollama launch), RAG and embeddings, vision, and any app that would rather call a local HTTP API than a hosted one. Ollama states it does not see your prompts when you run locally.

Is Ollama free?

Yes. The server and CLI are open source under the MIT license, and local models cost nothing beyond hardware. Cloud models and the web search API need an Ollama account (ollama signin); check ollama.com for current plans.

Which operating systems does Ollama run on?

macOS 14 Sonoma or later, Windows 10 or later, Linux (script, tarball or systemd service) and Docker. Models live in ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux and C:\Users\%username%\.ollama\models on Windows.

How do I pick a model?

Start with the quickstart default, gemma4 (default tag e4b, 9.6 GB, 128K context, text and images), then size to your RAM or VRAM. The library tags models as tools, thinking, vision or embedding: qwen3-coder for code, deepseek-r1, qwen3 or gpt-oss for reasoning, nomic-embed-text or mxbai-embed-large for embeddings. Models too big for the machine carry a :cloud tag.

Does Ollama need a GPU?

No. It falls back to CPU, fine for small models and slow for large ones. Accelerated paths: NVIDIA compute capability 5.0+ (driver 550 or newer), AMD via ROCm v7 on Linux and Windows, Apple Metal and the MLX engine on Apple silicon, and Vulkan, on by default on Windows and Linux.

Ollama vs llama.cpp vs LM Studio?

llama.cpp is the C/C++ inference engine (MIT) with llama-cli and an OpenAI-compatible llama-server. Ollama wraps an engine with a model registry, CLI, Modelfiles, GPU detection and the REST API. LM Studio is a desktop GUI running llama.cpp and MLX behind OpenAI-like endpoints on Apple silicon Macs, Windows and Linux. Scripts and servers: Ollama. Full control over quantisation and flags: llama.cpp. A GUI for non-developers: LM Studio.

Final Thoughts

Ollama is a powerful tool for developers who want to integrate LLMs into their applications seamlessly. With support for Docker, easy model import, OpenAI compatibility, tool calling features, and an intuitive API, Ollama provides all the tools you need to get up and running with LLMs. Whether you're building chatbots, data extraction workflows, or deploying models as microservices, Ollama makes it easy to create context-augmented applications that take full advantage of language models.

Get started with Ollama today, and take your AI projects to the next level.

Charafeddine MouzouniNovember 13, 2024