Ollama Integrations: Open WebUI, LiteLLM, LangChain

Ollama serves an OpenAI-compatible API at http://localhost:11434/v1/, which is why it slots into a production stack without custom glue: Open WebUI as the front end, LiteLLM as the gateway that routes one OpenAI-style endpoint to local and cloud models, and LangChain through the langchain-ollama package. This guide walks through each integration, custom models with a Modelfile, and where Ollama stops and a dedicated serving engine such as vLLM begins.

Updated September 2026: verified against Ollama v0.33.3, Open WebUI v0.11.3, langchain-ollama 1.1.0 and the LiteLLM Ollama docs; "what changed" notes above every outdated code block, plus a FAQ. Original text and code kept below.

Stitching these integrations into something a real team can run on Monday is the focus of Cohorte's AI Engineering Foundations course (E1).

While Ollama’s core function is running LLMs locally, its value grows when you start to use it as part of a larger system. Whether you want a sleek web interface for your local models, or you wish to blend local and cloud AI services, Ollama can often be a central piece of the puzzle. Let’s explore these aspects step by step.

Exploring Ollama’s Advanced Features

Beyond the basic commands to pull and run models, Ollama offers advanced functionality that can significantly enhance your AI workflow:

• Versioning and Model Management: As mentioned earlier, Ollama lets you manage multiple model versions easily​. For advanced users, this means you can A/B test different model versions or maintain separate models for different tasks. For example, you might have a “llama2-chat” model for conversational tasks and a “llama2-code” model fine-tuned for coding assistance, both present on your system. Using ollama show <model> will display details about a model (such as its parameters, version, and installation info)​, which is useful for tracking and documentation.

• Custom Models and Fine-Tuning: Ollama provides a way to create new models derived from existing ones via ollama create <new_model>​. This command is used to fine-tune or modify a model. While full fine-tuning of large models may require additional tools and data preparation, Ollama’s design (built on llama.cpp) can allow lightweight fine-tuning or applying delta weights to models. Advanced users can integrate Ollama with fine-tuning libraries or use low-rank adaptation (LoRA) files to specialize a model for a particular domain, then load it through Ollama for serving. Essentially, Ollama can serve as a local model hub – not only running pre-trained models but also ones you’ve custom-built or fine-tuned.

• Enhanced Prompting and Memory: In interactive mode, models have a conversation history they maintain (up to a context length). Advanced usage might involve adjusting the context length or system prompts. Ollama allows configuring system-level instructions or persona for a model by editing its configuration. For instance, you could set a default system prompt like “You are an AI assistant expert in finance.” so that every session with that model starts with that context. This configuration can be done by modifying model config files (if supported) or via commands in the REPL (/system or similar, depending on implementation). Additionally, the context window (how much conversation history the model remembers) can sometimes be increased by using larger context models or setting environment variables (e.g., OLLAMA_CONTEXT_LENGTH) when starting ollama serve​. Advanced users who need long conversations or who want to feed long documents can look into models with 8K or 16K token context and configure Ollama accordingly.

• Structured Outputs: A very powerful advanced feature introduced in late 2024 is structured output support. Ollama allows you to constrain a model’s output to a specific JSON schema​. This means you can ask the model to output data in a structured format (like JSON) and Ollama will help enforce that structure. For developers, this is incredibly useful when you want the model’s output to be machine-readable (for example, getting a response as an object with specific fields). This is part of a broader trend (OpenAI calls it function calling / JSON output). With Ollama, you can define a schema and the model will attempt to conform to it, reducing the need to parse and validating the outputs.

• Tool Usage (Function Calling): Ollama has introduced tool calling support for models​. Certain models (like some LLaMA 3 variants) can be configured with “tools” – external functions the model can call to perform actions (e.g., do a web search, or calculation) as part of responding. With Ollama’s support, these function calls are recognized and handled, allowing the model to interact with the outside world in a controlled way. This is similar to giving the model plugins or abilities beyond text generation. For example, you could equip a model with a calculator tool; when asked a complex math question, the model can invoke the calculator via a special token sequence, and Ollama will compute the result and feed it back to the model. This advanced feature turns Ollama into a framework for building AI agents that do more than chat – they can execute tasks.

• Multimodal Models: As of late 2024, Ollama also started supporting certain multimodal models like Llama 3.2 Vision​. These models can handle not just text but images (e.g., describing an image). Running them requires additional setup (like providing image input as base64 or via a UI), but it means Ollama isn’t limited to text-only models. For advanced use, you could run an image-capable model to do tasks like captioning images or answering questions about a picture, all locally.

• Embedding Models for RAG: Another advanced use case is Retrieval-Augmented Generation (RAG), where you use a smaller embedding model to convert text into vector embeddings for similarity search. Ollama added support for embedding models (which return vector representations instead of text)​. This allows you to generate embeddings of your documents locally and build a semantic search or FAQ system on top. For example, you could use an Ollama-served embedding model to index a corpus of PDFs, then when a query comes, find relevant text by embeddings, and feed that into a larger LLM also hosted on Ollama. The fact that this can be done locally with open models is a big win for data-sensitive applications.

In summary, Ollama’s advanced features make it much more than a basic CLI for running models. It is evolving into a full-fledged local AI platform with support for custom models, structured outputs, tool use, and more. These capabilities are particularly useful for developers who want to build applications on top of Ollama, which brings us to using Ollama in production scenarios.

Using Ollama in Production

One common question is: can Ollama be used in a production environment? The answer is nuanced. Ollama is primarily designed for local development and experimentation, and the maintainers caution that it’s not originally intended for high-load production use (for example, the documentation notes that the API is not meant for heavy production usage)​. However, that doesn’t mean it can’t be part of a production workflow in the right circumstances. Many users have successfully deployed Ollama in controlled production scenarios, especially when serving a limited number of users or using it as an internal service.

Considerations for using Ollama in production:

• Single-User vs Multi-User: Ollama is ideal for single-user or low concurrency use cases​. If you plan to have just one model serving one request at a time (for example, an internal bot that only a few people query, or batch jobs one after another), Ollama can handle it well. In scenarios where thousands of users might hit the model concurrently (like a public API or a large-scale web app), you may face limitations. High throughput and parallel request handling are not Ollama’s strengths out of the box​.

• Resource Management: In production, you’d typically run ollama serve on a server with the model pre-loaded for faster responses. One thing to note is that by default Ollama may unload a model after some idle time or after a request (to free memory). For production, you’d likely want the model to stay in memory to serve subsequent requests quickly. This behavior can be configured. For instance, you might run ollama serve with environment variables that control unloading or keep the process alive. Some community discussions indicate ways to pin a model in memory or simply design your service to send a dummy prompt periodically to keep it loaded.
• Scaling: If you need to handle more load, you can run multiple instances of Ollama behind a load balancer. Since Ollama is relatively lightweight (compared to heavyweight servers), spinning up a few processes (each maybe handling one model or one request at a time) could be a strategy. For example, you could run 3 Docker containers each with ollama serve and have a simple round-robin dispatch for incoming requests. This isn’t as efficient as a purpose-built multi-threaded server, but it can work for moderate traffic.
• Stateless API: Ollama’s API (which we’ll discuss soon) is stateless in the sense that each request is independent (unless you build a conversation by resending history). This fits well with typical HTTP request/response patterns in production. You’ll need to design your application to send the full context on each request if you want continuity (like sending conversation history as part of the prompt). Alternatively, a production app might maintain a session and keep an open interactive Ollama session per user, but that’s more complex. Most likely you’ll use the stateless API calls.
• Deployment Environment: You can deploy Ollama on a powerful server or even in the cloud. Some users have run Ollama on cloud VMs with GPUs (for example, on AWS or Azure) to serve models with more horsepower. It’s also possible to containerize Ollama (there are Docker images available, or you can create one) so it can run in Kubernetes or similar platforms. For instance, one could deploy Ollama on Google Cloud Run or Azure Container Instances to serve a model in a serverless fashion​ (with some tinkering for GPU support, or using CPU with smaller models). The trade-off is that beyond a certain scale, using an enterprise-grade serving solution (like OpenLLM from BentoML or HuggingFace’s Text Generation Inference) might be more appropriate​. But for small to medium workloads, Ollama is certainly feasible.

• Realistic Use Cases: An example of a real production use of Ollama is integrating it into an internal company tool. Imagine a company has a knowledge base and they want an AI assistant to answer employee questions based on that data, but they don’t want to send data to an external service. They could use Ollama to host a moderately sized LLM internally. Employees ask questions via a chat interface; behind the scenes, the app queries Ollama’s API. Only employees on the network can access it, and the data stays internal. This is a production use in the sense that it’s deployed for real users, though it might not be serving millions of unknown users on the internet. Another scenario is using Ollama in production pipelines – say for content generation as part of a product workflow (e.g. generating draft text that will be reviewed by humans). Here, reliability and consistency matter, and the cost savings of not relying on external API are a big plus.

It’s worth noting that some users on forums have reported success using Ollama in production for their specific needs, often citing that if you already have the hardware, it’s a cost-effective solution​. The key is to understand the constraints: Ollama shines in controlled, perhaps smaller-scale environments, and might struggle or require extra engineering for high-scale cloud deployment. In those latter cases, you might treat Ollama as a stepping stone – it proves out your concept locally, and if you need to scale up dramatically, you could transition to a more scalable serving stack later (the skills and code you develop with Ollama will still transfer quite well).

In summary, using Ollama in production is possible and practical for certain cases, especially when data privacy is paramount and scale is moderate. You should run thorough tests, monitor performance, and be prepared to tune the setup (in terms of concurrency and memory) to ensure it meets your production requirements.

Integrations with Open WebUI, LiteLLM, and Other Frameworks

One of the great things about Ollama is that it can integrate with other tools to provide better user experiences or broader functionality. Let’s discuss a few notable integrations:

Open WebUI – A GUI for Your Local LLMs

Open WebUI (sometimes called Ollama WebUI) is an open-source web interface that works with local LLM backends like Ollama​. If you prefer chatting with your model through a browser or want a shareable interface for non-technical users, Open WebUI is ideal. Essentially, Open WebUI acts as a frontend, providing a ChatGPT-like chat experience, while Ollama runs in the backend serving the model.

How it works: Open WebUI runs as a web application (you can launch it via Docker very easily) and connects to the Ollama API. It supports multiple backends, but in our case we’d use the Ollama backend. Once set up, you access a local website (e.g., http://localhost:3000) where you can log in, select available models, and start chatting.

Setup: Assuming you have Docker installed, getting Open WebUI running is often a one-liner. For example, from a dev community guide, you can run:

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway \
    -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:main

This command pulls and starts the Open WebUI container, mapping it to port 3000​. The --add-host=host.docker.internal:host-gateway part ensures that the Docker container can communicate with Ollama running on the host machine (by resolving host.docker.internal to the host’s IP). Once Open WebUI is up, you typically create an account (just a local login for the interface) and then you can configure models.

Connecting to Ollama: Open WebUI needs to know how to talk to Ollama. Usually, in its settings, you’d specify the base URL of the Ollama API (by default http://host.docker.internal:11434 if using the above Docker setup on localhost). The nice thing is Open WebUI by design supports Ollama out of the box, so it may automatically detect the local service if networked correctly. In some configurations, you might set an environment variable or a config in docker-compose that points Open WebUI to use Ollama. For example, setting OPENAI_API_BASE_URL to the Ollama server and an OPENAI_API_KEY dummy value (since Ollama doesn’t need a key for local calls)​. This essentially tricks the WebUI to use Ollama as if it were the OpenAI API.

Once connected, any model you have pulled in Ollama becomes available in the web UI. Open WebUI even provides features like a model selection dropdown, prompt presets, and conversation saving. It’s a much richer experience for chatting compared to the raw terminal. You get things like message history in a nice format, the ability to edit or retry prompts, etc.

Use case: Imagine you have colleagues who aren’t comfortable with CLI. You can set up Open WebUI on a machine with Ollama and they can access a chat interface through their browser to use the models you’ve hosted. This could be on your local machine or a shared server. It’s a quick way to create a private ChatGPT alternative in your network, with a user-friendly interface.

Additionally, Open WebUI allows connecting to cloud APIs too (like an OpenAI API key), and you can switch between local and cloud models in one interface​. This integration of local & cloud gives flexibility: for some tasks you use your private local model, for others maybe a call to GPT-4, all from one place.

LiteLLM – Bridging APIs and Hybrid Workflows

LiteLLM is another tool that often comes up alongside Ollama and Open WebUI. LiteLLM acts as a proxy that provides an OpenAI-compatible API on one side and translates calls to various backends on the other (be it Ollama, Azure OpenAI, Amazon Bedrock, etc.)​. It’s essentially a middleware that can make different AI services look like the standard OpenAI API.

In practice, LiteLLM is useful if you want to integrate multiple AI providers in one interface. For example, in Open WebUI, if you want to use both your local Ollama model and a model deployed on Azure’s OpenAI service, you can’t directly call Azure OpenAI from Open WebUI because it expects the OpenAI API spec. LiteLLM can sit in between: Open WebUI sends a request to LiteLLM (which it thinks is “OpenAI”), and LiteLLM routes it to Azure’s endpoint (with the appropriate changes in endpoint URL, keys, etc.), then returns the result back in the standard format​. Similarly, LiteLLM can route to Ollama itself or other sources.

For hybrid usage, LiteLLM allows scenarios like: “If the user selects GPT-4 (which we have via Azure) in the UI, use that; if they select Llama-2 (which is local via Ollama), use that.” Both appear through a unified API. This is powerful in enterprise contexts where you might have some models in cloud and some on-prem, and want one interface to access all.

Integrating LiteLLM: In an Ollama + Open WebUI setup, you’d deploy LiteLLM (it can be a small server or container) configured with the routes to your desired endpoints. For instance, configure it so that requests with model name “azure/gpt4” go to Azure OpenAI, and requests with model name “ollama/llama2” go to your local Ollama. Open WebUI would then point to LiteLLM as its backend (instead of directly to Ollama). A lot of this can be orchestrated with Docker Compose: you’d have one service for Open WebUI, one for LiteLLM, and maybe one for Ollama, all networked together. In fact, a Docker Compose example from a blog shows how they linked Open WebUI and LiteLLM, setting OPENAI_API_BASE_URL to the LiteLLM service and an extra_hosts entry so that the WebUI container can find the Ollama host machine​.

The end result is a flexible AI stack: Open WebUI for UI, LiteLLM for intelligent routing, and Ollama (plus possibly others) for actual model serving​. This setup means users can switch between models (local or cloud) seamlessly. For example, you might primarily use the local model to save costs, but if it fails or if you need a second opinion, you switch to a cloud model via the same interface.

Other Frameworks and Integrations

Beyond Open WebUI and LiteLLM, Ollama can integrate with numerous other tools and frameworks:

• LangChain: LangChain is a popular framework for building applications with LLMs (chaining prompts, tools, memory, etc.). Ollama can be used with LangChain by leveraging its compatibility with the OpenAI API. Essentially, you can use LangChain’s OpenAI (or rather openai.ChatCompletion) integration and just point it to your local Ollama endpoint. For example, in Python:

What changed since this was written: the openai package moved to a client object in v1.0 (openai.api_base and openai.ChatCompletion.create are gone; responses are objects, not dicts), and Ollama's OpenAI-compatible endpoint lives under /v1 with an API key that is required but ignored. Current call:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")
response = client.chat.completions.create(
    model="llama3.1",
    messages=[{"role": "user", "content": "Hello, world!"}],
)
print(response.choices[0].message.content)

Original:

import openai
openai.api_base = "http://localhost:11434"  # Ollama's default API endpoint
openai.api_key = "ignored"  # Ollama doesn't require a key, but the client needs one set
response = openai.ChatCompletion.create(
    model="llama2",
    messages=[{"role": "user", "content": "Hello, world!"}]
)
print(response["choices"][0]["message"]["content"])

• With the above configuration, any OpenAI-compatible client (which LangChain uses under the hood) will send requests to Ollama instead of OpenAI’s servers​. This means you can integrate local models into complex chains, use LangChain’s memory and tools with a local LLM as the brain, etc. Many developers find this extremely useful for prototyping AI apps without incurring API costs.

• Custom UIs and Bots: If you’re building a custom application (say a Slack bot, or a VS Code extension), you can integrate Ollama via its API or client libraries. For instance, Raycast, a productivity tool on macOS, has a community plugin that uses Ollama to run an AI assistant from the launcher​. Similarly, the “Continue” VS Code extension (an open-source code assistant) supports Ollama as a backend​, allowing code completions and chat inside the IDE using local models.

• BentoML OpenLLM: BentoML’s OpenLLM is a framework for serving LLMs in production. Interestingly, one can consider using OpenLLM to deploy the same models that Ollama runs, but in a more production-oriented manner. There’s a bit of overlap in purpose, but if you developed locally with Ollama and later wanted to scale up, OpenLLM could be an integration of sorts (you’d package the model for OpenLLM). BentoML’s blog contrasts Ollama vs cloud serving, indicating that beyond single-user scenarios, a dedicated serving infra might be better​. However, BentoML also allows you to containerize an Ollama model easily. So one way to integrate is to use BentoML to build a Docker image containing an Ollama-served model and deploy that. This leverages the simplicity of Ollama with the scalability of cloud infrastructure.

• API and SDKs: Ollama provides its own Python and JavaScript SDKs for integration​. The Python SDK (installable via pip install ollama) allows you to call the local models with simple function calls (we’ll demonstrate that in a later article). If you’re a developer writing an app in Python, you might prefer using this SDK rather than making raw HTTP calls. It supports features like streaming responses and even structured outputs natively. The JavaScript SDK likewise can be used in Node.js apps to interact with Ollama’s service. These libraries make integration as easy as using OpenAI’s official libraries, but everything happens against your local Ollama server.

• Other GUIs: Apart from Open WebUI, there are other community UIs (like Text Generation Web UI, Oobabooga’s UI, etc.). Some of these can work with Ollama if configured appropriately (usually by treating Ollama as an “API endpoint”). For instance, Text Generation Web UI could call Ollama’s API via its API mode. Additionally, projects like KoboldAI or LoLLMS WebUI might integrate or at least allow usage of models that Ollama can host. The specifics vary, but the key idea is Ollama speaks a common language (OpenAI API and its own REST API), so any tool that can be configured to use a custom endpoint for an LLM can potentially use Ollama.
• Cloud Integrations: We mentioned Azure and Bedrock via LiteLLM. There’s also the scenario of using Ollama on cloud VMs or even as part of cloud functions. For example, someone integrated Ollama into an Azure Functions or AWS Lambda-like environment for serverless usage​ (though loading a big model on a cold start is challenging – this might work with smaller models or in a warmed container).

• Workflow Managers: Tools like LangFlow or Flowise (which provide a UI to build LangChain flows) could also indirectly use Ollama if you choose an OpenAI-compliant node and point it to Ollama. Similarly, orchestrators like Airflow could call Ollama as part of a pipeline (for generating text as one step of a data pipeline).

The bottom line is that Ollama is quite interoperable. Its design to have a standard API makes it a plug-and-play component in many systems. Whether you want a nicer interface (Open WebUI), a combination of AI sources (via LiteLLM), or integration into coding frameworks and applications (SDKs, LangChain), there’s likely a way to use Ollama for it. This versatility means you can start locally in your terminal, but eventually use the same local model in a web app or a custom tool without completely changing your setup.

Code Snippet: Using Ollama with a Web UI (Integration Example)

To illustrate one integration, here’s a small snippet of how you might use Open WebUI with Ollama using Docker Compose (a hypothetical example combining services):

What changed since this was written: Open WebUI reads the Ollama address from OLLAMA_BASE_URL, not OPENAI_API_BASE_URL (that one is for OpenAI-compatible endpoints such as a LiteLLM proxy). Current compose:

services:
  ollama:
    image: ollama/ollama:latest
    volumes:
      - ollama-data:/root/.ollama
  webui:
    image: ghcr.io/open-webui/open-webui:main
    depends_on:
      - ollama
    ports:
      - "3000:8080"
    environment:
      OLLAMA_BASE_URL: "http://ollama:11434"
    volumes:
      - open-webui:/app/backend/data
volumes:
  ollama-data:
  open-webui:

Original:

# docker-compose.yml
version: '3'
services:
  ollama:
    image: ollama/ollama:latest  # assume an Ollama docker image
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama  # persist models
    command: ollama serve

  webui:
    image: ghcr.io/open-webui/open-webui:main
    depends_on:
      - ollama
    ports:
      - "3000:8080"
    environment:
      OPENAI_API_BASE_URL: "http://ollama:11434"    # point WebUI to Ollama service
      OPENAI_API_KEY: "not_used_but_required"
      WEBUI_AUTH: "false"  # disable auth for simplicity
    extra_hosts:
      - "ollama:127.0.0.1"  # ensure the container can resolve the name (if needed)

In this configuration, we have two services: ollama and webui. The WebUI is configured to talk to the Ollama’s API. By running docker-compose up, you would get a local Ollama server and the Open WebUI all set up together. Then you can open http://localhost:3000 and use the interface, which will send your queries to the Ollama model. (In practice, ensure the network names/addresses are correct; this is a simplified example.)

Code Snippet: Hybrid API via LiteLLM (Integration Example)

Another snippet demonstrating LiteLLM configuration (pseudo-code) might look like:

What changed since this was written: the image and port. LiteLLM's quick start runs docker.litellm.ai/berriai/litellm:latest on port 4000 with the config passed as --config:

litellm:
  image: docker.litellm.ai/berriai/litellm:latest
  command: ["--config", "/app/config.yaml"]
  ports:
    - "4000:4000"
  volumes:
    - ./litellm_config.yaml:/app/config.yaml
  environment:
    LITELLM_MASTER_KEY: "sk-1234"

Original:

litellm:
  image: litellm/proxy:latest
  ports:
    - "8000:8000"
  volumes:
    - ./litellm_config.yml:/app/config.yml
  environment:
    LITELLM_CONFIG: "/app/config.yml"

And in litellm_config.yml, configure routes:

What changed since this was written: the providers/routes file below was pseudo-code, never LiteLLM's syntax. The real config is a model_list: each entry maps a public model_name to a litellm_params.model whose prefix picks the provider. For Ollama use ollama_chat/<model> (/api/chat, recommended by the LiteLLM docs) rather than ollama/<model> (/api/generate). A cloud model is another entry, so http://localhost:4000/v1/chat/completions serves both:

model_list:
  - model_name: llama3.1
    litellm_params:
      model: ollama_chat/llama3.1
      api_base: http://host.docker.internal:11434
      keep_alive: "8m"
    model_info:
      supports_function_calling: true
  - model_name: gpt-cloud
    litellm_params:
      model: openai/<your-openai-model>
      api_key: os.environ/OPENAI_API_KEY

Original:

providers:
  azure_openai:
    type: "azure_openai"
    api_base: "https://your-azure-endpoint.openai.azure.com/"
    api_version: "2023-05-15"
    api_key: "AZURE_API_KEY"
  ollama_local:
    type: "ollama"
    base_url: "http://host.docker.internal:11434"

routes:
  - path: "/v1/chat/completions"
    # If model name starts with "azure:" route to Azure, else to Ollama
    target: "azure_openai" if model.startswith("azure:") else "ollama_local"

With such a config, a request coming in to LiteLLM (at localhost:8000/v1/chat/completions) with model "azure:gpt-35-turbo" will be sent to Azure’s OpenAI, whereas model "llama2" will be sent to the local Ollama. Open WebUI or any client can simply use http://localhost:8000 as if it were the OpenAI API, and LiteLLM handles dispatching to the correct backend. This example is illustrative; actual LiteLLM config syntax might differ, but it conveys the concept of conditional routing.

Before this meets real users: The Agent Eval Playbook is the evaluation method we run on every agent before production. Free PDF.

What changed since this was written

As of September 2026 (the notes above each affected code block carry the new calls):

• Ollama v0.33.3 (September 2, 2026). The /v1 API covers chat completions, completions, models, embeddings and a stateless /v1/responses (added in v0.13.3). Context length is not settable through /v1: use PARAMETER num_ctx in a Modelfile or OLLAMA_CONTEXT_LENGTH on the server.
• Open WebUI v0.11.3 (August 31, 2026). Ollama address: OLLAMA_BASE_URL; OPENAI_API_BASE_URL is for OpenAI-compatible endpoints such as LiteLLM.
• LiteLLM: config is a model_list; Ollama models are ollama_chat/<model> (recommended) or ollama/<model>; proxy image docker.litellm.ai/berriai/litellm, port 4000.
• LangChain: langchain-ollama (from langchain_ollama import ChatOllama) replaces the old langchain.llms import and the OpenAI-wrapper trick.
• Ollama Python SDK: Client(host=...), chat(..., stream=True); no chat_stream.
• Modelfile: FROM, PARAMETER, TEMPLATE, SYSTEM, ADAPTER, LICENSE, MESSAGE, REQUIRES; FROM takes a model name, a GGUF file, or a Safetensors directory.

FAQ

Can Ollama serve an OpenAI-compatible API?

Yes. Point any OpenAI client at http://localhost:11434/v1/ with api_key="ollama" (required but ignored). Chat completions, completions, models, embeddings and a stateless Responses endpoint are covered; streaming, JSON mode, vision and tools work, logprobs, image URLs and tool_choice do not. Alias a local model with ollama cp for tools that expect a name like gpt-3.5-turbo.

How do I put Ollama behind LiteLLM?

Add a model_list entry whose litellm_params.model is ollama_chat/<model>, with api_base pointing at the Ollama host, start the proxy with --config, and call http://localhost:4000/v1/chat/completions with the public model_name. Cloud models are more entries in the same list.

Which LangChain package for Ollama now?

langchain-ollama (1.1.0, April 2026): pip install -qU langchain-ollama, then from langchain_ollama import ChatOllama. It supports bind_tools() and structured output, and as a standard LangChain chat model it is what you hand to LangGraph as well.

Can I run several models at once?

Yes. OLLAMA_MAX_LOADED_MODELS caps loaded models (default 3 per GPU, or 3 on CPU), OLLAMA_NUM_PARALLEL sets parallel requests per model (default 1; RAM scales with parallelism times context length), OLLAMA_MAX_QUEUE (default 512) bounds the queue, and OLLAMA_KEEP_ALIVE sets how long models stay in memory.

How do I create a custom model with a Modelfile?

Start with FROM llama3.2 (or a local GGUF file or Safetensors directory), add PARAMETER lines (temperature, num_ctx, top_p, stop, seed, num_predict) and a SYSTEM prompt, then run ollama create my-model -f ./Modelfile. This is also how you fix the context window for OpenAI-compatible clients, since /v1 has no num_ctx setting.

Ollama vs vLLM for serving?

Ollama for a workstation, an internal tool, or moderate concurrency where simplicity wins. vLLM (PagedAttention, continuous batching, prefix caching, tensor and pipeline parallelism, an OpenAI-compatible server via vllm serve <model> on port 8000) when many users share one GPU pool and throughput per dollar is the metric. Both speak the OpenAI API, so a LiteLLM gateway lets you switch without touching clients.

Real-World Case Studies and Examples

To ground our discussion, let’s look at a few real-world examples where Ollama and its integrations have been employed:

• Private ChatGPT for Company X: A mid-sized tech company wanted an internal chatbot trained on their documentation (policies, FAQs, etc.). They used Ollama to run a fine-tuned Llama model on their premises. To make it user-friendly, they set up Open WebUI as an interface and embedded it in an internal site. Employees could ask questions like “How do I reset my VPN password?” and get answers sourced from internal docs. The system integrated an embedding model for document retrieval (all running via Ollama). This allowed the company to have a 24/7 assistant without exposing data to external AI services, and with one-time hardware investment, no ongoing API costs.
• Academic Research Group: A university research group focusing on NLP integrated Ollama into their workflow for evaluating model behaviors. They would quickly spin up different models (Llama, Mistral, etc.) with Ollama to test on their datasets. For ease of use, they built a small web dashboard (using Streamlit) which allowed researchers to select a model and input some test queries. The backend of this dashboard simply forwarded the request to the local Ollama API and displayed the results. The researchers appreciated this setup since they could compare outputs from multiple models rapidly, and even the less technical members could use the web interface to interact with models.
• Hybrid Cloud Application (Minions project): In a more experimental vein, Stanford’s Hazy Research group developed a system nicknamed “Minions” where a local model works in tandem with a larger cloud model​. The local model (running via Ollama on a user’s device) would handle a good chunk of queries and only when needed, defer to a powerful cloud model (like GPT-4) for especially complex parts. This showcased an approach to reduce cloud usage by leveraging local models. The integration relied on Ollama for the local part and OpenAI API for the cloud part, orchestrated by custom logic deciding when to call which. This kind of case study highlights how Ollama can be part of multi-tier AI systems, optimizing for cost and speed by intelligent routing.

• Visual Studio Code AI Assistant: The “Continue” open-source project (AI assistant in VSCode) allows running entirely local by connecting to Ollama​. Developers using Continue in VSCode have reported being able to get code completions and chat assistance without any internet. In practice, a developer working on sensitive code (who cannot use cloud AI for privacy reasons) can still benefit from AI pair programming by using an Ollama-hosted code model (like Code Llama or PolyCoder). This integration is a real example of how tools developers already use can incorporate local AI seamlessly.

• Edge Devices & IoT: There’s interest in running models on edge devices (like the NVIDIA Jetson series). In one case, enthusiasts managed to use Ollama on a Jetson Orin (an edge GPU device) to run smaller LLMs for IoT applications, like a voice assistant that doesn’t send data to the cloud. They integrated it with voice recognition (speech-to-text), then passed the text to Ollama, got a response, and used text-to-speech to reply – achieving a fully offline voice assistant gadget. This might not yet be a widespread production scenario, but it’s a real example of integrating multiple components (speech pipeline + Ollama for NLU) to create a complex application.

Each of these examples underscores a different aspect of Ollama’s versatility: internal Q&A bots, research tooling, hybrid cloud-offloading strategies, IDE integration, and even edge deployment. The common theme is leveraging local models for privacy, cost, or offline reasons, and using Ollama’s integrations to make the experience more user-friendly or to combine with other systems.

One more notable case: Google’s Firebase GenAI Kit announced support for Ollama​. This means developers using Google’s tooling for AI app deployment could incorporate Ollama as a backend. It’s a strong signal that even major platforms see the value in local-first AI solutions for production apps, likely for scenarios where developers want an option to run open models.

Code Snippets for Integrations

We’ve sprinkled some code examples above, but let’s add a couple more brief snippets that illustrate integration in code:

Using the Ollama Python SDK (Advanced Integration):

For a Python application, instead of calling HTTP endpoints directly, you can use the official SDK:

What changed since this was written: the client takes host, not base_url, and the SDK has no chat_stream; streaming is chat(..., stream=True), per the current README:

# pip install ollama
from ollama import Client

client = Client(host="http://localhost:11434")
messages = [
    {"role": "system", "content": "You are a helpful travel assistant."},
    {"role": "user", "content": "Suggest a 1-week itinerary for Japan."},
]
for chunk in client.chat(model="llama3.1", messages=messages, stream=True):
    print(chunk["message"]["content"], end="", flush=True)

Original:

# Install via: pip install ollama
from ollama import Client

client = Client(base_url="http://localhost:11434")  # assume Ollama is running

# Example: streaming chat completion using a local model
messages = [
    {"role": "system", "content": "You are a helpful travel assistant."},
    {"role": "user", "content": "Suggest a 1-week itinerary for Japan."}
]
for response in client.chat_stream(model="llama2", messages=messages):
    # Each response is a chunk of the streaming answer
    chunk_text = response.message.content if response.message else ""
    print(chunk_text, end="", flush=True)

This uses Ollama’s Python client to send a chat prompt and stream back the result chunk by chunk (useful for showing incremental output in a UI)​. The SDK handles constructing the HTTP requests under the hood. Note that the first message defines a role (system) which sets context, showing that the API supports role-based messages similar to OpenAI’s ChatCompletion format​. This is very developer-friendly for advanced apps.

Integration with LangChain (pseudo-code):

What changed since this was written: from langchain.llms import OpenAI is the pre-split LangChain layout and the fake-key trick is no longer needed. The current integration is langchain-ollama (1.1.0 on PyPI, April 2026) with ChatOllama, which supports bind_tools() and structured output:

# pip install -qU langchain-ollama
from langchain_ollama import ChatOllama

llm = ChatOllama(model="llama3.1", temperature=0)
print(llm.invoke("Q: What is 5+7?\nA:").content)

Original:

from langchain.llms import OpenAI

# Point LangChain's OpenAI wrapper to Ollama
import os
os.environ["OPENAI_API_BASE"] = "http://localhost:11434"
os.environ["OPENAI_API_KEY"] = "something"  # dummy

llm = OpenAI(model_name="mistral")  # LangChain will call our local model
prompt = "Q: What is 5+7?\nA:"
result = llm(prompt)
print(result)

LangChain’s OpenAI class will internally use the openai package which reads the environment variables. In this way, LangChain thinks it’s talking to OpenAI but actually it’s querying Ollama. Advanced chains (with tools, memory) can be built on top of this llm object, enabling powerful workflows entirely with a local model.

Final Thoughts

Ollama’s advanced use cases and integrations demonstrate how a local LLM runner can be embedded into larger systems. From providing a user-friendly interface via Open WebUI to acting as a component in a sophisticated multi-provider setup with LiteLLM, Ollama proves to be highly adaptable. We discussed that while Ollama isn’t tailored for massive-scale production out-of-the-box, it certainly can be utilized in production-like environments where its strengths (privacy, control, cost saving) shine, as long as one is mindful of its limitations.

The integrative capability means you don’t have to use Ollama in isolation. If you need a feature it doesn’t have (like a UI or a certain cloud model), odds are you can combine it with another tool to get the best of both worlds. The growing ecosystem – including official SDKs, community UIs, and third-party frameworks – is turning Ollama into a cornerstone of local AI development.

In conclusion, advanced users can leverage Ollama to build real applications, not just experiments. Whether it’s an internal chatbot, a development assistant, or a component of a hybrid cloud solution, Ollama provides the local inference engine that powers it. In the next part of this series, we will focus specifically on using Ollama for AI model serving: how to set up Ollama as a persistent service, optimize it for performance, and apply it to serve AI models as a backend for applications.

Charafeddine MouzouniMarch 18, 2025