Selenium AI Agent with LangGraph: Build a Web Scraper

A Selenium AI agent is a LangGraph graph whose nodes drive a real browser: one node navigates and pulls the page, one asks a model to turn the text into a typed record, one stores it, and a conditional edge decides whether to retry or stop. As of September 2026 that means langgraph 1.2.x (StateGraph, START, END), selenium 4.48 (Selenium Manager fetches the driver, so no more chromedriver path), and langchain.agents.create_agent when you want a tool-calling loop instead of a fixed graph. This guide gives the working 2026 version of the code, the architecture we use in production, the anti-bot and legal checks, and when Playwright or a browser-agent library is the better tool.

Updated September 2026: current LangGraph and Selenium versions and imports, a working StateGraph example, the four-node architecture, anti-bot and legal checks, Selenium vs Playwright vs browser agents, FAQ. The original walkthrough and code are kept below.

Scraping agents are fragile by default; the patterns that keep them auditable and reliable in production are exactly what we teach in Cohorte's Building Accountable AI Agents course (E3).

What LangGraph and Selenium each do

LangGraph is an agent framework designed to streamline the creation and orchestration of AI agents. It allows you to define modular "nodes" or tasks that your agent can execute. Each node can encapsulate a specific function—like data extraction, processing, or even decision-making. By integrating tools like Selenium, LangGraph agents can not only process data internally but also interact with the web dynamically. Selenium acts as a powerful browser automation tool, enabling your agent to navigate web pages, simulate user interactions, and extract relevant information in a smart and automated way.

Getting Started

Installation and Setup

Before diving into code, ensure you have Python installed and then install the necessary packages. Open your terminal and run:

pip install langgraph selenium

Additionally, download the appropriate WebDriver (e.g., ChromeDriver) for your browser and ensure it's accessible in your system’s PATH or specify its location directly in your code.

First Steps: Setting Up Selenium

Selenium requires a WebDriver to interact with your browser. Here’s a snippet to initialize a Selenium WebDriver:

What changed since this was written: you no longer download ChromeDriver by hand. Since Selenium 4.6, Selenium Manager ships inside the bindings and, when no driver is provided, it discovers, downloads and caches the right driver (Chrome via Chrome for Testing as of 4.11.0, Firefox as of 4.12.0, Edge as of 4.14.0). With selenium 4.48.0 (released August 27, 2026, Python 3.10+) the whole setup is:

from selenium import webdriver

def init_driver():
    options = webdriver.ChromeOptions()
    options.add_argument("--headless=new")
    # No executable_path: Selenium Manager (bundled since Selenium 4.6)
    # finds or downloads a matching driver the first time you run this.
    return webdriver.Chrome(options=options)

The original snippet with an explicit Service(executable_path=...) still works if you want to pin a driver yourself; it is just no longer required. Kept as written:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service

def init_driver():
    # Update the path to your chromedriver
    service = Service(executable_path='/path/to/chromedriver')
    driver = webdriver.Chrome(service=service)
    return driver

This simple function sets up Selenium, making it ready for use in your agent.

Step-by-Step Example: Building a Simple Agent

Let’s build a basic web-scraping agent using LangGraph and Selenium. The goal is to navigate to a given URL, extract the page title, and then return that information.

Step 1: Import Libraries and Initialize the Framework

First, import the necessary libraries and define your agent class based on LangGraph’s structure:

What changed since this was written: the documented LangGraph API is a StateGraph built from plain Python functions and a TypedDict state, compiled and then invoked. There is no Agent or Node base class to subclass in the current docs, and the class-based pattern in the original snippets below does not match them. The imports for langgraph 1.2.11 (released August 11, 2026) are:

from typing_extensions import TypedDict
from langgraph.graph import StateGraph, START, END
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

The original block is kept for reference:

from langgraph import Agent, Node
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
import time

# Initialize Selenium WebDriver
def init_driver():
    service = Service(executable_path='/path/to/chromedriver')  # Set your path
    driver = webdriver.Chrome(service=service)
    return driver

Step 2: Define a Scraping Node

Create a node that will be responsible for scraping data. In this example, the node will extract the page title:

class ScrapingNode(Node):
    def process(self, input_data):
        url = input_data.get("url", "https://example.com")
        driver = init_driver()
        driver.get(url)
        time.sleep(2)  # Wait for the page to load completely
        # Extract page title as a demonstration
        title = driver.title
        driver.quit()
        return {"title": title}

Step 3: Build the AI Agent

Now, define your AI agent by incorporating the scraping node. This agent can then orchestrate various tasks, but for simplicity, it will just handle the scraping process:

What changed since this was written: here is the same example as a compiled StateGraph, which is what runs against today's langgraph and selenium releases. The node is a function that receives the state and returns the keys it updates; the graph must be compiled before invoke. We also replaced the fixed time.sleep(2) with an explicit wait, which the Selenium docs recommend over sleeping (and warn not to mix with implicit waits).

class ScrapeState(TypedDict):
    url: str
    title: str

def init_driver():
    options = webdriver.ChromeOptions()
    options.add_argument("--headless=new")
    return webdriver.Chrome(options=options)

def scrape(state: ScrapeState):
    driver = init_driver()
    try:
        driver.get(state["url"])
        # explicit wait instead of time.sleep(2)
        WebDriverWait(driver, timeout=10).until(lambda d: d.title != "")
        return {"title": driver.title}
    finally:
        driver.quit()

builder = StateGraph(ScrapeState)
builder.add_node("scrape", scrape)
builder.add_edge(START, "scrape")
builder.add_edge("scrape", END)
graph = builder.compile()

if __name__ == "__main__":
    result = graph.invoke({"url": "https://example.com"})
    print("Page title:", result["title"])

The original class-based version follows, unchanged:

class WebScraperAgent(Agent):
    def __init__(self):
        super().__init__()
        self.add_node(ScrapingNode("scrape"))
    
    def run_agent(self, url):
        input_data = {"url": url}
        result = self.process_node("scrape", input_data)
        return result

if __name__ == "__main__":
    agent = WebScraperAgent()
    result = agent.run_agent("https://example.com")
    print("Page title:", result.get("title"))

This code creates an instance of WebScraperAgent that, when executed, uses Selenium to navigate to a specified URL, extract the page title, and print it out.

What changed in LangGraph and Selenium since this was written

This post was written in March 2025. Three things moved.

LangGraph is on the 1.x line. The current release is langgraph 1.2.11 (August 11, 2026). The graph primitives (state, nodes, edges) and the runtime are unchanged across 1.x, which is why the StateGraph example above runs as-is. What did change is where the prebuilt agent lives: create_react_agent in langgraph.prebuilt now carries a deprecation decorator whose message reads "create_react_agent has been moved to langchain.agents. Please update your import to from langchain.agents import create_agent." The LangGraph v1 release notes say the same: deprecated in favor of LangChain's create_agent, which "provides a simpler interface, and offers greater customization potential through the introduction of middleware". So for a scraping agent you have two shapes: a hand-built StateGraph (this post) or create_agent(model=..., tools=[...]) from langchain 1.3.18 (August 27, 2026) when the model should pick tools at run time.

Selenium manages its own drivers. selenium 4.48.0 (August 27, 2026) requires Python 3.10+. Selenium Manager has shipped with every release since 4.6 and runs as a fallback: if you provide no driver, it discovers, downloads and caches one that matches the installed browser. The "download ChromeDriver and put it on your PATH" step in the original guide is gone, which removes the most common cause of "this tutorial does not run".

The old snippets do not match the documented API. The class-based Agent / Node pattern and from langgraph import Agent, Node are not what the LangGraph docs describe; the documented way is from langgraph.graph import StateGraph, START, END, a TypedDict state, functions as nodes, compile(), then invoke(). The "What changed" notes above each original block give the replacement.

Architecture: the graph that scrapes safely

A scraping agent has four responsibilities: navigate, extract, structure, store. We keep them as separate steps in the graph but fold the first two into one node, because the WebDriver session must be opened and closed inside a node; graph state gets checkpointed and should hold only data (URL, text, record, error count), never a live browser. A conditional edge after the fetch decides between "go structure it", "retry with backoff" and "give up". The model appears in exactly one place, the structure node, where it turns page text into a validated Pydantic record. Everything else is deterministic Python, which is what makes the run reproducible and reviewable.

import time
from typing import Optional
from typing_extensions import TypedDict
from pydantic import BaseModel, Field
from langgraph.graph import StateGraph, START, END
from langchain.agents import create_agent
from langchain.agents.structured_output import ToolStrategy
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

class Product(BaseModel):
    name: str = Field(description="Product name as shown on the page")
    price: Optional[str] = Field(default=None, description="Price with currency, or null if absent")

class State(TypedDict):
    url: str
    text: str
    record: Optional[dict]
    error: Optional[str]
    attempts: int

def init_driver():
    options = webdriver.ChromeOptions()
    options.add_argument("--headless=new")
    return webdriver.Chrome(options=options)

def fetch(state: State):
    """navigate + extract: one node, because the driver must not live in state"""
    driver = init_driver()
    try:
        driver.get(state["url"])
        WebDriverWait(driver, timeout=10).until(
            lambda d: d.find_element(By.TAG_NAME, "main")
        )
        return {"text": driver.find_element(By.TAG_NAME, "main").text, "error": None}
    except Exception as exc:
        attempts = state.get("attempts", 0) + 1
        time.sleep(min(2 ** attempts, 30))   # backoff before the retry edge fires
        return {"text": "", "error": str(exc), "attempts": attempts}
    finally:
        driver.quit()

extractor = create_agent(
    model="provider:model-id",              # any chat model installed in your environment
    response_format=ToolStrategy(Product),  # validated Pydantic output
)

def structure(state: State):
    result = extractor.invoke({"messages": [
        {"role": "user",
         "content": "Extract the product from this page text:\n\n" + state["text"][:20000]}
    ]})
    return {"record": result["structured_response"].model_dump()}

def store(state: State):
    # replace with your database write; keep url, record, and a timestamp
    print(state["url"], state["record"])
    return {}

def after_fetch(state: State):
    if state.get("error") is None:
        return "structure"
    return "fetch" if state["attempts"] < 3 else END

builder = StateGraph(State)
builder.add_node("fetch", fetch)
builder.add_node("structure", structure)
builder.add_node("store", store)
builder.add_edge(START, "fetch")
builder.add_conditional_edges("fetch", after_fetch)
builder.add_edge("structure", "store")
builder.add_edge("store", END)
graph = builder.compile()

graph.invoke({"url": "https://example.com/product/1", "attempts": 0})

Why a fixed graph rather than letting an agent roam? Because a scraper that can invent its own next step is a scraper you cannot audit. With a graph, every run visits the same nodes in the same order, the retry policy is a function you can read, and the only non-deterministic step returns a schema-validated object. The failure modes are then honest ones: the site changed its markup (fetch fails the wait), the model could not fill the schema (validation error), or the target blocked you (see next section). Log the URL, the attempt count, the raw text length and the record on every run, and you have the trail an auditor will ask for. This separation of the model's job from the plumbing is the core of what we teach in the accountable-agents course linked above.

Anti-bot, rate limits and the legal check

Before the first request, do the boring checks, in this order.

• robots.txt and terms. Fetch /robots.txt and read the terms of service. If the path you want is disallowed or the terms forbid automated access, stop and ask for permission or an API. robots.txt is not a statute, but a scraper that ignores it is hard to defend in front of a client, a regulator or a court.
• Personal data. Names, emails, profiles and anything that identifies a person fall under GDPR the moment you store them, whatever the source. If the record can contain personal data, you need a lawful basis, a retention rule and a way to delete, before the first run.
• Identify yourself. Set a user agent that names your project and a contact address. Sites that would have blocked an anonymous bot often tolerate a polite, identifiable one.
• Rate and concurrency. One or two sessions, a pause between pages, and exponential backoff on HTTP 429 and 503 (the time.sleep(min(2 ** attempts, 30)) in the fetch node is the minimal version). Cache fetched pages so a bug in the structure node does not re-hit the site.
• Explicit waits, not sleeps. WebDriverWait(driver, timeout=10).until(...) waits for the condition you care about and fails fast when the page changed; a fixed sleep hides both. The Selenium docs also say not to mix implicit and explicit waits.
• Prefer the API. If the site exposes an API, a feed or a data export, use it. It is faster, more stable and unambiguous on permission.

What we will not tell you how to do: solve CAPTCHAs, rotate residential proxies to dodge a ban, spoof fingerprints or impersonate a logged-in user. A CAPTCHA or a block is the site saying no. An agent that is engineered to get past that answer is not "smart scraping", it is unauthorized access, and it is the first thing a security review or a procurement questionnaire will find. Build the agent so that a block ends the run with a logged error and a human decides what to do next.

Selenium vs Playwright vs browser agents in 2026

Three tool families now sit behind the query "web scraping agent", and they solve different problems.

Playwright as a library is two commands and a few lines:

pip install playwright
playwright install

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://playwright.dev")
    print(page.title())
    browser.close()

Swap the fetch node in the architecture above for that block and the rest of the graph is unchanged; LangGraph does not care which browser driver sits inside the node. That is the argument for keeping the browser behind a node boundary: you can change the driver without touching the agent.

browser-use is the other direction: instead of you writing the graph, an agent receives a task string and an LLM and decides what to click. Its quickstart is Agent(task="...", llm=...) then await agent.run(). It is the right tool for a one-off exploration or a task that spans many unknown sites, and the wrong tool for a nightly job that must hit the same 5,000 product pages and produce identical schemas, where a fixed graph is cheaper, faster and auditable. Our rule: fixed target and fixed schema, write the graph; unknown target, let a browser agent explore, then freeze what it found into a graph.

FAQ

Can an AI agent use Selenium?

Yes. Wrap the browser work in a function (navigate, wait, read the DOM) and register it as a LangGraph node, or as a tool for a create_agent loop. The model decides what to extract and what to do with it; Selenium does the driving. With selenium 4.48 and Selenium Manager there is no driver to install, and the driver object should be created and quit inside the node, never stored in graph state.

Selenium or Playwright for agents?

Both drive a real browser from Python. Playwright (1.62.0, Python 3.10+) installs its own browsers with playwright install and has a compact page API; Selenium (4.48.0, Python 3.10+) uses the browsers already on the machine, fetches drivers through Selenium Manager and has the larger legacy footprint, including Selenium Grid. For new agent code we default to Playwright unless the team already runs Selenium infrastructure.

How do I avoid getting blocked, legitimately?

Read robots.txt and the terms before the first request, identify your client honestly, keep one or two concurrent sessions, back off exponentially on 429 and 503 responses, cache what you already fetched and prefer the site's API or data export when one exists. Do not bypass CAPTCHAs, spoof residential IPs or impersonate a normal user to evade a block; a block is the site's answer, and an agent that ignores it is not one you can defend in an audit.

Does LangGraph replace LangChain agents?

No, they merged. Since LangGraph 1.0, create_react_agent in langgraph.prebuilt is deprecated with the message that it has moved to langchain.agents; the current factory is from langchain.agents import create_agent, which adds middleware and structured output. Use StateGraph when the pipeline is fixed (scraping usually is) and create_agent when the model should choose tools at run time.

How do I structure scraped data with an LLM?

Define a Pydantic model for the record, pass it as response_format=ToolStrategy(Model) to create_agent, and read result["structured_response"], which is validated against the model. Keep the raw page text next to the structured record so a reviewer can check the extraction, and make missing fields Optional rather than letting the model guess.

Is web scraping legal?

It depends on the jurisdiction, the site's terms, how you access it and what you collect. Publicly visible is not the same as free to take, and personal data triggers GDPR obligations regardless of how you obtained it. robots.txt is not a law, but ignoring it is hard to defend. For anything commercial, get counsel before you build; we are engineers, not lawyers.

Final Thoughts

Building an AI agent using a framework like LangGraph in tandem with Selenium provides a robust foundation for more complex automation tasks. While this guide covers a simple use case, consider expanding your agent with additional nodes that can:

• Process extracted data with natural language processing tools.
• Integrate with databases for storing and analyzing information.
• Handle dynamic web pages with advanced Selenium techniques.
• Incorporate error handling and logging for more robust operation.

By leveraging modular frameworks, you can quickly iterate and expand your agent’s capabilities, making it a versatile tool in your data collection and analysis toolkit. Remember, ethical considerations are paramount when scraping web data—always respect a website's terms of service and robot exclusion protocols.

This guide should serve as a stepping stone into the world of AI-driven web automation, offering a blend of practical code examples and conceptual understanding to help you build your own intelligent agents. Happy coding!

Tega AdeyemiMarch 28, 2025