Monitor web changes with LangChain

Two ways in: a tool and a retriever

An on-demand agent pulls changes through a tool. A retrieval chain instead ingests the stream continuously so the vector store behind its retriever is always fresh. Both consume the same deduplicated, already-extracted events; pick by whether the agent should ask for changes or simply retrieve over current content.

Expose Hypeline as tools

# pip install hypeline langchain
from langchain_core.tools import tool
from hypeline import Hypeline

hy = Hypeline("hype_...")

@tool
def watch_source(url: str) -> str:
    """Start watching a URL for genuinely-new content. Returns a cursor."""
    watch = hy.create_watch(url)
    return watch.cursor

@tool
def get_changes(cursor: str) -> str:
    """Fetch deduplicated changes since a cursor."""
    page = hy.get_changes_since(cursor=cursor)
    return "\n".join(f"{e.title}: {e.url}" for e in page.events) or "no new changes"

The hypeline client is a thin wrapper over the HTTP API: create_watch is POST /v1/sources, get_changes_since is GET /v1/events, and stream() opens the GET /v1/stream Server-Sent Events endpoint. You can call those directly with curl if you prefer.

Keep a retriever fresh from the stream

Run a background ingest that upserts each real change into your vector store, keyed by the event id, then expose that store as a retriever like any other.

from langchain_core.documents import Document

def ingest(store):
    for event in hy.stream():
        store.add_documents([
            Document(
                page_content=event.content,   # extracted main content
                metadata={"url": event.url, "title": event.title, "id": event.id},
            )
        ], ids=[event.id])                    # id => idempotent upsert

retriever = store.as_retriever(search_kwargs={"k": 5})

Because events are deduplicated on the extracted content, the store never fills with near-identical chunks, and the id makes a re-delivered change replace its document instead of adding a duplicate.

Why Hypeline, not a page-change checker

• Deduplicated on extracted content. Tools and retrievers see real changes to the main text, not diffs full of template or timestamp churn.
• One schema across every source. Feeds, watched pages, and streaming sources arrive as one event shape, so one tool and one ingest loop cover them all.
• Cursor resume. A tool resumes from the cursor it last saw, and an ingest worker backfills in order after a restart, so nothing is missed or repeated.
• Signed events. Content-layer Ed25519 signatures let a chain verify provenance before acting on or retrieving a change.