Keep your RAG pipeline current with live web changes

The mental model: continuous, deduplicated ingest

Instead of a nightly crawl, run one long-lived ingest worker that consumes the stream and upserts a document per real change. Each event is the extracted main content, already deduplicated, so you embed the item that changed and nothing else. Key every document by the event id and the upsert is idempotent: a re-delivered change replaces its chunk rather than adding a copy.

Ingest the stream into a vector store

# pip install hypeline
from hypeline import Hypeline
from your_vectorstore import embed, upsert   # any store: pgvector, Pinecone, etc.

hy = Hypeline("hype_...")

# Sources the pipeline should keep current.
for url in [
    "https://docs.example/changelog",
    "https://competitor.example/pricing",
    "https://regulator.example/filings",
]:
    hy.create_watch(url)

cursor = load_cursor()  # resume where the last run stopped

for event in hy.stream(since=cursor):
    upsert(
        id=event.id,                     # idempotent: replaces on re-delivery
        vector=embed(event.content),     # extracted main content, boilerplate removed
        metadata={
            "url": event.url,
            "title": event.title,
            "source_type": event.source_type,
            "emitted_at": event.emitted_at,
        },
    )
    save_cursor(event.id)                # durable resume point

The saved cursor is what makes this safe to run forever: if the worker restarts, it reconnects from the last event it stored, and the stream replays in order with no gap and no duplicate work. The hypeline client is a thin wrapper over the HTTP API: create_watch is POST /v1/sources and stream(since=...) opens the GET /v1/stream Server-Sent Events endpoint (the cursor rides the Last-Event-ID header), so you can run the same loop from plain curl.

Retrieval sees current content

Downstream retrieval is unchanged. Your generator queries the same vector store, but now the passages it retrieves reflect the latest genuine changes to each source, not a stale snapshot from the last crawl. Because template and timestamp churn never produced an event, the store stays lean and the nearest-neighbor results stay relevant.

Why Hypeline, not a page-change checker

• Deduplicated on extracted content. You embed real changes to the main text, so the index does not bloat with near-identical chunks.
• One schema across every source. Feeds, watched pages, and streaming sources arrive as one event shape, so a single ingest loop covers a mixed corpus.
• Cursor resume. A durable cursor turns an always-on ingest worker into a gap-free, exactly-resumable process.
• Signed events. Each event carries an Ed25519 signature, so you can verify a passage's provenance before it enters retrieval.