Monitor web changes with LlamaIndex

The mental model: ingest the stream

Point Hypeline at the sources your index should cover, then consume the stream as a continuous ingest. Each event is the extracted main content of a real change, deduplicated, so you insert one document per genuinely-new item instead of re-embedding an entire page on every crawl. Key each document by the event id and you get idempotent upserts for free.

Keep an index fresh from the stream

# pip install hypeline llama-index
from hypeline import Hypeline
from llama_index.core import VectorStoreIndex, Document

hy = Hypeline("hype_...")

# Sources whose content should stay retrievable and current.
for url in [
    "https://docs.example/changelog",
    "https://competitor.example/pricing",
    "https://status.example/",
]:
    hy.create_watch(url)

index = VectorStoreIndex.from_documents([])

# Ingest continuously. One document per real, deduplicated change.
for event in hy.stream():
    doc = Document(
        doc_id=event.id,                 # stable id => idempotent upsert
        text=event.content,              # extracted main content, boilerplate removed
        metadata={
            "url": event.url,
            "title": event.title,
            "source_type": event.source_type,
            "emitted_at": event.emitted_at,
        },
    )
    index.insert(doc)

Because Hypeline emits an event only when the extracted content really changed, the index never fills with near-duplicate chunks from timestamp or template churn. The doc_id makes a re-delivered event replace rather than duplicate its document. The hypeline client is a thin wrapper over the HTTP API: create_watch is POST /v1/sources and stream() opens the GET /v1/stream Server-Sent Events endpoint, so the same ingest works from plain curl.

Query the fresh index

engine = index.as_query_engine()
answer = engine.query("What changed in the competitor's pricing this week?")
print(answer)

Why Hypeline, not a page-change checker

• Deduplicated on extracted content. You ingest the extracted main text of real changes, so embeddings stay clean and your index does not bloat with boilerplate.
• One schema across every source. Feeds, watched pages, and streaming sources arrive as one event shape, so a single ingest loop covers a mixed corpus.
• Cursor resume. An ingest worker that restarts reconnects with its last cursor and backfills in order, so the index has no gaps.
• Signed events. Each event carries an Ed25519 signature, so you can verify a document's provenance before it enters retrieval.