Monitor web changes with CrewAI

The mental model: a Tool the crew can call

An agent inside a crew is invoked on demand, so it fits the pull pattern. You expose two thin tools over Hypeline: one that starts watching a URL and returns a cursor, and one that fetches deduplicated changes since a cursor. The agent decides when to call them. A long-running crew can also subscribe to the stream, but the tool shape is the most natural way to hand a crew eyes on the web.

Wrap Hypeline as a custom Tool

# pip install hypeline crewai
from crewai.tools import BaseTool
from hypeline import Hypeline

hy = Hypeline("hype_...")

class WatchSource(BaseTool):
    name: str = "watch_source"
    description: str = "Start watching a URL for genuinely-new content. Returns a cursor."

    def _run(self, url: str) -> str:
        watch = hy.create_watch(url)
        return f"watching {url} cursor={watch.cursor}"

class GetChanges(BaseTool):
    name: str = "get_changes"
    description: str = "Fetch deduplicated changes since a cursor. Returns titles and URLs."

    def _run(self, cursor: str) -> str:
        page = hy.get_changes_since(cursor=cursor)
        lines = [f"{e.title}: {e.url}" for e in page.events]
        return "\n".join(lines) or "no new changes"

The hypeline client is a thin wrapper over the HTTP API: create_watch is POST /v1/sources and get_changes_since is GET /v1/events, so the same calls work from plain curl if you prefer.

Hand the tools to an agent

from crewai import Agent, Task, Crew
from tools import WatchSource, GetChanges

analyst = Agent(
    role="Competitive analyst",
    goal="Notice and summarize real changes on tracked pages",
    tools=[WatchSource(), GetChanges()],
    backstory="Watches rival pricing and changelog pages for the team.",
)

watch = Task(
    description="Watch https://competitor.example/pricing, then report any genuinely-new content.",
    agent=analyst,
    expected_output="A short note per real change, with the source URL.",
)

Crew(agents=[analyst], tasks=[watch]).kickoff()

Or trigger a crew from a signed webhook

For a crew that should run the moment something changes, deliver events by webhook instead of polling. You create an alert (POST /v1/alerts with a filter and the sources to watch), then add a webhook destination to it (POST /v1/alerts/{alert_id}/destinations with a body whose kind is webhook and url is your endpoint), which returns a whsec_ signing secret once. Point that destination at a small endpoint, verify the HMAC signature, then call kickoff with the event as input. This keeps the crew idle until there is real work.

Why Hypeline, not a page-change checker

• Deduplicated on extracted content. Fingerprinting the extracted main text filters template and timestamp churn, so a tool call returns real changes, not raw diffs.
• One schema across every source. Feeds, watched pages, and streaming sources return the same event shape, so a single tool covers them all.
• Cursor resume. The crew hands back the cursor it last saw, and the next call returns strictly newer changes, so nothing repeats.
• Signed events. Content-layer Ed25519 signatures and HMAC-signed webhooks let the crew trust the input it acts on.