Mechanism

How change detection works

"Alert me when this page really changes" sounds simple and is not: pages mutate on every load without saying anything new. Here is the actual mechanism Hypeline runs, stage by stage, to emit an event only when content is genuinely new.

01

Fetch only when something might have changed

Conditional requests (ETag and Last-Modified) let an unchanged page answer with an empty 304 before any content downloads. Pages that ship only a JavaScript shell are rendered in a real browser engine first, so single-page apps are compared on what a person actually sees.
02

Extract the content, drop the chrome

A readability-class extraction engine isolates the main text from navigation, ads, cookie banners, and template furniture. Everything downstream reasons over this extracted content, never the raw HTML. This single step is where most false alarms die.
03

Rule out the identical, instantly

An exact content hash catches the common case, a page that has not meaningfully changed, in constant time. No further work happens for it, and no event is emitted.
04

Tell edits from noise with fuzzy fingerprints

For content that did change, a simhash fingerprint (a compact 64-bit sketch of the text) is compared against the previous one. A reshuffled timestamp or swapped ad moves the fingerprint barely at all; genuinely new writing moves it past the threshold. Only the latter becomes an event.
05

Diff listings item by item

Index and listing pages change constantly by design. Instead of comparing the whole page, each item is fingerprinted separately, so one new entry in a list of fifty produces exactly one event, for the new entry.

The order matters: extraction before fingerprinting is the load-bearing decision. Fingerprints of raw HTML degrade into noise on modern pages; fingerprints of extracted content stay meaningful. Everything after that is engineering to make it cheap at thousands of sources.

FAQ

Frequently asked questions

Why not just hash the HTML and compare?

Because almost every page changes on every load: timestamps, session tokens, rotating ads, A/B variants. A raw checksum would fire constantly. Extracting the main content first, then comparing, is what separates a real change from page noise.

What is a simhash, in one paragraph?

A fingerprint where similar texts get similar fingerprints. Classic hashes change completely when one character changes; a simhash moves only a little. Measuring how far the new fingerprint sits from the old one (the Hamming distance) tells you whether the text was edited or merely re-dressed.

Can it miss a real change?

The thresholds are tuned so cosmetic churn stays silent while substantive edits pass. A one-character typo fix inside a long page sits near the boundary by design: the system optimizes for "new content someone should act on", not forensic diffing of every byte.

Does this work on JavaScript-rendered pages?

Yes. When a plain fetch returns an empty JavaScript shell, the page is escalated to a real browser engine, rendered, and then the same extract-and-fingerprint pipeline runs on the hydrated result.

What does the consumer actually receive?

One event per genuine change, with the source URL, a title, timestamps, and an Ed25519 signature, on the same versioned schema as events from feeds and push sources.