Introducing Octen Extract: A Fetch API That Tells You What the Page Is

AI agents read the web all day: pulling docs, working through issue threads, chasing a stack trace, checking a fact. But a surprising share of what they fetch isn't content at all: login walls, CAPTCHA interstitials, dead links, cookie banners. The agent can't tell until it has already read the page into context, so it swallows everything, and sometimes mistakes a login wall for "there's nothing here," then quietly heads off in the wrong direction.

Across customer conversations, the same ask kept coming back: not cleaner markdown, but knowing what a page is before spending anything on reading it.

Clean markdown, full page content, metadata parsing: Firecrawl, Jina, Exa, everyone has these. They're table stakes now. The whole field is competing on the same thing: returning the page's content in better shape.

But that's not where agents get stuck. They get stuck after the content arrives: can this page be trusted? Does it belong to my task? Is it worth the context window? Today, every one of those judgments lands on the model. You pay to read the whole page before you find out whether it was worth reading.

Extract moves that judgment to the source. Every page comes back with two labels attached: what kind of page it is, and what domain it belongs to. These calls come from the page-understanding engine that Octen's own search stack is built on: every page passes through it before anything reaches our index. Now they ship with your results.

And all of it (content, page structure, category, highlights) comes at one flat price: $1 per 1,000 pages.

Pricing in this market is all over the place: credits with their own exchange rates, a "basic" tier and an "advanced" one, structured output sold as a second subscription. What almost none of them do is fold the judgment — what is this page? — into the fetch itself, at the same single price. We build it in: nothing to convert, no tiers, no second bill.

Calling it takes one request:

curl https://api.octen.ai/extract \
  -X POST \
  -H 'X-Api-Key: octen-YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"urls": ["https://www.mayoclinic.org/diseases-conditions/diabetes/symptoms-causes/syc-20371444", "https://www.instagram.com/instagram/"]}'

And here's what comes back:

// response — fields match the live API; content truncated
[
  {
    "url": "https://www.mayoclinic.org/diseases-conditions/diabetes/symptoms-causes/syc-20371444",
    "page_structure": "Content Page / Article",
    "category": "Health / Health Conditions and Concerns",
    "full_content": "Diabetes mellitus refers to a group of diseases…"
  },
  {
    "url": "https://www.instagram.com/instagram/",
    "page_structure": "No Main Content / Operation Page",
    "category": "Computers, Electronics & Technology / Social Media Networks",
    "full_content": "Log into Instagram …"
  }
]

What's in the response

page_structure: what kind of page this is

Every URL comes back labeled with what it actually is: an article, a product listing, an index page, or one of the things that should never reach a model: login walls, verification interstitials, error pages, cookie banners.

Hard failures (4xx/5xx, timeouts) already announce themselves. The pages that cause real trouble are the ones that return a 200 with a login wall or CAPTCHA behind it: fake content pages. They're the ones most likely to fool an agent into concluding "this site has nothing," wasting a whole crawl. page_structure exists to call their bluff.

A few live examples: a pure login page (Instagram) comes back as No Main Content / Operation Page; a dead Medium post comes back as No Main Content / Other. Both are caught before they touch your context window. It's careful about the word "log in": an X profile plastered with sign-in prompts but containing actual posts is labeled Index Page / Personal Page, content returned in full.

Coding agents pay the highest price here. They spend all day fetching GitHub issues, library docs, Stack Overflow threads, blog posts pasted into error messages, and a good share of those land on login walls, archived 404s, verification pages. Without this label, all of it pours into context, burning tokens and a reasoning step. And with usage-billed fetch APIs, you paid to download the wall itself, too. Worse, and more quietly: an agent that reads a login wall as "no information here" will walk off in the wrong direction without ever raising an error, and that costs far more than a few wasted tokens.

Moving this filter down into the API layer means those pages never get in. The cost doesn't disappear. It just stops being yours.

category: what domain this page belongs to

Every response also carries a topical classification, drawn from a taxonomy of 160+ categories.

What domain a page belongs to is a signal that crawl-and-index pipelines rely on every day: what to keep, what to drop, which lane to route it down.

And this isn't a classifier we bolted on for the launch, nor a byproduct we skimmed off our index. It's the other way around: this is the page-understanding engine Octen runs across tens of billions of pages, and our search index is built downstream of its judgments: what gets indexed, and how, depends on them. Extract hands you that engine directly. Our own search is simply its first customer.

Research agents are the clearest case. To answer one question, they ingest a batch of mixed sources in one go: government reports, financial filings, news, corporate blogs, forum threads. With categories attached, that batch gets sorted before it ever reaches the model: official and regulatory sources are treated as primary facts and extracted aggressively; financial content goes down a finance-specific extraction path; corporate blogs are flagged as positioned material, held for balancing; forum posts count as leads, routed to secondary verification; restricted or sensitive domains stay out of the run entirely.

That routing used to cost an extra model call per page. Now the classification arrives with the response, and the triage is done before the URL gets anywhere near the model.

Categories let an agent decide how to handle a page before it reads it.

highlights: which passages matter

Pass a query alongside the URL, and Extract returns relevance-ranked passages instead of the full page. Three examples, each using one query to pin down one specific fact:

In all three, the highlights give exactly the answer the full page would, at a fraction of the tokens. These are representative examples, not a benchmark: they show what highlights look like on find-one-fact queries. For queries that synthesize across sections, or where the answer is scattered through the page, we still recommend taking the full content.

Page structure, category, and highlights all arrive in the same call: no extra round trip, no extra charge. And the highlights engine shares its retrieval stack with Octen Search, the same low-latency system.

Extract also reads PDFs natively, returning the same clean markdown with page structure and category. More document formats are on the way.

From returning content to returning understanding

Put the three signals side by side and they turn out to be one thing: what fetch hands back is no longer just the page's content: it's an understanding of the page.

What web pages look like has always tracked who's reading them. They were written for people first, all layout and styling. Then programs became the readers, and we got markdown and structured metadata: content reshaped into something machines could parse. Now the reader is an agent, and parseable content isn't enough. Before it acts, an agent needs to know: what kind of page is this, what domain does it belong to, which part matters.

Content says what the page wrote. Understanding says what the page is. Today's fetch APIs hand you clean content and stop there. The "what is this page" judgment is left to you. Octen finishes that step and returns it with the content. A judgment that used to be scattered across countless calls, redone every single time, now lives in the API layer.

For an agent, the ideal fetch was never "get a pile of text back, then figure it out." It's content that arrives already knowing what it is, whether it's worth reading, and where to look. That's what Extract does now.

Get started

Extract is generally available: $1 per 1,000 pages, everything included, up to 20 URLs per call, and only successful URLs are billed. Elsewhere, fetching the content and understanding it are billed as two separate things. Here, it's one bill.

Try Extract · Read the docs · MCP server on GitHub