Engine

@cuitty/scrape/engine exports runScrape(request, deps). It is the pipeline orchestrator behind both server.run() and the HTTP run manager.

Pipeline

The current pipeline is:

1. Plan the task with ScrapeAiBridge.plan().
2. Resolve auth references or hydrate a captured session.
3. Navigate the browser session to the target URL.
4. Bifurcate the page into Markdown, HTML, and annotated elements.
5. Detect blockers and raise interaction.required when needed.
6. Extract from Markdown, drive the actor loop, or do both for hybrid mode.
7. Emit a reusable artifact unless emit.artifact is false.

Calling runScrape Directly

Use the direct engine API for tests, workers, and embedding where you want full control over dependencies.

import { runScrape } from "@cuitty/scrape/engine";
import { createMockScrapeAiBridge } from "@cuitty/scrape/ai";
import { createMockBrowserSession } from "@cuitty/scrape/driver";
import { createMockCredentialProvider } from "@cuitty/scrape/auth";

const result = await runScrape(
  {
    url: "https://example.com",
    intent: "Read the page headline",
    mode: "content",
  },
  {
    bridge: createMockScrapeAiBridge(),
    session: createMockBrowserSession(),
    credentials: createMockCredentialProvider(),
    onEvent: (event) => console.log(event.type),
  },
);

console.log(result.status, result.path, result.answer);

Actor Loop

Interactive and hybrid runs use an observe -> act -> heal -> escalate loop. On each step the engine reuses the latest bifurcated view, asks the bridge for the next action, resolves elementId to selectors, performs click or fill, and retries once with bridge.heal() if the selector misses.

The shipped driver currently drives click and fill. Other action verbs are represented in types and recorded by the engine, but full browser implementations are tracked in the engine spec.

EngineDeps Interface

EngineDeps is intentionally explicit:

Field	Required	Purpose
bridge	Yes	AI planning, extraction, action selection, and healing.
session	Yes	Browser session abstraction.
credentials	Yes	Credential and session storage provider.
clock	No	Deterministic timestamp injection.
idgen	No	Deterministic ID generation.
onEvent	No	Event sink for run progress.
interaction	No	Human-in-the-loop resolver.
runId, jobId	No	Pre-assigned IDs from the server.

Step Budget

The actor loop reads request.budget?.maxSteps and defaults to 8. maxTokens and maxMs are part of the public request contract but are not enforced by the current engine implementation yet.

Bifurcation

session.bifurcate() returns the dual view. Content mode extracts from Markdown. Interactive mode uses annotated elements and selectors. Hybrid mode acts through the DOM and then extracts from a fresh Markdown view.