POST /crawl

POST

crawl

{
  "url": "https://example.com/docs",
  "limit": 500,
  "allowSubdomains": false,
  "crawlEntireDomain": true,
  "output": ["markdown"]
}

{
  "url": "https://example.com/docs",
  "limit": 500,
  "output": ["markdown"],
  "email": "you@example.com",
  "webhook": {
    "url": "https://your-server.example.com/hooks/crawl",
    "events": ["completed", "failed"]
  }
}

{
  "url": "https://example.com/docs",
  "limit": 200,
  "scrape": {
    "js_render": true,
    "proxy_strategy": "premium",
    "wait_selector": "main",
    "timeout": 45000,
    "tag": "docs-crawl",
    "output": ["markdown", "screenshot_full_page"]
  }
}

{
  "success": true,
  "id": "5f0d6c1e-3a52-4b6e-9c40-1c7f2b8a9d11",
  "url": "/crawl/5f0d6c1e-3a52-4b6e-9c40-1c7f2b8a9d11"
}

{
  "detail": "Invalid crawl request."
}

{
  "detail": "Crawling is temporarily unavailable. Retry shortly."
}

Start a durable, multi-page crawl beginning at url. The crawl runs as a background job — use the returned id to poll GET /crawl/{id} for status and results, or cancel it with DELETE /crawl/{id}. By default the crawl discovers URLs via sitemap first (same semantics as POST /map) and then follows links found on each page across the whole domain. Set crawlEntireDomain to false to stay below the starting URL’s path, and allowSubdomains to include subdomains.

Authorization

Send your pline API key in the X-API-Key header on every request.

Body

string

required

Starting URL for the crawl.

string

Address to notify by email once the crawl completes. Must be a valid email address.

string[]

Only crawl URLs whose path matches one of these regex patterns (not globs). For example, ["^/docs/"] keeps the crawl to paths starting with /docs/.

string[]

Skip URLs whose path matches one of these regex patterns.

integer

Maximum link-following depth from the starting URL. Unlimited when omitted.

any

default:"include"

Sitemap handling for initial URL discovery, same semantics as POST /map.

boolean

default:"false"

Whether URLs with query parameters should be treated as duplicates.

integer

default:"10000"

Maximum number of pages to crawl (max 10,000).

boolean

default:"true"

Follow links anywhere on the domain. Set to false to stay below the starting URL’s path.

boolean

default:"false"

Follow links on subdomains of the starting URL’s registrable domain.

string[]

Output formats to return for each crawled page, as a list. Defaults to markdown only when omitted. At least one format is required when you set it.

object

Optional webhook notified on crawl lifecycle events.

Show webhook properties

string

required

Webhook endpoint to notify. Must be HTTPS, point to a public host (no localhost or private IPs), and not contain credentials.

object

Extra headers to send with each webhook call.

any

Arbitrary JSON echoed back in every webhook payload.

string[]

Events to send. Empty means all events. One of started, page, completed, failed.

object

Options applied to every page the crawl scrapes (also accepted as scrapeOptions). Field names are snake_case, as on POST /scrape. Any other POST /scrape field, such as geolocation, actions, or only_main_content, is also accepted. url, method, and body are ignored, because the crawler sets them for each page.

Show scrape properties

string[]

Output formats for each page, as a list: markdown, html, clean_html, links, screenshot, or screenshot_full_page. Used only when the top-level output object isn’t set; if both are present, the top-level output wins. screenshot_full_page is available only here. If neither is set, pages come back as markdown only.

boolean

Whether JavaScript must run. true uses a browser from the first attempt (+2 credits). false forbids a browser: fastest and cheapest, but a page that needs one fails instead of escalating, and it can’t be combined with actions or screenshot output. Omit it to let pline choose automatically.

string

Proxy cost tier: auto, basic, or premium. Omit it, or use auto, to start cheap and escalate only as far as the site requires. basic uses only the cheapest tier and fails instead of escalating. premium uses a premium proxy from the first attempt (+7 credits); use it for heavily protected sites. See Credits and usage.

string | string[]

Labels applied to each page scrape, for filtering and tracing requests in Request history. Accepts one string or an array. Up to 25 labels, each 1–25 characters of letters, digits, ., _, or -.

integer

Per-page scrape timeout in milliseconds (1,000–60,000).

string

CSS selector to wait for before returning a browser-rendered page.

integer

Extra browser settle wait in milliseconds before returning (0–60,000).

Response

boolean

required

Whether the crawl job was started.

string

required

Crawl job ID. Use this to poll status (GET /crawl/{id}) or cancel (DELETE /crawl/{id}).

string

required

Relative path to poll for this crawl’s status.

Response codes

Code	Meaning
`202`	Crawl job started
`422`	Invalid crawl request.
`503`	Crawling is temporarily unavailable. Retry shortly.

{
  "url": "https://example.com/docs",
  "limit": 500,
  "allowSubdomains": false,
  "crawlEntireDomain": true,
  "output": ["markdown"]
}

{
  "url": "https://example.com/docs",
  "limit": 500,
  "output": ["markdown"],
  "email": "you@example.com",
  "webhook": {
    "url": "https://your-server.example.com/hooks/crawl",
    "events": ["completed", "failed"]
  }
}

{
  "url": "https://example.com/docs",
  "limit": 200,
  "scrape": {
    "js_render": true,
    "proxy_strategy": "premium",
    "wait_selector": "main",
    "timeout": 45000,
    "tag": "docs-crawl",
    "output": ["markdown", "screenshot_full_page"]
  }
}

{
  "success": true,
  "id": "5f0d6c1e-3a52-4b6e-9c40-1c7f2b8a9d11",
  "url": "/crawl/5f0d6c1e-3a52-4b6e-9c40-1c7f2b8a9d11"
}

{
  "detail": "Invalid crawl request."
}

{
  "detail": "Crawling is temporarily unavailable. Retry shortly."
}

{
  "url": "https://example.com/docs",
  "limit": 500,
  "allowSubdomains": false,
  "crawlEntireDomain": true,
  "output": ["markdown"]
}

{
  "url": "https://example.com/docs",
  "limit": 500,
  "output": ["markdown"],
  "email": "you@example.com",
  "webhook": {
    "url": "https://your-server.example.com/hooks/crawl",
    "events": ["completed", "failed"]
  }
}

{
  "url": "https://example.com/docs",
  "limit": 200,
  "scrape": {
    "js_render": true,
    "proxy_strategy": "premium",
    "wait_selector": "main",
    "timeout": 45000,
    "tag": "docs-crawl",
    "output": ["markdown", "screenshot_full_page"]
  }
}

{
  "success": true,
  "id": "5f0d6c1e-3a52-4b6e-9c40-1c7f2b8a9d11",
  "url": "/crawl/5f0d6c1e-3a52-4b6e-9c40-1c7f2b8a9d11"
}

{
  "detail": "Invalid crawl request."
}

{
  "detail": "Crawling is temporarily unavailable. Retry shortly."
}

Scrape

Batch scrape

Map

Crawl

SERP

Authorization

Body

Response

Response codes

​Authorization

​Body

​Response

​Response codes

Authorization

Body

Response

Response codes