Skip to main content
GET
/
web
/
crawl
/
{jobId}
import { Supadata } from '@supadata/js';

const supadata = new Supadata({
  apiKey: 'YOUR_API_KEY',
});

// Start a crawl job
const crawl = await supadata.web.crawl({
  url: 'https://example.com',
  limit: 10,
});

// Get crawl job results
// This automatically handles pagination and returns all pages
const crawlResults = await supadata.web.getCrawlResults(crawl.jobId);
console.log(crawlResults);
{
  "status": "completed",
  "pages": [
    {
      "url": "https://supadata.ai",
      "content": "# Supadata\n## What is Supadata?\nSupadata is an API platform for data extraction for LLM training.",
      "countCharacters": 12300,
      "name": "Supadata: Web & YouTube to text API for developers",
      "description": "Supadata is one stop-shop API for developers to read web and YouTube content, ready for AI training and retrieval.",
      "ogUrl": "https://supadata.ai/opengraph-image.png"
    }
  ],
  "next": "https://api.supadata.ai/v1/crawl/123e4567-e89b-12d3-a456-426614174000?skip=1"
}

Authorizations

​
x-api-key
string
header
required

Path Parameters

​
jobId
string
required

The crawl job ID

Query Parameters

​
skip
number
Required range: x >= 0

Response

Successfully retrieved crawl status

​
status
enum<string>
required

The status of the crawl job

Available options:
scraping,
completed,
failed,
cancelled
Example:

"completed"

​
pages
object[]

List of pages that were crawled (only present when status is completed)

​
next
string

URL for fetching the next page of results

Example:

"https://api.supadata.ai/v1/crawl/123e4567-e89b-12d3-a456-426614174000?skip=1"