Crawl Status

import { Supadata } from '@supadata/js';

const supadata = new Supadata({
  apiKey: 'YOUR_API_KEY',
});

// Start a crawl job
const crawl = await supadata.web.crawl({
  url: 'https://example.com',
  limit: 10,
});

// Get crawl job results
// This automatically handles pagination and returns all pages
const crawlResults = await supadata.web.getCrawlResults(crawl.jobId);
console.log(crawlResults);

{
  "status": "completed",
  "pages": [
    {
      "url": "https://supadata.ai",
      "content": "# Supadata\n## What is Supadata?\nSupadata is an API platform for data extraction for LLM training.",
      "countCharacters": 12300,
      "name": "Supadata: Web & YouTube to text API for developers",
      "description": "Supadata is one stop-shop API for developers to read web and YouTube content, ready for AI training and retrieval.",
      "ogUrl": "https://supadata.ai/opengraph-image.png"
    }
  ],
  "next": "https://api.supadata.ai/v1/crawl/123e4567-e89b-12d3-a456-426614174000?skip=1"
}

{
  "error": "invalid-request",
  "message": "Invalid Request",
  "details": "The request is invalid or malformed",
  "documentationUrl": "https://supadata.ai/documentation/errors#invalid-request"
}

{
  "error": "invalid-request",
  "message": "Invalid Request",
  "details": "The request is invalid or malformed",
  "documentationUrl": "https://supadata.ai/documentation/errors#invalid-request"
}

GET

web

crawl

{jobId}

import { Supadata } from '@supadata/js';

const supadata = new Supadata({
  apiKey: 'YOUR_API_KEY',
});

// Start a crawl job
const crawl = await supadata.web.crawl({
  url: 'https://example.com',
  limit: 10,
});

// Get crawl job results
// This automatically handles pagination and returns all pages
const crawlResults = await supadata.web.getCrawlResults(crawl.jobId);
console.log(crawlResults);

{
  "status": "completed",
  "pages": [
    {
      "url": "https://supadata.ai",
      "content": "# Supadata\n## What is Supadata?\nSupadata is an API platform for data extraction for LLM training.",
      "countCharacters": 12300,
      "name": "Supadata: Web & YouTube to text API for developers",
      "description": "Supadata is one stop-shop API for developers to read web and YouTube content, ready for AI training and retrieval.",
      "ogUrl": "https://supadata.ai/opengraph-image.png"
    }
  ],
  "next": "https://api.supadata.ai/v1/crawl/123e4567-e89b-12d3-a456-426614174000?skip=1"
}

{
  "error": "invalid-request",
  "message": "Invalid Request",
  "details": "The request is invalid or malformed",
  "documentationUrl": "https://supadata.ai/documentation/errors#invalid-request"
}

{
  "error": "invalid-request",
  "message": "Invalid Request",
  "details": "The request is invalid or malformed",
  "documentationUrl": "https://supadata.ai/documentation/errors#invalid-request"
}

Authorizations

x-api-key

string

header

required

Path Parameters

jobId

string

required

The crawl job ID

Query Parameters

skip

number

Required range: x >= 0

Response

Successfully retrieved crawl status

status

enum<string>

required

The status of the crawl job

Available options:

scraping,

completed,

failed,

cancelled

Example:

"completed"

pages

object[]

List of pages that were crawled (only present when status is completed)

Show child attributes

string

URL for fetching the next page of results

Example:

"https://api.supadata.ai/v1/crawl/123e4567-e89b-12d3-a456-426614174000?skip=1"

Crawl Get Account Information

Using the API

Universal Endpoints

YouTube Endpoints

Web Endpoints

Account Endpoints

Authorizations

Path Parameters

Query Parameters

Response