Getting Started

Quick Start

Get started with the crawler.dev API in just a few minutes. This guide will walk you through making your first API call to extract text from a document or webpage.

Prerequisites

Before you begin, make sure you have:

• ✅ A crawler.dev account and API key (Get one here)
• ✅ cURL, Postman, or your preferred HTTP client
• ✅ A document file or webpage URL to test with

Step 1: Get Your API Key

If you haven't already, sign up for an account and grab your API key from your dashboard. It should look like something like this:

Code
 
IzylAzLAK28EvhxSeF9xW9jPbHwNljqz2skz1Z44

Step 2: Extract Text from a File

Let's extract text from a document file. Save this as a cURL command:

Code
 
curl -X POST https://api.crawler.dev/v1/extract/file \
  -H "x-api-key: YOUR_API_KEY" \
  -F "file=@/path/to/your/document.pdf"

Response

Code
 
{
  "filename": "document.pdf",
  "contentType": "application/pdf",
  "sizeBytes": 1024,
  "text": "This is the extracted text from your document..."
}

Step 3: Extract Text from a URL

You can also extract text directly from web pages:

Code
 
curl -X POST https://api.crawler.dev/v1/extract/url \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "cleanText": true
  }'

Response

Code
 
{
  "url": "https://example.com/article",
  "finalUrl": "https://example.com/article",
  "statusCode": 200,
  "contentType": "text/html",
  "sizeBytes": 2048,
  "text": "This is the extracted text from the webpage..."
}

Code Examples

JavaScript/Node.js

Code
 
const FormData = require('form-data');
const fs = require('fs');
const fetch = require('node-fetch');

async function extractTextFromFile(filePath) {
  const form = new FormData();
  form.append('file', fs.createReadStream(filePath));

  const response = await fetch('https://api.crawler.dev/v1/extract/file', {
    method: 'POST',
    headers: {
      'x-api-key': 'YOUR_API_KEY',
      ...form.getHeaders()
    },
    body: form
  });

  const result = await response.json();
  console.log(result.text);
}

Python

Code
 
import requests

def extract_text_from_file(file_path):
    url = "https://api.crawler.dev/v1/extract/file"
    headers = {
        "x-api-key": "YOUR_API_KEY"
    }
    
    with open(file_path, 'rb') as file:
        files = {'file': file}
        response = requests.post(url, headers=headers, files=files)
    
    result = response.json()
    return result['text']

# Usage
text = extract_text_from_file("document.pdf")
print(text)

PHP

Code
 
<?php
$curl = curl_init();

curl_setopt_array($curl, array(
  CURLOPT_URL => 'https://api.crawler.dev/v1/extract/file',
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_POST => true,
  CURLOPT_HTTPHEADER => array(
    'x-api-key: YOUR_API_KEY'
  ),
  CURLOPT_POSTFIELDS => array(
    'file' => new CURLFile('/path/to/document.pdf')
  ),
));

$response = curl_exec($curl);
curl_close($curl);

$result = json_decode($response, true);
echo $result['text'];
?>

Testing with Postman

1. Create a new POST request to https://api.crawler.dev/v1/extract/file
2. Add x-api-key header: YOUR_API_KEY
3. Select Body → form-data
4. Add a key named file with type File
5. Upload your document and click Send

Supported File Types

Our API supports extracting text from:

Category	File Types
Documents	PDF, DOC, DOCX, RTF, TXT
Spreadsheets	XLS, XLSX, CSV
Presentations	PPT, PPTX
Web Content	Any
Other	ODT, ODS, ODP

Common Response Codes

Status Code	Meaning	Action
200	Success	Text extracted successfully
400	Bad Request	Check your request format
401	Unauthorized	Verify your API key
413	Payload Too Large	File size exceeds limit
422	Unprocessable Entity	Unsupported file type
429	Too Many Requests	Rate limit exceeded
500	Server Error	Try again or contact support

Rate Limits

Be aware of rate limits based on your plan:

• Free: 100 requests/hour
• Starter: 1,000 requests/hour
• Pro: 10,000 requests/hour

Next Steps

Now that you've made your first API call, here's what to explore next:

1. Browse the full API reference for all available endpoints
2. Check out examples for real-world integration patterns
3. Learn about error handling for robust applications
4. Explore advanced features like batch processing

Need Help?

• 📖 Full API docs: /api/v1
• 💬 Support: Contact us if you get stuck
• 🎯 Examples: See real-world examples

You're all set! Happy text extracting! 🚀