Guides

Error Handling

The crawler.dev API uses standard HTTP status codes and returns detailed error information to help you build robust applications. This guide covers all error scenarios and how to handle them effectively.

HTTP Status Codes

Our API returns standard HTTP status codes to indicate the success or failure of requests:

Success Codes (2xx)

Code	Status	Description
200	OK	Request successful
201	Created	Resource created successfully

Client Error Codes (4xx)

Code	Status	Description
400	Bad Request	Invalid request format or missing required fields
401	Unauthorized	Invalid or missing API key
403	Forbidden	API key doesn't have required permissions
404	Not Found	Resource not found
413	Payload Too Large	File size exceeds maximum limit
415	Unsupported Media Type	File type not supported
422	Unprocessable Entity	Valid request but unable to process
429	Too Many Requests	Rate limit exceeded

Server Error Codes (5xx)

Code	Status	Description
500	Internal Server Error	Unexpected server error
502	Bad Gateway	Upstream service error
503	Service Unavailable	Service temporarily unavailable
504	Gateway Timeout	Request timeout

Error Response Format

All error responses follow a consistent JSON format defined in our OpenAPI specification:

Code
 
{
  "error": {
    "code": "STANDARDIZED_ERROR_CODE",
    "message": "Human-readable error message",
    "details": "Additional error details (optional)",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

Standardized Error Codes

The API uses standardized error codes for consistent error handling across all endpoints:

Error Code	Description	HTTP Status	Common Scenarios
NO_FILE	No file provided in request	400	File upload endpoint called without file
EMPTY_BUFFER	File content is empty	400	Empty file uploaded
UNSUPPORTED_CONTENT_TYPE	File type not supported	415	Unsupported file format uploaded
PAYLOAD_TOO_LARGE	File size exceeds limit	413	File larger than 50MB
INVALID_URL	Malformed or invalid URL	400	Invalid URL format provided
UNSUPPORTED_SCHEME	URL protocol not supported	400	Non-HTTP/HTTPS URLs
SSRF_BLOCKED	URL blocked for security	400	Internal/local network URLs
CONTENT_TOO_LARGE	Content size exceeds limit	413	Webpage content too large
UNSUPPORTED_CONTENT_TYPE	Content type not supported	415	Unsupported webpage format
FETCH_FAILED	Failed to fetch URL content	400	Network errors, blocked sites
TIMEOUT	Request timed out	408	Request processing timeout
EXTRACTOR_TIMEOUT	Text extraction timed out	408	Content processing timeout
EXTRACTOR_ERROR	Text extraction failed	422	Processing error during extraction
MISSING_API_KEY	No API key provided	401	Missing x-api-key header
INVALID_API_KEY	Invalid API key format	401	Malformed API key
QUOTA_EXHAUSTED	Rate limit exceeded	429	Too many requests
INTERNAL	Internal server error	500	Unexpected server issues
VALIDATION_ERROR	Request validation failed	400	Invalid request parameters
NOT_FOUND	Resource not found	404	Endpoint or resource missing

Common Error Scenarios

Authentication Errors

Missing API Key

Code
 
{
  "error": {
    "code": "MISSING_API_KEY",
    "message": "No x-api-key header provided",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

Invalid API Key

Code
 
{
  "error": {
    "code": "INVALID_API_KEY",
    "message": "Invalid API key format",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

API Key Not Found

Code
 
{
  "error": {
    "code": "INVALID_API_KEY",
    "message": "API key not found or has been disabled",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

File Upload Errors

No File Provided

Code
 
{
  "error": {
    "code": "NO_FILE",
    "message": "No file provided in the request",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

File Too Large

Code
 
{
  "error": {
    "code": "PAYLOAD_TOO_LARGE",
    "message": "File size exceeds maximum limit of 50MB",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

Unsupported File Type

Code
 
{
  "error": {
    "code": "UNSUPPORTED_CONTENT_TYPE",
    "message": "File type not supported for text extraction",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

Empty File

Code
 
{
  "error": {
    "code": "EMPTY_BUFFER",
    "message": "File content is empty",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

URL Processing Errors

Invalid URL

Code
 
{
  "error": {
    "code": "INVALID_URL",
    "message": "The provided URL is not valid",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

Unsupported URL Scheme

Code
 
{
  "error": {
    "code": "UNSUPPORTED_SCHEME",
    "message": "URL scheme not supported. Only HTTP and HTTPS are allowed",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

SSRF Blocked URL

Code
 
{
  "error": {
    "code": "SSRF_BLOCKED",
    "message": "URL blocked for security reasons",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

Fetch Failed

Code
 
{
  "error": {
    "code": "FETCH_FAILED",
    "message": "Failed to fetch content from the provided URL",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

Content Too Large

Code
 
{
  "error": {
    "code": "CONTENT_TOO_LARGE",
    "message": "Content size exceeds processing limits",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

Processing Errors

Extractor Timeout

Code
 
{
  "error": {
    "code": "EXTRACTOR_TIMEOUT",
    "message": "Text extraction timed out",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

Extractor Error

Code
 
{
  "error": {
    "code": "EXTRACTOR_ERROR",
    "message": "Failed to extract text from the provided content",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

Unsupported Content Type

Code
 
{
  "error": {
    "code": "UNSUPPORTED_CONTENT_TYPE",
    "message": "Content type not supported for text extraction",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

Rate Limiting Errors

Code
 
{
  "error": {
    "code": "QUOTA_EXHAUSTED",
    "message": "Rate limit exceeded. Try again later.",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

Validation Errors

Code
 
{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Request validation failed",
    "details": "Required field 'url' is missing",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

Error Handling Best Practices

1. Check HTTP Status Codes

Always check the HTTP status code before processing the response:

Code
 
const response = await fetch('https://api.crawler.dev/v1/extract/file', {
  method: 'POST',
  headers: {
    'x-api-key': apiKey
  },
  body: formData
});

if (!response.ok) {
  const error = await response.json();
  console.error('API Error:', error);
  
  // Handle specific error codes
  switch (error.error.code) {
    case 'MISSING_API_KEY':
    case 'INVALID_API_KEY':
      // Handle auth errors (redirect to login, refresh token, etc.)
      break;
    case 'QUOTA_EXHAUSTED':
      // Handle rate limiting (retry with backoff)
      break;
    case 'VALIDATION_ERROR':
      // Handle validation errors (show user-friendly message)
      break;
    case 'PAYLOAD_TOO_LARGE':
      // Handle file size errors
      break;
    case 'UNSUPPORTED_CONTENT_TYPE':
      // Handle unsupported file types
      break;
    default:
      // Handle unexpected errors
      break;
  }
  
  throw new Error(error.error.message);
}

2. Implement Retry Logic

For transient errors (5xx status codes), implement exponential backoff:

Code
 
async function extractTextWithRetry(requestData, maxRetries = 3) {
  let lastError;
  
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      const response = await makeAPIRequest(requestData);
      return response;
    } catch (error) {
      lastError = error;
      
      // Don't retry client errors (4xx)
      if (error.status >= 400 && error.status < 500) {
        throw error;
      }
      
      // Don't retry on last attempt
      if (attempt === maxRetries) {
        break;
      }
      
      // Exponential backoff: 1s, 2s, 4s
      const delay = Math.pow(2, attempt - 1) * 1000;
      console.log(`Retry attempt ${attempt}/${maxRetries} in ${delay}ms...`);
      await sleep(delay);
    }
  }
  
  throw lastError;
}

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

3. Handle Rate Limits

Respect rate limits and implement proper backoff:

Code
 
async function handleRateLimit(response) {
  const retryAfter = response.headers.get('Retry-After');
  
  if (retryAfter) {
    const waitTime = parseInt(retryAfter) * 1000; // Convert to milliseconds
    console.log(`Rate limited. Waiting ${waitTime}ms before retry...`);
    await sleep(waitTime);
  } else {
    // Default backoff if no Retry-After header
    await sleep(60000); // Wait 1 minute
  }
}

4. Validate Input Before API Calls

Validate files and URLs before sending to the API:

Code
 
function validateFile(file) {
  const maxSize = 50 * 1024 * 1024; // 50MB
  const supportedTypes = [
    'application/pdf',
    'application/msword',
    'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
    'text/plain'
  ];
  
  if (file.size > maxSize) {
    throw new Error(`File size (${file.size} bytes) exceeds maximum of ${maxSize} bytes`);
  }
  
  if (!supportedTypes.includes(file.type)) {
    throw new Error(`File type ${file.type} is not supported`);
  }
  
  if (file.size === 0) {
    throw new Error('File cannot be empty');
  }
}

function validateURL(url) {
  try {
    new URL(url);
  } catch {
    throw new Error('Invalid URL format');
  }
  
  if (!url.startsWith('http://') && !url.startsWith('https://')) {
    throw new Error('URL must use HTTP or HTTPS protocol');
  }
  
  // Check for common SSRF patterns
  const blockedPatterns = [
    /^https?:\/\/localhost/,
    /^https?:\/\/127\.0\.0\.1/,
    /^https?:\/\/10\./,
    /^https?:\/\/172\.(1[6-9]|2[0-9]|3[0-1])\./,
    /^https?:\/\/192\.168\./
  ];
  
  for (const pattern of blockedPatterns) {
    if (pattern.test(url)) {
      throw new Error('URL blocked for security reasons');
    }
  }
}

5. Graceful Error Messages

Show user-friendly error messages based on standardized error codes:

Code
 
function getUserFriendlyErrorMessage(error) {
  const errorMessages = {
    'MISSING_API_KEY': 'Authentication required. Please check your API key.',
    'INVALID_API_KEY': 'Invalid API key. Please verify your credentials.',
    'PAYLOAD_TOO_LARGE': 'File is too large. Maximum size is 50MB.',
    'UNSUPPORTED_CONTENT_TYPE': 'File type not supported. Please use PDF, DOC, DOCX, or TXT files.',
    'EMPTY_BUFFER': 'File cannot be empty. Please select a file with content.',
    'QUOTA_EXHAUSTED': 'Too many requests. Please wait before trying again.',
    'INVALID_URL': 'Invalid URL format. Please check the URL and try again.',
    'UNSUPPORTED_SCHEME': 'URL must use HTTP or HTTPS protocol.',
    'SSRF_BLOCKED': 'URL blocked for security reasons. Please use a public URL.',
    'FETCH_FAILED': 'Unable to access the provided URL. Please check the URL and try again.',
    'CONTENT_TOO_LARGE': 'Content is too large to process. Please try a smaller file or URL.',
    'EXTRACTOR_TIMEOUT': 'Processing timed out. Please try again with a smaller file.',
    'EXTRACTOR_ERROR': 'Failed to extract text. Please try a different file or URL.',
    'VALIDATION_ERROR': 'Invalid request. Please check your input and try again.',
    'NOT_FOUND': 'Resource not found. Please check the URL or endpoint.',
    'INTERNAL': 'Service temporarily unavailable. Please try again later.'
  };
  
  return errorMessages[error.error.code] || error.error.message || 'An unexpected error occurred.';
}

Monitoring and Debugging

Check Response Headers

Monitor these headers for additional information:

Code
 
const response = await fetch(apiUrl, options);

// Rate limit information
console.log('Rate Limit:', response.headers.get('X-RateLimit-Limit'));
console.log('Remaining:', response.headers.get('X-RateLimit-Remaining'));
console.log('Reset Time:', response.headers.get('X-RateLimit-Reset'));

// Request tracking
console.log('Request ID:', response.headers.get('X-Request-ID'));

Logging Errors

Log errors with context for debugging:

Code
 
function logError(error, context) {
  console.error('API Error:', {
    timestamp: new Date().toISOString(),
    errorCode: error.error.code,
    errorMessage: error.error.message,
    errorDetails: error.error.details,
    errorTimestamp: error.error.timestamp,
    context: context,
    requestId: error.requestId,
    userAgent: navigator.userAgent
  });
}

Error Recovery Strategies

Implement appropriate recovery strategies for different error codes:

Code
 
async function handleAPIError(error, originalRequest) {
  switch (error.error.code) {
    case 'MISSING_API_KEY':
    case 'INVALID_API_KEY':
      // Refresh token or redirect to login
      await refreshAuthToken();
      return retryRequest(originalRequest);
      
    case 'QUOTA_EXHAUSTED':
      // Wait and retry
      await handleRateLimit(error);
      return retryRequest(originalRequest);
      
    case 'EXTRACTOR_ERROR':
    case 'UNSUPPORTED_CONTENT_TYPE':
    case 'EMPTY_BUFFER':
      // Don't retry, show user error
      throw new UserError(getUserFriendlyErrorMessage(error));
      
    case 'INTERNAL':
    case 'EXTRACTOR_TIMEOUT':
      // Retry with backoff
      return retryWithBackoff(originalRequest);
      
    default:
      throw error;
  }
}

Testing Error Scenarios

Test your error handling with these scenarios:

1. Invalid API key: Use a fake API key to test INVALID_API_KEY errors
2. Large files: Test with files over 50MB to trigger PAYLOAD_TOO_LARGE errors
3. Empty files: Test with empty files to trigger EMPTY_BUFFER errors
4. Unsupported files: Test with image files to trigger UNSUPPORTED_CONTENT_TYPE errors
5. Invalid URLs: Test with malformed URLs to trigger INVALID_URL errors
6. Blocked URLs: Test with localhost URLs to trigger SSRF_BLOCKED errors
7. Network issues: Simulate network timeouts to trigger FETCH_FAILED errors

Error Code Reference

For quick reference, here are all the standardized error codes organized by category:

Authentication & Authorization

• MISSING_API_KEY - No API key provided
• INVALID_API_KEY - Invalid or disabled API key

File Processing

• NO_FILE - No file in request
• EMPTY_BUFFER - Empty file content
• UNSUPPORTED_CONTENT_TYPE - Unsupported file format
• PAYLOAD_TOO_LARGE - File too large

URL Processing

• INVALID_URL - Malformed URL
• UNSUPPORTED_SCHEME - Non-HTTP/HTTPS URL
• SSRF_BLOCKED - Security-blocked URL
• FETCH_FAILED - Failed to fetch content
• CONTENT_TOO_LARGE - Content too large

Processing

• EXTRACTOR_TIMEOUT - Text extraction timeout
• EXTRACTOR_ERROR - Text extraction failure
• UNSUPPORTED_CONTENT_TYPE - Unsupported content format

System

• QUOTA_EXHAUSTED - Rate limit exceeded
• INTERNAL - Internal server error
• VALIDATION_ERROR - Request validation failed
• NOT_FOUND - Resource not found
• TIMEOUT - General request timeout

Remember: Robust error handling using these standardized error codes is crucial for a good user experience and helps with debugging issues in production.