Scraping Products Instructions
Scraper API
Webhooks

Webhooks

Webhooks allow you to receive real-time notifications when scraping operations complete, fail, or start. Instead of polling for results, configure a webhook endpoint to receive instant updates about your scraping jobs.

Overview Permalink to Overview

Webhooks are HTTP callbacks that deliver event notifications to your configured URL. Use webhooks to:

  • Get notified when scrapes complete — Receive results automatically
  • Handle failures gracefully — React to errors in real-time
  • Integrate with chat platforms — Send notifications to Discord or Slack
  • Build event-driven workflows — Trigger downstream processes automatically

Supported Platforms Permalink to Supported Platforms

Platform Description
Discord Send rich embeds to Discord channels via webhook URLs
Slack Post messages to Slack channels using incoming webhooks
Custom Send JSON payloads to any HTTP endpoint with HMAC signature verification

Webhook Configuration Permalink to Webhook Configuration

Parameters Permalink to Parameters

Parameter Type Required Description
url string Yes Your webhook endpoint URL
type string Yes Webhook type: "discord", "slack", or "custom"
events array Yes List of events to subscribe to
secret string No Secret key for HMAC signature (custom webhooks only)

Available Events Permalink to Available Events

Event Description
started Job has been queued and started
completed Job finished successfully
failed Job encountered an error and failed

Usage Examples Permalink to Usage Examples

With Scrape Requests Permalink to With Scrape Requests

curl -X POST "https://scrape.evomi.com/api/v1/scraper/realtime" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "mode": "browser",
    "webhook": {
      "url": "https://your-server.com/webhook",
      "type": "custom",
      "events": ["completed", "failed"],
      "secret": "your-secret-key"
    }
  }'

With Crawl Requests Permalink to With Crawl Requests

curl -X POST "https://scrape.evomi.com/api/v1/scraper/crawl" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "example.com",
    "max_urls": 100,
    "webhook": {
      "url": "https://discord.com/api/webhooks/...",
      "type": "discord",
      "events": ["completed"]
    }
  }'

With Map Requests Permalink to With Map Requests

curl -X POST "https://scrape.evomi.com/api/v1/scraper/map" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "example.com",
    "sources": ["sitemap", "commoncrawl"],
    "webhook": {
      "url": "https://hooks.slack.com/services/...",
      "type": "slack",
      "events": ["completed", "failed"]
    }
  }'

With Search Requests Permalink to With Search Requests

curl -X POST "https://scrape.evomi.com/api/v1/scraper/search" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "e-commerce platforms",
    "max_urls": 20,
    "webhook": {
      "url": "https://your-server.com/webhook",
      "type": "custom",
      "events": ["completed"]
    }
  }'

With Schedules Permalink to With Schedules

curl -X POST "https://scrape.evomi.com/api/v1/account/schedule" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Daily Price Monitor",
    "config_id": "cfg_abc123",
    "interval_minutes": 1440,
    "webhook": {
      "url": "https://discord.com/api/webhooks/...",
      "type": "discord",
      "events": ["completed", "failed"]
    }
  }'

Platform-Specific Integration Permalink to Platform-Specific Integration

Discord Webhooks Permalink to Discord Webhooks

Discord webhooks post rich embeds to a channel. Get your webhook URL from channel settings:

  1. Go to Channel Settings → Integrations → Webhooks
  2. Click New Webhook and copy the URL
# Discord webhook example
"webhook": {
  "url": "https://discord.com/api/webhooks/123456789/abc123...",
  "type": "discord",
  "events": ["completed", "failed"]
}

Discord embeds include:

  • Job status with color coding (green for success, red for failure)
  • URL scraped
  • Credits used
  • Timestamp

Slack Webhooks Permalink to Slack Webhooks

Slack webhooks post messages using incoming webhook URLs. Create one in your Slack app:

  1. Create a Slack App or use an existing one
  2. Enable Incoming Webhooks
  3. Create a new webhook URL
# Slack webhook example
"webhook": {
  "url": "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXX",
  "type": "slack",
  "events": ["completed"]
}

Slack messages include:

  • Status emoji (✅ or ❌)
  • Job details in attachment format
  • Link to view results

Custom Webhooks Permalink to Custom Webhooks

Custom webhooks send JSON payloads to any HTTP endpoint. Use the secret parameter for HMAC signature verification:

# Custom webhook with HMAC signature
"webhook": {
  "url": "https://your-server.com/api/webhooks/evomi",
  "type": "custom",
  "events": ["started", "completed", "failed"],
  "secret": "your-webhook-secret"
}

Payload Structure Permalink to Payload Structure

Custom Webhook Payload Permalink to Custom Webhook Payload

When using custom webhooks, Evomi sends a POST request with the following JSON structure:

{
  "event": "completed",
  "timestamp": "2026-03-09T12:00:00Z",
  "data": {
    "task_id": "task_abc123",
    "task_type": "scrape",
    "url": "https://example.com",
    "status": "success",
    "credits_used": 5,
    "result": {
      "content": "...",
      "metadata": {}
    }
  }
}

Payload Fields Permalink to Payload Fields

Field Type Description
event string Event type: started, completed, or failed
timestamp string ISO 8601 timestamp of the event
data.task_id string Unique identifier for the task
data.task_type string Type: scrape, crawl, map, or search
data.url string URL that was scraped (for scrape tasks)
data.status string success or failed
data.credits_used number Credits consumed by the operation
data.result object Scraping results (for completed events)
data.error string Error message (for failed events)

HMAC Signature Verification Permalink to HMAC Signature Verification

For custom webhooks, verify the request authenticity using HMAC-SHA256 signatures.

How It Works Permalink to How It Works

  1. Evomi creates a signature using your secret and the request body
  2. The signature is sent in the X-Evomi-Signature header
  3. Your server verifies the signature before processing

Verification Example (Node.js) Permalink to Verification Example (Node.js)

const crypto = require('crypto');

function verifyWebhook(req, secret) {
  const signature = req.headers['x-evomi-signature'];
  const payload = JSON.stringify(req.body);
  
  const expectedSignature = 'sha256=' + 
    crypto.createHmac('sha256', secret)
      .update(payload)
      .digest('hex');
  
  return signature === expectedSignature;
}

// Usage
app.post('/webhooks/evomi', (req, res) => {
  if (!verifyWebhook(req, 'your-webhook-secret')) {
    return res.status(401).send('Invalid signature');
  }
  
  // Process webhook
  console.log('Event:', req.body.event);
  res.status(200).send('OK');
});

Verification Example (Python) Permalink to Verification Example (Python)

import hmac
import hashlib
import json

def verify_webhook(headers, body, secret):
    signature = headers.get('X-Evomi-Signature', '')
    expected = 'sha256=' + hmac.new(
        secret.encode(),
        json.dumps(body).encode(),
        hashlib.sha256
    ).hexdigest()
    
    return hmac.compare_digest(signature, expected)

# Usage in Flask
@app.route('/webhooks/evomi', methods=['POST'])
def handle_webhook():
    if not verify_webhook(request.headers, request.json, 'your-webhook-secret'):
        return 'Invalid signature', 401
    
    # Process webhook
    event = request.json.get('event')
    print(f'Event: {event}')
    return 'OK', 200

Best Practices Permalink to Best Practices

1. Respond Quickly Permalink to 1. Respond Quickly

Return a 200 OK response within 5 seconds. Long-running processes should be handled asynchronously:

@app.route('/webhook', methods=['POST'])
def webhook():
    # Queue the work for background processing
    process_webhook_async.delay(request.json)
    return 'OK', 200

2. Implement Retry Logic Permalink to 2. Implement Retry Logic

If your endpoint is temporarily unavailable, Evomi will retry:

  • Up to 5 attempts
  • Exponential backoff
  • 24-hour retention

3. Verify Signatures Permalink to 3. Verify Signatures

Always verify HMAC signatures for custom webhooks to ensure requests come from Evomi.

4. Handle Duplicate Events Permalink to 4. Handle Duplicate Events

Events may be delivered more than once. Use the task_id to deduplicate:

processed_tasks = set()

def handle_webhook(payload):
    task_id = payload['data']['task_id']
    if task_id in processed_tasks:
        return  # Already processed
    processed_tasks.add(task_id)
    # Process the event

5. Use HTTPS Permalink to 5. Use HTTPS

Webhook URLs must use HTTPS with a valid SSL certificate.

Troubleshooting Permalink to Troubleshooting

Webhooks Not Received Permalink to Webhooks Not Received

  1. Check firewall settings — Ensure your server accepts connections from Evomi IPs
  2. Verify SSL certificate — HTTPS is required with a valid certificate
  3. Review server logs — Look for incoming requests from Evomi
  4. Test manually — Use curl to test your endpoint

Test Your Endpoint Permalink to Test Your Endpoint

curl -X POST "https://your-server.com/webhook" \
  -H "Content-Type: application/json" \
  -H "X-Evomi-Signature: sha256=..." \
  -d '{
    "event": "completed",
    "timestamp": "2026-03-09T12:00:00Z",
    "data": {
      "task_id": "test_123",
      "task_type": "scrape",
      "status": "success"
    }
  }'

What’s Next? Permalink to What’s Next?

Rate LimitsError Handling