SourceSync.ai

SourceSync.ai es una plataforma de Retrieval Augmented Generation (RAG) como servicio que te ayuda a crear aplicaciones de IA con tus propios datos. Esta guía explica cómo usar Firecrawl con SourceSync.ai para funciones de web scraping.

Configuración

Primero, obtiene tu clave de API de Firecrawl desde tu panel de Firecrawl.
Configura tu espacio de nombres de SourceSync.ai para usar Firecrawl como proveedor de web scraping:

curl -X PATCH https://api.sourcesync.ai/v1/namespaces/YOUR_NAMESPACE_ID \
  -H "Authorization: Bearer YOUR_SOURCE_SYNC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "webScraperConfig": {
      "provider": "FIRECRAWL",
      "apiKey": "YOUR_FIRECRAWL_API_KEY"
    }
  }'

const response = await fetch(
  'https://api.sourcesync.ai/v1/namespaces/YOUR_NAMESPACE_ID',
  {
    method: 'PATCH',
    headers: {
      Authorization: `Bearer ${SOURCE_SYNC_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      webScraperConfig: {
        provider: 'FIRECRAWL',
        apiKey: 'YOUR_FIRECRAWL_API_KEY',
      },
    }),
  }
)

import requests

response = requests.patch(
    'https://api.sourcesync.ai/v1/namespaces/YOUR_NAMESPACE_ID',
    headers={
        'Authorization': f'Bearer {SOURCE_SYNC_API_KEY}',
        'Content-Type': 'application/json',
    },
    json={
        'webScraperConfig': {
            'provider': 'FIRECRAWL',
            'apiKey': 'YOUR_FIRECRAWL_API_KEY',
        },
    },
)

Uso

Una vez configurado, puedes usar los puntos de conexión de Web scraping de SourceSync.ai junto con las capacidades de Firecrawl. Estos son los principales métodos de ingesta:

Ingesta de lista de URL

Extrae contenido de URL específicas:

curl -X POST https://api.sourcesync.ai/v1/ingest/urls \
  -H "Authorization: Bearer YOUR_SOURCE_SYNC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "namespaceId": "YOUR_NAMESPACE_ID",
    "ingestConfig": {
      "source": "URLS_LIST",
      "config": {
        "urls": [
          "https://example.com/page1",
          "https://example.com/page2"
        ],
        "scrapeOptions": {
          "includeSelectors": ["article", "main"],
          "excludeSelectors": [".navigation", ".footer"]
        }
      }
    }
  }'

const response = await fetch('https://api.sourcesync.ai/v1/ingest/urls', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${SOURCE_SYNC_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    namespaceId: 'YOUR_NAMESPACE_ID',
    ingestConfig: {
      source: 'URLS_LIST',
      config: {
        urls: ['https://example.com/page1', 'https://example.com/page2'],
        scrapeOptions: {
          includeSelectors: ['article', 'main'],
          excludeSelectors: ['.navigation', '.footer'],
        },
      },
    },
  }),
})

response = requests.post(
    'https://api.sourcesync.ai/v1/ingest/urls',
    headers={
        'Authorization': f'Bearer {SOURCE_SYNC_API_KEY}',
        'Content-Type': 'application/json',
    },
    json={
        'namespaceId': 'YOUR_NAMESPACE_ID',
        'ingestConfig': {
            'source': 'URLS_LIST',
            'config': {
                'urls': ['https://example.com/page1', 'https://example.com/page2'],
                'scrapeOptions': {
                    'includeSelectors': ['article', 'main'],
                    'excludeSelectors': ['.navigation', '.footer'],
                },
            },
        },
    },
)

Rastreo del sitio web

Rastrea todo un sitio web con reglas personalizadas:

curl -X POST https://api.sourcesync.ai/v1/ingest/website \
  -H "Authorization: Bearer YOUR_SOURCE_SYNC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "namespaceId": "YOUR_NAMESPACE_ID",
    "ingestConfig": {
      "source": "WEBSITE",
      "config": {
        "url": "https://example.com",
        "maxDepth": 3,
        "maxLinks": 100,
        "includePaths": ["/docs", "/blog"],
        "excludePaths": ["/admin"],
        "scrapeOptions": {
          "includeSelectors": ["article", "main"],
          "excludeSelectors": [".navigation", ".footer"]
        }
      }
    }
  }'

const response = await fetch('https://api.sourcesync.ai/v1/ingest/website', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${SOURCE_SYNC_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    namespaceId: 'YOUR_NAMESPACE_ID',
    ingestConfig: {
      source: 'WEBSITE',
      config: {
        url: 'https://example.com',
        maxDepth: 3,
        maxLinks: 100,
        includePaths: ['/docs', '/blog'],
        excludePaths: ['/admin'],
        scrapeOptions: {
          includeSelectors: ['article', 'main'],
          excludeSelectors: ['.navigation', '.footer'],
        },
      },
    },
  }),
})

response = requests.post(
    'https://api.sourcesync.ai/v1/ingest/website',
    headers={
        'Authorization': f'Bearer {SOURCE_SYNC_API_KEY}',
        'Content-Type': 'application/json',
    },
    json={
        'namespaceId': 'YOUR_NAMESPACE_ID',
        'ingestConfig': {
            'source': 'WEBSITE',
            'config': {
                'url': 'https://example.com',
                'maxDepth': 3,
                'maxLinks': 100,
                'includePaths': ['/docs', '/blog'],
                'excludePaths': ['/admin'],
                'scrapeOptions': {
                    'includeSelectors': ['article', 'main'],
                    'excludeSelectors': ['.navigation', '.footer'],
                },
            },
        },
    },
)

Procesamiento de sitemaps

Procesa todas las URL de un sitemap:

curl -X POST https://api.sourcesync.ai/v1/ingest/sitemap \
  -H "Authorization: Bearer YOUR_SOURCE_SYNC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "namespaceId": "YOUR_NAMESPACE_ID",
    "ingestConfig": {
      "source": "SITEMAP",
      "config": {
        "url": "https://example.com/sitemap.xml",
        "scrapeOptions": {
          "includeSelectors": ["article", "main"],
          "excludeSelectors": [".navigation", ".footer"]
        }
      }
    }
  }'

const response = await fetch('https://api.sourcesync.ai/v1/ingest/sitemap', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${SOURCE_SYNC_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    namespaceId: 'YOUR_NAMESPACE_ID',
    ingestConfig: {
      source: 'SITEMAP',
      config: {
        url: 'https://example.com/sitemap.xml',
        scrapeOptions: {
          includeSelectors: ['article', 'main'],
          excludeSelectors: ['.navigation', '.footer'],
        },
      },
    },
  }),
})

response = requests.post(
    'https://api.sourcesync.ai/v1/ingest/sitemap',
    headers={
        'Authorization': f'Bearer {SOURCE_SYNC_API_KEY}',
        'Content-Type': 'application/json',
    },
    json={
        'namespaceId': 'YOUR_NAMESPACE_ID',
        'ingestConfig': {
            'source': 'SITEMAP',
            'config': {
                'url': 'https://example.com/sitemap.xml',
                'scrapeOptions': {
                    'includeSelectors': ['article', 'main'],
                    'excludeSelectors': ['.navigation', '.footer'],
                },
            },
        },
    },
)

Funcionalidades

Al usar Firecrawl con SourceSync.ai, tienes acceso a:

Compatibilidad con renderizado de JavaScript
Limitación automática de la tasa
Extracción de contenido basada en selectores CSS
Rastreo recursivo con control de profundidad
Procesamiento de sitemaps

Recursos

Para obtener soporte adicional:

Correo: support@sourcesync.ai
Discord: Únete a nuestra comunidad

Primeros pasos

Endpoints principales

Más

Inicios rápidos

Guías para desarrolladores

Webhooks

Casos de uso

Otros

Contribuciones

Configuración

Uso

Ingesta de lista de URL

Rastreo del sitio web

Procesamiento de sitemaps

Funcionalidades

Recursos

​Configuración

​Uso

​Ingesta de lista de URL

​Rastreo del sitio web

​Procesamiento de sitemaps

​Funcionalidades

​Recursos

Configuración

Uso

Ingesta de lista de URL

Rastreo del sitio web

Procesamiento de sitemaps

Funcionalidades

Recursos