Web data · Convex component

@sholajegede/convex-bright-data-datasets

A convex bright data datasets component for Convex.

npm GitHub Convex directory Ask about it

npm install @sholajegede/convex-bright-data-datasets

1downloads last week

674downloads last 12 months

v0.1.3latest, Mar 14, 2026

4releases since Mar 14, 2026

Downloads

Download history

Loading download history…

Build this

Build with @sholajegede/convex-bright-data-datasets

The install and setup steps from the package documentation.

Installation

npm install @sholajegede/convex-bright-data-datasets

Add the component to your convex/convex.config.ts:

import { defineApp } from "convex/server";
import convexBrightDataDatasets from "@sholajegede/convex-bright-data-datasets/convex.config.js";

const app = defineApp();
app.use(convexBrightDataDatasets);

export default app;

Setup

1. Instantiate the client in your Convex functions:

// convex/brightDatasets.ts
import { components } from "./_generated/api.js";
import { BrightDatasets } from "@sholajegede/convex-bright-data-datasets";

export const brightDatasets = new BrightDatasets(components.convexBrightDataDatasets, {
  BRIGHTDATA_API_TOKEN: process.env.BRIGHTDATA_API_TOKEN!,
});

2. Mount the webhook handler in convex/http.ts:

import { httpRouter } from "convex/server";
import { components } from "./_generated/api.js";
import { createWebhookHandler } from "@sholajegede/convex-bright-data-datasets";

const http = httpRouter();

http.route({
  path: "/webhooks/brightdata",
  method: "POST",
  handler: createWebhookHandler(components.convexBrightDataDatasets),
});

export default http;

3. Set your Convex environment variable:

npx convex env set BRIGHTDATA_API_TOKEN your_token_here

Your Convex HTTP actions URL (the webhook endpoint to register in Bright Data) is:

code

https://<your-deployment>.convex.site/webhooks/brightdata

You can find this by running npx convex dev and looking for VITE_CONVEX_SITE_URL in your .env.local.

Usage

Trigger an async collection

// convex/myFunctions.ts
import { action, query } from "./_generated/server.js";
import { components } from "./_generated/api.js";
import { brightDatasets } from "./brightDatasets.js";
import { v } from "convex/values";

// Trigger a LinkedIn profile collection
export const collectProfiles = action({
  args: { urls: v.array(v.string()) },
  handler: async (ctx, args) => {
    return await brightDatasets.trigger(ctx, {
      datasetId: "gd_l1viktl72bvl7bjuj0", // LinkedIn profiles dataset
      inputs: args.urls.map((url) => ({ url })),
      webhookUrl: process.env.CONVEX_SITE_URL + "/webhooks/brightdata",
    });
    // Returns: { snapshotId: "s_...", status: "pending" }
  },
});

// Reactive query — subscribe to snapshot status from the frontend
export const getSnapshot = query({
  args: { snapshotId: v.string() },
  handler: async (ctx, args) => {
    return await ctx.runQuery(components.convexBrightDataDatasets.lib.getSnapshot, {
      snapshotId: args.snapshotId,
    });
  },
});

// Reactive query — subscribe to records as they arrive
export const getRecords = query({
  args: { snapshotId: v.string() },
  handler: async (ctx, args) => {
    return await ctx.runQuery(components.convexBrightDataDatasets.lib.getRecords, {
      snapshotId: args.snapshotId,
    });
  },
});

tsx

// React — subscribes reactively, re-renders when status or records update
const snapshot = useQuery(api.myFunctions.getSnapshot, { snapshotId });
// snapshot.status   — "pending" | "collecting" | "digesting" | "ready" | "failed" | "canceled"
// snapshot.recordCount — number of records received so far

const records = useQuery(api.myFunctions.getRecords, { snapshotId });
// records — array of structured records from Bright Data, parsed from NDJSON

Synchronous scrape (small jobs)

export const scrapeProfile = action({
  args: { url: v.string() },
  handler: async (ctx, args) => {
    return await brightDatasets.scrape(ctx, {
      datasetId: "gd_l1viktl72bvl7bjuj0",
      inputs: [{ url: args.url }],
    });
    // Returns: { records: [...], status: "ready" }
    // If job exceeds 1 min: { records: [], snapshotId: "s_...", status: "running" }
  },
});

Poll for status

export const checkStatus = action({
  args: { snapshotId: v.string() },
  handler: async (ctx, args) => {
    return await brightDatasets.pollStatus(ctx, args.snapshotId);
    // Fetches from Bright Data, updates snapshot in Convex, returns current status
  },
});

Cancel a collection

export const cancelJob = action({
  args: { snapshotId: v.string() },
  handler: async (ctx, args) => {
    return await brightDatasets.cancel(ctx, args.snapshotId);
  },
});

List all snapshots

export const listJobs = query({
  args: {},
  handler: async (ctx) => {
    return await ctx.runQuery(components.convexBrightDataDatasets.lib.listSnapshots, {
      limit: 20,
    });
  },
});

Discovery mode

// Discover Amazon products by keyword
export const discoverProducts = action({
  args: { keywords: v.array(v.string()) },
  handler: async (ctx, args) => {
    return await brightDatasets.trigger(ctx, {
      datasetId: "gd_l7q7dkf244hwjntr0",
      inputs: args.keywords.map((keyword) => ({ keyword })),
      discoveryMode: "discover_new",
      discoverBy: "keyword",
      limitPerInput: 10,
      webhookUrl: process.env.CONVEX_SITE_URL + "/webhooks/brightdata",
    });
  },
});