Concepts
Data sources, connectors, jobs, executions — and how events differ from signals.
Data sources
A data source is a stored, tested connection to an external system. It
carries a connector_type, a config with two halves — connection
(host, database, endpoint) and credentials (passwords, keys, tokens) —
plus a status (active or error) and the result of its last connection
test (test_status, last_tested_at, test_error).
Credentials are write-only: every read endpoint strips credentials and
sensitive connection fields (password, secret, api_key, token,
private_key) before returning the source.
Connectors
The connector_type decides how data is pulled (or received):
| Family | Types | How data moves |
|---|---|---|
| Databases | postgres, postgresql, mysql, sqlserver, bigquery | Scheduled queries; incremental sync via a watermark column |
| Files | file (csv, excel, json, parquet), s3 | One-off uploads or bucket reads |
| APIs & streams | rest, graphql, kafka | Scheduled polls with pagination and incremental params |
| Webhooks | webhook | The provider pushes to you — no schedule, no polling |
| SaaS providers | e.g. stripe | Curated resource catalogs, usually connected via OAuth |
Databases and SaaS providers expose a tabular schema, so inspect returns ready-to-sync, auto-mapped tables. REST and GraphQL sources are sampled instead: the platform reads one response and detects the record shape, field types, and timestamp automatically.
Jobs & schedules
A job binds a source to one dataset (a table, an endpoint, a file) and says how its fields map to signal keys and when to sync. Three schedule shapes:
// Cron — fixed times
{ "type": "cron", "cron_expression": "0 2 * * *", "timezone": "UTC", "enabled": true }
// Interval — every N seconds
{ "type": "interval", "interval_seconds": 300, "enabled": true }
// Once — run only when triggered manually
{ "type": "once", "enabled": false }The connect flow accepts friendly frequency tokens instead
(realtime, 5m, 15m, 1h, daily, weekly, manual, …) and
translates them to these shapes for you.
Jobs with a watermark_column sync incrementally: each run records the
highest value it saw and the next run only pulls rows past it. Jobs
without one re-pull in full. Scheduling priority comes from your plan, not
from the request.
Executions
An execution is one run of a job: queued → processing →
completed, failed, or cancelled. Each record carries progress
(percentage), stats (rows processed, duration, throughput), and — on
failure — a structured error with a message and the phase that failed.
Failed executions can be retried;
running ones can be cancelled.
Events vs signals
Ingested data lands as one of two things:
- Signals — named numeric or categorical time series, keyed
stream.field(e.g.orders.amount). This is the default for every mapped column, and what segments and pipelines consume. - Events — free-text occurrences ("checkout failed", a tweet, a
support ticket) with an
occurred_attimestamp. Fields flaggedis_eventin a mapping — or delivered to a webhook's/eventsURL — take this path and enrich forecasts as context rather than as series. Each one is scored automatically and lands under an event stream — see Signals → Events for how to read them back.
The same source can feed both: a webhook source exposes separate
/events and /signal delivery URLs, and any poll job can mark one text
column is_event while the rest of its columns become signals.
Cost
Ingested volume is metered per workspace. Bulk imports and streaming pushes each have an included monthly allowance — every workspace gets its own — and beyond it, ingestion bills per GB. Rates and your current usage live in Billing.