Sources & Integrations
All seven Thyme source connectors - when to use each, parameters, and authentication.
A source connects an external data system to a Thyme dataset. Thyme ships seven connectors, split into polling (incremental reads against a cursor) and streaming (continuous consumption).
For the @source decorator semantics - cursor, every, max_lateness, cdc - see Sources concept. This section is the per-connector parameter reference.
All connectors
| Connector | Type | Use case |
|---|---|---|
| PostgresSource | Polling | Postgres tables |
| KafkaSource | Streaming | Kafka topics |
| KinesisSource | Streaming | AWS Kinesis streams |
| S3JsonSource | Polling | JSON / JSONL files in S3 |
| SnowflakeSource | Polling | Snowflake tables |
| BigQuerySource | Polling | BigQuery tables |
| IcebergSource | Polling | Apache Iceberg tables |
Polling connectors require cursor and every on @source for incremental reads. Streaming connectors (Kafka, Kinesis) consume continuously - setting cursor or every on them raises a validation error.
Two field categories
Every connector splits its kwargs into two groups:
- Per-dataset (required, no env fallback) - identifies what to read:
table,topic,stream_arn,prefix, etc. Mistyping silently turning into""would mask a typo, so these stay strict. - Connection-level (env-defaulted) - identifies where to read from:
host,port,region, etc. When omitted, the connector readsTHYME_<TYPE>_<FIELD>from the environment. Explicit kwargs always override the env-var fallback.
This means you can keep credentials and connection details out of the Python file:
# All connection params come from env vars
PostgresSource(table="orders")
# Or override per-source if you need to
PostgresSource(table="orders", host="prod-db.internal")Secrets
Credential fields (password, sasl_password, role_arn, credentials_json) accept either a plain string or a Secret reference, so secrets stay out of code:
from thyme import Secret
from thyme.connectors import PostgresSource
PostgresSource(table="orders", password=Secret(env="PG_PASSWORD"))See the per-connector pages for the exact list of Secret-capable fields.