Common sources of big data include business transactions, application events, connected devices, documents and media, and external datasets. These are useful starting groups, not a complete or mutually exclusive classification. A sale can appear as a database row, an application event and an invoice.
Start with the question the data should answer. For an orders dashboard, that might be the value of current orders by region. Then identify which system owns the relevant facts, how changes reach you and how you will know the result is complete enough to use.
Five sources of big data
| Source | Examples | First ingestion decision |
|---|---|---|
| Business transactions | Orders, payments, stock movements and CRM records | Do you need a periodic snapshot or every relevant change, including deletions? |
| Application events | Product interactions, service logs and business events | What does each event mean, and how are identity, ordering and replay handled? |
| Connected devices | Equipment readings, meter observations and location reports | How will you handle device clocks, outages, late arrivals and calibration changes? |
| Documents and media | Invoices, forms, images, audio and video | Which original files, extracted fields and review decisions must be retained? |
| External datasets | Supplier feeds, licensed market data and published datasets | What may you collect and reuse, and how will you detect corrections or withdrawals? |
Social-media content and public-web records fit within the last two groups. Public availability does not establish permission to collect, retain or republish everything. Check the provider’s access terms and the intended use before designing collection.
Volume alone does not make a source useful. A small, authoritative order table may matter more to the orders question than a large set of loosely related social posts.
Separate the source from its storage platform
An order-management application produces order records. Its database stores them. A change-capture connector can deliver them to a warehouse, where a reporting model turns them into a daily view.
A cloud bucket can be the endpoint from which you ingest files, but “cloud” does not tell you who produced those files or how to interpret them. The same applies to a lake, warehouse or streaming platform. Record the business origin as well as the immediate endpoint.
For each dataset, identify the owning system and team, the record or event being represented, its keys, permitted uses and change mechanism. Keep the choice of storage and processing technology alongside those decisions. It cannot substitute for them.
Choose a change mechanism before choosing tools
| Mechanism | A useful fit | What must be resolved |
|---|---|---|
| Scheduled files or queries | A periodic report whose source can supply a consistent extract | Snapshot consistency, files that arrive twice, missing files and delete reconciliation |
| Database change data capture (CDC) | Keeping a downstream representation current as source rows change | Initial snapshot, log position, updates and deletes, restart behavior and source-log retention |
| Application events | A producer can publish a defined business occurrence | Meaning, identity, ordering scope, versioning and the consistency between the business write and publication |
| API polling or webhooks | A source exposes a supported integration interface | Pagination, rate limits, retry/backoff, cursor retention and a way to find missed or changed records |
These mechanisms can be combined. A CDC pipeline often starts with a snapshot, and an API may deliver a batch.
CDC describes changes to database rows. An application event might instead mean that an order was accepted for fulfillment. Those are different promises, even when they originate in the same application.
The PostgreSQL 18 and Debezium 3.6 documentation describe the relevant behavior. For PostgreSQL CDC, the initial snapshot and subsequent log position must fit together. A connector such as Debezium documents that handoff. PostgreSQL also warns that logical decoding can resend recent changes after a crash. Plan for replay at the consumer. Debezium’s PostgreSQL connector does not emit DDL change events. Its schema refresh is not a sink migration or a decision about what a renamed field means to your report. Debezium PostgreSQL connector, PostgreSQL logical decoding.
A concrete example with order changes
Suppose an operations team needs the current order value for a dashboard. Its proposed path is a PostgreSQL order table, a CDC connector, a restricted ingestion process and a reporting table. The reporting table holds the latest accepted state of each order. Summing every arriving update would count the same order repeatedly. This fixture sums nondeleted order amounts; it does not calculate recognized revenue or reconcile payments.
The companion below isolates the consumer part of that design. It uses invented, normalized change records and SQLite. It does not connect to PostgreSQL, run Debezium or test a warehouse. Its event envelope is an example adapter format, not the Debezium wire format.
An order key identifies the entity. A stable event ID identifies one change for replay detection. A per-order revision establishes which accepted version is newer in this example. Production adapters need an ordering rule supported by their actual source; timestamps alone are not a safe substitute.
Write down the contract
This sample contract combines field rules with operating decisions. The source owner approves record meaning and schema changes; the ingestion owner manages failures; a designated operator reviews corrections. Those are proposed roles, not people authenticated by the lab.
| Agreement | Rule in this example |
|---|---|
| Transport | One immutable, contiguous delivery sequence starts at offset 1. This differs from per-order revisions, which may arrive out of order. |
| Identity and ordering | One source, stable event IDs and order keys, increasing per-order revisions. Upserts contain full row images. |
| Field shape and units | Required envelope fields and validated order fields. Amounts are integer USD minor units within the declared range. |
| Schema evolution | Versions 1.0 and 1.1 are supported. Version 1.1 adds optional channel. Unknown versions or fields are rejected. |
| Deletion and replay | Deleted order keys are not reused. Retained delete markers and identity receipts protect against old replays. |
| Rejection and recovery | Retain rejected records before advancing. Preserve the original, correction, reason and outcome when repairing. |
| Freshness and access | Record age uses a fixed test clock. Production lag objectives, permissions and retention must be agreed and tested separately. |
The field definition is in contract.schema.json. Python checks it explicitly; the lab is not a general JSON Schema validator. Here is one complete accepted synthetic delivery from events.json:
{
"offset": 7,
"event_id": "e-200-1",
"schema_version": "1.1",
"order_id": "200",
"revision": 1,
"op": "upsert",
"occurred_at": "2026-09-17T11:59:30Z",
"data": {
"amount_minor": 2500,
"currency": "USD",
"status": "paid",
"channel": "web"
}
}
Version 1.1’s channel field works because this consumer explicitly supports it. That does not prove a 1.0-only consumer would accept the change. A version 2.0 record using a decimal amount string is quarantined until an explicit correction maps the sample value to supported integer minor units.
The contract is a project-specific agreement, not a claim of compliance with a universal data-contract standard. It makes the producer, consumer and recovery rules explicit. The full companion includes the contract, fixtures and executable checks.
Make failure visible and recovery repeatable
The consumer keeps its reporting state, processed-event record and checkpoint in one SQLite transaction. A crash before commit must leave none of those changes behind. If the commit succeeds but the delivery acknowledgement is lost, replay must not apply the change twice.
Event records rejected by the consumer’s field, timestamp or version checks go into a durable quarantine with a reason. Malformed JSON and invalid transport offsets stop intake before that path. This example advances its transport checkpoint only after the rejected record is retained. That keeps later transport work moving, but it does not make the reporting data complete. The unresolved quarantine must remain visible to the operator.
Repair is a separate, audited action. It must be safe to retry, respect later versions and preserve delete markers. Silently editing an old event file and restarting is not an adequate recovery record.
The tests inject Python exceptions before and after commit, then reopen temporary database connections. They do not kill the process or simulate power loss. Both tested environments passed all 19 checks: Python 3.13.7 with SQLite 3.50.4, and Python 3.14.7 with SQLite 3.53.4. Checks were recorded on 17 September 2026.
| Test | Observed result |
|---|---|
| Exception after writes, before commit | No partial state remains; checkpoint stays at 0. |
| Exception after commit, before acknowledgement | Checkpoint is 1; repeating the delivery leaves one applied event. |
| Duplicate event at another transport offset | The event is not applied a second time. |
| Invalid amount, malformed timestamp or unsupported schema | The record is retained with a rejection reason. |
| Approved synthetic correction retried | One repair is retained; a repeated repair makes no further change. |
| Old revision delivered after deletion | The order is not restored. |
At the end of the 13-delivery fixture, the checkpoint is 13, with four live orders and two retained delete markers. Four deliveries reached quarantine; two were repaired and two remain unresolved. Reaching the end of the input does not certify a complete reporting dataset.
The test results apply to the supplied synthetic records and local transaction logic. They are not throughput measurements or a claim of exactly-once delivery across an entire pipeline. Connector snapshot behavior, source failover, broker retention and downstream permissions need their own integration tests.
Check freshness, quality and access separately
Define freshness as an elapsed interval between named events, with a reporting objective chosen by the business. A connector heartbeat is not evidence that the dashboard includes every accepted source change. Monitor progress through the pipeline and unresolved records as well as record age. During a quiet period, an old business timestamp alone does not prove the pipeline is broken.
The example uses a fixed clock to make its age checks reproducible. It validates the declared record shape and field rules, keeps old revisions from replacing newer ones and distinguishes a durable rejection from a successful application. It cannot establish whether an accepted order amount is correct in the source system.
Keep collection permissions separate from reporting permissions. Scope the capture account to the required tables and capture functions, restrict raw changes and quarantine records, and give report readers only the fields they need. Check the connector’s actual privilege requirements rather than assuming a normal read-only SQL user can perform CDC. Replication slots also need monitoring because lagging consumers can retain source WAL. PostgreSQL replication security, slot WAL retention.
These access controls are deployment requirements. The SQLite example does not test database roles, tenant isolation, encryption, retention or authorization.
Run the failure and recovery checks
Download the ingestion lab, place it in a new local folder and run:
python3 -m zipfile -e data-sources-ingestion-lab.zip .
cd data-sources-ingestion-lab
python3 run_demo.py
python3 -m unittest -v
The opening output demonstrates the two failure points:
Before commit failure: checkpoint=0, applied_events=0; all writes rolled back.
After commit failure: checkpoint=1, applied_events=1; acknowledgement was lost.
Retry: redelivery, applied_events=1; no second application.
The ZIP contains the inputs, field schema, repair records, implementation, tests and execution receipts. It needs Python’s standard library and SQLite, with no package installation or network connection. The README records tested versions and the limits of the example. The fixture was prepared with AI assistance and checked by automated tests and a separate AI review; it is not a customer deployment record.
Run this in an isolated local folder with Python, using only the supplied synthetic files. No source-system credentials or production data are required. Inspect the input records, expected results and test code rather than treating a passing summary as evidence about your own system.
Before applying the design to a real source, reproduce a connector restart across a snapshot or log boundary, a missed API page where applicable, an incompatible field change, a deletion and a recovery after a long outage. Verify both the data that appears and the data that should no longer appear. Establish who owns each exception and when the report must be marked incomplete.
Where implementation work fits
When the requirement is a governed reporting dataset, Allerin’s Data & Analytics Platform service describes source-to-warehouse connectors, KPI definitions, quality checks and operating runbooks. The source inventory, contract and failure cases above give a team something concrete to scope before building a dashboard.
When the central problem is keeping operational systems in agreement, Integration FastTrack describes contract tests, replay fixtures, idempotency and reconciliation. That is a different decision from collecting every available source for analysis.
Bring a representative record, a documented change or deletion, the freshness requirement and an example failure. Those inputs help establish what the implementation must preserve and what a useful acceptance test should prove.
