{"id":14902,"date":"2017-11-26T17:04:51","date_gmt":"2017-11-26T11:34:51","guid":{"rendered":"https:\/\/www.allerin.com\/blog\/?p=14902"},"modified":"2026-09-17T14:10:28","modified_gmt":"2026-09-17T08:40:28","slug":"data-sources-ingestion-review-6276","status":"publish","type":"post","link":"https:\/\/www.allerin.com\/blog\/data-sources-ingestion-review-6276\/","title":{"rendered":"5 sources of big data and how to ingest them"},"content":{"rendered":"<p>Common <strong>sources of big data<\/strong> include business transactions, application events, connected devices, documents and media, and external datasets. These are useful starting groups, not a complete or mutually exclusive classification. A sale can appear as a database row, an application event and an invoice.<\/p>\n<p>Start with the question the data should answer. For an orders dashboard, that might be the value of current orders by region. Then identify which system owns the relevant facts, how changes reach you and how you will know the result is complete enough to use.<\/p>\n<nav style=\"margin: 24px 0; padding: 18px; background: #edf4f0; border-radius: 6px;\" aria-label=\"Article sections\">\n<p style=\"margin: 0 0 10px;\"><strong>In this guide<\/strong><\/p>\n<ul style=\"margin: 0; padding-left: 20px;\">\n<li><a href=\"#five-sources-of-big-data\">Five sources of big data<\/a><\/li>\n<li><a href=\"#separate-the-source-from-its-storage-platform\">Separate the source from its storage platform<\/a><\/li>\n<li><a href=\"#choose-a-change-mechanism-before-choosing-tools\">Choose a change mechanism before choosing tools<\/a><\/li>\n<li><a href=\"#a-concrete-example-with-order-changes\">A concrete example with order changes<\/a><\/li>\n<li><a href=\"#run-the-failure-and-recovery-checks\">Run the failure and recovery checks<\/a><\/li>\n<li><a href=\"#where-implementation-work-fits\">Where implementation work fits<\/a><\/li>\n<\/ul>\n<\/nav>\n<h2 id=\"five-sources-of-big-data\" style=\"scroll-margin-top: 110px;\">Five sources of big data<\/h2>\n<div style=\"overflow-x: auto; max-width: 100%; margin: 20px 0;\" tabindex=\"0\" role=\"region\" aria-label=\"Five representative data origins\">\n<table style=\"width: 100%; min-width: 540px; table-layout: auto; border-collapse: collapse; font-size: 16px; line-height: 1.55;\">\n<caption style=\"text-align: left; font-weight: 600; margin-bottom: 10px;\">Five representative data origins<\/caption>\n<thead>\n<tr>\n<th style=\"padding: 12px; border: 1px solid #c5d6d0; background: #edf4f0; text-align: left; vertical-align: top;\" scope=\"col\">Source<\/th>\n<th style=\"padding: 12px; border: 1px solid #c5d6d0; background: #edf4f0; text-align: left; vertical-align: top;\" scope=\"col\">Examples<\/th>\n<th style=\"padding: 12px; border: 1px solid #c5d6d0; background: #edf4f0; text-align: left; vertical-align: top;\" scope=\"col\">First ingestion decision<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Business transactions<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Orders, payments, stock movements and CRM records<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Do you need a periodic snapshot or every relevant change, including deletions?<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Application events<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Product interactions, service logs and business events<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">What does each event mean, and how are identity, ordering and replay handled?<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Connected devices<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Equipment readings, meter observations and location reports<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">How will you handle device clocks, outages, late arrivals and calibration changes?<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Documents and media<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Invoices, forms, images, audio and video<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Which original files, extracted fields and review decisions must be retained?<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">External datasets<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Supplier feeds, licensed market data and published datasets<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">What may you collect and reuse, and how will you detect corrections or withdrawals?<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>Social-media content and public-web records fit within the last two groups. Public availability does not establish permission to collect, retain or republish everything. Check the provider&#8217;s access terms and the intended use before designing collection.<\/p>\n<p>Volume alone does not make a source useful. A small, authoritative order table may matter more to the orders question than a large set of loosely related social posts.<\/p>\n<h2 id=\"separate-the-source-from-its-storage-platform\" style=\"scroll-margin-top: 110px;\">Separate the source from its storage platform<\/h2>\n<p>An order-management application produces order records. Its database stores them. A change-capture connector can deliver them to a warehouse, where a reporting model turns them into a daily view.<\/p>\n<p>A cloud bucket can be the endpoint from which you ingest files, but \u201ccloud\u201d does not tell you who produced those files or how to interpret them. The same applies to a lake, warehouse or streaming platform. Record the business origin as well as the immediate endpoint.<\/p>\n<p>For each dataset, identify the owning system and team, the record or event being represented, its keys, permitted uses and change mechanism. Keep the choice of storage and processing technology alongside those decisions. It cannot substitute for them.<\/p>\n<h2 id=\"choose-a-change-mechanism-before-choosing-tools\" style=\"scroll-margin-top: 110px;\">Choose a change mechanism before choosing tools<\/h2>\n<div style=\"overflow-x: auto; max-width: 100%; margin: 20px 0;\" tabindex=\"0\" role=\"region\" aria-label=\"Choosing a change mechanism\">\n<table style=\"width: 100%; min-width: 540px; table-layout: auto; border-collapse: collapse; font-size: 16px; line-height: 1.55;\">\n<caption style=\"text-align: left; font-weight: 600; margin-bottom: 10px;\">Choosing a change mechanism<\/caption>\n<thead>\n<tr>\n<th style=\"padding: 12px; border: 1px solid #c5d6d0; background: #edf4f0; text-align: left; vertical-align: top;\" scope=\"col\">Mechanism<\/th>\n<th style=\"padding: 12px; border: 1px solid #c5d6d0; background: #edf4f0; text-align: left; vertical-align: top;\" scope=\"col\">A useful fit<\/th>\n<th style=\"padding: 12px; border: 1px solid #c5d6d0; background: #edf4f0; text-align: left; vertical-align: top;\" scope=\"col\">What must be resolved<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Scheduled files or queries<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">A periodic report whose source can supply a consistent extract<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Snapshot consistency, files that arrive twice, missing files and delete reconciliation<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Database change data capture (CDC)<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Keeping a downstream representation current as source rows change<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Initial snapshot, log position, updates and deletes, restart behavior and source-log retention<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Application events<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">A producer can publish a defined business occurrence<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Meaning, identity, ordering scope, versioning and the consistency between the business write and publication<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">API polling or webhooks<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">A source exposes a supported integration interface<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Pagination, rate limits, retry\/backoff, cursor retention and a way to find missed or changed records<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>These mechanisms can be combined. A CDC pipeline often starts with a snapshot, and an API may deliver a batch.<\/p>\n<p>CDC describes changes to database rows. An application event might instead mean that an order was accepted for fulfillment. Those are different promises, even when they originate in the same application.<\/p>\n<p>The PostgreSQL 18 and Debezium 3.6 documentation describe the relevant behavior. For PostgreSQL CDC, the initial snapshot and subsequent log position must fit together. A connector such as Debezium documents that handoff. PostgreSQL also warns that logical decoding can resend recent changes after a crash. Plan for replay at the consumer. Debezium&#8217;s PostgreSQL connector does not emit DDL change events. Its schema refresh is not a sink migration or a decision about what a renamed field means to your report. <a href=\"https:\/\/debezium.io\/documentation\/reference\/3.6\/connectors\/postgresql.html\" target=\"_blank\" rel=\"noopener\">Debezium PostgreSQL connector<\/a>, <a href=\"https:\/\/www.postgresql.org\/docs\/18\/logicaldecoding-explanation.html\" target=\"_blank\" rel=\"noopener\">PostgreSQL logical decoding<\/a>.<\/p>\n<h2 id=\"a-concrete-example-with-order-changes\" style=\"scroll-margin-top: 110px;\">A concrete example with order changes<\/h2>\n<p>Suppose an operations team needs the current order value for a dashboard. Its proposed path is a PostgreSQL order table, a CDC connector, a restricted ingestion process and a reporting table. The reporting table holds the latest accepted state of each order. Summing every arriving update would count the same order repeatedly. This fixture sums nondeleted order amounts; it does not calculate recognized revenue or reconcile payments.<\/p>\n<p>The companion below isolates the consumer part of that design. It uses invented, normalized change records and SQLite. It does not connect to PostgreSQL, run Debezium or test a warehouse. Its event envelope is an example adapter format, not the Debezium wire format.<\/p>\n<p>An order key identifies the entity. A stable event ID identifies one change for replay detection. A per-order revision establishes which accepted version is newer in this example. Production adapters need an ordering rule supported by their actual source; timestamps alone are not a safe substitute.<\/p>\n<h3 id=\"write-down-the-contract\" style=\"scroll-margin-top: 110px;\">Write down the contract<\/h3>\n<p>This sample contract combines field rules with operating decisions. The source owner approves record meaning and schema changes; the ingestion owner manages failures; a designated operator reviews corrections. Those are proposed roles, not people authenticated by the lab.<\/p>\n<div style=\"overflow-x: auto; max-width: 100%; margin: 20px 0;\" tabindex=\"0\" role=\"region\" aria-label=\"A sample ingestion contract\">\n<table style=\"width: 100%; min-width: 540px; table-layout: auto; border-collapse: collapse; font-size: 16px; line-height: 1.55;\">\n<caption style=\"text-align: left; font-weight: 600; margin-bottom: 10px;\">A sample ingestion contract<\/caption>\n<thead>\n<tr>\n<th style=\"padding: 12px; border: 1px solid #c5d6d0; background: #edf4f0; text-align: left; vertical-align: top;\" scope=\"col\">Agreement<\/th>\n<th style=\"padding: 12px; border: 1px solid #c5d6d0; background: #edf4f0; text-align: left; vertical-align: top;\" scope=\"col\">Rule in this example<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Transport<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">One immutable, contiguous delivery sequence starts at offset 1. This differs from per-order revisions, which may arrive out of order.<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Identity and ordering<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">One source, stable event IDs and order keys, increasing per-order revisions. Upserts contain full row images.<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Field shape and units<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Required envelope fields and validated order fields. Amounts are integer USD minor units within the declared range.<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Schema evolution<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Versions 1.0 and 1.1 are supported. Version 1.1 adds optional <code>channel<\/code>. Unknown versions or fields are rejected.<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Deletion and replay<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Deleted order keys are not reused. Retained delete markers and identity receipts protect against old replays.<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Rejection and recovery<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Retain rejected records before advancing. Preserve the original, correction, reason and outcome when repairing.<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Freshness and access<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Record age uses a fixed test clock. Production lag objectives, permissions and retention must be agreed and tested separately.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>The field definition is in <code>contract.schema.json<\/code>. Python checks it explicitly; the lab is not a general JSON Schema validator. Here is one complete accepted synthetic delivery from <code>events.json<\/code>:<\/p>\n<pre style=\"max-width: 100%; overflow-x: auto; white-space: pre; padding: 16px; background: #f3f5f4; border: 1px solid #c5d6d0; border-radius: 6px; font-size: 14px; line-height: 1.5;\" tabindex=\"0\" aria-label=\"Code example\"><code class=\"language-json\">{\r\n  \"offset\": 7,\r\n  \"event_id\": \"e-200-1\",\r\n  \"schema_version\": \"1.1\",\r\n  \"order_id\": \"200\",\r\n  \"revision\": 1,\r\n  \"op\": \"upsert\",\r\n  \"occurred_at\": \"2026-09-17T11:59:30Z\",\r\n  \"data\": {\r\n    \"amount_minor\": 2500,\r\n    \"currency\": \"USD\",\r\n    \"status\": \"paid\",\r\n    \"channel\": \"web\"\r\n  }\r\n}\r\n<\/code><\/pre>\n<p>Version 1.1&#8217;s <code>channel<\/code> field works because this consumer explicitly supports it. That does not prove a 1.0-only consumer would accept the change. A version 2.0 record using a decimal amount string is quarantined until an explicit correction maps the sample value to supported integer minor units.<\/p>\n<p>The contract is a project-specific agreement, not a claim of compliance with a universal data-contract standard. It makes the producer, consumer and recovery rules explicit. The full companion includes the contract, fixtures and executable checks.<\/p>\n<h3 id=\"make-failure-visible-and-recovery-repeatable\" style=\"scroll-margin-top: 110px;\">Make failure visible and recovery repeatable<\/h3>\n<p>The consumer keeps its reporting state, processed-event record and checkpoint in one SQLite transaction. A crash before commit must leave none of those changes behind. If the commit succeeds but the delivery acknowledgement is lost, replay must not apply the change twice.<\/p>\n<p>Event records rejected by the consumer\u2019s field, timestamp or version checks go into a durable quarantine with a reason. Malformed JSON and invalid transport offsets stop intake before that path. This example advances its transport checkpoint only after the rejected record is retained. That keeps later transport work moving, but it does not make the reporting data complete. The unresolved quarantine must remain visible to the operator.<\/p>\n<p>Repair is a separate, audited action. It must be safe to retry, respect later versions and preserve delete markers. Silently editing an old event file and restarting is not an adequate recovery record.<\/p>\n<p>The tests inject Python exceptions before and after commit, then reopen temporary database connections. They do not kill the process or simulate power loss. Both tested environments passed all 19 checks: Python 3.13.7 with SQLite 3.50.4, and Python 3.14.7 with SQLite 3.53.4. Checks were recorded on 17 September 2026.<\/p>\n<div style=\"overflow-x: auto; max-width: 100%; margin: 20px 0;\" tabindex=\"0\" role=\"region\" aria-label=\"Executed synthetic failure and recovery checks\">\n<table style=\"width: 100%; min-width: 540px; table-layout: auto; border-collapse: collapse; font-size: 16px; line-height: 1.55;\">\n<caption style=\"text-align: left; font-weight: 600; margin-bottom: 10px;\">Executed synthetic failure and recovery checks<\/caption>\n<thead>\n<tr>\n<th style=\"padding: 12px; border: 1px solid #c5d6d0; background: #edf4f0; text-align: left; vertical-align: top;\" scope=\"col\">Test<\/th>\n<th style=\"padding: 12px; border: 1px solid #c5d6d0; background: #edf4f0; text-align: left; vertical-align: top;\" scope=\"col\">Observed result<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Exception after writes, before commit<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">No partial state remains; checkpoint stays at 0.<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Exception after commit, before acknowledgement<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Checkpoint is 1; repeating the delivery leaves one applied event.<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Duplicate event at another transport offset<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">The event is not applied a second time.<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Invalid amount, malformed timestamp or unsupported schema<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">The record is retained with a rejection reason.<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Approved synthetic correction retried<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">One repair is retained; a repeated repair makes no further change.<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">Old revision delivered after deletion<\/td>\n<td style=\"padding: 12px; border: 1px solid #c5d6d0; text-align: left; vertical-align: top;\">The order is not restored.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>At the end of the 13-delivery fixture, the checkpoint is 13, with four live orders and two retained delete markers. Four deliveries reached quarantine; two were repaired and two remain unresolved. Reaching the end of the input does not certify a complete reporting dataset.<\/p>\n<p>The test results apply to the supplied synthetic records and local transaction logic. They are not throughput measurements or a claim of exactly-once delivery across an entire pipeline. Connector snapshot behavior, source failover, broker retention and downstream permissions need their own integration tests.<\/p>\n<h3 id=\"check-freshness-quality-and-access-separately\" style=\"scroll-margin-top: 110px;\">Check freshness, quality and access separately<\/h3>\n<p>Define freshness as an elapsed interval between named events, with a reporting objective chosen by the business. A connector heartbeat is not evidence that the dashboard includes every accepted source change. Monitor progress through the pipeline and unresolved records as well as record age. During a quiet period, an old business timestamp alone does not prove the pipeline is broken.<\/p>\n<p>The example uses a fixed clock to make its age checks reproducible. It validates the declared record shape and field rules, keeps old revisions from replacing newer ones and distinguishes a durable rejection from a successful application. It cannot establish whether an accepted order amount is correct in the source system.<\/p>\n<p>Keep collection permissions separate from reporting permissions. Scope the capture account to the required tables and capture functions, restrict raw changes and quarantine records, and give report readers only the fields they need. Check the connector&#8217;s actual privilege requirements rather than assuming a normal read-only SQL user can perform CDC. Replication slots also need monitoring because lagging consumers can retain source WAL. <a href=\"https:\/\/www.postgresql.org\/docs\/18\/logical-replication-security.html\" target=\"_blank\" rel=\"noopener\">PostgreSQL replication security<\/a>, <a href=\"https:\/\/www.postgresql.org\/docs\/18\/runtime-config-replication.html#GUC-MAX-SLOT-WAL-KEEP-SIZE\" target=\"_blank\" rel=\"noopener\">slot WAL retention<\/a>.<\/p>\n<p>These access controls are deployment requirements. The SQLite example does not test database roles, tenant isolation, encryption, retention or authorization.<\/p>\n<h2 id=\"run-the-failure-and-recovery-checks\" style=\"scroll-margin-top: 110px;\">Run the failure and recovery checks<\/h2>\n<p><a href=\"https:\/\/www.allerin.com\/downloads\/data-sources-ingestion-lab.zip\">Download the ingestion lab<\/a>, place it in a new local folder and run:<\/p>\n<pre style=\"max-width: 100%; overflow-x: auto; white-space: pre; padding: 16px; background: #f3f5f4; border: 1px solid #c5d6d0; border-radius: 6px; font-size: 14px; line-height: 1.5;\" tabindex=\"0\" aria-label=\"Code example\"><code class=\"language-sh\">python3 -m zipfile -e data-sources-ingestion-lab.zip .\r\ncd data-sources-ingestion-lab\r\npython3 run_demo.py\r\npython3 -m unittest -v\r\n<\/code><\/pre>\n<p>The opening output demonstrates the two failure points:<\/p>\n<pre style=\"max-width: 100%; overflow-x: auto; white-space: pre; padding: 16px; background: #f3f5f4; border: 1px solid #c5d6d0; border-radius: 6px; font-size: 14px; line-height: 1.5;\" tabindex=\"0\" aria-label=\"Code example\"><code class=\"language-text\">Before commit failure: checkpoint=0, applied_events=0; all writes rolled back.\r\nAfter commit failure: checkpoint=1, applied_events=1; acknowledgement was lost.\r\nRetry: redelivery, applied_events=1; no second application.\r\n<\/code><\/pre>\n<p>The ZIP contains the inputs, field schema, repair records, implementation, tests and execution receipts. It needs Python&#8217;s standard library and SQLite, with no package installation or network connection. The README records tested versions and the limits of the example. The fixture was prepared with AI assistance and checked by automated tests and a separate AI review; it is not a customer deployment record.<\/p>\n<p>Run this in an isolated local folder with Python, using only the supplied synthetic files. No source-system credentials or production data are required. Inspect the input records, expected results and test code rather than treating a passing summary as evidence about your own system.<\/p>\n<p>Before applying the design to a real source, reproduce a connector restart across a snapshot or log boundary, a missed API page where applicable, an incompatible field change, a deletion and a recovery after a long outage. Verify both the data that appears and the data that should no longer appear. Establish who owns each exception and when the report must be marked incomplete.<\/p>\n<h2 id=\"where-implementation-work-fits\" style=\"scroll-margin-top: 110px;\">Where implementation work fits<\/h2>\n<p>When the requirement is a governed reporting dataset, Allerin&#8217;s <a href=\"https:\/\/www.allerin.com\/services\/data-analytics-platform\">Data &amp; Analytics Platform service<\/a> describes source-to-warehouse connectors, KPI definitions, quality checks and operating runbooks. The source inventory, contract and failure cases above give a team something concrete to scope before building a dashboard.<\/p>\n<p>When the central problem is keeping operational systems in agreement, <a href=\"https:\/\/www.allerin.com\/services\/integration-fasttrack\">Integration FastTrack<\/a> describes contract tests, replay fixtures, idempotency and reconciliation. That is a different decision from collecting every available source for analysis.<\/p>\n<p>Bring a representative record, a documented change or deletion, the freshness requirement and an example failure. Those inputs help establish what the implementation must preserve and what a useful acceptance test should prove.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Five common data sources, ingestion tradeoffs and a tested synthetic order-change example covering contracts, schema changes, replay, quarantine and recovery.<\/p>\n","protected":false},"author":1,"featured_media":6464,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[126,3],"tags":[468],"class_list":["post-14902","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-big-data","category-technology","tag-big-data-sources"],"_links":{"self":[{"href":"https:\/\/www.allerin.com\/blog\/wp-json\/wp\/v2\/posts\/14902","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.allerin.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.allerin.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.allerin.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.allerin.com\/blog\/wp-json\/wp\/v2\/comments?post=14902"}],"version-history":[{"count":2,"href":"https:\/\/www.allerin.com\/blog\/wp-json\/wp\/v2\/posts\/14902\/revisions"}],"predecessor-version":[{"id":14906,"href":"https:\/\/www.allerin.com\/blog\/wp-json\/wp\/v2\/posts\/14902\/revisions\/14906"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.allerin.com\/blog\/wp-json\/wp\/v2\/media\/6464"}],"wp:attachment":[{"href":"https:\/\/www.allerin.com\/blog\/wp-json\/wp\/v2\/media?parent=14902"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.allerin.com\/blog\/wp-json\/wp\/v2\/categories?post=14902"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.allerin.com\/blog\/wp-json\/wp\/v2\/tags?post=14902"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}