High-Velocity Data Ingestion
How Clickerwayne Zelle collects, verifies, and signs high-volume data across multiple collection workloads during a nightly ingestion window. Live deployment: Wholesale Dito Store.
Document Information
| Document version | 1.0 |
|---|---|
| Owner | Clickerwayne Zelle Solutions Inc |
| Ingestion window | 23:59 to 06:00 nightly |
| Processing model | Nightly batch processing |
| Live deployment | Wholesale Dito Store |
| Source of truth for live status | https://www.wholesaledito.store/provenance |
Scope
This document describes the ingestion system that collects data from external sources, normalizes it, and signs it with the Provenance Engine before it reaches a platform. It covers the nightly ingestion window, the crawler, the signing step, and the safety limits.
The system runs multiple collection workloads. The workloads themselves are not listed here. What this document covers is the part that is common to all of them: the window, the crawler behavior, the deterministic parsing requirement, the rate limits, the blocked address ranges, and the signing step. Price and product data are used as the worked example throughout because that workload is already publicly documented. The mechanics apply to every workload the system runs.
It does not describe the pricing, sourcing, or commercial agreements behind any source. It does not describe the Provenance Engine itself, which is documented separately.
The Ingestion Window
All ingestion work runs as a nightly batch between 23:59 and 06:00. Nothing outside that window competes with live traffic on the platforms being read or on the platforms being written to. Within the window, the system works through its source lists, verifies each record, signs it, and stages it for publication.
Each workload runs on a schedule. The end time of every workload is known in advance, and every workload is sized to finish inside the window. The schedule is fixed; the runtime of each pass depends on the number of sources and records in each workload.
Pipeline Stages
Every record passes through the same sequence. No stage is skipped, and no record is published without passing the signing step.
Stage 1: Source selection
The ingestion worker reads from fixed source lists, one per workload. Sources are added to a list by Clickerwayne Zelle, not discovered automatically. The lists are not public.
Stage 2: Collection
A headless crawler fetches each source page and parses the relevant fields. The parsing path uses pattern matching and DOM traversal. No generative AI touches the parsing path.
Stage 3: Normalization
Parsed fields are normalized into a common record shape. The shape is described in the Data Fields section below, using price and product data as the example. Other workloads use the same shape with their own field names.
Stage 4: Signing
Each record is signed with the Provenance Engine at the point of ingestion. The signature covers the record content and a timestamp. Signed manifests are staged for publication. The algorithm, canonicalization rule, and signature format are documented on the Provenance Engine page.
Stage 5: Publication
Signed records are published to the target platform. The public manifest is linked from the verification page so a third party can check the record against the published public key.
The Crawler
The collection stage uses a headless crawler. Two crawler profiles are documented here: SnifferBot for site auditing, and Sniff3rBot for collection workloads. Both use the same core approach.
Deterministic parsing
Requests are fetched and the response is parsed with pattern matching and DOM traversal. No generative AI, no inference, and no interpretation in the parsing path. The same input produces the same output.
Rate-limited
Every crawler is rate-limited. The limit is configured per crawler and is not shared across workloads. The published figure of one scan per minute and ten per hour per session applies to the audit crawler only. Collection workloads each have their own limit, set to match the source and the workload's tolerance.
Blocked address ranges
Private, loopback, link-local, and cloud metadata address ranges are blocked before any outbound request. The crawler cannot be pointed at internal infrastructure.
Verifiable User-Agent
Traffic from the audit crawler identifies itself as SnifferBot/1.0. Traffic from the collection crawler identifies itself as Sniff3rBot. Webmasters can verify both in server logs.
Full history kept
Every run is stored in the database for audit history. Display surfaces collapse to the most recent record per source. The history is not exposed publicly by default.
Signs at ingestion
The signature is generated when the record enters the system, not when it is published. A record that is signed but not published still exists as a verifiable manifest.
Data Fields
The table below shows the record shape using the price and product workload as the example. Other workloads use the same shape with their own field names.
| Field | Description |
|---|---|
| product_name | Normalized name of the product. |
| brand | Brand of the product, where available. |
| specification | Size, weight, or other unit-defining attribute. |
| price | Price as recorded from the source at the time of collection. |
| unit | Unit the price is expressed in. |
| source | Identifier of the source page. |
| region | Region the source is associated with, where applicable. |
| market_or_store | Specific market or store the price was collected from, where applicable. |
| collected_at | Timestamp of collection, in UTC. |
| signed_at | Timestamp of signing, in UTC. |
| signature | Base64-encoded DER signature from the Provenance Engine. |
| instance_id | Instance identifier for revocation checks. The revocation list is refreshed daily. |
Safety and Determinism
Two properties are maintained across every workload and every run: the crawler cannot be pointed at internal infrastructure, and the parsing logic is deterministic. Neither property is negotiable.
No generative AI is used to parse, summarize, or interpret a record. The parsing is pattern matching and DOM traversal. The same page produces the same record on every run. Determinism is a requirement for signing, because a signature over a non-deterministic record is not reproducible.
The rate limits and blocked address ranges described under The Crawler apply to every outbound request, across every workload. Rate limits are set per crawler, not shared. The blocked ranges are checked before any request is dispatched.
Failure Handling
When a source fails to respond, returns a malformed page, or returns a record that cannot be parsed, the ingestion worker aborts the record and backs off. The failed record is not signed and not published. The backoff and the abort apply per record, so a single failing source does not stop the rest of the workload.
Ingestion runs as a nightly batch. A workload that completes inside the window publishes its signed records in the normal way. The schedule for each workload is set so that the run finishes inside the window.
Operational Questions
When does ingestion run?
As a nightly batch between 23:59 and 06:00. The window is fixed. Each workload runs on a schedule and is sized to finish inside the window.
Is the data signed?
Yes. Every record is signed with the Provenance Engine at the point of ingestion. A record that has not been signed is not published.
Can a third party verify an ingested record?
Yes. The signed manifest is published alongside the record. A verifier uses the public key documented on the Provenance Engine page to check the signature. No account is required.
What sources are used?
The source lists are fixed and maintained by Clickerwayne Zelle. They are not public.
How does revocation work?
A manifest that is later found to contain an error is added to the revocation list. The revocation list is refreshed daily. A verifier checks a manifest's instance_id against the list before treating the manifest as current.
Boundaries
The following statements are deliberate. They describe what the ingestion system does not do.
- The signature proves that a record was issued by the holder of the private key and has not changed since it was signed. It does not prove that the data recorded is correct at the source. If a source published a wrong value and the crawler recorded it, the signature will pass.
- The signature does not prove that the live source page matches the recorded manifest at every moment. The source page can change between runs.
- The system does not infer, predict, or generate values. Every recorded value was read from a source.
- The system does not disclose its collection workloads. The document describes the mechanics that are common to all of them. It does not list which workloads run.
- The system does not guarantee that an AI system will cite an ingested record. Signing removes one class of doubt. It does not produce visibility.
- The system does not read from internal Clickerwayne infrastructure. The blocked address ranges prevent it.