Checking whether your data feeds deliver what you pay for

The researcher’s job runs every morning. It calls the vendor’s API, gets HTTP 200, parses the response, and finishes. On release day, it still returns last month’s observation. A file feed fails the same way: the prices file lands before six with its usual number of rows, and some of them repeat yesterday’s closing prices. Nothing in the job’s success status tells the researcher that the input to their analysis is missing.

A check that the file arrived or the API answered shows only that something was delivered. Checking data delivery requires an expectation about what the delivery should contain. For a financial dataset, that might be a scheduled release, the securities your subscription covers, or a revision the vendor has announced.

Check each feed the way your jobs receive it: the same request with the same entitlement, or the same file from the same location. Google’s SRE book calls this black-box monitoring: “testing externally visible behavior as a user would see it,” as opposed to white-box monitoring of a system’s internal metrics. Monitoring Distributed Systems It is the same idea as synthetic monitoring, which runs automated tests against a live production system on a regular basis, applied to the data in the delivery rather than only to whether it arrived. A vendor’s status page reports on the vendor’s own systems. It can show every stage healthy while a stale cache or a transfer that stopped partway leaves your copy short. A check that receives what your jobs receive sees what they see.

A team without these checks can learn of a bad delivery from a number: a signal that didn’t move on release day, or a backtest that gives a different answer after the vendor restated its history. By then no record shows what the feed contained when the job read it, and the conversation with the vendor starts from memory.

Consider a fictional activity index that your team receives through a data vendor’s API. The publisher releases its August value, 102.4, on time on September 3, 2026, at 12:30 UTC. The vendor’s documentation says releases are available through its API within two minutes of publication. The record, schedule, deadline, and observations below are invented to illustrate the method.

A missing release needs an expected record

Before release time, an empty response to the August query is ordinary. After the delivery deadline, the same response conflicts with what the vendor documents. A check needs both the request and the release calendar to tell those situations apart, and it needs to know what identifies the release. A recent response timestamp wouldn’t help if the body still contained the previous month’s data.

A check definition can carry all of that:

check: activity-index-august-2026
credential: monitor-standard       # the entitlement your research jobs use
request:
  path: /v1/observations
  params:
    series_id: activity-index
    period_start: "2026-08-01"
    period_end: "2026-09-01"
expect:
  observation_id: obs_aug26        # from the vendor's documented identifiers
  revision_number: 1
  value_from: publisher_release    # captured independently at publication
release:
  scheduled: 2026-09-03T12:30:00Z
  deadline: 2026-09-03T12:32:00Z   # the vendor's documented two minutes
schedule:
  - every: 30s
    from: 2026-09-03T12:29:00Z
    until: 2026-09-03T12:45:00Z
  - every: 15m                     # between releases

Each expectation needs a source other than the feed under examination. The identity comes from the vendor’s documentation. The value, 102.4 in index points, comes from the publisher’s own release, captured when it publishes. The response being checked cannot serve as its own answer key. Check each expectation against its source before the first scheduled run.

Source publication needs evidence too. A calendar shows when a release was scheduled, not that the publisher released on time. Keep the reference that confirms publication, with its time precision. Without it, a late delivery could be the publisher’s delay as easily as the vendor’s, and the finding should say that it can’t tell which.

Polling establishes observations at particular times

Suppose the checks run every thirty seconds around this release, and delivery is late. These are selected rows from an illustrative check history; all times are UTC on September 3.

The check at the deadline missed the record, and the next one returned it. For this credential, the record became retrievable after 12:32:00 and by 12:32:31; the thirty-second interval is the precision of the finding. A job with another entitlement, or served from another region, may have seen something else.

Each row of that history is a stored observation:

{
  "check": "activity-index-august-2026",
  "due_at": "2026-09-03T12:32:00Z",
  "started_at": "2026-09-03T12:32:00.104Z",
  "finished_at": "2026-09-03T12:32:01.187Z",
  "credential": "monitor-standard",
  "http_status": 200,
  "request_id": "req_7f3a91",
  "pages": 1,
  "traversal_complete": true,
  "returned": [],
  "outcome": "absent",
  "evidence": "responses/activity-index-august-2026/20260903T123200Z.json"
}

Request start and finish stay separate, because the server selected its data somewhere between them. credential names the entitlement profile, never the key itself. The response body, or a reference to it where retention is limited, lets an engineer inspect the last absence and the first presence without reconstructing them from a chart. outcome carries the distinctions the rest of this article depends on: present, absent, incomplete when the traversal could not finish, and request_failed. An absence becomes a finding only after the deadline.

Polling a file location bounds a file’s arrival the same way. The file’s modification time doesn’t: a transfer can carry over the time the vendor wrote the file, as sftp -p does, so it may predate the file’s arrival. A file can also be read before it’s finished. If the vendor uploads under a temporary name and renames the file when it’s done, or sends a marker file after it, check for that; otherwise treat a file whose size is still changing as not yet delivered.

Equal row counts can hide missing records

The release check follows one record. A daily file carries thousands. Suppose your vendor’s prices file should cover 3,100 securities, and today’s has 3,100 rows, as usual. A count passes. But one security, B, appears twice and another, C, is missing, so every job reading the file has no price for C.

Compare identities, not counts. List the identifiers the subscription covers, and report missing and repeated identities separately. The expected set needs its own source, such as the coverage list in your contract or the universe file you send the vendor. Yesterday’s file shows what changed; it can’t show what should be there.

For a file, the check definition names where it lands and where the expected set comes from:

check: daily-prices-2026-09-04
location: sftp://files.vendor.example/outgoing/prices_20260904.csv
expect:
  identities_from: coverage/2026-09.csv   # the securities the subscription covers
  key: security_id
  price_date: "2026-09-03"
release:
  deadline: 2026-09-04T10:00:00Z          # 6:00 a.m. in New York
  complete_when: marker_file              # prices_20260904.csv.done
schedule:
  - every: 5m
    from: 2026-09-04T09:30:00Z
    until: 2026-09-04T12:00:00Z

Repeated values need a softer test. A security whose price matches yesterday’s may simply not have traded. Count the rows unchanged from the previous delivery and compare that count with an ordinary day’s. A jump from a dozen to 212 is worth raising with the vendor, with both files attached, but it remains a suspicious change until the vendor confirms which prices are stale.

An API download spanning many pages has the same trap and one more: the pages come from a view that can change between requests. The traversal must follow the API’s documented snapshot and pagination rules, since combining pages from changing views can itself produce a discrepancy. If the third page fails, the check is incomplete; it cannot call every unseen record missing. Without an authoritative expected set, a drop in count is an observation to investigate, not a confirmed loss.

A finding from the prices file states that the file first found by the check at 5:55 a.m. New York time held 3,100 rows covering 3,099 securities, with B repeated and C absent, against the coverage list for September. It attaches the file and the list, and says that any job reading the file had no price for C. Why C was missing stays open until the vendor investigates.

Give each finding the command that reproduces its check, and once the vendor makes a change, add the later check in which the expected delivery arrived. Keep what the evidence shows separate from what it can’t, so the vendor can see where its own investigation needs to begin.

Your own record shows what the vendor changed

An announced revision changes the expectation. From September 10, when the publisher revises August to 102.1, a response containing 102.4 no longer satisfies a query for the current value. If the vendor’s API answers historical queries, one with a September 4 cutoff should still return 102.4. A check that merely flags changed values would mistake the announced revision for a defect.

A vendor that changes history without notice is different: values for past dates in today’s delivery differ from what an earlier delivery contained, and nothing in the feed marks the change. Only a record of the earlier delivery can show it. Keep each delivery as you received it, or the response behind each observation, and compare the history in each new delivery with what you already hold. Report each difference with both deliveries attached, and treat those that match an announced revision as expected.

The same record has a second use. A backtest run on the vendor’s current history uses values your team didn’t have at the time. Run on the deliveries you kept, it uses what you actually received. The timestamps that make financial data usable explains why a backtest needs each number as it was known. Check what your vendor contract lets you store before keeping full deliveries. Where it limits retention, keep the identifiers, times, and a hash of each delivered record. They show which records changed, though not what the earlier values were.

The monitoring can lose coverage too

A stopped monitor leaves no response to inspect. Its silence must remain visible as a gap in observation. A request that ran and received an error is a different event, and a successful request whose response lacked the release is different again. Collapsing all three into “data unavailable” would hide whether the monitor had evidence for that claim.

That is why each observation records due_at as well as when it ran. Compare the schedule with the stored rows, and a due time with no row is a period the monitor didn’t observe. An expired credential, a throttled request, and a process that never started all reduce coverage, though each calls for a different response from the operator.

Polling more often narrows the gaps between observations and uses more of the request allowance in your vendor contract. Schedule checks closely around an expected delivery and less often between deliveries. Retries need to fit that allowance too; a delayed traversal should not silently overlap several copies of itself. When the intended schedule can’t be met, the stored rows keep the actual check times.

Useful monitoring starts with a scope this concrete: the feeds checked, the deliveries and revisions expected during the period, the credentials or file locations used, and the observation period. Each added feed or delivery channel is another path to check. A report covering these feeds says nothing about the ones it didn’t check.

A vendor can run the same checks on its own delivery with a customer’s access. They show what its customers receive, which dashboards built on the vendor’s internal metrics can miss.

Checks like these, run on the feeds an investment team pays for, are the data delivery monitoring I set up. If you’ve found a late or short delivery only after a model used it, tell me about it.