# historicaldata.net - U.S. Stock Bars (daily and one-minute)

This file describes everything needed to read, interpret, and independently verify the data. It ships with every download so the data explains itself without reference to any website. Sections 1-9 describe the full product; a free sample zip carries its own opening section, generated from the sample's contents, ahead of this text.

---

## 1. What you have

Plain CSV files, no archive to unpack. The same bars are published in two arrangements; take either or both.

History starts <!-- COVERAGE_START:2003-10-01 --> **2003-10-01**, which is itself a trading day.
A security's own file starts on its first trading day, which is usually later.
Exchange test symbols (NTEST, CTEST, ZVZZT and their kin) are excluded from every file here,
including `symbols.csv`: the SIP feeds carry them, but they are not securities. The exclusion list
is fixed and is checked before every publish.

| Path | Contents |
|---|---|
| `day_by_symbol/{SYMBOL}_day.csv` | One file per currently listed security, its whole history. 17 columns, header row included. |
| `day_by_symbol/{SYMBOL}_day_delisted_{YYYY-MM-DD}.csv` | One file per delisted lifecycle. A symbol can be reassigned to a different company, so each life is a separate file. |
| `day_by_date/{YYYY-MM-DD}_day.csv` | One file per trading day, every security that traded that day. Same 17 columns plus `symbol`. **Rolling window - roughly the last 25 calendar days, not the whole history.** |
| `*/manifest.json` | The file list for that collection: name, byte size, SHA-256, and which trading days are missing. |
| `day_by_date/last-21-days_manifest.json` | The same, restricted to the trading days inside the last 21 calendar days. Subscribers fetch this one. |
| `symbols.csv` | One row per lifecycle: name, security type, exchange, delisting date, CIK, FIGI. |
| `README.md` | This document. |
| `update.py` | Keeps a local library current from your subscription. |

### File names

    {SYMBOL}_{grain}.csv                    A_day.csv
    {SYMBOL}_{grain}_delisted_{date}.csv    AACT.WS_day_delisted_2025-09-25.csv
    {date}_{grain}.csv                      2025-01-02_day.csv

`grain` is `day` or `minute`, and it sits in the same position in a listed and a
delisted name, so one rule reads either. It is part of the name and not only of
the directory, so the two granularities can be kept in one folder without
colliding.

**Underscore separates the fields; a symbol never contains one.** A symbol can
contain a **dot** - `AAC.U`, `EBR.B`, `AACT.WS`, 211 of them in this dataset. Strip
the literal trailing `.csv` (Python's `os.path.splitext` does this correctly - it
takes only the final `.csv`), then split the rest on `_`; never split on dots.

**Which one do you want?** `day_by_symbol` is the archive - a time series per security, covering the whole period, and the shape you want for backtesting one name. `day_by_date` is a cross-section per day, the shape you want for ranking the whole market on a given date.

**`day_by_date` only holds a rolling window**, roughly the last 25 calendar days. It is a delivery channel rather than a second archive: `update.py` merges each day's cross-section into a `day_by_symbol` library, so a subscriber downloads one small file a day and ends up holding the full history. Nothing is lost by the rotation - once a day has settled (see below) the raw columns in `day_by_symbol` never change, and a dated file's `adj_*` columns equal its raw ones by design, so any past cross-section can be cut back out of `day_by_symbol` exactly as it was published.

**One difference between them, and it matters.** In `day_by_date` the `adj_*` columns equal the raw columns. A cross-section file is a point-in-time record: it contains no information from after its own date, so a later dividend or split never rewrites it. Two things can still change while a day sits inside the window - its own record. First, the consolidated feed keeps revising a day's `volume`, `transactions` and occasionally `vwap` for about two trading weeks after it (late-reported and cancelled trades; prices do not move), so we refetch the last 15 trading days every night and republish any day whose rows changed. Measured across five years, nothing changes after that settling period - a day older than two weeks is final. Second, if the exchange revises or withdraws a dividend that fell on that date, we correct it, and the corrected file replaces the one you downloaded. That is why `update.py` re-checks the days you already hold against the published checksums instead of matching on file names alone. Once a day has rolled out of the window it can no longer be refetched, so corrections to it travel a different way: the dated manifests carry a `corrections` list (section 6) covering the last 90 days of them, and `update.py` applies it to your library. Adjustment looks backwards from today, so it lives in `day_by_symbol` (and in the library `update.py` builds for you, which it recomputes whenever a dividend or split arrives). If you need adjusted prices, use `day_by_symbol` or let `update.py` build it - do not read `adj_close` out of a dated file and expect it to be adjusted.

**Security type (`CS`, `ETF`, `PFD`, ...) is not in the file name.** Read it from `symbols.csv`. Do not infer it from the symbol.

Throughout this dataset the identifier column is named `symbol`, matching the
exchanges' own listing directories and FIX tag 55. Where you see the word
"ticker" elsewhere in the industry, it means the same thing.

---

## 2. Column dictionary (17 columns, fixed order)

| # | Column | Type | Meaning |
|---|---|---|---|
| 1 | `date` | date | Trading day, `YYYY-MM-DD`, U.S. Eastern. |
| 2 | `open` | number | Session open, as traded. Never adjusted. |
| 3 | `high` | number | Session high, as traded. |
| 4 | `low` | number | Session low, as traded. |
| 5 | `close` | number | Session close, as traded. |
| 6 | `volume` | number | Shares traded. Counts every trade, including those outside the regular session (section 7). Whole shares until 2026-02-22; fractional from 2026-02-23 (section 7). |
| 7 | `vwap` | number | Volume-weighted average price for the day. |
| 8 | `transactions` | integer | Number of trades. |
| 9 | `adj_open` | number | Column 2 adjusted for splits and cash distributions (section 4). |
| 10 | `adj_high` | number | Column 3 adjusted. |
| 11 | `adj_low` | number | Column 4 adjusted. |
| 12 | `adj_close` | number | Column 5 adjusted. |
| 13 | `adj_volume` | number | Column 6 divided by the row's adjustment factor - the **same** factor that multiplies the prices, cash distributions included - so adjusted price x adjusted volume equals raw price x raw volume. Six decimals. Derived from the `volume` printed here; section 4 shows the reproduction and its rounding. |
| 14 | `adj_vwap` | number | Column 7 adjusted. |
| 15 | `dividend` | number | Cash per share with this row's date as the ex-date. Blank when there was none. |
| 16 | `dividend_type` | text | Passed through from the source's classification: `CD` regular, `SC` special, joined with `+` when several fall on the same ex-date. A trailing `+CUR:XXX` is our own marker: column 15 is denominated in `XXX`, not USD - see section 4. **The regular/special distinction is reliable in recent years but not historically** - the source only began labeling `SC` consistently around 2023 (Costco's 2012-2020 special dividends all carry `CD`; the 2023 one carries `SC`). Amounts and ex-dates are unaffected, and the adjustment math treats both types identically, so `adj_*` is correct either way. Treat the label as advisory for older rows. |
| 17 | `split` | text | Split ratio written `from:to`, e.g. `1:4`. Blank when there was none. |

Rows are ordered by `date`, ascending. Files are ASCII with CRLF line endings and a header row.

Files in `day_by_date/` carry one extra column, `symbol`, inserted as column 2; columns 1 and 3 through 18 are the table above, in the same order. Rows there are ordered by `symbol`.

### Minute files (18 columns)

`minute_by_symbol/{SYMBOL}_minute.csv` uses the same 17 columns in the same order, with two differences:

- column 1 is `timestamp` instead of `date`: `YYYY-MM-DD HH:MM:SS`, the start of the minute for a priced bar, U.S. Eastern wall-clock time (EST or EDT as the date dictates);
- column 18, `session`, is appended: `pre` before 09:30, `reg` from 09:30 to 15:59, `post` from 16:00. On an early-close day the regular session ends at 13:00, so `reg` runs to 12:59 and `post` starts at 13:00. `event` marks a dated corporate action on a day with no minute bar; its `00:00:00` timestamp is a date marker, not a trade or a priced minute.

So the minute header is `timestamp,open,high,low,close,volume,vwap,transactions,adj_open,adj_high,adj_low,adj_close,adj_volume,adj_vwap,dividend,dividend_type,split,session`. A priced row describes one minute; a minute with no trade has no priced row. If a corporate action falls on a day with no minute bar for that security, an `event` row carries it once at `00:00:00`: all price, quantity and transaction fields are blank. This row is not an intraday observation. No minute bar does **not** prove zero daily trading: daily and minute aggregates use different rules, so a printed zero here would invent a fact. `vwap` and `adj_vwap` are **blank in every minute row** because the historical minute feed does not carry VWAP. An action on a day with minute bars appears only on its first bar. Minute adjustments use the prior regular-session minute close, which can differ from the daily close; the event date and amount agree with the daily file, but the two grains need not have identical numeric factors. `minute_by_date/{date}_minute.csv` inserts `symbol` as column 2, like the daily cross-sections.

Every column answers to one test: an algorithm consumes it. `dividend_type` is here because
adjustment conventions branch on it - excluding special distributions from a capital-events-only
series, or from a dividend-yield factor, requires knowing which is which.

Recorded cash dividends are included on their ex-dates in the per-security CSV files.
Payment dates, record dates, declaration dates, currency and payment frequency are not
published. No separate dividend master table is included.

---

## 3. The blank rule - read this before writing any parser

> **A blank field means "not available". It never means zero.
> We never write 0 to stand in for a missing value.**

This is the central design commitment of this dataset, and it is the same commitment across every product we publish. Where an upstream source uses `0` as a missing-value marker, we detect it and write a blank instead.

| Column | Blank means | It does **not** mean |
|---|---|---|
| `dividend`, `dividend_type` | No ex-dividend on this date | A distribution of $0.00 |
| `split` | No split on this date | A 1:1 split |
| `vwap`, `transactions` | The value was not carried for this bar | No trading occurred |
| `adj_*` | The adjustment could not be derived for this row | An adjustment factor of 1.0 |

If you require a numeric column, substitute your own sentinel deliberately - do not assume blank can be read as 0. A blank `dividend` and a `dividend` of 0.000000 are different statements: the first says nothing happened, the second says a distribution of zero was declared.

---

## 4. How the adjusted columns are computed

Each adjusted column is the raw column multiplied by one factor per row: the product of every corporate action dated **after** that row.

| Action | Factor |
|---|---|
| Split, written `from:to` | `from / to` - a 1:4 split gives 0.25 |
| Cash distribution | `(P - D) / P`, where `P` is the close on the last trading day before the ex-date and `D` is the cash per share on the ex-date |
| Volume | `adj_volume = volume / f`, where `f` is the row's factor above - splits **and** cash distributions - so `adj_close x adj_volume = close x volume`. Written to six decimals. |

For minute archives, the cash-distribution reference price is the last recorded
`reg` close before the event date. Pre-market and after-hours trades do not
replace it; a date without a regular-session trade leaves the last available
regular close in effect. Daily archives use their own daily close. These prices
can differ, so the two granularities can have different adjustment factors.

Nothing else is applied. No smoothing, no back-filling, no vendor-specific conventions.

**Reproducing `adj_volume`, and how close you can get.** KO on 2022-09-15, an
ex-dividend date (0.44 cash) with no split anywhere in the file:

    volume 11451929   adj_volume 12790928.203067   close 59.53   adj_close 53.2982

The factor is the product of every later distribution, and the printed prices
reproduce it only to four decimals: `adj_close / close = 0.8953166...`, and
`volume / 0.8953166 = 12790926.02` - within 2.2 shares (1.7e-7) of the printed
`adj_volume`. That residue is the rounding of `adj_close` to four places, not a
different formula: `adj_volume` was computed with the full-precision factor. It
is a real distribution effect - the file holds no split, so the split-only reading
of this column would predict `adj_volume = volume`. `verify.py` checks whether
one factor fits the printed rounding intervals of every adjusted price and the
six-decimal adjusted volume; it does not treat the rounded `adj_close / close`
ratio as the exact factor.

Two cases are worth stating outright, because they look like errors and are not:

- **Several actions on one day.** Cash distributions sharing an ex-date are summed and
  produce a single factor, `(P - D1 - D2) / P`. Splits sharing a date are written joined
  with `+` (`3000:1+1:3000`) and their factors multiply. That pairing is the ordinary shape
  of a going-private transaction and its net effect is 1, so the prices correctly do not move.
- **A distribution larger than the price.** A liquidating payout can exceed the prior close,
  and `(P - D) / P` would then be zero or negative. No positive factor exists, so the
  distribution is written into the row but **not** applied to earlier prices. The event stays
  visible; the adjustment is the thing that cannot be defined.
- **A distribution declared in another currency.** Cross-listed companies - Canadian banks
  and pipelines, mainly - declare in their home currency while their shares here trade in
  USD. Column 15 then holds the declared amount and column 16 carries a `CUR:` tag, e.g.
  `CD+CUR:CAD`. `P` is in USD and `D` is not, so `(P - D) / P` is not a number we can form,
  and the conversion actually applied to holders is done by the depositary at the pay date,
  at a rate with a spread, which is not knowable on the ex-date.

  **A row's factor is the product of every event after it**, so one unknown factor makes every
  earlier row unknown as well. Those rows are published with **blank** `adj_*` columns. This is
  deliberate and it is the same commitment as section 3: a number that quietly omitted one
  term would look entirely ordinary, and you would have no way to notice. Where the adjusted
  columns are blank, use the raw prices and apply your own FX series.

  Where an upstream record exists in both the declared currency and USD for the same
  distribution, we use the USD one and adjust normally; no tag appears.

Every factor is re-derivable from the `dividend` and `split` columns inside these files, and `verify.py` does exactly that: it recomputes the adjustment chain from the file's own contents and compares it against the `adj_*` columns. **The arithmetic can be checked here, without trusting us and without network access.**

---

## 5. Trading days, and how to tell a gap from a holiday

We do not ship a general trading calendar. It is public reference data available from several maintained libraries, and a data vendor reprinting it adds nothing. What only we can tell you is where **our** data has holes, and that is in the manifest.

There are four cases, and the manifest separates them:

| Case | A file for that day? | Where it is declared |
|---|---|---|
| Market closed (weekend, holiday) | No | Nowhere - it was never expected |
| Trading day, but we received no data | No | `trading_days_missing` |
| Normal | Yes, in `files` | `files` |
| Market closed, but a corporate action is dated that day | Yes, **event rows only** | `files` *and* `event_only_days` |

So a date with no file and no entry in `trading_days_missing` was a market holiday, and you can stop looking. A date in `trading_days_missing` is a real gap in what we hold, stated plainly rather than smoothed over.

**No file is ever written with a header and no rows.** If your loop reads a dated file, it has rows.

### The fourth case, and why we do not quietly fix it

An exchange cannot set an ex-date on a day it is closed, so when a source dates one there, something may be wrong upstream. The days are listed under `event_only_days` in the relevant `day_by_date` and `minute_by_date` manifests. The largest historical cluster is 2012-10-29, when Hurricane Sandy closed the market after ex-dates had already been announced; that is not itself a data error.

We considered moving such an event to the neighbouring trading day and measured whether we could tell which one. We could not: reconstructing each issuer's own ex-date and record-date convention decided a handful in each direction and left most of them undecidable. Any single rule would silently move a portion of them the wrong way, so **the date is left exactly as the source recorded it**, and the rows for that day exist so the event still reaches you. They are event rows only - never a price - and the day is listed in `event_only_days` so a trading-day loop can skip it while a merge tool still picks the event up.

One consequence, stated plainly: for whichever of these events truly belonged to the *preceding* trading day, that one preceding bar carries one adjustment too many. Correcting them would require guessing which ones, so we do not. Every day involved is named in `event_only_days`, so you can locate the affected bars yourself and exclude them if your work cannot tolerate them.

---

## 6. What is in `manifest.json`

Each collection has its own manifest, and it describes **files**, not their contents:

```json
{
  "product": "stock_day_by_date",
  "files_count": 405,
  "bytes_total": 490286907,
  "trading_days_missing": [],
  "corrections": [
    { "symbol": "GBDC", "date": "2024-08-16",
      "dividend": "", "dividend_type": "", "split": "" }
  ],
  "files": [
    { "name": "2025-01-02_day.csv", "bytes": 1092870, "sha256": "d99c6c..." },
    { "name": "2025-01-08_day.csv", "bytes": 1092709, "sha256": "d9c24a...",
      "delisted": ["CPTN", "CPTNW"] }
  ]
}
```

Every field passes one test: **can you know it without opening the file?** Row counts and date ranges fail it, and they are not republished anywhere - they are in the files themselves, one pass away. Security type, exchange and the identifiers fail it too but are properties of the security rather than of our data, so they live in `symbols.csv`. What remains here is what you need in order to download and to check: the name, the size, and the digest.

- `delisted` appears on a day when one or more securities were delisted as of it. It is the exception that proves the rule - it is *not* in the file, so opening the file will not tell you. `update.py` uses it to seal `{SYMBOL}_day.csv` as `{SYMBOL}_day_delisted_{date}.csv` at the correct point in time, which matters if you fall behind: a symbol reassigned to a new company while you were away must not have both companies' rows merged into one file.
- `trading_days_missing` is scoped to the days that manifest actually covers, so the dated manifests speak only for the rolling window. The same field on `day_by_symbol/manifest.json` covers the **whole** period - that is where to look for a gap in the archive.
- `corrections` appears on the dated manifests and carries the last **90 days** of corrections to days that have already rolled out of the window. Each entry is the three event columns as we publish them **now** for that security and date - a state, not an instruction, so applying the whole list again changes nothing. An entry whose three columns are empty means the exchange withdrew what it had announced; if that day carried no trading, the row itself goes. `update.py` applies these and recomputes the affected security's adjusted columns. This is the one thing a dated file cannot tell you: it stops being downloadable after about 25 days, so a revision after that would otherwise never reach a library you maintain yourself. Run at least every 90 days and you stay in step; if you have been away longer, re-download the securities you care about rather than trusting the merge.

There is deliberately **no version number, no build timestamp and no generator string**. A version field is a bill paid by every reader for a future that may not arrive; the absence of one is itself the marker of this format. To tell two builds apart, compare `sha256`.

### `symbols.csv`

One row per **lifecycle**, not per symbol - a reassigned symbol has one row per life.

Every column is a property of the **security**. None of them describes what we hold. That is why a
single file serves both collections: nothing in it changes with the grain you bought.

| Column | Meaning |
|---|---|
| `symbol` | The symbol as of the end of this lifecycle. |
| `symbol_history` | Every symbol this company traded under, `SYMBOL:from_date` joined with `\|`, oldest first, current one included. Example: `FB:2012-05-18\|META:2022-06-09`. Blank means **no symbol change is recorded for this security** - usually because there was none. It is not a guarantee that none ever happened. |
| `name`, `type`, `exchange` | Company name, security type (`CS`, `ETF`, `PFD`, ...), listing exchange. |
| `status` | `active`, `delisted`, or `renamed`. A `renamed` row is a **previous holder** of the same symbol that left it by renaming rather than delisting (Alcoa Inc handed AA to Alcoa Corp in 2016 this way). Its rows live in the current holder's file - there is no delisting date to name a separate file by - so this row exists to let you look the handover up: the identity here is the old company's, and the takeover date is in the current holder's `symbol_history`. |
| `delisted_at` | Delisting date, blank when still listed. |
| `cik`, `figi` | Regulatory and instrument identifiers, for joining to other datasets. |

**There is no file-name column.** Build the name from this row and the grain you want:

    delisted_at empty  ->  {symbol}_{grain}.csv
    otherwise          ->  {symbol}_{grain}_delisted_{delisted_at}.csv

`grain` is `day` or `minute`. `symbol` plus `delisted_at` identifies one lifecycle, which is exactly
what the file name encodes, so the result is exact rather than a guess. Whether that file is
published is a separate question, and the manifest of that collection is the authority - the minute
collection covers fewer securities than the day collection.

**There are no coverage columns either** - no first date, last date, or row count. Those describe our
data, not the security, so they would differ between the two collections while this file does not.
They are in the files you downloaded: one pass gives you all three, computed from the data rather
than taken on trust.

---

## 7. Known limitations

Stated plainly, because you should know them before building on the data:

1. **Daily volume and the sum of minute volume do not agree, and cannot be made to.** They are
   built by two different pipelines. Daily bars follow the end-of-day aggregate guidelines published
   by the Securities Information Processors; minute bars are assembled from individual trades under
   the intraday rules that decide which sale conditions may update a bar. Our upstream states plainly
   that the sum of intraday bars may not match the daily bar and that this is intentional.

   Measured here on 2026-08-14: adding up **every** minute bar of the day - pre-market, regular
   session and post-market together - reaches 21.6M for AAPL, 12.2M for MSFT and 5.9M for KO, while
   the daily bars for the same securities read 28.2M, 16.2M and 9.2M. The shortfall is 23% to 35%,
   and it is not explained by session coverage: the extended-hours bars are already included in
   those sums.

   Practical consequence: use daily volume for daily work and minute volume for intraday work, and
   do not treat either as a check on the other. Different vendors also resolve these rules
   differently - two authoritative sources we compared disagreed with each other by 17% on the same
   day - so a volume that differs from another provider is not by itself evidence of an error.
2. **`volume` becomes fractional on 2026-02-23, and that is a real break in the series.** Until 2026-02-20 a trade with a fractional component had to be reported to the tape as a whole number of shares, so every volume in this dataset before that date is a whole number. FINRA's fractional-share reporting took effect on Monday 2026-02-23, and the consolidated tapes began disseminating fractional quantities to six decimal places; from that day about 85% of securities show a fractional volume on a typical day. We publish what the tape publishes and do not round it away - rounding would hide a real change behind a second, invented one. The difference is always under one share, so it is immaterial to almost any calculation, but if you compare turnover across that boundary you are comparing two definitions.
3. **Securities with no trading inside the covered window are not included.** Absence from this dataset means "no bars in this window", not "never existed".
4. **Adjustment can only reflect the corporate actions recorded in these files.** Two different failures hide behind that sentence, and they are worth separating. The first is us losing an event between our source table and your file; we reconcile the two - ex-date by ex-date, amount by amount - and that is our job, not yours. The second is an event our source never carried at all, and that one cannot be ruled out from this data by anyone: a file and the table it was built from say the same thing, so publishing the table alongside it would prove nothing it does not already prove. It is why `verify.py` reports a dividend-history warning when a security's payments stop or change cadence - a heuristic, offered as one, rather than a proof we cannot give. Pay date, record date, declaration date, currency and frequency are **not published**; they take no part in any adjustment formula, and this dataset carries what a calculation uses.
5. **A reassigned symbol is split across files, and matching them up is your decision.** `ANET_day_delisted_2024-12-03.csv` and a later `ANET_day.csv` may be unrelated companies. `symbols.csv` carries the CIK and FIGI needed to tell, and `symbol_history` covers the other direction - the same company under an earlier symbol.
6. **Early closes are not marked in daily files.** Half-day sessions are ordinary trading days there; nothing in a daily row distinguishes a 1pm close from a 4pm one. Minute files do carry the boundary: on a half session `reg` ends at 12:59 and `post` begins at 13:00 (section 2), and `verify.py` checks it against the exchange's early-close list.
7. **The same action can give different daily and minute numeric factors.** Both files carry the event date and amount, including a non-trading `event` marker in minute history where needed. The dividend factor uses the last regular-session minute close in a minute file and the published daily close in a daily file; those closes can differ. Compare events by date and amount, then verify each factor against its own stated prior-close convention. Event markers never invent a price and do not change raw bars. An unavailable distribution factor leaves earlier adjusted values blank in both grains.
8. **Securities that distribute in a foreign currency have no adjusted prices before their most recent such distribution.** Section 4 explains why. This affects cross-listed names - Canadian banks, pipelines and miners in particular. The raw columns are complete and unaffected; only `adj_*` is blank, and `dividend_type` tells you exactly which distribution caused it.

---

## 8. Verifying what you received

Every collection ships a `manifest.json` listing each file's byte size and
SHA-256. It is the check that matters: it proves the bytes you hold are the
bytes we published, offline, years from now. `verify.py`, included here, does
that check and goes further: it tests every sentence in section 9 against every
row and re-derives the adjustment chain from each file's own dividend and split
columns, without trusting us and without network access. It needs only Python 3
(3.7 or later, standard library) and reads nothing outside the paths you give it.

Run it from the directory that holds `verify.py`, naming the files, a directory
(searched recursively; a `manifest.json` found in it covers the files below it),
or a zip:

    python verify.py day_by_symbol/AAPL_day.csv       # one file
    python verify.py --strict day_by_symbol           # a directory you downloaded
    python verify.py --strict --extract .             # the free sample, unzipped
    python verify.py --extract stocks_sample_2022H2.zip

On Windows PowerShell use `python` (or `py`); on macOS and Linux `python3` may
be the command. Paths are the same on both.

Two switches decide how strict the reading is:

- `--strict` treats anything that could not be checked as a failure: a column a
  check needs is absent, no `manifest.json` covers the files, a file is present
  that the manifest does not list. Use it on files we delivered; leave it off
  when the folder also holds other data.
- `--extract` declares that the files are a slice of a longer history, as the
  free sample is. Corporate actions after the slice still apply inside it, so I8
  (adjusted equals raw after the last corporate action) is reported with the
  final `close / adj_close` ratio instead of enforced; every other invariant is
  enforced exactly as on a full file.

The report names each section 9 invariant it checked, by number, per file, and
counts data files separately from manifest and archive reports. Exit code 0
means every check passed; 1 means at least one failed; **2 means nothing was
verified** - no data file of ours was found - which is not a pass.

`update.py`, also included, applies the same arithmetic when it maintains your
library, and runs `verify.py` over whatever a run wrote, so the check happens
whether or not you remember to ask for it.

## 9. What every file guarantees

Below is everything we promise about the contents of a delivered file. It is a
closed list: `verify.py` carries this table verbatim and checks each sentence on
every row of every file, reporting each by its number; nothing here goes
unchecked, and its report says which numbers applied to the file at hand (I13 to
I15 are about minute files, I4 about daily files). The checksum, calendar and
adjustment-chain checks it also performs are reported separately, as "other
checks". A property not stated here is not a promise - it may hold today and
stop holding tomorrow without that being a defect.

Each was measured against the whole published archive before being written
down, not inferred from how the files are produced.

| # | Applies to | Guarantee |
|---|---|---|
| I1 | all files | Dates are written YYYY-MM-DD, and minute timestamps YYYY-MM-DD HH:MM:SS, in US Eastern time. |
| I2 | per-security files | Rows are ordered by date, ascending, with no repeated date. |
| I3 | all files | low <= open <= high and low <= close <= high. |
| I4 | day files | Every published price is greater than zero. |
| I5 | all files | volume is never negative. |
| I6 | all files | A row that carries prices carries a volume. |
| I7 | all files | On a row carrying prices, the adjusted columns are either all present or all blank. |
| I8 | per-security files | From the most recent corporate action onward, adj_close equals close, unless the row is marked ADJ:UNKNOWN. |
| I9 | per-security files | Rows dated before a distribution paid in a foreign currency carry no adjusted values. |
| I10 | all files | split is written from:to, with several on one date joined by +. |
| I11 | all files | Cash dividend and cash dividend type are either both present or both blank; ADJ:UNKNOWN is a separate availability tag. |
| I12 | all files | A published dividend is greater than zero. |
| I13 | minute files | A priced minute has session pre, reg or post; a non-priced corporate-action marker has session event, a midnight timestamp, blank prices and quantities, and a dated action or ADJ:UNKNOWN status. |
| I14 | minute files | Bars from 09:30 to 15:59 ET carry session reg, and to 12:59 on a half session. |
| I15 | minute files | Minute files carry no vwap: both vwap columns are empty in every row. |

Three of these read differently from what you might expect, and the difference
is the point:

**I6 does not have a converse.** A daily row can carry a volume and no prices.
On an ex-dividend date with no daily trading, its event row has `volume` 0
and blank prices. A minute `event` marker instead leaves volume blank: lack
of a minute bar does not establish that no trade occurred in the daily tape.

**I7 is scoped to rows carrying prices.** A zero-volume daily event row can
carry `adj_volume` 0 with adjusted prices blank. A minute `event` marker has
no observed quantity, so its `adj_volume` is blank as well.

**I9 says before, not on.** A row is adjusted by the events dated after it, never
by its own, so the row on the distribution date itself is unaffected and keeps
its adjusted values; the blanks begin with the row before it.

## Unavailable adjustments and reviewed corrections

`ADJ:UNKNOWN` in `dividend_type` marks a row whose adjusted values cannot be
certified. All six `adj_*` fields on that row are blank. This is an availability
tag, not a cash payment, split ratio, ex-date, or adjustment factor. A cash type
may precede it, for example `CD+ADJ:UNKNOWN`. Raw price and volume observations
are retained.

The accompanying `adjustment-status.json` identifies affected files and date
ranges, the missing evidence, and source references. A known spin-off without
a verified valuation makes earlier adjusted values unavailable. If the exact
ex-date is also uncertain, the reviewed historical range is conservatively
marked unavailable; no replacement ex-date is invented. A successful file
verification confirms the stated availability and arithmetic, not that a
missing company action has been fully repaired.

The current `update.py` preserves this tag and the missing adjusted values. In
a minute file, a tag on the first row of a day applies to that entire day. Older
copies of the updater or verifier may not understand this status; use the
scripts supplied with the corrected release. The download client does not
recalculate adjusted values.

For this historical repair, replace the affected complete per-security files
with the reviewed release before resuming incremental updates. The rolling
dated files and cash-correction list alone cannot retrofit old availability
intervals or move observations between security lifecycles. Availability tags
are excluded from cash/split correction comparisons, and later cash corrections
preserve an existing unavailable status.

Security identity is case sensitive. Some historical codes were transferred
from a parent to a different company. The identity intervals in
`adjustment-status.json` distinguish those securities; they do not certify a
continuous total-return bridge. Do not join predecessor and successor files
only because their symbol text matches. A reported distribution or a supplied
split encoding is not, by itself, an independently verified valuation factor.


<!-- MCP:BEGIN -->
## Use these files with an AI agent

`mcp.py` is an optional local MCP server for AI clients. The same script supports
stock daily bars, stock one-minute bars and the 34-column options format,
including stock files organized by date. It works with files you have already
downloaded; it does not sign in to an account or fetch paid data.

### Set up once

1. Install Python 3.10 or later. No additional packages are required.
2. Save `mcp.py` beside this README. Keep your CSVs in a dedicated data folder.
3. In an AI client that supports local MCP over stdio, configure Python as the
   command and pass the script path and data folder as arguments. For example:

```json
{
  "mcpServers": {
    "historicaldata-local": {
      "command": "python",
      "args": ["C:/data/historicaldata/mcp.py", "--root", "C:/data/historicaldata"]
    }
  }
}
```

Replace the paths with your own; macOS and Linux use the same arguments with
their local paths and Python executable (often `python3`). Configuration keys
depend on your client. The client launches the process and communicates through
standard input/output. No network port or separate terminal is needed.
To check the script version, run `python mcp.py --version`.

### What your agent can do

- `list_files`: list relative CSV filenames and sizes in the configured folder.
- `read_csv`: read one file, select columns, filter by an inclusive date range
  and up to four exact text matches, and return up to 100 matching rows per page.
- `summarize_csv`: calculate populated and missing counts, min/max, sums and
  arithmetic means for numeric columns. Group by one text field and/or by day
  or month. Empty numeric cells are excluded; all-empty aggregates are null.

Example requests (use the actual filenames returned by `list_files`):

- Stocks: "Show raw close, adjusted close and split in this TSLA daily file
  for 24-26 August 2022. Keep missing values visible."
- Minutes: "For 25 August 2022 in this TSLA minute file, summarize volume by
  session and show the populated and missing counts."
- Options: "From this daily options file, select SPY calls expiring on
  16 September 2022. Show contract, strike, bid, ask, IV and IV flag."

Each analysis returns the actual file SHA256, query and row counts. Later pages
must supply the previous hash as `expected_sha256`; a changed file requires
restarting from the first page. CSV values remain strings, including empty cells;
summaries use decimal arithmetic. Filenames and symbol case are preserved.
If the downloader assigned a numbered filename to avoid a Windows collision,
use that local filename and its supplied filename map. Do not infer a security
identity by stripping the number, joining reused tickers or uppercasing symbols.

### Scope and limits

The script reads one complete file per analysis, without modifying it. Default
limits are 512 MiB per file, 30 seconds per scan, 100 groups and 1 MiB per result.
An exceeded limit or a file changing during the scan returns an error rather
than a partial summary. Larger files can use explicit `--max-file-mb` (up to
4096) and `--max-seconds` (up to 300), subject to the client's own timeout.
Cross-file joins, chart generation and arbitrary code execution are not provided.

Calculations run locally, but requested rows and results go to your AI client
and may be sent to its model provider. Use an environment permitted by your
data license and policies. Do not put download credentials into prompts.
The tool does not extend download access or change your license.

A source hash identifies the bytes read; it does not prove current publication,
complete coverage or that previously reported data defects have been repaired.
Use the field definitions, adjustment notes and verification instructions in
this README when interpreting results. The MCP tool does not repair data or
replace `verify.py`.
<!-- MCP:END -->