U.S. Bureau of Labor Statistics

BLS connector: the US CPI for All Urban Consumers, at 400 item codes.

BLS publishes the CPI as plain tab-separated flat files that need no key and no registration -- unlike its v2 JSON API, which needs one, and its v1, which is capped at 25 queries a day. The files carry the whole item structure, from "All items" down to "Bananas" and "Gasoline, unleaded regular".

723 series and 433,828 observations: 400 item codes for the U.S. city average, not seasonally adjusted, plus the 323 of them BLS also seasonally adjusts, with history reaching 1913.

Six things this connector had to get right.

cu.data.0.Current is a trap, and the obvious file to reach for. It is the one file containing every series, 48 MB, and it holds only 1997 onward. Measured on the all-items index: 384 monthly values there against 1,476 in cu.data.1.AllItems, which is 84 years thrown away by choosing the convenient file. The eleven topic files in catalog.BLS_DATA_FILES carry the full history instead, and fetch() verifies they cover every series in scope rather than trusting that they do.

M13 is not a thirteenth month. BLS writes the annual average into the period column as M13 -- its own documentation says so, in one line, in cu.txt. A parser reading M13 as a month files the year's average under a date that does not exist, twelve times too high in the count and wrong in the level. It is taken here as a published annual series in its own right, which is why one item code yields both an .M and an .A series.

The data files are not self-describing. A data row is series id, year, period, value -- no name, no units, no base. Everything else lives in cu.series and cu.item, so parse() reads those from the snapshot directory beside the data file it is given. That is a deliberate deviation from the other connectors, where one file is one complete answer; the alternative was to invent labels from the series id, which would lose the item names that make 400 codes searchable at all. It stays offline, pure, and reproducible from the archive, which is what the guarantee actually requires.

BLS is not COICOP and must not pretend to be. Eight major groups against COICOP's twelve, with different boundaries -- COICOP separates restaurants from food, BLS puts shelter and furnishings together under housing. The item codes are kept verbatim under taxonomy: 'bls'. The one equivalence asserted is the all-items aggregate, SA0, which carries coicop: '_T' like every other headline. The United States' COICOP divisions come from the IMF, are already held, and are current -- checked before this was written, so nothing is lost by declining to map.

Both seasonal adjustments, and the id has to say which. BLS publishes 400 items unadjusted and 323 of them adjusted, for the same item code and the same month. They are different series and neither is a correction of the other, so the seasonal fragment sits in the concept id, before the variant so that catalog.break_span still reads INDEX off the end.

Every series carries its own base, and they differ: 1982-84=100 for most, DECEMBER 1997=100 for 377 of them, and five other bases besides. Recorded per series as reference_period, because an index level means nothing without it and serve.py rebases on read anyway.

What is deliberately left out:

3,901  semi-annual series        periodicity 'S', and this system's
                                 frequency vocabulary is A, Q and M. There
                                 is no honest code for it and inventing one
                                 would touch every consumer of the field
3,944  the other 57 areas        regions, size classes and metro areas.
                                 BLS itself warns these "cannot be used to
                                 measure interarea differences in price
                                 levels", and they are breadth rather than
                                 the depth this connector was for. The area
                                 is already in the concept id, so adding
                                 them later renumbers nothing
   85  alternate-base series     the same concepts rerun on an old base
                                 under their own item codes (AA0 for SA0),
                                 which serve.py's rebasing makes redundant
    2  purchasing power of the   the reciprocal of an index, in dollars --
       consumer dollar           a level wearing an index's shape