Instituto Brasileiro de Geografia e Estatistica

IBGE connector: the Brazilian IPCA, disaggregated to individual products.

This is the first source that goes below COICOP division level. The IMF gives twelve divisions; SIDRA gives the general index, 9 grupos, 19 subgrupos, 51 itens and 377 subitens -- reaching things like "Arroz" and "Produto para barba".

FOUR TABLES, one series. IBGE splits the IPCA by era, and each table is a window on the same thing rather than a different measurement:

1737   1979-12 ->            the headline alone, no categories, and the
                             only place IBGE publishes an index NUMBER
2938   2006-07 .. 2011-12    categories; no 12-month rate exists in it
1419   2012-01 .. 2019-12    categories
7060   2020-01 ->            categories

They feed the SAME series ids, because they are the same series. Each was validated before being wired in, and against something outside itself: 1737 against 7060's monthly rate over their 79-month overlap (0.000000 pp), and each category table against 1737 by aggregating its grupos with the monthly weights it publishes (worst 0.0067 pp, on a method calibrated at 0.0066 pp where the answer was already known). See catalog.IBGE_RESOURCES.

A table is named in a snapshot's file name by ONE LETTER, which can never be reassigned: an archived file must keep parsing to the table it came from.

Five things worth knowing before changing anything here.

SIDRA caps a request at 50,000 values. All categories across all periods is ~36,000 per variable, so it is one request per variable and the four cannot be unioned into one call. Asking for all four returns HTTP 400 with the count in the message, which is at least an honest error.

Row 0 of every response is a header row, a dict of column descriptions rather than data. It must be skipped, and it looks exactly like a data row to anything that does not check.

The category code lives inside the label, not the code field. D4C is SIDRA's internal id; the IPCA code is the prefix of D4N, as in "1101002.Arroz". The general index is the one exception -- D4N is just "Indice geral" with no prefix -- so a parser that assumes the prefix silently mangles the single most important series in the table.

No ETag, no Last-Modified, Cache-Control: no-cache, so the conditional request gate can never fire here either. The content hash is what stops an unchanged response from becoming a vintage. The cost of that is real: a poll downloads ~44 MB whether or not anything changed. Full history is fetched every time on purpose, because a short window cannot distinguish a revision from a period the source never mentioned.

fetch() emits its resources in sorted order, and that is load-bearing. ingest.py parses a snapshot in fetch order while rebuild.py replays the same directory alphabetically. Several of these tables describe the same series, and the last one parsed supplies its metadata -- so the two orders have to be the same order, or a rebuild stops matching live for a reason nothing in the data would explain.