The availability of web-scraped and scanner data sets provides central banks with unprecedented access to real-time data on individual product prices. However, to use these data for inflation nowcasting and forecasting, analysts need to classify products according to statistical conventions. In the absence of reliable, scalable classification methods, inflation analysts are flooded with data but lack actionable insight.
Product classification at the scale of web-scraped data represents a major challenge. Manually processing this amount of data is not feasible. Classification using large language models (LLMs) is promising, but with the LLM models currently available, the processing time and cost become prohibitively high. Project Spectrum used the European Central Bank’s Daily Price Dataset (DPD), which contains billions of price-product daily observations for 34 million unique products. At the time of writing, classifying this data set using GPT-5 would take over six months of computing time at a cost exceeding EUR 0.5 million.
Project Spectrum – a collaboration between the Bank for International Settlements (BIS), the Deutsche Bundesbank and the European Central Bank – explored an alternative approach where artificial intelligence (AI) was used only to transform product descriptions into high-dimensional text embeddings. These were then classified into product categories using classic machine learning algorithms. Text embedding is a foundational AI technique used by many natural language processing applications. This method achieved accuracy levels comparable to LLM prompting, but at a fraction of the cost: the entire DPD was classified in just five days for approximately EUR 1,500.
Besides classifying all records in the current DPD, the project has developed a production pipeline solution that can classify new products as they are added to the DPD. In addition, to ensure continuous improvement, an iterative algorithm was implemented to gradually expand the reference data set. By selectively adding manually labelled data to the reference and validation sets, this algorithm systematically refines the classification logic, enhances overall predictive accuracy and adapts to a changing product range.
By turning raw, fragmented product descriptions into structured data, Project Spectrum equips analysts and policymakers with timely, detailed insights into price developments. Ultimately, the project contributes to an emerging new generation of AIpowered analysis, where data abundance can be translated more easily into actionable economic understanding.
This report is intended for monetary policy analysts who utilise high-frequency data for inflation nowcasting and data scientists within central banks looking for cost-effective alternatives to LLMs for large-scale classification. It also serves as a technical reference for statistical agencies seeking to automate the categorisation of scanner and web-scraped data into official indices. Finally, it provides a methodological framework for economic researchers studying price-setting behaviour at the individual product level