Institutional Books - Visual Elements: An open-source pipeline for extracting, classifying, deduplicating, and captioning visual elements from digital book collections
An open-source end-to-end pipeline for detecting, classifying, deduplicating, and captioning visual elements from historical book collections is introduced and an initial dataset of 22.6 million visual elements extracted from the Institutional Books: Harvard Library dataset is released.
Abstract
Historical book collections contain rich visual elements - such as illustrations, photographs, engravings, and decorative art - that are frequently under-explored in large-scale digitization projects. While Optical Character Recognition (OCR) has standardized the extraction of textual content, these visual components offer a layer of nuance and context that remains largely untapped by automated text extraction workflows. This technical report introduces Institutional Books - Visual Elements, an open-source end-to-end pipeline for detecting, classifying, deduplicating, and captioning visual elements from historical book collections. Alongside this pipeline, we release an initial dataset of 22.6 million visual elements extracted from the 983,004 scanned volumes that comprise the Institutional Books: Harvard Library dataset. This work contributes to ongoing, community-wide efforts to enable new use cases for digitized library collections through computational access, from artificial intelligence model training to digital humanities research.
Released in 2025, Institutional Books: Harvard Library (IB-HL) is a collection of 983,004 volumes (242B o200k_base tokens), originally digitized through Harvard Library's participation in the Google Books Library project. As researchers and developers have begun to use IB-HL, a tension has emerged between standard large-scale preprocessing practices and the goals of careful information stewardship. Many existing pipelines optimize for web text: as a result, they tend to aggressively filter, deduplicate, restrict by language, and sometimes discard meaningful metadata. Meanwhile, researchers seeking to use IB-HL duplicate effort while performing similar processing and analysis. We describe an approach that we call Enriched Text. Instead of producing a single'complete'stream of tokens, we normalize the text while preserving metadata through annotations. We separate endmatter, detect per-paragraph language, identify clusters of duplicate paragraphs, and compute per-paragraph bits-per-byte scores. We provide this information through HTML-like annotations layered on top of the text. By parsing these annotations, users can tailor the output to their own needs instead of accepting a global editorial decision on content. The pipeline applies to all $\approx$250 languages in the collection. This report describes this project's goals, implementation, and design rationale. The release includes IB-HL-ET (an enriched-text version of IB-HL containing 217B o200k_base tokens across 983,003 volumes, organized into 1.39B annotated subtopic paragraphs) and the pipeline that produced it. These serve to make the collection easier for machines to parse and for humans to study.
David Lowry-Duda, Matteo Cargnelutti, Catherine Brobston et al.· 0 citations
High-quality book metadata improves digital libraries by supporting cataloging, searchability and automated classification. However, metadata generation from scanned books remains challenging due to limited annotated data sets and optical character recognition (OCR) limitations. This study aims to propose a deep learning-based framework for automatic metadata extraction from digitized books by integrating computer vision and natural language processing (NLP).
The framework uses MobileNetV2 to classify title pages, table of contents (ToC) pages and content pages. EfficientDet detects metadata-related regions, such as titles and author information. OCR extracts text from these regions, followed by named entity recognition (NER) and regular expressions to refine the extracted metadata. The framework was evaluated using a custom data set of 188 books published between 1800 and 2021, comprising 857 annotated pages.
The page classification model achieved 97.16% accuracy. For object detection, the model obtained average precision (AP) and average recall (AR) scores of 71.7% and 42.0% for title pages, 55.1% and 33.4% for ToC pages and 87.9% and 54.4% for content pages, respectively.
This study contributes a scalable framework that combines page classification, object detection, OCR, NER and rule-based refinement to reduce manual metadata extraction in digital library management. Future research can expand multilingual data sets, improve robustness to OCR noise and explore end-to-end learning for richer bibliographic metadata extraction.
Amalia Amalia, B. Siregar, Khairul Imam· International Journal of Per...· 0 citations
An end-to-end approach for automated historical postcard indexing that integrates computer vision and natural language processing techniques is presented and effective integration of multiple AI techniques for automated heritage document analysis is demonstrated.
Matthieu Pélingre, Salvatore Tabbone· Journal on Computing and Cul...· 0 citations
The novel task of cross-modal query suggestion is introduced, which interactively guides users by suggesting textual refinements based on visual clusters identified in the retrieval results, and the creation of CroQS, a benchmark dataset comprising 50 diverse queries and 295 semantic clusters in generic domain.
Giacomo Pacini, Nicola Messina, Nicola Tonellotto et al.· 0 citations
An open, documented workflow for layout segmentation and Automatic Text Recognition of mid-twentieth-century typescripts from the Prefecture of Naples for TEI encoding and named-entity extraction is presented.
A methodologically sound, AI-assisted framework for unlocking complex, multimodal historical archives is proposed by proposing a methodologically sound, AI-assisted framework for unlocking complex, multimodal historical archives.