Research repositories contain a large amount of scientific knowledge, but access to structured articles and specialised information, such as datasets or software metadata, remains limited. In this paper, we present the INRIA DataLake project, which provides an ecosystem of scalable and interconnected pipelines for preparing scientific literature, extracting structured information, and applying specialised treatments. Using a large-scale shared infrastructure, Grid'5000/ABACA, we demonstrate our ecosystem through a concrete use case: extracting software mentions from scientific articles deposited daily and visualising them after validation in the HAL research portal. Our results show that the system can efficiently process large volumes of scientific literature while supporting user validation and interoperability with external systems. Designed to grow by integrating additional pipelines and sharing the preparation effort across research groups, this project already contributes to open science through improved visibility and tracking of research software.
Luca Foppiano, Vipul Gupta, Samuel Scalbert et al.· 0 citations
Evaluations should do more than measure a models current performance. They should tell us what to fix for the next model iteration and provide a way to generate targeted post training data. Most evaluation pipelines identify weak examples, topics, or categories, but they leave the underlying capability failure implicit: they say where a model fails, not why. We introduce CRAFT, a method that converts any rubric based evaluation dataset into a model specific diagnosis of weak capabilities. CRAFT treats each grading criterion as a capability probe: it extracts a capability description from every prompt rubric pair, clusters these descriptions into a hierarchical capability tree, scores the target model at every node, and selects low performing nodes dynamically across tree levels, at the granularity where each failure is clearest. The selected weak capabilities then direct the generation of targeted supervised finetuning data. Holding the data generation, finetuning, and evaluation setup fixed, we compare CRAFT against prompt level EvalTree clustering and untargeted random generation on four open source models, two professional domains (finance and legal), and 13 held out benchmarks disjoint from the diagnostic data. CRAFT achieves the strongest finance domain average for all four models under repeated temperature decoding; on legal domain, it is strongest for three of four models and remains within the decoding variance bands of the best baseline on the fourth. Diagnosing weaknesses at the level of rubric criteria, rather than prompts or categories, thus yields both a sharper picture of what a model cannot do and measurably better models after finetuning on that diagnosis.
Vipul Gupta, Zihao Wang, Razvan-Gabriel Dumitru et al.· 0 citations
This work introduces an interaction-centric taxonomy that localizes failures to the interactions in which they originate and identifies the responsible component, and organizes 41 failure modes by assigning each to an edge between two components and a fault side indicating where the repair belongs.
Harsh Raj, Vipul Gupta, Anas Mahmoud et al.· 0 citations