OPAL: On-demand Progressive Accelerated Scientific Lossy Compression

Abstract

Scientific applications generate massive amounts of data, necessitating effective data reduction techniques like error-bounded lossy compression. While state-of-the-art error-bounded lossy compressors effectively reduce storage footprint, they typically treat the compressed data as monolithic blobs, forcing analysts to decompress large volumes just to access specific data modalities. Conversely, existing storage solutions designed for flexible access lack the efficiency of modern compression. To bridge this gap between efficient compression and flexible data retrieval, we propose OPAL, a unified system that harmonizes high-performance compression with flexible, on-demand data access. Our first contribution is the OPAL universal framework, an adaptive architecture that endows generic lossy compressors with on-demand retrieval capabilities, enabling flexible and progressive access across multiple spatial regions, resolutions, and precisions. Our second contribution is OPALI, a customized compression algorithm designed natively with flexible data access in mind. By leveraging novel mechanisms such as shadow anchors, hybrid storage, and grouped bit-plane coding, OPALI eliminates the overhead inherent in generic wrappers. Extensive evaluations on real-world scientific datasets demonstrate that OPAL transforms standard compressors into flexible retrieval engines, while OPALI achieves superior compression ratios and retrieval performance, significantly accelerating scientific discovery.