Typeset Replacement of Handwritten Text and Mathematics on Lecture Slides Using Vision-Language Models

Abstract

University lecturers often annotate printed slides during class with handwritten notes and mathematical derivations. Generic OCR skips this ink or returns unrelated strings, and no public dataset fits the task: German handwriting corpora hold prose or historical script, handwritten-mathematics corpora are language-neutral stroke data, and none captures a single lecturer’s annotations where German text and free-form mathematics share a slide. We ask whether a general vision-language model (VLM) can outperform purpose-built OCR and handwritten-mathematics recognisers in this data-scarce setting, and drive a pipeline that typesets the recognised annotations back onto the slide. A YOLOv8x detector locates handwritten regions and labels them as text or math. Qwen3-VL-8B transcribes each class through a tailored prompt, and a LaTeX+dvipng stage renders the output into the original ink footprint. We hand-annotated 165 slides from one professor, evaluated every stage on held-out crops, and ran the pipeline over a 579-slide corpus. On handwritten mathematics the VLM reaches roughly an 18% character error rate, less than half the next-best system (PaddleOCR-VL) and far below SmolDocling, TrOCR, Pix2Tex and TAMER, which all exceed 80% raw CER; on German text it reaches under 6% with no fine-tuning. The pipeline renders 95% of detected math regions, and a 100-region audit judged 83% of rendered replacements fully correct. Two further findings: a warm-started detector beats a newer architecture trained from scratch, and meta-learning on out-of-domain handwriting fails to transfer, where a small adapter is cheaper. Code, configurations, evaluation scripts and annotated evaluation sets are released at github.com/ragav1n/ink2digital