Swift: Turning Your Hand into a Writing Pad with Unmodified Smartwatches

Abstract

Smartwatches remain constrained by small touchscreens. To enable expressive text input without extra hardware, we introduce Swift, an on-skin writing technique that treats the back of the hand as a passive handwriting pad to capture handwritten English letters and gestures with an unmodified commercial smartwatch. The core design rationale is that Swift measures high-frequency finger-on-skin friction acoustics and complements them with active ultrasonic probing to better recognize characters with similar stroke trajectories. A lightweight cross-attention fusion model is designed to reliably integrate the two modalities, while robustness is ensured via various augmentation techniques. Across an offline technical evaluation in lab and everyday environments (N=22), Swift achieves a macro F1-score of 0.91 after leave-one-subject-out training and few-shot personalization. A real-time deployment study (N=8) demonstrates phrase composition on the watch with users reporting the system as learnable and engaging. A Wizard-of-Oz comparison study (N=8) further indicates users generally prefer Swift over touchscreen handwriting, with preference increasing for more demanding scenarios such as heads-up text entry.