More than a Typing Method: Rethinking the Usability Evaluation of Knowledge-Based Authentication in Virtual Reality

Abstract

Knowledge-based authentication methods for Virtual Reality (VR) are typically evaluated using performance metrics such as entry time and recall rate. While this practice creates an appearance of standardization, it may obscure meaningful procedural differences across studies and, more critically, fails to capture the complexity of real-world authentication. Drawing on findings from a remote and longitudinal study of VR authentication conducted in participants' everyday environments, we show that the commonly adopted objective performance metrics can be misleading: the method with the slowest entry time was perceived as the most efficient due to less failed attempts, and users adhered to significantly slower entry method even with faster alternative available. These findings highlight the limitations of laboratory-based, repeated-entry paradigms and motivate a broader reconsideration of how usability is evaluated in this space. We encourage usability evaluations in future research to emphasize subjective and behavioral measures due to better alignment with usability, and call for more ecologically valid study designs, including longitudinal and in-the-wild deployments, to better reflect authentic authentication experiences.