MODEL AND METHOD OF INCLUSIVE INTERACTION WITH VISUAL AND TEXTUAL CONTENT BASED ON CONTEXTUAL GESTURE INTERPRETATION

Abstract

Background. Contactless gesture control, as an auxiliary channel of interaction with an interface, has now become widely used in various areas of life, as it provides a convenient, safe, and, if necessary, inclusive way of interacting with visual, graphical, and textual content. The subject of the study is the process of contactless control of basic navigation scenarios for working with visual, textual, and web content using hand tracking methods and contextual interpretation of gesture events. The objective of the study is to develop and experimentally validate a method of contactless interaction with visual and textual content based on RGB hand tracking and contextual interpretation of gestures for implementing practical user scenarios. The main research task of the study is to develop a conceptual model of context-adaptive inclusive user interaction with an information system and the external environment. Methods. Functional modeling, computer vision methods, analysis of hand keypoints, geometric rules for gesture recognition, and experimental testing were used. Results. A method is proposed in which a gesture event is converted into a command only after determining the type and analyzing the context, namely the technical and scenario context. The experimental validation was carried out using five basic gestures and three scenarios. Conclusions. The obtained results confirmed the operability of the method as an auxiliary channel of contactless interaction with visual and textual content.