An RGBD Dataset of American Sign Language Homework-Assignment Videos by Students and by Fluent ASL Signers

Abstract

This dataset contains a video-recording corpus of students (and fluent signers) performing ASL phrases and sentences, as a part of homework assignments. This new dataset was collected to support the design of technologies to fundamentally advance partial-recognition of some aspects of ASL. For example, identifying linguistic and performance attributes of ASL without necessarily determining the entire sequence of signs, or automatically determining if a performance is fluent or contains errors made by ASL students. This research effort was aimed at enabling future computer-vision technologies to support educational tools to assist ASL learners in achieving fluency, with an automatic system providing feedback on their signing. This is a novel dataset that has been collected as a part of a collaborative project between Rochester Institute of Technology, The City College of New York, and Hunter College. This material is based upon work supported by the National Science Foundation under award Nos. 1462280, 1400802, and 1400810. Access: After becoming an authorized user of Databrary, please contact Matt Huenerfauth if you have difficulty accessing this volume. Organization of the Dataset: The dataset is organized into sub-folders, with codenames such as "P01" or "F02" etc. These codenames refer to specific human signers who were recorded in this dataset. Participant codes beginning with "P" refer to non-fluent students learning ASL, and participant codes beginning with "F" refer to fluent ASL signers. Folders with names beginning with "H" refer to specific homework assignment prompts that participants were responding to. Sub-folders with names beginning with "U" refer to individual utterances which were recorded in response to that prompt; for instance, some of the homework assignment prompts asked participants to produce several short videos. Filetypes: *.eaf: The videos were annotated using ELAN. Please see the detailed instructions and annotation guide document provided with this dataset release for documentation of the various tiers of annotation. *.mp4, *_dep.bin: The dataset has been captured by using a Kinect 2.0 RGBD camera. The output of this camera system includes multiple channels which include RGB and depth data. The video resolution produced is 1920 x 1080 pixels for the RGB channel and 512 x 424 pixels for the depth channels respectively. Due to limitations in the acceptable filetypes for sharing on Databrary, it was not permitted to share binary *_dep.bin files directly produced by the Kinect v2 camera system on the Databrary platform. If your research requires the original binary *_dep.bin files, then please contact Matt Huenerfauth.