Sightation Counts: Leveraging Sighted User Feedback in Building a BLV-aligned Dataset of Diagram Descriptions

Created by MG96

External Public cs.AI cs.CV cs.HC

Statistics

Citations
0
References
0
Last updated
Loading...
Authors

Wan Ju Kang Eunki Kim Na Min An Sangryul Kim Haemin Choi Ki Hoon Kwak James Thorne
Project Resources

Name Type Source Actions
ArXiv Paper Paper arXiv
Abstract

Often, the needs and visual abilities differ between the annotator group and the end user group. Generating detailed diagram descriptions for blind and low-vision (BLV) users is one such challenging domain. Sighted annotators could describe visuals with ease, but existing studies have shown that direct generations by them are costly, bias-prone, and somewhat lacking by BLV standards. In this study, we ask sighted individuals to assess -- rather than produce -- diagram descriptions generated by vision-language models (VLM) that have been guided with latent supervision via a multi-pass inference. The sighted assessments prove effective and useful to professional educators who are themselves BLV and teach visually impaired learners. We release Sightation, a collection of diagram description datasets spanning 5k diagrams and 137k samples for completion, preference, retrieval, question answering, and reasoning training purposes and demonstrate their fine-tuning potential in various downstream tasks.

Note:

No note available for this project.

No note available for this project.
Contact:

No contact available for this project.

No contact available for this project.