Abstract
A method generates illustrations of fingerspelled letters and other handshape-defined units of signed languages (Auslan, BSL, ASL) and checks each one automatically before it is accepted, so that accessible interfaces, accessibility testing and inclusive user research with Deaf and hard-of-hearing participants use correct signs. A target hand configuration is posed on a three-dimensional rigged hand model and rendered. A generative image model produces a candidate illustration conditioned on the render. A hand-pose estimator trained on synthetic line drawings of the same rig reads the articulation back. Target and estimate are compared in a sign-language phonological feature space: per-digit handshape, thumb, palm orientation, finger direction, finger count, chirality and a typed contact graph. Differences are tested against hard gates and per-unit tolerances learned from expert decisions and calibrated to bound false acceptance. Failing features select targeted regeneration. Moving letters are interpolated in joint space and checked frame by frame.
The problem this solves Sign-language illustrations are used every day in accessible websites and apps, onboarding and help content, education, and the materials used in accessibility testing and user research with Deaf and hard-of-hearing people. They need to be exactly right: in Auslan, BSL, ISL and ASL, a single wrong finger, thumb position or point of contact turns one letter into another. AI image generators are now widely used to produce these assets, but they are known to draw hands badly, adding or dropping fingers and confusing which hand is which. The errors are easy to miss for anyone who doesn't sign, so incorrect illustrations can reach products and research materials unnoticed. The only reliable check has been a fluent signer reviewing every image by hand, which doesn't scale and makes accurate sign-language content slow and expensive to produce. This method closes that gap. It checks each generated illustration against the intended handshape using the features that carry meaning in a signed language, accepts only images within tolerances set by Deaf reviewers, and fixes failures in a targeted way. The result is a practical route to accurate, inclusive sign-language content for accessible design, inclusive user research and testing with diverse and disabled people, while keeping Deaf signers in charge of what counts as correct.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
McDermott, Katherine; University of Sydney; Cao, Kejia; Cao, Kaiyi; Lu, Yushu; Yin, Donghai; Chen, Junyang; and Song, Yueyi, "Pose-verified generation of sign-language illustrations: closed-loop checking of hand articulation in a phonological feature space, with feature-targeted regeneration", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/12813