Prototype · Apache-2.0 · browser demo
ayaka-hand
A prosthesis control system that pre-shapes the grip before contact. A wrist camera looks at the object being reached for and forms the correct grasp on the way in.
The problem
Roughly 23 % of upper-limb myoelectric prostheses are abandoned in real-world use. A recurring reason is cognitive load: conventional control asks the wearer to consciously modulate muscle contractions to pick a grip and meter force — for every object, all day.
The split here is deliberate. The muscle signal is reduced to a single binary GO trigger. The wearer decides when to close. The vision system decides how.
How it works
Two tiers, teacher and reflex. A large vision-language model (Qwen3-VL-32B-Instruct, Apache-2.0) reasons about what an object is and what state it is in, and emits a structured grasp label. It is used offline, to label the training set. It is never in the real-time control loop.
What runs in the loop is a MobileNetV2 with two heads, distilled from those labels: 2,232,839 parameters, 8.9 MB of ONNX, executing in the browser through ONNX-Runtime Web. Its only input is wrist-camera pixels. Nothing tells it what it is looking at.
A grasp has to be committed inside 125 ms to feel like part of the body.
A cloud model cannot live in that budget, so it does not.
Results
The baseline is deliberately generous. It is handed the ground-truth object class name — a perfect detector — and looks up the canonical grip for it. The student gets only pixels.
| Metric | Lookup baseline · gets true class name | Student · 2.23M, pixels only |
|---|---|---|
| Grip accuracy | 70.5 % | 72.3 % |
| Grip macro-F1 | 0.581 | 0.659 |
| Grasping-force accuracy | 65.4 % | 81.1 % |
| Grasping-force macro-F1 | 0.595 | 0.806 |
The baseline scores 0.000 on pinch
No object class maps cleanly to a pinch grip — you have to look at the thing. A class name cannot get there in principle, which is why the baseline scores exactly zero F1 on that class. The student reaches 0.323.
Force gains 15.7 points
How hard to grip depends on an object's state, not its category. A full cup and an empty cup are the same class name and a different amount of force. Pixels carry that difference; a label does not.
What it is not
Not a medical device. This is a research prototype from a hackathon submission (AMD Developer Hackathon, Act II). No claim is made that it prevents injury, and none should be inferred.
No physical hand has been built. The demo renders a simulated hand in the browser. Everything measured here was measured against recorded imagery, not against a device on a person.
The hazard flags do nothing. The teacher model can emit advisory flags — hot, sharp, fragile. They are display-only. They are not distilled into the student and they are never wired to actuation. The system does not detect or prevent harm and must not be relied on to.
Data and licensing
Trained on 11,004 teacher-labelled COCO 2017 crops, filtered by licence: only CC BY, CC BY-SA and no-known-restriction images, with all NonCommercial and NoDerivatives material excluded. Validated on 613 held-out crops.
Code and documentation are Apache-2.0.