Appendix — input
Wave a hand up and this whole page scrolls up; wave down and it goes down.
Each flick gives the page a push proportional to how hard you waved, and it coasts to a stop
— so your arm rests between gestures instead of holding a pose. Switch to
Hold for the throttle behaviour, where height above or below centre sets a steady
speed, or point the sensor at the demo panel instead of the page with the target switch.
Every stage of the signal is on the right, from the raw landmark to the pixels per second
going into scrollTop.
Frames arrive at about 30 fps; the display wants 60. Detection runs on
requestVideoFrameCallback — the camera's own cadence — while
scrolling is integrated once per animation frame. Polling the tracker from the render
loop stalls both of them.
The model returns a full hand skeleton. This sensor reads the wrist and the middle knuckle and throws the rest away. Fingertips are the jitteriest landmarks on the hand — they swing with any wrist rotation, so steering from one feels broken even when tracking is perfect.
A selfie camera flips X, so a hand moving right reports moving left. Y is untouched. Since this sensor only reads vertical motion the mirror never enters the control law — but the preview is flipped back in CSS so the picture matches the person.
A One Euro filter smooths hard when the hand is nearly still, where jitter shows and lag does not, and backs off when it moves fast, where lag shows and jitter does not. A plain lerp strong enough to kill the shake also adds lag — which reads as slow and inaccurate at the same time.
A hand held still is never actually still. Without a neutral zone around centre the panel creeps constantly and the sensor feels broken, however good the tracking is. The band is the difference between a control and a drift.
Offset is raised to a power before it becomes velocity. That buys fine control near the band — small deliberate nudges stay slow — while the extremes still reach a few thousand pixels a second. A linear mapping gives you neither end well.
Velocity accumulates into position: scrollTop -= v * dt, with
dt clamped so a stalled tab cannot fling the panel several seconds' worth of
scroll on its first frame back. Position is never assigned from the hand directly —
that is what makes it hold steady instead of twitching.
Hands leave frame constantly. A hard cutoff on the first missed detection makes the panel stutter, so misses are counted before the signal is dropped, and velocity eases to zero rather than snapping. Recovery should be invisible.
Flick up or down — each wave gives the page a push
How it works — the signal chain
A camera frame is a long way from a scroll position. Each stage below throws away information the next one does not need, and the order matters — smoothing after the dead band, for instance, would smooth a signal that has already been snapped to zero.
A 640×480 stream at 30 fps. Detection is driven by
requestVideoFrameCallback so it runs once per camera frame, never
twice on the same one.
MediaPipe's hand landmarker returns 21 3-D points. Everything except the wrist and the middle knuckle is discarded on the spot.
out → landmarks[21]The two survivors are averaged into a single vertical coordinate. A midpoint is far steadier than any single landmark, and rotating your wrist barely moves it.
out → rawY · 0–1A One Euro filter adapts its cutoff to speed — hard smoothing when still, light when moving. Fixed smoothing has to pick one, and gets the other wrong.
out → y · de-jitteredOffset from centre is measured, and anything inside ±0.08 becomes exactly zero. Without this the panel creeps forever, because a still hand is never still.
out → offset or 0What survives is normalised and raised to a power, buying fine control near the band while leaving the extremes fast. This is the stage that decides how the sensor feels.
out → velocity · px/sVelocity is eased toward its target, multiplied by frame time and accumulated into
scrollTop. Position is never written from hand position directly.
The control law
This is the whole sensor. The curve below is drawn from the same function that drives the demo, so moving any slider up there redraws it — and while the camera is running, the marker shows exactly where your hand sits on it.
Response curve Dead band — velocity forced to zero Your hand, live
y is the hand's height in the frame, 0 at the top. Because a hand held high
gives a small y, 0.5 − y comes out positive — so the sign of
the offset is the scroll direction, with no branching. The four bold names are the
four sliders.
Wave mode — the return stroke
Waving is the obvious gesture and the treacherous one. Flick your hand up and your arm has to come back — so the camera sees two motions of opposite sign, and a detector that simply reads direction cancels the first with the second. The page ends exactly where it started, and the sensor looks broken while working perfectly.
The fix is to make the detector edge-triggered rather than continuous. Three gates do it, and removing any one of them brings the problem back:
Vertical hand speed has to cross a threshold before anything fires, so drifting, repositioning and general fidgeting never count as a gesture at all.
rejects → slow driftOnce a flick fires, the detector stops listening for 340 ms. A return stroke launched immediately after the flick lands inside that window and is never seen.
rejects → fast returnAfter the window closes the hand must actually slow down before the detector re-arms. A leisurely return that outlasts the window still cannot fire, because it never gets slow enough to re-arm mid-motion.
rejects → slow return
What survives is one push per deliberate flick, however untidily the arm comes back. The
push is scaled by how hard you waved — up to 2.2× — and then simply
coasts: velocity is multiplied by the glide factor every frame, raised to
dt × 60 so the distance covered is the same on a 60Hz and a
144Hz display rather than stopping twice as fast on the faster one.
Wave or hold? Waving is better for travelling — a few flicks cross the document and your arm rests between them. Holding is better for arriving, because you can creep at ten pixels a second and stop exactly where you meant to. The switch is in the panel above.
Why a throttle
The obvious mapping — hand height is scroll position — is the one that falls apart first. The problem is range: an arm covers about 40 cm, and the content does not care.
| Mapping | How it behaves | Why it was or wasn't used |
|---|---|---|
| Absolute | Hand height maps straight onto scroll position, top of frame to top of document. | 1:1 and obvious, but 40 cm of arm has to span the entire document — on a long page that is roughly 25 px of scroll per millimetre of hand. Tracking jitter is amplified by the same factor. |
| ThrottleIn use | Hand offset sets scroll speed; velocity is integrated into position. | Range stops mattering — any distance is reachable from one comfortable arm position, and the dead band gives a real resting state. The cost is that it needs a progress readout, since the hand no longer says where you are. |
| Pinch and drag | Pinch to grab the page, move to drag it, release to let it coast. | The most physical of the three and the closest to a real touch gesture, but pinch is the flakiest signal on the hand — a missed release leaves the page stuck to your palm. |
Using it
The camera never starts on its own. Nothing is requested until you press the button, the video is processed entirely on this machine, no frame leaves the page, and Esc cuts the stream at any time — as does the Stop button in the corner readout, which stays on screen for exactly that reason while your hand is driving the page. The wheel, trackpad and arrow keys keep working throughout; nothing is captured or locked, so you can always take back control by hand. Hand tracking needs a network connection the first time — the model is fetched from Google's CDN, then cached by the browser.