It started with a question that was probably not worth spending two days on:
Could I use my fingers to control the lights in my room from a distance?
I didn't know much about computer vision. I had barely touched AVFoundation, Vision, or Core Media. I had never built a gesture recognition system.
Which, naturally, made me want to try it.
The idea behind HandKit was simple: point my iPhone's front camera at myself, recognize a few hand gestures, and translate them into actions in my HomeKit home.
The first two interactions I wanted were:
- Pinch my thumb and index finger together to toggle a light.
- Move my index finger horizontally to control brightness from 0 to 100%.
The final interaction looks almost trivial.
A pinch. A lamp turns on.
But getting from a raw camera feed to an intentional, reliable gesture turned out to involve camera pipelines, pixel buffers, coordinate systems, noisy measurements, a little geometry, signal filtering, state machines, hysteresis, Swift Concurrency, and finally HomeKit.
This article is a walkthrough of how I built it, but more importantly, why the architecture and algorithms evolved the way they did.
[VIDEO: First successful pinch controlling the physical lamp]
Starting with the camera
Before detecting a hand, I needed access to something much simpler: images.
On iOS, that starts with AVFoundation.
The capture pipeline can be simplified to:
AVCaptureDevice
↓
AVCaptureDeviceInput
↓
AVCaptureSession
│
├────────→ AVCaptureVideoPreviewLayer
│ ↓
│ Preview
│
└────────→ AVCaptureVideoDataOutput
↓
Video frames
Each component has a different responsibility.
AVCaptureDevice represents the physical capture device. In my case, the iPhone's front-facing camera.
AVCaptureDeviceInput turns that device into an input that can be attached to a capture session.
AVCaptureSession coordinates the capture pipeline.
From there, I needed the video in two different places.
The first path goes to an AVCaptureVideoPreviewLayer, so I can see the camera feed on screen.
The second goes to an AVCaptureVideoDataOutput, because displaying the video isn't enough. I need access to the individual frames if I want to analyze what my hand is doing.
That distinction became the foundation of HandKit:
Camera
│
├──→ human sees the preview
│
└──→ application analyzes the frames
Managing the capture session with Swift Concurrency
I didn't want camera lifecycle operations scattered throughout the UI, so I introduced a CameraManager.
It owns the capture session and is responsible for configuring, starting, and stopping it.
I implemented it as an actor.
actor CameraManager {
let session = AVCaptureSession()
// ...
}
Actors are useful here because they provide isolation around mutable state.
One thing I had to properly understand while building this was that an actor is not a thread.
An actor defines an isolation domain. Its executor determines where jobs isolated to that actor execute.
For the capture session, I wanted serialized execution on a dedicated queue, so I gave CameraManager a custom serial executor.
Conceptually:
CameraManager actor
↓
Custom SerialExecutor
↓
camera.session queue
↓
AVCaptureSession
A simplified version looks like this:
actor CameraManager {
private let sessionQueue =
DispatchSerialQueue(label: "camera.session")
nonisolated var unownedExecutor: UnownedSerialExecutor {
sessionQueue.asUnownedSerialExecutor()
}
// ...
}
This gave the camera pipeline a clear execution context without turning the MainActor into the place where capture work happens.
I also kept configuration inside the manager.
From outside, starting the camera shouldn't require knowing:
configure first
then start
but don't configure twice
That is an internal invariant.
So CameraManager tracks whether the session has already been configured and exposes a higher-level start() operation.
This was one of the first architectural principles that kept appearing throughout the project:
The caller should express an intention, not reproduce the internal procedure required to achieve it.
Keeping session management and frame processing separate
The next step was receiving frames from AVCaptureVideoDataOutput.
AVFoundation provides them through AVCaptureVideoDataOutputSampleBufferDelegate.
I initially considered making CameraManager the delegate too, but there was both a technical and architectural reason not to.
The delegate participates in Objective-C's NSObjectProtocol, while my camera manager is an actor. More importantly, managing the lifecycle of the camera and processing every video frame are different responsibilities.
So I introduced VideoOutputDelegate.
CameraManager
│
├── configure / start / stop
├── AVCaptureSession
│
└── VideoOutputDelegate
↓
frames
↓
Vision
I also separated their execution queues.
camera.session
├── configure
├── start
└── stop
frame.session
├── frame
├── frame
├── frame
└── Vision processing
This matters because analyzing an image can take time. I don't want that work unnecessarily blocking operations on the capture session itself.
It also gave the code a much cleaner conceptual boundary:
CameraManager manages capture.
VideoOutputDelegate interprets what was captured.
A video is just a lot of images arriving very quickly
The delegate eventually receives:
func captureOutput(
_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer,
from connection: AVCaptureConnection
)
The important value here is the CMSampleBuffer.
I initially thought of a camera frame as essentially an image, but the media pipeline is richer than that.
A CMSampleBuffer is a container for a media sample. For video, it carries the sample and associated information such as timing and format metadata.
For Vision, what I really needed was the image buffer inside it:
guard let pixelBuffer =
CMSampleBufferGetImageBuffer(sampleBuffer)
else {
return
}
That gives me a CVPixelBuffer.
The pipeline had now become:
Camera
↓
AVCaptureVideoDataOutput
↓
CMSampleBuffer
↓
CVPixelBuffer
↓
Vision
There is no need to turn every frame into a UIImage.
The pixel buffer can go directly into Vision.
Asking Vision to find a hand
Apple's Vision framework provides VNDetectHumanHandPoseRequest.
Instead of recreating the request for every frame, I keep one instance around and ask it to detect at most one hand.
private let handPoseRequest: VNDetectHumanHandPoseRequest = {
let request = VNDetectHumanHandPoseRequest()
request.maximumHandCount = 1
return request
}()
For each pixel buffer, I create a request handler:
let handler = VNImageRequestHandler(
cvPixelBuffer: pixelBuffer,
orientation: .right,
options: [:]
)
try handler.perform([handPoseRequest])
I'll come back to that .right, because getting there involved one of my favorite debugging moments in this experiment.
Once Vision detects a hand, it returns a VNHumanHandPoseObservation.
From that observation, I can request specific landmarks:
let indexTip = try hand.recognizedPoint(.indexTip)
let thumbTip = try hand.recognizedPoint(.thumbTip)
Vision doesn't return screen pixels.
The locations are normalized coordinates, which makes them independent of a specific image resolution.
So now I had something incredibly useful:
index fingertip → (x, y)
thumb fingertip → (x, y)
That sounds like enough to build gestures.
It wasn't.
Vision gives estimates, not truth
Each recognized point also has a confidence value.
Very quickly, I discovered why.
I printed the index coordinates while moving my finger around. When Vision was confident, I would get values like:
x: 0.578
y: 0.300
confidence: 0.872
x: 0.577
y: 0.311
confidence: 0.867
x: 0.575
y: 0.328
confidence: 0.855
But occasionally the tracking would produce a large jump with very low confidence.
If I interpreted every coordinate as equally trustworthy, a tracking mistake could look like an extremely fast gesture.
My first filter therefore became:
guard indexTip.confidence >= 0.7 else {
return
}
0.7 isn't a universal Vision constant.
It was an experimental starting point based on the data I was observing.
That distinction matters throughout HandKit: most thresholds aren't presented as magic values that should work for everybody. They're values I measured and tuned while experimenting with my own setup.
But confidence filtering wasn't enough.
Even when Vision was confident and my finger looked completely still to me, its coordinates continued to move slightly.
That led to the first genuinely interesting algorithmic problem.
How do you measure movement when "not moving" still moves?
To know whether my finger is moving, I need more than its current position.
I need its previous one.
For two consecutive measurements:
previous = (x₁, y₁)
current = (x₂, y₂)
I calculate:
Δ x = x_2 - x_1
Δ y = y_2 - y_1
But I also wanted one value representing how much the point moved regardless of direction.
That's just the hypotenuse of the triangle formed by the two deltas:
d = sqrt((Δ x)^2 + (Δ y)^2)
Swift already gives me exactly that operation:
let movementDistance = hypot(deltaX, deltaY)
This was my first little return to the Pythagorean theorem for a project involving smart lights.
The interesting discovery came when I kept my index as still as possible.
movementDistance wasn't zero.
I saw values around:
0.001
0.003
0.002
0.005
0.001
...
and occasional larger spikes.
The system needed to distinguish:
actual intentional movement
from:
natural tracking jitter
A single frame wasn't enough to make that decision.
Filtering the tracking signal
Instead of considering one movement measurement at a time, I started keeping a sliding window containing the five most recent distances.
[d1, d2, d3, d4, d5]
When a sixth arrives:
[d1, d2, d3, d4, d5, d6]
I remove the oldest:
[d2, d3, d4, d5, d6]
In Swift:
recentMovementDistances.append(movementDistance)
if recentMovementDistances.count > 5 {
recentMovementDistances.removeFirst()
}
Then I calculate the median.
For five sorted values:
[0.001, 0.002, 0.003, 0.005, 0.050]
↑
median
the median is 0.003.
That was useful because an isolated bad measurement such as 0.050 doesn't drag the result upwards as strongly as it would affect an average.
This is deliberately a very small and simple filter. HandKit isn't trying to be a sophisticated signal-processing library.
But it was enough to introduce an important idea:
A noisy measurement becomes much more useful when you reason about it over time instead of treating every sample independently.
And that observation led directly to state machines.
Detecting movement with hysteresis
My first instinct could have been:
movement > threshold → moving
movement < threshold → idle
But imagine the signal oscillating around that threshold:
0.0069
0.0071
0.0068
0.0072
The state could rapidly become:
IDLE
MOVING
IDLE
MOVING
IDLE
Instead, I introduced two different thresholds.
From my experiments, I started with:
median > 0.01 → start moving
median < 0.005 → become idle
Between them, I keep the current state.
> 0.01
┌───────────────────────┐
│ ↓
IDLE MOVING
↑ │
└───────────────────────┘
< 0.005
This is hysteresis.
The gap between the two thresholds prevents the state from constantly flipping when the signal sits near a boundary.
In code, I represent the state explicitly:
enum MovementState {
case idle
case moving
}
rather than hiding the model behind a boolean.
That small choice became useful because I wasn't really trying to answer:
Is there movement in this exact frame?
I was answering:
Has an intentional movement started, and are we currently inside it?
That is a temporal question.
A gesture is something that happens over time
Originally, I started thinking about recognizing a swipe.
A single position obviously can't represent a swipe.
Even two consecutive positions aren't enough.
A gesture has a beginning, a trajectory, and an end.
So when transitioning from .idle to .moving, I store:
startPosition
startTime
using a ContinuousClock for elapsed time measurement.
When movement eventually ends, I can compare:
startPosition → endPosition
startTime → endTime
and derive:
totalDeltaX = endX - startX
totalDeltaY = endY - startY
plus the gesture duration.
This gave me the information required to reason about a swipe:
- Did it travel far enough horizontally?
- Did it remain reasonably horizontal?
- Did it happen quickly enough?
But testing this exposed a completely different bug.
The horizontal gesture that somehow moved vertically
I performed horizontal movements and looked at my data.
Instead of getting:
|Δx| >> |Δy|
I repeatedly got large changes on Y.
Then I performed an intentionally vertical gesture.
This time X changed dramatically.
My coordinate system was effectively rotated relative to what I was seeing.
The culprit was this:
orientation: .up
I had initially told Vision to interpret the pixel buffer as if it were already oriented upright.
The preview looked fine, which made this easy to miss.
But the orientation of what the user sees and the orientation in which Vision interprets the raw image buffer aren't necessarily the same thing.
I tested .right.
Then repeated the horizontal gestures.
This time I got measurements such as:
Δx = -0.217
Δy = +0.013
Δx = +0.263
Δy = -0.040
Δx = -0.278
Δy = +0.018
Exactly what I wanted:
|Δx| >> |Δy|
That bug produced one of my favorite lessons from the project:
Before tuning an algorithm around strange results, make sure the coordinate system feeding it is actually correct.
Otherwise, I could have spent a lot of time building increasingly complicated gesture logic to compensate for incorrect input data.
I also explicitly tested direction with the front-facing camera.
In my current configuration:
Δx > 0 → visual movement towards the left
Δx < 0 → visual movement towards the right
Again, I preferred measuring the behavior rather than assuming what mirroring would do.
Realizing I didn't actually want a swipe
While designing the swipe, I realized something else.
For brightness control, a swipe is the wrong interaction.
A swipe is a discrete gesture:
gesture
↓
recognized
↓
perform action once
That's great for something like:
swipe → next track
Brightness is continuous.
What I actually wanted was a gesture-controlled slider:
finger starts moving
↓
keep tracking horizontal position
↓
brightness changes continuously
↓
finger stops
So the same .idle / .moving state machine remained useful, but the meaning of the moving state changed.
Instead of waiting for the gesture to end before doing something, I calculate brightness continuously while movement is active.
There's another subtle decision here.
I could update brightness using each frame's delta:
brightness += currentDelta
But then tracking errors can accumulate over time.
Instead, at the beginning of the gesture I store:
startPosition
startBrightness
Every subsequent value is calculated relative to those fixed references:
Δ x = currentX - startX
brightness = startBrightness + f(Δ x)
This means that if I move away and then return exactly to my starting position, I should return to the original brightness regardless of the intermediate path.
That avoids cumulative drift.
The current prototype maps the normalized horizontal displacement to a value between 0 and 100 and clamps the result:
let newBrightness = startBrightness + brightnessDelta
brightness = min(max(newBrightness, 0), 100)
I also built a small visual indicator in SwiftUI so I could test the interaction before connecting it to a real light.
[VIDEO: Hand movement controlling the on-screen brightness indicator]
That separation was intentional.
First prove:
gesture → correct numeric value
Then connect:
numeric value → physical device
One unknown at a time.
Detecting a pinch with geometry
The second gesture was simpler conceptually.
I already had the index fingertip.
Vision can also give me the thumb tip.
let indexTip = try hand.recognizedPoint(.indexTip)
let thumbTip = try hand.recognizedPoint(.thumbTip)
After applying the same confidence filtering, I have two normalized points.
So detecting how close the fingers are is again a geometry problem:
Δ x = indexX - thumbX
Δ y = indexY - thumbY
distance = sqrt((Δ x)^2 + (Δ y)^2)
or:
let distance = hypot(deltaX, deltaY)
The confidence check needs to cover both points. A reliable index position doesn't help if the thumb position is uncertain:
guard indexTip.confidence >= 0.7,
thumbTip.confidence >= 0.7 else {
return
}
let deltaX = indexTip.location.x - thumbTip.location.x
let deltaY = indexTip.location.y - thumbTip.location.y
let distance = hypot(deltaX, deltaY)
This time, the distance compares two landmarks in the same frame. Earlier, I used the same calculation to compare one landmark across consecutive frames. The geometry is identical; the question is different.
Before deciding what counted as a pinch, I measured the distance while opening and closing my fingers. In my setup, the observations were roughly:
Fingers apart: 0.15 to 0.21
Fingers touching: 0.005 to 0.015
Contact didn't produce a perfect zero. Vision estimates the locations of landmarks in a two-dimensional image, and those estimates still contain noise when the fingers touch.
These are distances in normalized image coordinates, not centimeters or measurements of physical contact. Normalizing the coordinates removes the dependence on pixel resolution; it doesn't make the measurement independent of hand size, camera distance, or image geometry. Moving my hand farther from the camera changes its apparent size and therefore the distance between the detected points.
The measurements were enough to establish a useful threshold for this prototype, but they weren't a general calibration for every hand and camera setup.
[IMAGE: Thumb-tip and index-tip landmarks, with the measured distance shown for open fingers and a pinch]
A pinch needs memory too
The movement detector had already taught me what happens when a noisy measurement sits near one threshold.
For the pinch, I reused hysteresis with a separate state machine:
enum PinchState {
case open
case pinched
}
The rules became:
OPEN -- distance < 0.015 --> PINCHED
PINCHED -- distance > 0.03 --> OPEN
Between the two thresholds, I preserve the current state. At either exact boundary, the strict comparisons also leave the state unchanged.
That gap matters. After recognizing a pinch, I don't want a small increase from 0.014 to 0.016 to mean that the hand has opened again. The fingers have to separate more clearly before another pinch can begin.
But there was one more distinction to make: the state and the event are different things.
If I called the toggle action on every frame where the distance was small enough, holding my fingers together would repeatedly switch the lamp on and off.
The event I care about is the transition:
OPEN -> PINCHED -> emit one toggle
The core logic can be expressed as:
switch pinchState {
case .open:
if distance < 0.015 {
pinchState = .pinched
onToggle()
}
case .pinched:
if distance > 0.03 {
pinchState = .open
}
}
Once the state becomes .pinched, subsequent frames stay in that branch until the fingers reopen. Releasing the pinch rearms the interaction; it doesn't toggle the light.
This gave me a specific behavior to test: pinch, hold, release, pinch again. Holding should produce no additional action, and the second pinch should produce exactly one new action.
Hysteresis stabilizes the transition, but it doesn't solve every tracking problem. For example, losing the hand while it is pinched raises a separate question about when to reset or rearm the recognizer. That is part of making the experiment more robust, beyond this first working interaction.
[CODE: PinchState and processPinch implementation, highlighting the single OPEN-to-PINCHED callback]
Finding the actual light with HomeKit
At this point, the gesture pipeline could produce a toggle event. I still needed to connect that event to something outside the phone.
HomeKit introduced another hierarchy to understand:
HMHomeManager
|
+-- HMHome
|
+-- HMAccessory
|
+-- HMService (Lightbulb)
|
+-- HMCharacteristic (PowerState)
+-- HMCharacteristic (Brightness)
HMHomeManager provides access to the configured homes. Each home contains accessories, and each accessory exposes services. The values I can inspect or change belong to the characteristics of those services. Apple's HMCharacteristic documentation describes that final layer.
I began by inspecting what my own home actually exposed. Among the discovered accessories were Lamp, Desktop, and Lanterne. For this experiment, I selected Lamp and looked for its lightbulb service.
That selection was deliberately specific to my setup. A user-facing version would need accessory selection and a stable identifier instead of depending on a display name.
Within the service, the two characteristic types I needed were:
HMCharacteristicTypePowerState
HMCharacteristicTypeBrightness
PowerState represents whether the light is on. Brightness represents its brightness as an integer percentage of the maximum, as defined by Apple's brightness characteristic documentation.
Finding those characteristics didn't yet prove that I could control the lamp. I first asked HomeKit to read their values.
The actual output was:
BRIGHTNESS: 100
POWER: 1
Now I knew that I had found the intended accessory and could read its state: the lamp was on, with brightness at 100%.
[IMAGE: HomeKit discovery output showing Lamp, Desktop, and Lanterne, followed by Lamp's PowerState and Brightness readings]
Testing one layer at a time
It was tempting to connect the pinch immediately. Instead, I worked through four small steps:
Read state -> Manual write -> Temporary button -> Gesture
The first read established that discovery worked.
The next test wrote false to the power characteristic. The physical lamp switched off. That established that the app could send a command to the accessory, independently of Vision or gesture recognition.
I removed that temporary write after the test. Leaving a command in the discovery path would make loading the home change the lamp's state, and discovery callbacks can happen more than once.
Then I added a temporary SwiftUI button:
Button("Toggle Lamp") {
lightController.toggle()
}
The button gave me a controlled way to test the full toggle operation. If it failed, I knew to investigate HomeKit, characteristic discovery, or the read/write sequence. I didn't have to wonder whether Vision had missed my fingers.
Only after that worked did I replace the button as the trigger with the pinch event.
This sequence was one of the most useful engineering decisions in the project. Each step reduced the number of possible explanations for a failure before I introduced another subsystem.
Giving HomeKit its own controller
I put HomeKit discovery and control in LightController.
Its job was to find Lamp, retain the characteristics the app needed, and expose an operation such as toggle() to the rest of the application.
The references begin as optional values because HomeKit discovery is asynchronous:
private var powerCharacteristic: HMCharacteristic?
private var brightnessCharacteristic: HMCharacteristic?
They refer to live HomeKit objects. They aren't preferences to persist in UserDefaults, and keeping them avoids traversing the entire home hierarchy on every gesture.
Once discovery had found the power characteristic, toggle() followed this sequence:
Power characteristic available?
|
v
Read current value
|
v
Convert to Bool and invert
|
v
Write new value
|
v
Check completion for an error
I read the value before toggling because the lamp could also be changed from the Home app or another controller. A local Boolean would only describe what HandKit last believed, which might no longer match the accessory.
There was also an Objective-C interoperability detail: HMCharacteristic.value is exposed as Any?. In my tests, the power value arrived as an NSNumber, so I used its boolValue before inverting it:
guard let value = powerCharacteristic.value as? NSNumber else {
return
}
let newState = !value.boolValue
This conversion belongs after a successful read, followed by the write and its own error handling. The prototype logged failures rather than treating an attempted command as a successful one.
A read followed by a write still isn't an atomic toggle. Another command could arrive between them, and rapid gestures could overlap requests. Serializing device commands is a further robustness step; the working demo doesn't establish that all concurrent-control cases are handled.
At this snapshot, retaining the brightness characteristic prepared the second interaction. It did not mean that the gesture-controlled brightness value was already being written to the lamp.
[CODE: LightController discovery and toggle(), including the PowerState read, NSNumber conversion, write, and error handling]
Keeping recognition independent from the action
The frame-processing code knows about landmarks, distances, and gesture states. It doesn't need to know what a HomeKit accessory is.
VideoOutputDelegate communicates through two callbacks:
private let onToggle: @Sendable () -> Void
private let onBrightnessChanged: @Sendable (CGFloat) -> Void
One emits a discrete event. The other emits a continuously updated value.
CameraManager receives the toggle callback from the application and passes it to the delegate. The application decides that this event should call LightController.toggle().
That boundary means I can inspect or log the gesture output without a connected lamp. It also means the same pinch recognizer could eventually trigger something else without changing the geometry or thresholds.
There is still a deliberate prototype-level coupling here: onBrightnessChanged names the value in terms of the current interaction. It abstracts away HomeKit, but it isn't a completely generic gesture API. If I extracted this into a reusable component, a normalized slider value or displacement event might be a better contract.
For HandKit, the useful separation was already clear:
Recognition: What did the hand do?
Application: What should that mean?
HomeKit: How do I ask the lamp to do it?
Crossing from the frame queue to the MainActor
These callbacks also exposed a concurrency boundary.
Frames arrive on frameQueue. The UI-facing model and, in this project, LightController are isolated to the MainActor.
Calling lightController.toggle() directly from the synchronous toggle callback produced an isolation error. Swift was pointing out an actual mismatch in the architecture: code running in the frame-processing context was trying to access main-actor-isolated behavior directly.
The callback needed an explicit handoff:
onToggle: {
Task { @MainActor in
lightController.toggle()
}
}
The brightness callback uses the same pattern:
onBrightnessChanged: { newBrightness in
Task { @MainActor in
viewModel.brightness = newBrightness
}
}
@Sendable describes a closure's ability to cross concurrency boundaries with safe captures. It doesn't dispatch the closure onto the main actor. The Task { @MainActor in ... } body establishes that execution context explicitly. The distinction follows Swift's concurrency and isolation model.
The values crossing this boundary are small: an event or a brightness number. The pixel buffers, Vision request, and recognition state remain in the frame-processing path.
Moving the callback onto the main actor also doesn't make the physical device operation synchronous. HomeKit still completes its read and write asynchronously. Actor isolation protects access to application state; it doesn't guarantee that the lamp has already reacted.
For this prototype, the callback handoff made the two paths explicit. If updates become more frequent than the UI or accessory can use, the next step is to coalesce them rather than create an ever-growing stream of pending work. Independent tasks also shouldn't be used as a guarantee of event ordering.
One shared model for the camera and the interface
The on-screen brightness indicator reads from a CameraViewModel that is both observable and main-actor isolated:
@MainActor
@Observable
final class CameraViewModel {
var brightness: CGFloat = 0
}
This reduced example shows the two separate responsibilities. @Observable lets SwiftUI track changes to the state it reads. @MainActor defines where that mutable state is accessed. Observation alone doesn't establish concurrency isolation.
There was an equally important ownership detail: ContentView and CameraManager needed the same model instance.
If each created its own CameraViewModel, the camera callback could update one object while SwiftUI observed another. The gesture calculations would be correct, but the indicator would never reflect them.
In the view's initialization, the wiring therefore follows this shape:
init() {
let viewModel = CameraViewModel()
let lightController = LightController()
self._viewModel = State(initialValue: viewModel)
self.lightController = lightController
self.cameraManager = CameraManager(
viewModel: viewModel,
onToggle: {
Task { @MainActor in
lightController.toggle()
}
}
)
}
This is an excerpt of the wiring, with the surrounding stored properties and view body omitted. The important part is that the local viewModel passed to CameraManager is the same reference retained for the view's state. Inside the manager, the brightness callback updates that reference on the main actor.
The resulting feedback path is:
Vision -> slider value -> callback -> MainActor
|
v
CameraViewModel
|
v
SwiftUI indicator
That shared instance is what connects a correct calculation to something visible on screen.
The complete architecture
By the end of this iteration, the system looked like this:
ContentView
creates and connects objects
|
v
CameraManager actor
|
custom SerialExecutor
|
sessionQueue
|
AVCaptureSession
|
+----------------+------------------+
| |
v v
AVCaptureVideoPreviewLayer AVCaptureVideoDataOutput
| |
v v
Camera preview frameQueue
|
v
VideoOutputDelegate
|
v
CMSampleBuffer -> CVPixelBuffer
|
v
Vision hand-pose request
|
v
Confidence-filtered points
|
+------------------+----------------+
| |
v v
Movement / slider Pinch state
| |
v v
onBrightnessChanged(value) onToggle()
| |
v v
Task { @MainActor in ... } Task { @MainActor in ... }
| |
v v
CameraViewModel LightController
shared with ContentView |
| v
v HomeKit PowerState
SwiftUI 0-100 indicator |
v
Physical Lamp
Next connection, not yet implemented at this snapshot:
slider value -> HomeKit Brightness -> physical light intensity
The boundaries each answer a practical question. The camera manager owns capture lifecycle. The delegate processes frames and maintains recognition state on its serial queue. The view model exposes UI state. The light controller handles accessory discovery and commands.
The architecture also makes the unfinished part visible: the brightness path reaches the interface, while the toggle path reaches the physical lamp.
Where the experiment stands
The pinch now controls a real HomeKit lamp. Bringing my thumb and index finger together triggers one toggle, and holding the pinch doesn't repeatedly switch it.
The continuous horizontal gesture also works as an on-screen slider. It produces and displays brightness values from 0 to 100 using a fixed starting position and starting brightness.
At the snapshot covered by this article, that slider was not yet connected to HomeKit's Brightness characteristic. I had discovered and read the characteristic, but the physical brightness-control path was still unfinished.
Those are two different milestones: recognizing and displaying the intended value, and successfully applying it to an accessory.
[VIDEO: First successful end-to-end pinch test, with the physical lamp and my reaction visible]
The video captures the part that a console log couldn't. After spending so much time looking at coordinates and state transitions, I brought two fingers together and something in the room changed.
I laughed. A lot.
What I learned
This project made several concepts concrete for me.
Actor isolation and execution are separate concerns. Making CameraManager an actor gave its mutable state an isolation boundary; choosing a serial executor gave its work a deliberate execution context. Neither choice removed the need to think about delegate callbacks and other queues.
A correctly displayed preview doesn't prove that image analysis is using the correct orientation. The .up to .right fix came from comparing expected movement with observed deltas. In this configuration, it resolved the mismatch. It isn't a universal orientation setting for every camera and device posture.
Confidence filtering, median filtering, and hysteresis solve different problems. Confidence rejects uncertain landmarks. The five-sample median limits the influence of isolated movement spikes. Hysteresis prevents repeated state changes near a boundary. Combining them was useful because no single threshold could do all three jobs.
Filtering also has a cost. A window of recent measurements adds inertia, and per-frame displacement depends on sampling cadence. The numbers that worked in my tests aren't a substitute for testing different frame rates, camera distances, and lighting conditions.
Gesture recognition requires memory. The movement path needs a beginning and an evolving trajectory; the pinch path needs to know whether the fingers were already together. Once I modeled those states explicitly, the behavior became easier to reason about.
The transition can be the useful event. A sustained pinch describes a condition. Entering that condition describes an action the application can respond to once.
A fixed reference simplifies continuous control. Mapping every position back to startPosition and startBrightness avoids accumulating frame-to-frame errors and gives the interaction a predictable return point.
Object identity matters as much as data flow. The brightness callback and SwiftUI needed to refer to the same model instance. Correct values sent to an unobserved object are still invisible to the user.
Finally, separating the layers made debugging manageable. A button let me validate HomeKit without a camera gesture. A visual indicator let me validate the brightness calculation without changing a lamp. The callbacks connected those tested pieces.
Learning with AI as a mentor
I used AI, including Codex, throughout the experiment as a mentor: to point me toward documentation, explain unfamiliar concepts, and challenge my reasoning when I couldn't explain what I was seeing.
The learning rule was to work through the challenges myself. I wanted to inspect the measurements, reason about the algorithm, write and test the behavior, and understand the architectural decisions. Explanations and small examples helped me move forward, but having the challenges completed for me would have removed the part I was doing this project to experience.
The most useful moments were the questions that made me look again: what is actually inside this buffer? Why does a stationary finger still move? Which coordinate system am I measuring? What should happen on the next frame if the fingers are still touching?
That is how a small experiment with a lamp became an excuse to understand much more of the iOS stack.
What's next
The immediate next step is to connect the existing slider value to the real brightness characteristic. That includes starting from the lamp's current brightness, converting the calculated value to the characteristic's expected integer representation, and handling write failures.
I also want to control the rate of those writes. Camera frames can produce updates much faster than a light needs them. Keeping the latest desired value, limiting intermediate commands, and ensuring the final value is applied would make that connection more useful than sending every frame directly to HomeKit.
Beyond that, the next experiments are about robustness:
- Test thresholds across different hand sizes, camera distances, and lighting conditions, and explore a distance measure relative to hand size.
- Define what happens when tracking confidence drops or the hand leaves the frame, including how to reset movement history and rearm a pinch.
- Decide how pinch and horizontal movement should interact when both occur together.
- Serialize device commands and make discovery, unavailable accessories, and command failures visible in the interface.
- Replace the hardcoded
Lampselection with an accessory picker. - Explore macOS camera support while reusing the gesture logic and revisiting orientation and capture assumptions.
These are the next steps, not features already demonstrated by the current prototype.
From a small question to a working lamp
I started HandKit because I wanted to know whether I could control my lights with my fingers. I ended this iteration with a working pinch, a brightness interaction I could see on screen, and a much clearer understanding of everything between a camera frame and a physical action.
The satisfying part was being able to explain that path. I could point to where the pixels arrived, where the landmarks became measurements, where a state transition became an event, and where that event finally reached HomeKit.
There is still work left. The brightness path needs its final connection, and the gesture logic needs to survive more than the conditions in which I first tuned it.
But the moment in that first successful video already says why I wanted to build this. I pinched my fingers, the lamp reacted, and I couldn't stop laughing. A question that had sounded a little ridiculous had led me through unfamiliar frameworks, a return to Pythagoras, and a room that now responded to something I had built.
That was a very good reason to spend two days on it.
