{"kind":"doc","doc":{"title":"Adventures in Machine Vision","section":{"slug":"posts","name":"Posts"},"rkey":"1780360240000-adventures-in-machine-vision","cid":"bafkreib5nnz3u65jjvmnyuea6jqy6xot4tqbm5kbxrmkeye7cpyd3gkufm","renderKey":"bafkreib5nnz3u65jjvmnyuea6jqy6xot4tqbm5kbxrmkeye7cpyd3gkufm@2026-06-02T00:30:40Z@2026-10-20T04:18:20Z@Posts|","href":"/posts/1780360240000-adventures-in-machine-vision","publishedAt":"2026-06-02T00:30:40Z","updatedAt":"2026-10-20T04:18:20Z","tags":["computer-vision","edge-computing","raspberry-pi","hardware","machine-learning","side-projects","quantified-self"],"description":"A Nat Friedman anecdote about an AI watching him drink water sent me digging for a spare Raspberry Pi. Now a camera and a sensor HAT watch my desk, a self-inflicted panopticon that logs who walks by and what the room is doing.","readingMinutes":3,"markdown":"A Nat Friedman anecdote recently caught my eye:\n\n![](blob://bafkreihzqgr2igbs5ggops54ad7vb5bjy6zlwnsnltb6nvzcqfegqs5gs4)\n\nMost of it is fluff. Friedman, Daniel Gross, and the Collisons are positioned to consume AI at a level most of us only read about: all the toys, all the tokens, early access, and people on payroll to wire it together. A lot of what gets demoed from that vantage point is a postcard from a budget I don't have. But the kernel of the idea is almost always cheap, you don't need their resources to point some consumer hardware at yourself and see what falls out.\n\nNow I need to preface this by making it clear that blanketing your house with cameras and letting an AI watch you 24/7 sounds dystopian.\n\n> I want you to walk to the kitchen right now and drink a bottle of water, and I'm going to watch to make sure you do it.\n\nMaybe dystopian is the wrong word. This is self-inflicted panopticon horror.\n\nLet's build our own.\n\n![](blob://bafkreib2gdou7dixtah6skgraok5zcha4xh7som56lfidxpsutd4tuyb64)\n\n## The build\n\nI had a spare Raspberry Pi Zero 2 W rattling around in a drawer, so I gave it a job. A 5MP Arducam (the OV5647, a native Pi sensor, so no driver wrangling) rides the camera ribbon, and a repurposed Waveshare Environment Sensor HAT sits on the GPIO header. The camera and the HAT use different connectors, so they coexist on one board on my desk.\n\nThe pipeline is small. `picamera2` pulls frames, a quantized TFLite SSD-MobileNet-v2 runs COCO object detection, and every confirmed sighting gets appended to a daily JSONL log. A read-only FastAPI service reads that log back out over my tailnet. The detector writes, the API reads, and they run as separate services so I can restart one without disturbing the other.\n\nThe Zero 2 W is the bottleneck: quad A53, 512 MB, around 2 to 6 FPS for full COCO. That's slow for video and plenty for \"who or what just walked by.\" It debounces per label, so it records presence over time instead of flooding the log every frame.\n\nIt stores no video. Frames stay in memory unless I ask for them, and what lands on disk is metadata: one JSON object per detection with the label, confidence, and bounding box. There's a live preview too, off by default and tailnet-only, plain MJPEG that renders into an `<img>` and encodes on the Zero 2 W's hardware block instead of the CPU.\n\nThe HAT writes its own stream, polling temperature, humidity, pressure, and light every thirty seconds. The board heats itself, so the temperature reads high. A live reading of the SoC temperature feeds a thermal-divider model that subtracts the self-heating and pulls the number back toward the actual room.\n\nRight now it isn't doing anything. It watches my desk and writes things down. But I have ideas:\n\n- track my water intake during the workday\n- turn the lights on and off depending on whether I'm actually at my desk\n- etc.\n\n```mermaid\nflowchart LR\n  accTitle: How the desk camera keeps its notes\n  accDescr: The camera feeds an object detector and the sensor board feeds room readings, both write to daily log files, and a read-only API serves those logs over the tailnet without storing any video.\n  C[Camera] --> D[Object detector]\n  D -->|sightings| L[Daily logs]\n  S[Sensor board] -->|room readings| L\n  L --> A[Read-only API]\n  A -->|tailnet| P[Phone or laptop]\n```\nOnly notes about what was seen land on disk, and the API can read them but never change them.","media":{"bafkreihzqgr2igbs5ggops54ad7vb5bjy6zlwnsnltb6nvzcqfegqs5gs4":{"mime":"image/png","width":1580,"height":560,"dominantColor":"#e2e7ed","blurhash":"LGR3czjYf,NH0it5RkWE9#t3WCWC","thumbCid":null},"bafkreib2gdou7dixtah6skgraok5zcha4xh7som56lfidxpsutd4tuyb64":{"mime":"image/png","width":1521,"height":1085,"dominantColor":"#b5b5b3","blurhash":"LJOgKM9Fof~qRjM{xux[D%%M9F9G","thumbCid":null}},"series":{},"related":[{"title":"Intern","href":"/posts/1781835006151-intern","section":"posts","description":"The Autonomous Intern ships as a Raspberry Pi 5 in a desk-toy case running a stack I didn't pick. I flashed it to Debian Trixie, swapped the whole thing for the Hermes agent with its own built-in memory, and folded it all into one setup script.","publishedAt":"2026-06-19T02:10:06Z"},{"title":"Now Playing","href":"/posts/1784736733544-now-playing","section":"posts","description":"Putting my Apple Music \"Now Playing\" on display with a Raspberry Pi and a 64x64 LED matrix.","publishedAt":"2026-07-25T17:18:48Z"},{"title":"Toy GPTs","href":"/posts/1780416704000-toy-gpts","section":"posts","description":"Two tiny GPTs out of Karpathy's microGPT. One keeps its weights on Ethereum. The other holds a séance with dead presidents.","publishedAt":"2026-06-02T16:11:44Z"},{"title":"Single-origin deployment with bhvr","href":"/posts/1747182886660-single-origin-deployment-with-bhvr","section":"posts","description":"Fried my Pi using a 10V solar panel without a buck converter.","publishedAt":"2025-05-13T17:34:00Z"},{"title":"Pure Internet: office---space","href":"/posts/1736536967495-pure-internet-office-space","section":"posts","description":"A Pi Zero 2 W and Waveshare Environment HAT record my office conditions every 60 seconds.","publishedAt":"2025-01-10T11:22:00Z"},{"title":"Now Playing","href":"/experiments/1789660555000-now-playing","section":"experiments","description":"A Raspberry Pi Zero 2 W drives a 64×64 LED matrix showing the album art of whatever is playing, decoded from an AirPlay 2 stream via shairport-sync, with an ambient video loop and sun-following brightness when idle.","publishedAt":"2026-09-20T05:15:33Z"}],"prev":{"title":"Farfield","href":"/posts/1779066375000-farfield"},"next":{"title":"Toy GPTs","href":"/posts/1780416704000-toy-gpts"}}}