Threshold

Outbound WebRTC audio path

Transcribe + on-device Whisper

Ports/adapters, fully offline-testable

430M+ people addressable (WHO)

Overview

The first video doorbell experience built for people who cannot hear it. Existing deaf doorbell products only solve the alert (brighter lights, louder chimes, vibrating pagers); none handle the conversation. Threshold builds the missing half: visitor speech becomes live captions on the resident's phone, and a tapped reply is spoken aloud out of the doorbell speaker, a two-way conversation at the door with no hearing required. The insight is that Ring's WebRTC audio channel is bidirectional (sendrecv) but Amazon's own sample uses only the inbound half; Threshold is built on the outbound channel everyone else leaves idle. Captions default to Amazon Transcribe streaming for accuracy, with an on-device Whisper path for Privacy Mode that keeps raw audio off the network. Amazon Bedrock generates smart reply suggestions in Clarity Mode. Every external dependency (Ring, Transcribe, Bedrock, push, datastore, on-device ASR, connectivity, vision, locale) sits behind a port with a fixture, so the app and stateless Node/Fastify backend are fully testable offline. The product logic is built and tested; the device, cloud, and Ring adapters are not yet implemented. Built for the Amazon Developer Hackathon 2026 (Ring track).

Architecture Diagram

System Overview

Inbound: Visitor Speech to Live Captions

Outbound: Tapped Reply Spoken at the Door

Edge and WebRTC Session

AI Services (Transcribe and Bedrock)

Data and Privacy Model

Two-Phone Demo Relay

Operations

Design Decisions

• →One WebRTC stack, owned by the WebView (decision ARCH-1). The entire value is a synthesized reply audible at the door, and that is only reachable through a browser WebRTC runtime (AudioContext to MediaStreamAudioDestinationNode to RTCRtpSender.replaceTrack). flutter_webrtc cannot inject outbound audio on any platform, so the live session runs inside an in-app WebView with exactly one peer connection (audio sendrecv, video recvonly) and Flutter draws captions and controls as overlays. This bet was made first and de-risked with a spike before any UI.
• →Ports and adapters for every external dependency. Ring, Transcribe, Bedrock, push, the datastore, the on-device ASR engine, connectivity, vision, and locale each sit behind an interface with a fixture. That is what lets the app and backend be developed and fully tested offline, and why the demo build can use deterministic golden responses while the outbound-audio path stays real.
• →Two caption engines behind one port. Clarity Mode uses Amazon Transcribe streaming for accuracy; Privacy Mode runs on-device Whisper (whisper_ggml via FFI, or WASM in the WebView) so raw audio never leaves the device. The UI maps caption confidence to weight and label, never color or audio alone, so the signal survives for colorblind and low-vision users.
• →Accessibility as a hard constraint, not a polish pass. A firm rule that no em-dash or en-dash appears in any user-facing string is enforced by a CI lint gate and a reply-rendering sanitizer. Distinct alert patterns use a haptic vocabulary plus strobe so a detection at the door becomes a felt and seen alert within 5 seconds.
• →Privacy by construction. The design keeps raw audio and video off persistent storage, and Privacy Mode transcribes entirely on-device with zero egress. Ring tokens live on the backend rather than the device; the current build encrypts them with a local symmetric helper in an in-memory store, with a KMS-backed token store planned.
• →The reusable heart extracted into a standalone MIT package. The caption and reply logic lives in flutter_ring_captions, consumed from the repo as a git dependency (publish_to intentionally set to none), so the core is extractable without coupling it to the app shell.

Deployment

The backend is a stateless Node 22 + Fastify service that handles OAuth, webhooks, the WebRTC signaling broker, the ASR relay, replies, and transcripts. It is deployed on AWS as an ECS Fargate container behind an ALB terminating HTTPS/WSS with a trusted certificate, and listens on port 8080 by default (override via PORT). The Android app is Flutter/Dart (iOS planned), with the live session running inside an in-app WebView that owns the single WebRTC peer connection. The two-phone demo relays ring, captions, and replies between a visitor web page and the resident app through the backend hub; both phones share one Device ID. Clarity Mode routes captions through Amazon Transcribe streaming and reply suggestions through Amazon Bedrock, while Privacy Mode keeps transcription on-device via Whisper. CI gates on the Flutter test suite (the primary correctness gate) and the backend's typecheck, lint, and tests; npm run dev runs the backend in local mode against fixtures only, with no AWS or real Ring. Full device testing requires a Ring developer account, an OAuth app, and a Ring Protection plan. The device, cloud, and Ring adapters are specified and the product logic is tested, but those adapters are not yet fully implemented.

Lessons Learned

The whole project is a study in letting one deliberate architectural bet order everything else. Amazon's Ring sample uses only the inbound half of a bidirectional WebRTC channel; Threshold's entire value is the outbound half, a synthesized reply audible at the door. That capability is only reachable through a browser WebRTC runtime, so the live session had to move into a WebView with a single peer connection, Flutter drawing captions and controls as overlays on top. Making that decision first, and de-risking it with a spike before building any UI, meant the rest of the app was designed around a path that was already proven to work rather than discovered to be impossible late. The second lesson was how much leverage ports and adapters give an accessibility product: putting Ring, Transcribe, Bedrock, push, the datastore, and the on-device ASR engine each behind an interface with a fixture is what let the product logic be fully built and tested offline while the real adapters stayed unimplemented, and it kept Privacy Mode (on-device Whisper, zero egress) and Clarity Mode (Transcribe plus Bedrock) as two implementations of one port instead of two code paths. Treating accessibility as a hard constraint also changed the engineering: a CI lint gate and a reply sanitizer enforce a no-dash rule on every user-facing string, and caption confidence is a semantic signal mapped to weight and label rather than color or sound, so it survives for users who cannot rely on either.