Adding a Mic

After I got the speaker working, I went on to add a microphone.

The plan was to turn it into something closer to an actual assistant: something I could talk to, interrupt, and ask questions without having to touch my computer. This would also bring it one step closer to becoming part of Cadence.

I initially experimented with a full-duplex audio pipeline that lets the agent listen while it speaks. In theory, this means I can interrupt it whenever I want instead of waiting for it to finish its entire response.

In practice, I accidentally created a rather interesting problem.

The agent started listening to itself.

While answering a question, it would pick up its own voice through the microphone, transcribe it, and start responding to the transcription. At one point, I ended up with an agent discussing the pronunciation of Scuderia Ferrari, apparently because it had heard itself talking about Ferrari.

It was a pretty good demonstration of why building a voice assistant is considerably more complicated than just connecting a microphone to a speaker.

Eventually, I decided to stop overcomplicating things and pivoted to a much simpler interaction model: push-to-talk. Instead of keeping the microphone open while the assistant speaks, I can just call its name(supposedly Cadence but I have to train it to recognize “Hey Cadence” so I am currently calling it “Hey Jarvis”), say something, and let it respond. No interruptions, no accidental self-conversations, and considerably less engineering overhead.

It's not quite the seamless assistant I originally envisioned, but it works. And honestly, there's something satisfying about getting rid of an entire class of problems by simply changing how the system works.

For now, the speaker can hear me, understand what I say, and respond. That's enough to make it feel considerably more alive.