wtf is pipecat-visemes?

pipecat-visemes adds server-side lip sync to voice bots built with Pipecat, a framework for building voice AI applications. Instead of asking the browser to analyze audio, or relying on a particular text to speech provider's timing information, the server itself listens to the audio being streamed out and works out how a mouth should move while speaking it, then sends that timing data to the client so it can animate an avatar's mouth in sync with the sound. The project is designed to work with any text to speech service Pipecat supports, since the analysis runs directly on the raw audio rather than depending on any one provider's word or phoneme timestamps. It is built entirely on top of the officially released Pipecat package using its normal extension points, so it does not require a custom fork. A processor sits between the text to speech step and the audio output, analyzing the sound using signal processing techniques borrowed from speech science, such as tracking formants and pitch, and produces short batches of mouth shape data. A second component then delivers those batches to the connected client at the right moment, timed to match when the matching audio actually plays, and drops any batches that become irrelevant if the bot is interrupted mid sentence. The repository includes a working example bot, a small web client built with React that connects over WebRTC and shows the animated mouth along with debugging tools, a test suite, and a benchmark that scores how accurate the lip sync is against reference recordings analyzed with the speech tool Praat. The current analyzer is described as the first of several planned tiers, with future versions meant to use provider supplied viseme events, timestamp based phoneme lookup, or a dedicated audio recognition model, while keeping the same output format so client code never has to change.