Hibiki: Real-time Voice-Preserving Language Translation

Hands-on with Open Source Real-Time Translation Models

Remember Star Trek's Universal Translator? Science fiction is becoming reality with two groundbreaking open-source models: Meta's Seamless Streaming and Hibiki. These models are revolutionizing real-time speech translation, and I've had the opportunity to test one firsthand.

The Open Source Revolution in Real-Time Translation

Until recently, real-time speech translation was dominated by closed systems from tech giants. Google's Interpreter Mode, Microsoft's Skype Translator, and DeepL Voice showcased impressive capabilities but kept their technology under wraps. That's changing with the release of two powerful open-source alternatives.

Seamless Streaming: A Universal Translator

Meta's Seamless Streaming is a multilingual powerhouse that I've personally tested. I even tested it with Armenian to challenge it. That data-set certainly needs some work, let’s just say. The model supports:

• Speech recognition in 96 languages

• Speech-to-text translation from 101 source languages into 96 target languages

• Speech-to-speech translation for 36 target languages

Using the model through Hugging Face's interface (try it yourself at huggingface.co/spaces/facebook/seamless-streaming), I experienced near real-time translation with impressively natural output. The latency is typically under .200 seconds - comparable to cell phone latency.

What sets Seamless Streaming apart is its intelligent handling of language differences. The model dynamically decides when to start translating based on the sentence structure of both languages, ensuring natural-sounding output even between very different languages like Japanese and English.

While the model is open-source, it comes with a CC BY-NC 4.0 license, limiting it to non-commercial use. This makes it perfect for research and personal projects but requires licensing for business applications.

Hibiki: High-Fidelity Voice-Preserving Translation

Hibiki takes a different approach, focusing on high-quality translation between specific language pairs (currently French-to-English) while preserving the speaker's voice characteristics. Its key features include:

• Decoder-only transformer architecture for efficient processing

• Real-time translation with minimal delay

• Voice preservation technology that maintains speaker identity

• MIT/Apache-2.0 licensed code and CC-BY 4.0 licensed models

What makes Hibiki particularly exciting is its commercial-friendly licensing and efficient architecture. It's designed to run on consumer hardware, potentially enabling offline translation devices or apps that don't require cloud connectivity.

There’s more to learn about this model, but seeing as it just was released about 6 hours ago, I have yet to play with it. I am definitely going to put this one through it’s paces and for my own purposes, will be testing it with Armenian to see how it does.

Sign up below to get an update when I am able to put this model through it’s paces.

Practical Applications and Future Potential

These models open up exciting possibilities for developers and researchers:

1. Educational Tools

• Language learning platforms with real-time feedback

• Multilingual classroom support

• Conference translation systems

2. Communication Tools

• Cross-language video chat applications

• Community forum translation

• International customer service solutions

3. Content Creation

• Live stream translation

• Multilingual podcast production

• Real-time video dubbing

The Road Ahead

While these models represent significant progress, challenges remain:

• Handling complex accents and dialects

• Managing context and cultural nuances

• Balancing latency with translation quality

• Scaling to more language pairs

However, the open-source nature of these projects means the entire community can contribute to solving these challenges. We're likely to see rapid improvements and new language pairs added as researchers and developers build upon these foundations.

Conclusion

The release of Seamless Streaming and Hibiki marks a turning point in speech translation technology. Their open-source nature democratizes access to advanced translation capabilities, enabling innovation beyond what any single company could achieve. Whether you're a developer, researcher, or just curious about the technology, these models provide an exciting glimpse into the future of human communication.