What problem does Mini-Omni solve?

1. Real-time voice interaction delay problem: Traditional models usually rely on a two-step process of generating text and then converting it into voice when generating voice, which causes significant delays and affects user experience. Mini-Omni uses parallel generation technology to generate text and voice at the same time, greatly reducing response time and achieving true real-time voice interaction.

2. Integration of speech and text reasoning capabilities: Most existing large language models perform well in text reasoning, but are relatively weak in speech reasoning. Mini-Omni retains the strong capabilities of language models in text reasoning through innovative training methods and model architectures, and extends these capabilities to speech processing and generation.

3. Reduce model complexity and resource requirements: Mini-Omni simplifies the process of integrating speech capabilities into large language models through the "Any Model Can Talk" approach. This approach requires less additional training data and model adjustments, allowing other models to quickly have speech interaction capabilities, reducing resource and time consumption.

Main Features of Mini-Omni

Real-time voice input and output

Speech Recognition and Generation

Multimodal Understanding and Generation

Parallel generation technology

“Any Model Can Talk” approach

Batch Parallel Inference

VoiceAssistant-400K dataset supports

Technical Approach of Mini-Omni

End-to-end speech generation architecture

• Mini-Omni adopts an end-to-end voice input and output architecture, directly from voice input to voice output, avoiding the traditional voice-to-text and voice-to-text steps, greatly reducing latency, and providing real-time voice conversation capabilities.

Thinking while talking

• Concept: During a conversation, Mini-Omni is able to generate audio while reasoning. This “thinking and talking” feature is achieved through delayed parallel generation.

• Working mechanism: When the model generates each layer of audio tokens, it uses a delay technique to generate text tokens first and then gradually generate audio tokens. After generating the first layer of text tokens, the multi-layer codebook of the audio encoder SNAC starts to generate audio tokens in parallel. This allows the model to generate high-quality audio with a short delay.

• Technical advantages: By delaying parallel generation, the model effectively solves the complexity of generating text and audio simultaneously while ensuring high-quality audio output.

Parallel generation technology

• Mini-Omni uses a parallel generation strategy to generate text and voice responses simultaneously, reducing the time it takes to generate speech and ensuring that users get feedback almost instantly. Parallel generation can also flexibly handle tasks in different modalities.

• Core idea: Mini-Omni introduces a text-guided parallel generation strategy, which generates text while generating audio, and uses text reasoning capabilities to improve the accuracy of audio generation.

1. Implementation: The model assumes that text has a high information density, so it achieves simultaneous output of text and audio by generating text first and then generating audio. Before generating audio, the model will first generate corresponding text tokens based on the text generation part, and then generate audio tokens. This reduces the waiting time for audio generation and achieves synchronous output of voice and text.Technical advantages : The parallel generation strategy solves the delay problem caused by the traditional method of first generating text and then generating audio, greatly speeding up the speech generation speed and improving real-time performance.

Batch Parallel Inference

• Core idea: Batch parallel generation technology is used to improve the efficiency and accuracy of the model when processing audio and text reasoning tasks.

• Implementation: This technology processes the model's input in batches, and each input sample needs to generate both text and audio. During the inference process, the model not only generates text output, but also generates corresponding audio output based on it. In order to enhance the model's reasoning ability when generating audio, two parallel sample generation strategies are adopted: one generates text and the other generates audio. In this process, the text output is embedded in the audio generation sample, forming a stronger reasoning ability.

• Technical advantages: This method effectively utilizes the model's powerful capabilities in text reasoning and transfers them to audio generation, greatly improving the model's performance in processing audio reasoning tasks with lower requirements for computing resources.

SNAC Audio Codec

• Core technology: Mini-Omni uses the SNAC audio encoder, which is an efficient music-grade encoder with an 8-layer codebook structure that can process a large number of audio tokens in a short time.

• How it works: The SNAC encoder efficiently encodes audio and discretizes the audio signal into multiple levels of codebooks. This encoding method greatly reduces the complexity of the model when processing audio while ensuring that the generated audio has high fidelity.

• Technical advantages: Through its multi-layer structure, the SNAC encoder allows the model to remain efficient in generating high-quality audio, avoiding the audio quality degradation problem commonly caused by low-bitrate encoders.

“Any Model Can Talk” Approach

• Concept: This is an innovative training and inference method designed to help other large language models quickly adapt to speech output capabilities.

• Implementation: This method is divided into three stages:
1. Modality alignment: First, align the model’s text and audio to ensure that the model can understand and generate speech. In this phase, the model is initially trained using speech recognition and speech synthesis data to improve its speech processing capabilities.
2. Adaptive training: Once the audio and text modalities are aligned, the model begins to focus on generating text given the audio input, and the audio output is achieved through simple text-to-audio synthesis. This stage uses data from speech question answering (Speech QA) and text question answering (Text QA) for training.
3. Multimodal fine-tuning: In the final stage, all weights of the model are unfrozen and comprehensive fine-tuning is performed using multimodal data to ensure that the model remains efficient in multimodal interactions.

• Technical advantages: This method greatly reduces the training cost of the model, allowing other language models to quickly obtain voice interaction capabilities with a small amount of additional data without making major changes to the model architecture.

Text directives delay parallel generation

• By adopting a text instruction delayed parallel generation strategy, the model first generates text and then generates speech based on the text. The efficiency of text reasoning is used to reduce the complexity of speech generation while maintaining high-quality speech output.

Audio Discretization and Coding

• Mini-Omni uses audio discretization technology to convert speech signals into discrete audio tokens for inference processing in language models. The SNAC encoder is used to ensure high-quality speech generation.

• Audio encoding: Mini-Omni uses advanced speech encoding technologies such as Whisper to discretize audio input into tokens for model processing.

• Audio decoding: When generating audio, the model decodes audio tokens through a multi-layer codebook technique to ensure that the generated audio is of high quality and low latency.

Three-stage training framework

• The training process of Mini-Omni is divided into three stages:
1. Modality expansion phase: training the model’s speech recognition and generation capabilities.
2. Adaptation training phase: Use speech recognition and text generation task data to further optimize the model's speech understanding and text generation capabilities.
3. Comprehensive fine-tuning stage: Perform multimodal fine-tuning on the model, optimize the voice output quality, and achieve flexible switching between voice and text.

VoiceAssistant-400K Dataset of Mini-Omni

Experimental Results of Mini-Omni

Automatic Speech Recognition (ASR) results

• Test set : LibriSpeech dataset, divided into four parts: test-clean, test-other, dev-clean, and dev-other.

• Evaluation indicator: error rate (WER, Word Error Rate).

2. Speech QA and Text QA Results

• Task Type:
• Text QA: The model generates text answers based on text input.
• Speech QA: The model generates spoken responses based on speech input, using a parallel generation strategy to achieve real-time response.

• Performance evaluation: Mini-Omni demonstrated efficient reasoning capabilities when processing Text QA and Speech QA tasks, especially when using batch parallel generation technology, the reasoning performance of speech output was significantly improved.

3. Effect of batch parallel generation strategy

• Improved reasoning capabilities: Batch parallel generation extends the model's reasoning capabilities from text reasoning to speech generation, significantly improving the model's performance in voice question-answering tasks.

• Improved audio quality: Through parallel generation technology, the model can generate higher quality audio, especially reducing latency in streaming output and improving user experience.

4. Speech Generation Quality Assessment

• Audio Clarity: With the SNAC audio encoder, the generated audio quality is on par with common TTS systems.

• Latency test: Although there may be a slightly longer delay due to network reasons when using Gradio for demonstration, the overall generated audio is smooth and of high quality.

5. Performance Summary

Summary

• Speech recognition capability: Mini-Omni's performance in speech recognition is close to that of the mainstream Whisper-small model, indicating that it has strong speech understanding capabilities.

• Speech generation capability: Through batch parallel generation and SNAC encoder, Mini-Omni can efficiently generate high-quality speech and significantly reduce the generation delay.

• Reasoning performance: The batch parallel generation strategy significantly improves the reasoning efficiency of the model, especially its performance in multimodal tasks, enabling it to maintain consistent and efficient reasoning capabilities in both voice question answering and text question answering.

Main Contributions of Mini-Omni

• End-to-end speech generation: Mini-Omni achieves real-time interaction between speech and text through a parallel generation strategy, reducing generation delays and making speech interaction more natural and smooth.

• “Any Model Can Talk” approach: Provides a path for other language models to quickly expand into the field of voice interaction with only a small amount of data and minimal model modifications.

• High-quality speech generation and multimodal reasoning: Mini-Omni not only performs well in speech recognition (ASR) and speech generation (TTS) tasks, but also has strong reasoning capabilities in multimodal tasks (such as TextQA and SpeechQA).

🔥

Don't Let Server Issues Slow You Down. RackNerd: Your Solution