FunAudioLLM is a set of speech processing models developed by Alibaba to improve speech interaction between humans and large language models. It consists of two main models: SenseVoice and CosyVoice.
SenseVoice : A speech recognition model that can recognize speech in multiple languages, identify the speaker’s emotions, and detect special events in audio (such as music, laughter, etc.). It can transcribe speech content quickly and accurately.
CosyVoice : Speech generation mode, this model mainly generates natural and emotionally rich speech. It can imitate different speakers and even clone a person’s voice with a few seconds of audio samples.
Through the combination of SenseVoice and CosyVoice, FunAudioLLM provides comprehensive speech understanding and generation capabilities, making the speech interaction between people and large language models more natural and rich.
FunAudioLLM
Key Features of SenseVoice and CosyVoice
SenseVoice focuses on multilingual speech recognition, emotion recognition, and audio event detection, providing high-precision, low-latency speech processing capabilities. CosyVoice focuses on natural speech generation and control, supports the generation of multiple languages, timbres, and speaking styles, and can achieve zero-sample learning and fine-grained speech control. The combination of the two enables FunAudioLLM to provide an excellent voice interaction experience in a variety of application scenarios.
SenseVoice Key Features
Multilingual speech recognition
SenseVoice-Small: Supports five languages: Chinese, English, Cantonese, Japanese and Korean. It adopts a non-autoregressive end-to-end architecture with extremely low recognition latency. It is 5 times faster than Whisper-small and 15 times faster than Whisper-large.
SenseVoice-Large: High-precision speech recognition supporting over 50 languages.
2. Emotion Recognition
Recognize emotions in speech, such as happiness, sadness, anger, etc., by detecting the pitch, rhythm, and intonation changes of speech.
3. Audio Event Detection
Detect special events in speech, such as music, laughter, applause, etc., and predict the start and end time of the event.
SenseVoice-Small can detect various human-computer interaction events, such as background music, applause, laughter, crying, coughing and sneezing.
4. Language Identification
Ability to identify the language used by the speaker to ensure accuracy of speech recognition and contextual understanding.
5. Inverse Text Normalization (ITN)
Provides punctuated and formatted transcription results to improve the readability and accuracy of the transcribed text.
Main Features of FunAudioLLM
Multilingual speech recognition: With more than 400,000 hours of training data, the recognition performance is better than the Whisper model.
Efficient inference: The SenseVoice-Small model uses a non-autoregressive end-to-end framework with extremely low inference latency. It only takes 70 milliseconds to process 10 seconds of audio, which is 15 times faster than Whisper-Large.
Emotion Recognition: On multiple test datasets, it achieved the results of the current best emotion recognition model.
Event detection: Supports multiple common audio event detections.
Convenient fine-tuning: Convenient fine-tuning scripts and strategies are provided, so users can easily solve the long-tail sample problem according to business scenarios.
Service deployment: Provides service deployment pipeline, supports multiple concurrent requests, and client languages include Python, C++, HTML, Java, and C#, etc.
Key Features of CosyVoice
Speech Generation
Supports multi-language speech generation, including Chinese, English, Cantonese, Japanese, and Korean.
Able to generate natural and emotionally rich speech, supporting different speaking styles and emotional expressions.
2. Diverse voice control
Timbre control: The timbre of the generated speech can be precisely controlled to match it to the voice of a specific speaker.
Speaking style control: Control the speaking style of the voice, such as emotion, speaking speed, pitch, etc., through text commands.
3. Zero-shot learning
Voice cloning from just a few seconds of audio samples, without the need for additional training data.
Supports cross-language voice cloning, allowing you to speak in one language using the voice of another language.
4. Fine-grained control of paralinguistic features
It supports inserting subtle speech features such as laughter, breathing, and modal particles to make the generated speech more natural and vivid.
Text command control: The speaker’s identity, emotion, and speaking style can be precisely controlled through text commands.
5. Multi-character dialogue
It can generate multi-character conversational speech, suitable for scenarios such as interactive podcasts and emotional chats.
