Traditional multimodal generative models usually require the design of specialized processing methods or models for different modalities such as text and images. For example, text uses a language model, while images use a diffusion model or other generative models. This approach requires multiple independent models, making it difficult to efficiently process and generate multiple types of data within the same framework.
Researchers from Meta and the University of Southern California developed a Transfusion model that solves this problem by processing both text and images in a unified model.
Transfusion can process and generate both discrete data (such as text) and continuous data (such as images) at the same time. The model combines the next token prediction task of the language model (for processing text) and the technology of the diffusion model (for processing images) to train a unified model that can handle multiple modalities.
Unified model architecture: Transfusion proposes a single Transformer architecture that can process both text and image data in the same model. This eliminates the need to use different model architectures for different modalities, thereby simplifying the processing of multimodal data.
Avoiding information loss: By applying the diffusion model directly on the image instead of quantizing the image into discrete tokens, Transfusion retains the complete information in the image. This enables the model to generate higher quality images and avoids the information loss caused by quantization.
Higher computational efficiency and generation quality: Transfusion demonstrates higher computational efficiency and generation quality when dealing with cross-modal tasks, especially in text-to-image and image-to-text tasks, where Transfusion outperforms traditional methods.
A series of experiments verified the performance of the Transfusion model in single-modal and cross-modal tasks, including text-to-text, image-to-text, and text-to-image generation tasks. In comparison with the Chameleon method, Transfusion showed better scalability and efficiency at various scales and computational loads, especially in image generation tasks, where its computational efficiency was 34 times better than Chameleon. In addition, Transfusion also outperformed Chameleon in text tasks, even though the two adopted similar approaches to text modeling.
Transfusion
Key Features of Transfusion
1. Multimodal Generation
Text-generated images: Transfusion can generate high-quality images based on the input text description, similar to text-to-image generation models (such as DALL-E). The model combines the sequence prediction of the language model and the image generation capabilities of the diffusion model during the generation process. The generated images not only conform to the text description, but also have high quality.
Image-generated text: Transfusion can also generate descriptive text based on the input image, such as generating a title or description of the image. This function has important applications in automatic annotation or understanding of image content.
Joint modality generation: The model can generate text and image content simultaneously, which is very useful in multimodal content creation, description generation and other application scenarios. It can insert an image into a text or generate a corresponding text description in the context of an image according to user needs.
Transfusion: Joint modality generation
2. Unified multimodal processing
Processing discrete and continuous data: Transfusion can process both discrete data (such as text) and continuous data (such as images) in the same model. Through the unified Transformer architecture, the model can understand and generate multimodal data without the need to design separate processing models for different modalities.
Mixed modality data training: The model can be trained on both text and image data, using the language model’s loss function for text prediction and the diffusion model’s loss function for image generation. This allows the model to effectively learn and comprehensively process data from different modalities.
3. Cross-modal generation
Generate images from text and text from images: Transfusion supports crossing different modalities in the generation process. For example, after generating a piece of text, it can continue to generate images related to it, and vice versa. This cross-modal generation capability makes it particularly suitable for complex multimodal tasks such as multimodal content creation or automated report generation.
4. Image compression and efficient generation
Image Compression and Generation: By using VAE (Variational Autoencoder) to encode images into compact patch representations, Transfusion is able to efficiently process image data and reduce computing resource consumption while maintaining high-quality image generation effects.
5. Scalability and performance optimization
Scalability: Transfusion can further improve performance by increasing the size of model parameters or training data. This allows the model to maintain efficient generation quality when processing larger-scale multimodal data.
Architecture flexibility: Transfusion uses an adjustable model architecture, such as the ability to choose different patch sizes or encoding and decoding layers, which allows the model to be optimized for specific application scenarios to balance performance and computational cost.
Flexible modality encoding and decoding: The model uses flexible modality encoding and decoding mechanisms, such as using U-Net layers to better encode and decode image data, thereby improving the quality of image generation. This flexibility allows the model to better adapt to different types of input data.
Technical Methods of Transfusion
1. Model architecture design
Model architecture of Transfusion
Unified Transformer Architecture
The Transfusion model uses a unified Transformer architecture to process data of different modalities. Both text and image data are processed by the same Transformer. This design enables the model to share parameters between different modalities, enhancing cross-modal understanding and generation capabilities.
Modality-specific encoding and decoding layers:
Text processing: Text data is converted into vector representation through a standard embedding layer and enters the Transformer for processing.
Image processing: Image data is first encoded into continuous patches (image blocks) through a variational autoencoder (VAE), and then these patches are further encoded into vectors suitable for Transformer processing through linear layers or U-Net layers.
2. Data representation and processing
Text representation: Text data is processed through standard tokenization, where each token is represented as a discrete integer, which is then converted into a vector representation for use by the Transformer.
Image representation: The image is first encoded into a low-dimensional continuous vector through VAE. Each image is divided into multiple patches, and each patch is represented as a continuous vector. These patches are arranged in order to form a sequence so that they can be mixed with text data.
Mixed-modal sequences: During training, text and image data are mixed in the same sequence, and the patch sequence of image data is surrounded by special start-of-image (BOI) and end-of-image (EOI) markers, indicating the start and end positions of the image.
3. Attention Mechanism
Causal Attention for Text: For text data, the model uses a standard causal attention mechanism to ensure that when generating each token, it can only focus on the content before the current token, thereby achieving the prediction of the next token.
Bidirectional Attention for Images: For image data, the model allows bidirectional attention between different patches within the same image. This means that each patch can pay attention to all other patches in the image, thereby better capturing global information when generating images.
Transfusion: Bidirectional Attention for Images
4. Training objectives (Loss Functions)
Language Modeling Objective (LM Loss): For text data, the model adopts the standard next token prediction task and optimizes the model by minimizing the cross entropy loss between the predicted token and the true token.
Diffusion Loss: For image data, the model generates a clear image by learning how to reverse a process of gradually adding noise. Specifically, the model learns how to gradually restore the image from the noise, which is achieved by minimizing the noise prediction error (mean square error).
Joint loss function: The overall loss function of the Transfusion model is the weighted sum of the language model loss and the diffusion model loss, where the language model loss and the diffusion model loss are optimized for text and image data respectively.
5. Reasoning process
Text generation: When generating text, the model follows the standard method of language models, sampling token by token from the model distribution until a complete sentence or paragraph is generated.
Image generation: When the model encounters a BOI marker in a sequence, it enters image generation mode. The model first inputs pure noise, then gradually restores the image through multiple diffusion steps until a complete image is generated. When image generation is complete, an EOI marker is added to the sequence, and the model returns to text generation mode.
Transfusion: Image generation
Experimental Results of Transfusion
1. Comparative experiment: Transfusion and Chameleon
Text generation quality: In text generation tasks (e.g., perplexity evaluation on the C4 and Wikipedia datasets, and accuracy evaluation on the Llama 2 task suite), Transfusion outperforms the Chameleon model in all tests. Even with the same parameters and computational effort, Transfusion performs significantly better in text generation.
C4 perplexity: The perplexity of the Transfusion model is lower than that of Chameleon, showing better text generation capabilities.
Llama 2 task accuracy: In the Llama 2 evaluation suite, Transfusion also has higher accuracy than Chameleon.
Image Generation Quality: Transfusion outperforms Chameleon at all model sizes on image generation tasks such as FID and CLIP scores on the MS-COCO benchmark.
FID score: In the MS-COCO test, Transfusion is significantly better than Chameleon in image generation quality, and achieves a 34-fold improvement in computational efficiency with the same amount of computation.
CLIP score: The images generated by Transfusion also outperform the Chameleon model in terms of semantic consistency with the text.
2. Extended Experimentation: Impact of Different Architecture Configurations
Impact of attention mechanism: By introducing the bidirectional attention mechanism in the image generation task, the FID score of the Transfusion model is significantly improved. Bidirectional attention allows different patches in the same image to pay attention to each other, thereby improving the quality of image generation.
Impact of patch size: Experiments have found that when using U-Net encoder/decoder, a larger patch size helps improve the performance of image generation, especially while reducing computing resource consumption while still maintaining a high image generation quality.
Comparison between U-Net and linear encoder: By comparing the image encoding and decoding methods of U-Net and simple linear layers, experiments show that the introduction of U-Net not only provides obvious performance improvements in small models, but also maintains a significant positive impact on the generation quality even in large models.
3. Large-scale model experiments
Comparison with existing state-of-the-art models: The research team trained a Transfusion model with 7B parameters and compared it with existing state-of-the-art image generation models on multiple benchmarks.
GenEval Scores: Transfusion performs close to high-performance models such as DeepFloyd and SD3 on the GenEval benchmark, and outperforms other smaller-scale image generation models such as SDXL.
Text Generation: Transfusion’s text generation capability is comparable to that of the Llama model, demonstrating its strong capability in processing plain text tasks.
4. Image Editing Experiment
Image editing capabilities: The experiment also demonstrated the potential of the Transfusion model in image editing tasks. After fine-tuning with a small amount of image editing data, Transfusion was able to generate modified images that met expectations based on the input image and editing instructions, showing the model's generalization ability in new tasks.
Transfusion: Image editing capabilities
5. Overall conclusion
Performance and Scalability: Transfusion performs well in multiple unimodal and multimodal benchmarks and demonstrates good scalability at various scales. By combining the advantages of language models and diffusion models, Transfusion significantly surpasses traditional multimodal generation methods in both computational efficiency and generation quality.
The study shows that the Transfusion model can be further extended to a larger parameter scale and can be applied to more types of continuous data (such as audio, video). Future research may explore the introduction of other continuous data types into this multimodal model to further enhance the model's multimodal processing capabilities.
