UniPortrait is a unified framework for portrait image personalization, focusing on providing highly editable image generation while maintaining identity consistency in single-person and multi-person scenarios. The framework was developed by the research team of Alibaba Group.
It is able to:
Single and multi-person image personalization: Unify the generation of personalized single and multi-person images to ensure identity consistency in complex scenarios.
High-fidelity identity preservation: When generating images, the facial features and identity information in the reference image can be accurately maintained.
Extensive facial editability: Allows users to flexibly edit and customize images based on text descriptions without losing original identity features.
Free-form input description: Supports the use of a variety of text prompts without pre-set layout or formatting restrictions.
Key Features of UniPortrait
High Fidelity
UniPortrait draws portraits that are extremely lifelike, clearly showing each person's unique facial details.
Highly editable
You can modify the portrait drawn by UniPortrait according to your own preferences, such as changing the hairstyle, expression, etc. UniPortrait can meet your needs very well.
Free creation
You can give full play to your imagination and describe in words what kind of portrait you want UniPortrait to draw. UniPortrait will try its best to understand your meaning and transform it into a vivid picture.
UniPortrait
What problem does UniPortrait solve?
Challenges of identity preservation
Traditional image personalization methods often have difficulty in accurately maintaining the facial shape and texture details of the original reference image when generating new images. These methods usually either lose spatial information or fail to focus on the facial area, resulting in poor identity consistency in the generated images.
By introducing an innovative ID embedding module and decoupling strategy, UniPortrait can flexibly edit and generate high-fidelity personalized images while maintaining facial shape and texture details.
Mixed identity issues
In multi-person image generation, traditional methods are prone to identity mixing, that is, the same generated face may contain multiple identity features at the same time. This will lead to ambiguity and inconsistency of identities in the generated images.
Through its ID routing module, UniPortrait adopts an adaptive identity assignment strategy to ensure that each facial region only receives specific identity information, thereby avoiding identity mixing and improving the identity fidelity of the image.
Layout and hint constraints for generated images
Many existing methods require users to follow a specific format when entering prompts, and often require the layout of the generated images to be pre-set, which limits the user's creative freedom.
UniPortrait supports free-form text description input, breaking this limitation and allowing more flexible and diverse image generation.
Key Features of UniPortrait
Personalized generation of single and multi-person images
Single-person image personalization: UniPortrait can generate personalized images of single people, ensuring that the generated image is highly consistent with the facial identity information in the reference image. It supports extensive editing of images while maintaining facial details such as shape and texture. This enables users to generate personalized portrait images with a specific style, expression or background based on text descriptions. For example, you can upload a photo of yourself and then generate a photo wearing different clothes or in a different scene through text description, but the person is still you.
Personalization of multi-person images: UniPortrait is not only able to process single-person images, but also to generate personalized images in multi-person scenarios. In this case, UniPortrait can ensure that each facial region only receives the corresponding identity information, avoiding the problem of multiple identities being mixed on the same face. This allows users to generate images containing multiple personalized characters, and the identity of each character can be accurately maintained and distinguished. For example, you can upload several photos of your friends, and then generate a group photo of everyone together against different backgrounds, and everyone's facial features can be accurately preserved without confusion.
High-fidelity identity preservation
Identity Embedding Module: This module extracts editable high-fidelity facial features through a decoupling strategy and embeds these features into the context space of the diffusion model. It is able to capture detailed facial structure information while maintaining the ability to edit facial identities.
By introducing an innovative ID embedding module, UniPortrait is able to highly preserve facial features, including key details such as face shape and skin texture, thereby generating images that are highly consistent with the original identity. This is especially important for generating portrait images that need to maintain realism (such as personal photos and portraits).
Multi-reference image fusion
UniPortrait supports extracting identity features from multiple reference images and fusing them to generate more representative and high-fidelity personalized images. This is particularly useful when dealing with identities that require the integration of multiple angles or expressions. That is, you can upload multiple photos of the same person, and UniPortrait will combine the characteristics of these photos to generate more accurate and realistic personalized photos.
High Fidelity and Editability
Through its unique architecture and training methods, UniPortrait is able to provide rich facial editing capabilities, such as modification of facial expressions, poses, etc., while maintaining the authenticity of facial identity.
Free Editing:UniPortrait supports free-form text input, allowing users to freely define the content, style, and layout of generated images through descriptive text without having to follow a specific format or preset. This feature greatly enhances the user's creative freedom and the diversity of image generation.
Face editability: UniPortrait allows users to make a variety of edits to the faces of people in the image through text prompts, such as changing expressions, adding accessories, adjusting age, etc. Despite these edits, the system is still able to keep the identity characteristics of the people unchanged, which provides great flexibility for personalized image generation.
Identity Gradient: UniPortrait supports interpolation between different identities, generating images that gradually transition from one identity to another through linear interpolation. This is very helpful for generating composite images with multiple identity features or exploring the gradient effect of identity features. You can use UniPortrait to perform gradient operations between two characters, such as generating a transitional photo that has both your features and those of your friend, which is very interesting.
Diverse layouts: UniPortrait can generate photos with different layouts based on your description. Whether you want single-person or multi-person photos, it can satisfy you, and the generated photos have rich layouts and content.
Compatibility with existing build control tools
UniPortrait has good extensibility and is compatible with existing generation control tools such as ControlNet and IP-Adapter, which makes it more flexible in generating controllable personalized images, allowing you to more precisely control the details in the photo, such as adjusting the person's posture or background.
Multiple application scenarios
UniPortrait can not only generate high-fidelity personalized portraits, but can also be used in a variety of scenarios such as facial feature modification, identity interpolation (smooth transition between multiple identities), and stylized generation of multi-identity images.
Text To Single ID
UniPortrait: Text To Single ID
Text To Multi-Identity
UniPortrait: Text To Multi-Identity
Technical Methods of UniPortrait
UniPortrait uses Stable Diffusion as its base model to generate high-quality images through a diffusion process. The diffusion model combines the autoencoder, CLIP text encoder, and U-Net architecture to generate corresponding images based on input text prompts.
UniPortrait relies mainly on two "secret weapons":
Identity embedding module: Capturing the identity information of people: This module is like the "eyes" of UniPortrait. It will carefully observe the photos you provide and extract the most important facial features of the people, including face shape, facial proportions, skin texture, etc., and then convert this information into a special code called identity embedding.
Unlike other AI painters, UniPortrait's "eyes" are more sensitive. It not only pays attention to the most prominent features of the people, but also pays attention to the subtle differences, such as moles and fine lines on the face, so as to remember each person's appearance more accurately.
Identity Routing Module: Put the right person in the right place:
When UniPortrait needs to draw multiple portraits, this module is like a "conductor". It will "assign" the identity embed code corresponding to each person to the correct position in the picture according to your instructions (that is, the text description you enter), ensuring that everyone appears where you want and will not be confused.
UniPortrait: Identity Routing Module
1. ID Embedding Module
The first important step of UniPortrait is to extract the facial features of the person from the reference photo. Unlike traditional methods, it does not only use the final image features, but uses the more detailed features extracted in the intermediate steps. This is done to better preserve the shape and texture of the face, making the generated image look more like the person in the reference photo.
This is one of the core modules of UniPortrait and is used to extract and embed facial features to ensure that the generated image is consistent with the identity in the reference image.
Decoupling strategy: UniPortrait uses a decoupling strategy to separate facial identity features from facial structural features, avoiding the mixing of identity-related information with other facial attributes (such as expression, posture, lighting, etc.). This is achieved in two steps:
Intrinsic ID Features: extracted from the penultimate layer of the face recognition network, contain identity-related spatial information, but are not sensitive to changes in facial expressions, postures, etc.
Face Structure Features: Extracted from shallow features of the CLIP image encoder and face recognition network, capturing the structural and texture information of the face to enhance facial similarity.
DropToken and DropPath regularization: In order to prevent the model from overfitting to identity-irrelevant information (such as lighting, posture, etc.) that may be contained in facial structure features, DropToken and DropPath regularization are introduced in the facial structure branch. This approach forces the model to rely more on intrinsic ID features, thereby achieving a balance between ID similarity and facial editability.
2. ID Routing Module
When generating photos containing multiple people, there is usually a problem: the facial features of different people may be mixed up. To avoid this, UniPortrait uses a "routing" technology to ensure that each person's facial features are not confused. It is like assigning a unique "identity tag" to each person's face, ensuring that each person's face in the generated image can maintain their own characteristics without being confused with others.
This module is mainly used for multi-person image generation and solves the problem of identity mixing in multi-person images.
Identity Assignment: During the image generation process, UniPortrait ensures that each facial region receives only one identity information through a routing network. This means that each person's face will not be confused with the features of others, thus avoiding the situation of multi-ID fusion.
Routing mechanism: This mechanism works by computing and assigning the best matching identity feature at each location in the latent space. This process is performed at each spatial location so that each facial region can be accurately matched to the corresponding identity information.
3. Two-Stage Training
The training process of UniPortrait is divided into two main stages:
Single person ID training phase: In this phase, the model mainly learns how to embed and maintain identity features in single person images. The goal of this phase is to ensure that the model can generate high-fidelity personalized images while maintaining a certain degree of editability.
Multi-ID fine-tuning stage: In this stage, the system introduces an ID routing module to specifically learn how to assign and manage identities in multi-ID images. This stage ensures that each identity corresponds to only one facial region to avoid identity mixing by adding routing regularization loss.
4. Regularization Techniques
In order to make the generated images more stable and less error-prone, UniPortrait adopts some technical means. For example, when generating images, it randomly removes some details that are easily disturbed. This can avoid the model from over-reliance on certain specific details and ensure that the generated images are more realistic.
UniPortrait uses several regularization methods to improve the performance and stability of the model:
DropToken and DropPath: These techniques are used on branches of facial structure features to ensure that the model does not over-rely on features that are prone to introducing noise, thereby enhancing the balance between identity preservation and image editing.
Routing Regularization Loss: In multi-ID generation, this loss ensures that each identity feature can only be assigned to one facial region, further reducing the risk of identity mixing.
Scenarios of UniPortrait
In experiments, UniPortrait demonstrates its superior performance in both single-ID and multi-ID personalization tasks. Compared with other methods such as FastComposer and InstantID, UniPortrait performs better in identity preservation, cue consistency, and image diversity.
UniPortrait can be applied in many fields, providing a variety of image personalization generation functions. The following are some of the main application scenarios:
Scenarios of UniPortrait
1. Personalized portrait generation
For example, you can upload a photo of yourself and use UniPortrait to generate avatars in different styles, which can be used for social media, game characters or virtual images. Whether it is cartoon style, oil painting style, or other creative styles, UniPortrait can maintain your identity characteristics while giving different artistic effects.
AI avatar generation: Users can generate customized AI avatars through text descriptions. These avatars can show different styles, expressions and postures while maintaining identity consistency.
Avatar and character creation: It can be used to create avatars, especially in games, virtual social platforms, or metaverse applications, to generate virtual characters that match the user's identity characteristics.
2. Portrait Editing
Facial attribute modification: You can edit specific facial attributes according to user requirements, such as modifying age, gender, expression, hairstyle, etc., to generate personalized portraits that meet specific needs.
Facial style transfer: UniPortrait can convert faces into different artistic styles, such as cartoon style, oil painting style, etc. by inputting text or image samples, which is suitable for creative design and artistic creation.
3. Multi-identity image generation
Multi-character scene construction: It can generate scene images containing multiple identities, such as family gatherings, group activities, etc., to meet the needs of film and television, advertising, and social media content creation.
Customized group generation: In education or business, customized group images can be generated for a group of people for use in promotional materials, team presentations, etc.
Digital Heritage Preservation:
UniPortrait can be used to generate digital versions of family photos, especially those of deceased loved ones. By digitizing old photos or photos from different eras, UniPortrait can generate high-fidelity images to help families preserve precious memories and recreate these images in different scenarios.
4. Identity Interpolation
Identity blending and transition: By interpolating between different identities, images with gradually transitioning identity features can be generated. This has broad application prospects in artistic creation and character animation production.
Identity transformation across age and gender: UniPortrait supports interpolation and transformation of identity features, and can generate images across age groups or genders. For example, you can generate photos of you at different ages, or explore your appearance after gender transformation. These features have application value in entertainment, research, and even psychotherapy.
5. Storyline Generation
Multi-scene story creation: By combining with text prompts, UniPortrait can generate multi-scene story images with coherent and consistent character images, which has great potential in the fields of comics, children's book illustrations, film storyboards, etc.
Emotional Expression and Narrative: Generate images that express different emotions by adjusting facial expressions and postures, which is suitable for affective computing and psychology research.
6. Virtual try-on and image customization
With UniPortrait, users can try out different clothing and accessories without actually wearing them. For example, you can upload a photo of yourself and then generate pictures of you wearing different clothes, hats or glasses through text descriptions. This is especially useful on e-commerce platforms to help users better choose products that are suitable for them.
Virtual Try-On: In the e-commerce and fashion sectors, UniPortrait can be used in virtual try-on applications to generate images of users wearing different clothing and accessories to help users make purchasing decisions.
Image design and consultation: Provide users with personalized image design suggestions and generate image design drawings such as hairstyle and makeup that suit the user's temperament and needs.
7. Content creation and advertising
Advertising creativity: Advertising companies can use UniPortrait to generate customized character images that match the brand image for use in advertising posters, social media promotions, and other content.
Social Media Content Creation: Content creators can use UniPortrait to generate unique, personalized images to increase the appeal and interactivity of their social media content.
8. Education and Research
Facial Recognition Research: UniPortrait’s generative capabilities can be used for testing and validation of facial recognition algorithms, generating high-fidelity datasets of images of different identities.
Psychological and behavioral research: By generating images of different facial expressions and postures, researchers can use these images in psychology and behavioral experiments to study human emotional responses and social behavior.
