Concept Sliders: Precisely Control Image Features

type

status

date

slug

summary

Key Features of Concept Sliders

Fine-tune the concept in the image

Fine-tune the diffusion model through the LoRA adapter

to create sliders for precise control of specific image features. For example, control age, gender, expression, etc.

Users can control certain elements in the image by simply sliding and adjusting parameters to achieve control over specific features in the generated or edited image. For example, you can make a person's eyes bigger or smaller, or adjust the light intensity in the picture. This control is continuous, and you can fine-tune it according to your needs until you are satisfied.

Text concept slider

Users can generate sliders by describing text. For example, if you enter "I want people to look happier", the model will generate a slider for adjusting the degree of "happiness".

Provides GPT-4 integrated function to generate sliders, making it easier to create sliders.

When you generate images from prompts, you often want to make more detailed adjustments to certain attributes, such as the transition between "young" and "old". Text concept sliders allow you to train the model to remember these concepts by inputting simple text descriptions, and then use sliders to control the strength of this concept. For example, you can make the person in the image gradually age as the slider moves, without changing other attributes.

concept slider: By scaling the slider factor, users can control the strength of the edit — **concept slider:** By scaling the slider factor, users can control the strength of the edit

By using a small set of textual descriptions of the attributes to control, Concept Sliders can be trained to allow finegrained control of generated images during inference. By scaling the slider factor, users can control the strength of the edit.

concept slider: image can be controlled using different sliders — **concept slider:** image can be controlled using different sliders

We show how several attributes of an image can be controlled using different sliders. We note that due to the low-rank formulation, the parameters are light weight, easy to share, and plug.

concept slider: weather sliders for “delightful”, “dark”, “tropical”, and “winter” — **concept slider:** weather sliders for “delightful”, “dark”, “tropical”, and “winter”

We demonstrate weather sliders for “delightful”, “dark”, “tropical”, and “winter”. For delightful, we notice that the model sometimes make the weather bright or adds festive decorations. For tropical, it adds tropical plants and trees. Finally, for winter, it adds snow.

concept slider: style sliders for “pixar”, “realistic details”, “clay”, and “sculpture” — **concept slider:** style sliders for “pixar”, “realistic details”, “clay”, and “sculpture”

We demonstrate style sliders for “pixar”, “realistic details”, “clay”, and “sculpture”.

Visual Concept Slider

Supports training sliders based on a set of images (e.g., before and after image pairs) to achieve precise control of specific visual features (e.g., eye size, facial features, etc.).

It can be used to make detailed edits to the properties of an image, such as enlarging eyes, changing facial expressions, etc.

Some visual elements may be difficult to describe in words, such as subtle changes in facial expressions or different lighting effects. For these concepts that are difficult to describe in words, Concept Sliders supports training sliders through paired images. You can select several pictures with contrasting effects to train the model to recognize these details, and then adjust these visual elements through sliders.

Sliders can be created for concepts that can not be described in words. These sliders are created by artists by using 6-8 pairs of images.

Stylespace latents can be transferred from styleGAN to Stable Diffusion XL.

Fix image defects

When generating images, the model sometimes has some common problems, such as unnatural hand shapes, blurry objects, or distorted lines. Concept Sliders provides a "healing slider" that can help fix these problems. By adjusting the slider, you can make the model generate clearer, more realistic images with less distortion or blurriness.

Concept Slider: healing slider — **Concept Slider:** healing slider

The repair slider enables the model to generate images that are more realistic and undistorted. The parameters under the control of this slider help the model correct some of the flaws in their generated outputs like distorted humans and pets in (a, b), unnatural objects in (b, c, d), and blurry natural images in (b,c).

Concept Slider: repair slider — **Concept Slider:** repair slider

We demonstrate the effect of our “repair” slider on fine details: it improves the rendering of densely arranged objects, it straightens architectural lines, and it avoids blurring and distortions at the edges of complex shapes.

We demonstrate a slider for fixing hands in stable diffusion. We find a direction to steer hands to be more realistic and away from “poorly drawn hands”.

Concept Slider: a slider for fixing hands in stable diffusion — **Concept Slider:** a slider for fixing hands in stable diffusion

Image Editing

Real-life image editing with sliders. Null Inversion technology allows you to modify specific areas of an image using sliders without text cues.

This greatly simplifies the image editing process and is suitable for processing various pictures and visual tasks.

To combine multiple sliders

If you want to control multiple image elements at the same time, Concept Sliders allows you to combine multiple sliders. For example, you can adjust multiple details such as lighting effects, weather, style, etc. at the same time, which can produce more complex and layered images. For example, you can adjust the weather slider to "Winter" and the style slider to "Pixar Style" at the same time to achieve a specific effect.

Concept Slider: “cooked” and “fine dining” food sliders — **Concept Slider:** “cooked” and “fine dining” food sliders

We show blending “cooked” and “fine dining” food sliders to traverse this 2D concept space. It is interesting how the model makes portion sizes small for “fine dining”.

Concept Slider: multiple sliders progressively up to 50 sliders at a time — **Concept Slider:** multiple sliders progressively up to 50 sliders at a time

We qualitatively show the effects of composing multiple sliders progressively up to 50 sliders at a time. We use far greater than 77 tokens (the current context limit of SDXL) to create these 50 sliders. This showcases the power of our method that allows control beyond what is possible through prompt-based methods alone.

Avoid interfering with other properties

Sometimes, when you adjust one attribute (such as changing a character's age), other unrelated attributes (such as the character's race) will also change, causing the image to be distorted. Concept Sliders introduces a method to keep certain attributes unchanged, such as keeping the character's race and gender unchanged when you change the age, thus avoiding the distorted image effect.

What can you do with Concept Sliders?

More precise creation: If you are creating a work of art, you can first generate a base image, and then gradually adjust the details in the image through the slider until it is exactly what you want, without having to constantly modify the prompt words and regenerate them.

Repair problem images: If your generated image has problems such as unnatural fingers and crooked building lines, you can use the repair slider to improve these details and improve the image quality.

Multiple style controls: You can adjust multiple sliders at the same time to generate mixed-style images, such as combining "cartoon style" and "retro style", or making the weather in the picture gradually transition from "sunny" to "winter".

Technical Highlights of Concept Sliders

Lightweight and fast

LoRA technology makes model adjustments lightweight. There is no need to retrain the entire model. You only need to modify a small number of parameters to quickly achieve the desired effect.

High-precision control

Concept Sliders allow you to make continuous, precise adjustments to specific details in your image without having to regenerate or enter complex prompts.

Adjust multiple attributes simultaneously

You can use multiple sliders to control different image attributes at the same time to achieve complex combination effects.

Project Website | Arxiv Preprint | Trained Sliders | Colab Demo | Huggingface Demo

🔥

High Performance, Low Cost: Get more bang for your buck with RackNerd