The new AI-driven workflow developed by Mickmumpitz is designed to easily composite videos onto any background . It can automatically cut out the subject, match the lighting of the background, repair edges, and add additional visual effects (VFX) effects such as depth of field blur and film grain.
The workflow is available as a free download and users can further optimize the project using traditional VFX tools such as Blender and After Effects, making it suitable for a variety of film projects such as music videos or sci-fi short films.
Key points:
Workflow Features
This AI workflow simplifies the process of video compositing by automating complex tasks, allowing you to composite any video onto any background with just one click.
Customize and Optimize
While the workflow automatically handles most tasks, effects can be further fine-tuned using additional VFX tools to achieve desired results.
1. Preliminary preparation
Before you begin, make sure you have the following:
The video file you want to synthesize
Target background image or video
Install the required tools: Blender, After Effects (for further VFX optimization)
Download and install ComfyUI and the AI compositing workflow
Installation Guide : You can get detailed installation steps by following the link below: Installation Guide Link
2. Start using workflow
2.1 Importing Materials
You can use any video as input, preferably a video of a subject with a simple background. This will help the AI model better identify and cut out the subject. The AI model can automatically cut out the subject, or you can specify what to cut out through prompts.
Open ComfyUI software.
Import the video file you want to process and the target background into the software.
2.2 Automated synthesis
First, use the AI model to automatically cut out the subject in the video. The effect will be better for the subject video with a simple background. There may be some edge flickering problems during the editing process, which will be solved in the subsequent steps.
Automatically cut out subjects, match lighting, repair edges, and add basic VFX using an AI compositing workflow.
The workflow is automatically processed to produce the final composite result in just a few minutes.
2.3 Matching Lighting
In order to make the edited subject blend more naturally with the background, you need to match the subject's lighting with the background lighting. To do this, you can use the IC Light tool to adjust the lighting to make it more consistent with the background.
2.4 Edge and background restoration
In order to fix the edge problems caused by the editing process, generative AI technology can be used to automatically create a blurred mask and generate new details such as hair. This step can significantly improve the integration of the subject and the background and avoid unnatural edges.
After the initial compositing, you may want to further adjust the edges and background to achieve a more perfect result.
Edge restoration settings
Light Diffusion: Keeps the subject and background image as close to the original as possible, correcting only the edges and reducing flicker.
Number of steps: 6
Configuration: 1.5-2
Sampler: dpmpp2m
Scheduler: sgm_uniform
Denoising: 0.65
Model: v3_sd15_mm
ControlNet:v11f1e_sd15_tile
Strength: 0.65
Background repair settings
High Diffusion: Makes the subject blend more seamlessly into the image. This setting changes the style to make it more AI-like. Use BG Mask to apply only to the background, or use Reactor Face Replacement to correct the subject's face.
Number of steps: 8
Configuration: 1.5-2
Sampler: lcms
Scheduler: sgm_uniform
Denoising: 0.65
Model: AnimateLCM_sd15_t2v
ControlNet:v11f1e_sd15_tile
Strength: 0.45-0.8 (lower values will make the picture more abstract AI effect)
3. Further optimization
After you’ve finished editing your subject and matching the lighting, you can add additional effects such as depth of field blur and film grain. These effects can enhance the overall look of your video, making it more professional and artistic.
Once your automated workflow is complete, you can further fine-tune your video using Blender or After Effects. Here are some suggestions:
Lighting and color matching: Make sure the lighting and color of the composited video matches the background to avoid any abruptness.
Add additional VFX: Use After Effects to add smoke, fire, or other special effects to make your video more visually appealing.
4. Complete the project and export
Once you have made all your adjustments, export and save your final composite video in a format suitable for publishing. You can now use this composite effect in your own film projects, music videos, or other creations.
Some examples. Only the subject and background videos were imported, and the workflow was executed: automatic compositing in one click!
👍🏼
Future-proof your business with scalable infrastructure
