KEEP is a method that focuses on making low-resolution videos clearer, especially making faces in videos look clearer and more realistic.
Generally speaking, to make faces in videos clearer, one can either use general video sharpening technology or process each image frame by frame. Both methods have problems: the former may not be able to restore facial details well, while the latter makes it difficult to maintain the smoothness between video frames.
It mainly solves two key problems in Video Face Super-Resolution (VFSR):
Stable restoration of facial details: When processing low-quality videos, how to restore clear and detailed faces is a difficult problem. Existing methods may not be able to preserve facial details well during the restoration process, resulting in blurred or distorted faces after restoration.
Temporal consistency problem: In a video, each frame is closely related to the previous and next frames. If each frame is processed separately, it will lead to inconsistency between frames. For example, the same face will look different in consecutive frames, causing a "flickering" or "jumping" effect. This temporal inconsistency will make the video look unnatural.
What problem does KEEP solve?
KEEP innovatively solves the above two problems by using the principle of "Kalman filtering":
Stable face restoration: KEEP uses the information of previously restored frames to help restore the current frame. This means that the facial detail restoration of each frame not only relies on the information of the current frame, but also refers to the information of the previous frame, thus ensuring the stability and consistency of facial details.
Temporal consistency: By integrating and propagating information between consecutive frames, KEEP ensures the consistency of faces in the video between different frames, so that the final video will not have "flickering" or unnatural facial changes.
Main Features of KEEP
Facial detail restoration
KEEP can restore clear and detailed faces from low-quality video frames. It helps to more accurately reconstruct facial details in the current frame by integrating video information from previous frames.
Temporal consistency preservation
KEEP ensures that the appearance of faces in a video remains consistent between consecutive frames. This means that faces in the video will not change or jump noticeably between different frames, making the video look more natural and smooth.
Anti-video degradation
KEEP has strong recovery capabilities for severely degraded videos (such as blur, noise, compression distortion, etc.), and can effectively restore facial details and maintain video coherence in extreme cases.
Kalman filter-based feature propagation
KEEP uses the principle of Kalman filtering to recursively use the information of previous frames to optimize the restoration of the current frame during video processing. This mechanism makes face restoration in the video more stable and accurate.
KEEP: Kalman filter-based feature propagation
Technical methods:
The KEEP framework uses a generative model to convert low-resolution images into high-resolution facial images. The key idea of KEEP is based on the principle of Kalman filtering. The model combines the function of Kalman filtering to maintain the stability of the image in time, so that the facial features in the video do not change significantly during playback.
Kalman filtering is a recursive algorithm for state estimation of dynamic systems. It can obtain a more accurate estimate of the current state by combining the current measurement value with the estimate value at the previous moment.
When we watch a video, each frame is actually a picture, and these pictures are played together. KEEP will use the information of the previous frames in the video to help restore the current frame. It's a bit like you are drawing a continuous painting, and each part of the painting refers to the previous part to ensure the consistency of the style and content of the picture.
In KEEP, this principle is used in video processing as follows:
State prediction: Use the information of the previous frame to predict the facial feature state of the current frame. This is equivalent to using the previously restored frame to predict the facial state of the current frame. Assuming that there are already several clear frames in the video, the Kalman filter will use these frames to predict the appearance of the face in the next frame.
State update: The actual observation of the current frame (i.e., the low-quality image of the current frame) is fused with the predicted state to update and obtain a more accurate state of facial features. This update process is similar to the "correction" step in the Kalman filter. When a new frame appears, the Kalman filter checks whether the frame is consistent with the one it predicted. If not, it will adjust according to the actual picture it sees to make the prediction more accurate. In this way, even if the picture is a little blurry or changes, the processed video will look more stable.
Explanation with examples
Kalman filtering is a mathematical method that, in simple terms, is used to predict and correct data. For example, suppose you are walking around a room and someone is using a camera to record your walking. The camera lens may shake, causing the captured image to be a little blurry. Kalman filtering is like a smart assistant. It will combine the previously captured images to predict where you will go next, and then correct the current blurry image to make the entire video look smoother and clearer.
Kalman filtering of KEEP
1. Feature Propagation
Feature propagation refers to the use of information between previous and next frames to improve video quality in video processing.
KEEP uses the mechanism of feature propagation to propagate the facial feature state of the previous frame to the current frame. This means that each frame in the video is not only processed independently, but also utilizes the useful information of the previous frames, which helps to maintain consistency between frames.
The role of feature propagation: When processing videos, the KEEP method remembers the facial features in each frame, such as the position and shape of the eyes and mouth, and then passes this information to the next frame. This is like when you draw a comic strip, the appearance of the characters in the previous picture will affect the appearance of the characters in the next picture, which can ensure the consistency of the character image.
2. Cross-Frame Attention (CFA)
Cross-frame attention is a technique that compares the similarities between different frames in a video.
How it works: Cross-frame attention compares adjacent frames in a video to find similar parts, such as the outline of a face, the position of the eyes, etc. It then optimizes based on these similar parts to make these features consistent throughout the video. This avoids sudden changes or distortions in facial details in the video.
The role of this mechanism is:
When processing the current frame, similar features in the previous frame are searched and matched, and these similar features are used to help restore the facial details of the current frame.
This method can avoid the problem of facial detail jumping caused by processing each frame independently.
In a video, each frame is not isolated. KEEP will use the information in the previous frame to help this frame become better. Specifically, when processing the current frame, KEEP will "find" similar parts in the previous frame and use this information to make the current frame more stable and delicate. It's like when you are doing a puzzle, you first find parts of similar colors and put them together, so that the whole picture will become more coherent.
3. Optical Flow and Spatial Warping
In the state prediction process, KEEP uses optical flow estimation technology to capture the motion information between the previous and next frames, and maps the estimation result of the previous frame to the current frame through spatial warping. This operation helps the model better capture dynamic changes while reducing information loss.
That is, when the face in the video moves, KEEP will calculate the trajectory of this movement, and then "distort" the information of the previous frame to the current frame, so that the changes in the face can be captured more accurately. This is like when you shoot a moving object, the camera will adjust the focus according to the movement of the object to keep the image clear.
4. Encoder-Decoder Structure
KEEP uses an encoder-decoder structure to process video frames:
Low quality encoder (LQ Encoder): responsible for feature extraction of the low quality current frame.
High-quality encoder (HQ Encoder): used to extract high-quality features for prediction.
Decoder: Decodes the updated feature state into a high-quality face image.
5. Quantization and Codebook
In terms of feature representation, KEEP uses a quantizer and code to compress continuous feature representation into a discrete code space. This quantization method can enhance robustness to severe degradation.
6. Training and loss function
KEEP training is divided into multiple stages, including pre-training codebook, training Kalman filter network, and final training after adding cross-frame attention mechanism. A variety of loss functions are used in the training process, including pixel loss, perceptual loss, adversarial loss, etc., to ensure the quality and consistency of the restored results.
Experimental results:
Experimental results demonstrate the effectiveness of the KEEP method in the video face super-resolution task. Whether in terms of image quality, temporal consistency or identity preservation, the KEEP method performs well and significantly outperforms other existing methods. This shows that the method is highly adaptable and robust when dealing with low-quality videos with various degrees of degradation.
results of KEEP
Local detail restoration: The KEEP method is significantly better than other methods in restoring facial details. For example, when processing details such as eyes, nose, and mouth, the effect of the KEEP method is more natural and clear.
Temporal consistency: Other methods may cause noticeable jitter or inconsistency between video frames after processing, while the KEEP method significantly reduces these problems and ensures the smoothness of the video.
Performance under severe degradation: In the most severe degradation, the KEEP method can still effectively restore facial details, while other methods often appear blurred and distorted.
Identity preservation and jitter:By comparing the fluctuations of the identity preservation score (IDS) and the key point distance (AKD), the experiments show that the KEEP method can better maintain the identity consistency of the face in the video and significantly reduce the jitter of the face position in the video. This means that the video processed by the KEEP method will perform better in applications such as facial recognition.
