From Short Clips to Real-Time Streams:Identity-Aware Supervision and Context-Efficient Generation for Video Face Swapping
21.174 FPSEnd-to-end throughput
1 RTX 4090Single-GPU inference
10 minutesMaximum evaluated video length
APPLICATION SHOWCASE
Applications
Real-time Streaming Test
Streaming Demonstration
We tested real-time streaming on a 19-minute live-streaming video sampled at 20 FPS.
The demos are for research and model evaluation only; harmful or unethical use is not supported.
OVERVIEW
Abstract
Video face swapping transfers the reference identity to a target video while preserving its other attributes and motion. Despite recent progress in reference-guided flow-based methods, jointly achieving faithful identity transfer, high fidelity, and real-time generation over long video streams remains challenging. To address this challenge, we rethink the task-specific structure of reference-guided video face swapping. First, face swapping changes only identity while preserving most visual content, yet flow-matching supervision is weakly sensitive to these changes. We therefore develop Joint Rec-ID Loss, which balances explicit identity supervision and latent reconstruction according to the noise level. Second, the target video already provides non-edited attributes and temporal dynamics, enabling generation with fewer spatial tokens and shorter temporal contexts through Face-RoI Generation and Short-Context Rollout Training. By integrating these designs, we propose StreamingSwap, a reference-guided flow-matching framework for real-time streaming video face swapping. We also introduce HumanFaceVid, comprising 1,000 hours of cinematic training data and evaluation sets of 200 short and 200 long videos. Experiments demonstrate that our method achieves competitive identity transfer and visual quality at 20 FPS on a single NVIDIA RTX 4090 GPU, with stable generation over videos up to 10 minutes long.
Abstract · pp. 1–2 · Original paper text
TASK-SPECIFIC ANALYSIS
What limits the capability and speed of reference-guided face swapping?
Q1 · IDENTITY SENSITIVITY
How strongly does the flow-matching objective penalize identity-transfer failures?
Face swapping changes only identity while preserving most target content, making latent reconstruction supervision weakly sensitive to identity changes. Identity similarity varies by more than 0.75, but mean face-region latent MSE varies by less than 0.05.
Figure 2(a). Reconstruction error is weakly sensitive to identity changes.
Design implication: Training needs to directly supervise source identity to complement the weak sensitivity of latent reconstruction to identity changes.
Q2 · CONTEXT REQUIREMENTS
How much spatial and temporal context is actually required to preserve fidelity?
The target video already supplies content and motion, allowing more compact spatial and temporal context.
Spatial context: focus on the face
Keeping 39% of spatial tokens largely preserves quality while increasing throughput from about 1.3 to 2.2 FPS.
Figure 2(b). Spatial coverage versus identity, fidelity, speed, and computation.
Temporal context: keep local continuity
Six neighboring latent frames retain most quality with less computation; reducing context to two causes a sharp quality drop.
Figure 2(c). Temporal context versus identity, fidelity, speed, and computation.
Design implication: Generation needs to exploit the target-provided content and motion to concentrate computation on face-region tokens and a short temporal context.
Section 3 · pp. 4–5 · Context-analysis speeds refer to the initialization model.
METHODOLOGY
Method Overview
Phase I restricts processing to face-region tokens and balances latent reconstruction with identity supervision according to noise level. Phase II uses self-rollout with short generated histories, enabling streaming inference while retaining bidirectional attention within each chunk.
01
Joint Rec-ID Loss
Balance explicit identity supervision with reconstruction according to noise level, preserving identity and target attributes together.
02
Face-RoI Generation
Generate face-region tokens and blend them with the target video, avoiding redundant background computation.
03
Short-Context Rollout Training
Train on generated history to reduce error accumulation; retain bidirectional attention within each short window.
Section 4 · pp. 6–7 · Data preparation in Appendix B
EXPERIMENTAL EVIDENCE
Comparisons with Existing Methods
HumanFaceVid: 200 short and 200 long videos. All speed measurements use a single RTX 4090.
Short Videos
HumanFaceVid · 200 videos
Table 1 · Short-video results · ↑ Higher is better · ↓ Lower is better
Short-video Visual Comparisons
Long Videos
HumanFaceVid · 200 videos
Table 1 · Long-video results · ↑ Higher is better · ↓ Lower is better
Long-video Visual Comparisons
StreamingSwap balances identity, target fidelity, and visual quality at 21.174 FPS. InSwapper and DreamID-V achieve higher identity scores, but the visual comparisons show that higher embedding similarity does not necessarily imply better perceptual quality or target fidelity.
Table 1 · FPS counts new output frames; latency is per inference run (Appendix D).
HUMANFACEVID
Data Construction
Built from OpenHumanVid, HumanFaceVid provides 1,000 hours of training data and 200 short / 200 long evaluation videos. Reversed-pair training uses synthetic inputs and original real videos as ground truth.
Figure 6. Data preparation and role-reversed training for HumanFaceVid.
