The Future of Portrait Editing: Harnessing ControlNet and LoRA Models

✅ Workflow Overview

This workflow, titled "Consistent Portrait Pose Transfer", aims to generate new poses for a reference portrait while maintaining consistency in facial features, style, and details. It achieves this by combining ControlNet with LoRA models to ensure that the generated image retains the original character's appearance despite the pose change.

Core functionalities:

• ✅ Loading base models and VAE

• 🎯 Using ControlNet for pose control

• 🖼️ Generating new poses while preserving the reference portrait’s appearance

• 🔥 Using LoRA models to maintain consistency

• ⚙️ Image preview and comparison

This workflow is particularly useful for character consistency, pose transfer, animation frame generation, and style transfer.

🔥 Core Models

1. Flux Base Model

   • Name: Flux Fill dev - fp8_黑森林官方重绘模型_fp8_e4m3fn

   • Function: A base model used for generating high-quality images, supporting style and detail consistency.

   • Loading Method: Loaded using the UNETLoader node.

2. LoRA Model

   • Name: ACE++_1.0

   • Function: Enhances portrait consistency, ensuring that the generated image resembles the reference in facial features, clothing, and style.

   • Weight: 1.0

   • Loading Method: Loaded via the LoraLoader node.

3. ControlNet Model

   • Name: FLUX.1-dev-ControlNet-Union-Pro-InstantX.safetensors

   • Function: Generates pose-guided images by applying the pose of the reference image to the generated portrait.

   • Loading Method: Loaded using the ControlNetLoader node.

4. Florence-2 Model

   • Name: microsoft/Florence-2-base

   • Function: A vision-language model used for image feature extraction.

   • Loading Method: Loaded using the Florence2ModelLoader node.

🔧 Node Explanation

Here’s a detailed explanation of the key nodes used in this workflow:

🌟 Base Model Loading

• UNETLoader

  • Function: Loads the base model.

  • Parameters:

    • Model name: Flux Fill dev - fp8_黑森林官方重绘模型_fp8_e4m3fn

    • Precision: fp8_e5m2

  • Output: Model data for image generation.

• VAELoader

  • Function: Loads the VAE (Variational Autoencoder) for image decoding.

  • Parameters:

    • Model name: ae.sft

  • Output: VAE data for image encoding and decoding.

🎯 LoRA Model Loading

• LoraLoader

  • Function: Loads the LoRA model to enhance portrait consistency.

  • Parameters:

    • Model name: ACE++_1.0

    • Weight: 1.0

  • Output:

    • Model Data: Used for generating consistent portraits.

    • CLIP Data: For text encoding.

🖼️ Reference and Pose Image Processing

• LoadImage (Reference Image)

  • Function: Loads the reference portrait.

  • Parameters:

    • File name: image.png

  • Output: Image data.

• LoadImage (Pose Image)

  • Function: Loads the new pose reference image.

  • Parameters:

    • File name: image.png

  • Output: Pose image data.

• DWPreprocessor

  • Function: Preprocesses the pose image to extract keypoints.

  • Parameters:

    • Resolution: 512

    • Detection models:

      • yolox_l.onnx (human detection)

      • dw-ll_ucoco_384_bs5.torchscript.pt (pose detection)

  • Output:

    • Pose Image

    • Pose Keypoints

🔥 Pose Control and Consistent Generation

• ControlNetLoader

  • Function: Loads the ControlNet model.

  • Parameters:

    • Model name: FLUX.1-dev-ControlNet-Union-Pro-InstantX.safetensors

  • Output: ControlNet data.

• SetUnionControlNetType

  • Function: Sets the ControlNet model type to openpose.

  • Output: ControlNet data.

• ControlNetApplyAdvanced

  • Function: Applies the ControlNet pose image to the generation process.

  • Parameters:

    • Strength: 0.9

    • Negative conditioning weight: 0.3

  • Output:

    • Positive Conditioning

    • Negative Conditioning

🔥 Sampling and Generation

• KSampler

  • Function: Generates the image using the base model, conditioning, and ControlNet pose guide.

  • Parameters:

    • Random seed: 597634329310649

    • Steps: 30

    • CFG Scale: 1

    • Sampler type: euler

    • Beta schedule: simple

  • Output: Latent image data.

• VAEDecode

  • Function: Decodes the latent image into the final image.

  • Inputs:

    • Latent image

    • VAE

  • Output: The final generated image.

🛠️ Image Processing and Comparison

• ImageResize+

  • Function: Resizes the image to match the reference dimensions.

  • Parameters:

    • Width: 1024

    • Height: 1024

    • Algorithm: lanczos

  • Output: Resized image.

• ImagePadForOutpaint

  • Function: Pads the image for seamless inpainting.

  • Parameters:

    • Direction: Right

    • Padding: 40px

  • Output:

    • Padded image

    • Mask

• ImageConcanate

  • Function: Concatenates the reference and generated images side by side.

  • Parameters:

    • Direction: Right

  • Output: Concatenated image.

• PreviewImage

  • Function: Displays the generated image and comparison image.

🔍 Workflow Structure

The workflow is divided into several Groups, each serving a specific purpose:

1. ✅ Base Model Loading

   • Location: Top left

   • Function: Loads the base model and VAE.

   • Input: Model path

   • Output: Model data.

2. 🎯 Reference and Pose Image Loading

   • Location: Bottom left

   • Function: Loads the reference and pose images.

   • Input: Image files

   • Output: Image data.

3. 🔥 LoRA Model Selection

   • Location: Center left

   • Function: Loads the LoRA model for character consistency.

   • Input: Base model

   • Output: Style-enhanced model.

4. 🛠️ Pose Detection and ControlNet Application

   • Location: Center

   • Function: Detects pose keypoints and applies ControlNet.

   • Input: Pose image

   • Output: Pose-conditioned data.

5. 🖼️ Image Output

   • Location: Right

   • Function: Generates and saves the final image.

   • Input: Latent image, VAE

   • Output: Final image file.

🔑 Inputs & Outputs

Input Parameters:

• Reference portrait (image.png)

• Pose reference image

• Flux model and LoRA model

• ControlNet model

• Random seed: 597634329310649

Output:

• Generated portrait with new pose

• Side-by-side comparison image

• Image preview

⚠️ Tips & Considerations

• ✅ Model Dependencies:

  • This workflow requires ControlNet and LoRA models. Ensure they are properly installed.

• ⚡ Performance Requirements:

  • A 12GB+ NVIDIA GPU is recommended for smooth performance.

• ⚠️ Output Variations:

  • Different seeds will generate different pose images.

  • Adjusting the LoRA weight fine-tunes consistency.

• 🔥 Optimization Tip:

  • Use fixed seeds and consistent resolution for repeatable, consistent results.