Anime-ify Your Videos: A Step-by-Step Guide to Studio Ghibli-Style Animations

CN
ComfyUI.org
2025-05-28 12:34:54

1. Workflow Overview

mb7xfwczib52jwk8it210dd0b71563ac5a1218770912bfeabe358d0ee212a7b208af71681b4af86127.gif
  • Purpose
    This workflow converts real-life videos into anime-style animations, featuring:

    • Input: Real videos (e.g., 10068-0228.mp4) → Output: Studio Ghibli-style animation.

    • Style control via LoRA (e.g., LiveActionAnim model).

    • Auto-generates prompts with MiniCPM_VQA_Polished or manual input.

  • Core Models

    1. Wan2.1-T2V-1.3B: Base video generation model (bf16 precision).

    2. Wan2.1_VAE_bf16: Video decoder.

    3. umt5-xxl-enc-bf16: T5XXL text encoder for prompt processing.

    4. LoRA: LiveActionAnim for anime-style enhancement.

2. Key Nodes

  • WanVideoSampler: Core sampler using unipc scheduler.

  • WanVideoLoraSelect: Loads LoRA model (weight=0.7).

  • MiniCPM_VQA_Polished: Analyzes input video to generate English descriptions (e.g., "Studio Ghibli style...").

  • BaiduTranslateNode: Translates Chinese prompts to English.

  • Dependencies

    • Models:

      • Place Wan2.1-T2V-1.3B.safetensors in models/wan_video/.

      • Download umt5-xxl-enc-bf16.safetensors to models/text_encoders/.

    • Plugins:

      • ComfyUI-WanVideoWrapper: Essential for video nodes.

      • comfyui-videohelpersuite: Video loading/combining.

3. Workflow Structure

  • Group 1: Input Processing

    • VHS_LoadVideo: Loads video (832x480, 16fps).

    • DepthAnythingV2Preprocessor: Extracts depth maps.

  • Group 2: Prompt Generation

    • Auto path: MiniCPM_VQA_Polished → Translation → Text encoding.

    • Manual path: Direct prompt input.

  • Group 3: Video Generation

    • WanVideoModelLoaderWanVideoSamplerWanVideoDecode.

4. Inputs & Outputs

  • Inputs:

    • Video: MP4 format (e.g., 10068-0228.mp4).

    • Prompts: Auto-generated or manual.

    • Resolution: 832x480 (adjustable via ImageResizeKJ).

  • Output:

    • Anime-style video (MP4, H.264 encoded).

5. Notes

  • Performance:

    • Requires ≥12GB VRAM; bf16 models recommended.

  • Troubleshooting:

    • Missing umt5-xxl-enc-bf16? Download from HuggingFace.

  • Compatibility:

    • NVIDIA GPU only (CUDA 12.1+).