All resources

SketchColour Pipeline for 2D Animations

Friday, 4 September 2026Blog
By Izma Khan

Well hellow fellow reader.

It is a bright Friday morning in the month of September as I write this. The sun’s been scorching beyond its limit for a couple of months now, but there’s a hint of rain in the sky. And this, my dear snozzwangers and whangdoodles, is a sign. A sign to replay the reunion of Taki and Mitsuha in Weathering With You or lose ourselves in the magical world of Spirited Away. I cannot help but imagine myself as a creator tucked away at Studio Ghibli, skipping meals and surviving purely on green tea just to color that one final scene that flips the entire plot upside down. Now, I can only wish I had that kind of artistic endurance, mostly because I live in the breakneck world of modern AI. As an animator who still enjoys eating three square meals a day, I am officially retiring the giganticus struggle. Say hello to my cheat code: SketchColour.



SketchColour?

SketchColour is the first sketch-to-colour pipeline for 2D animation built on a Diffusion Transformer (DiT) backbone (more on that as we proceed). Instead of making artists meticulously paint every single frame by hand, SketchColour automates the process and drastically cuts down the effort. Although it still is not the most efficient method to sustain a colouring pipeline especially in an industry that values detail and precision, it surely eases our work and evidently works comparatively better than the existing models.



Why Traditional Models Failed?

Automating colorisation sounds all sunshines and rainbows but its implementation has been a mammoth of a task. The earliest models relied on GANs (Generative Adversarial Networks, where a generator network creates images and a discriminator network evaluates them → pushing each other to improve) and U-Nets (networks structured like the letter U to iteratively denoise images). Since these early systems processed each frame in isolation they lacked temporal memory resulting in accumulation of error across entire scenes.

Later systems attempted to resolve this by adopting video diffusion models utilizing a U-Net backbone paired with ControlNet to ensure spatial guidance. While effective, this architecture introduced three major problems:

  • Parameter Bloating: Since ControlNet duplicates model architectures, we end up multiplying the number of trainable parameters by billions which requires significant memory and processing capabilities.

  • Colour Bleed: During fast motions, succeeding scenes diverge from the initial coloured scene (RGB), causing a failure to align correctly within the latent space (the rich features extracted using U-Net). This results in colour bleeding, where colours from the reference frame incorrectly leak onto unrelated objects.

  • Loss of Fine Details: Standard U-Net backbones downsample feature maps early in the processing pipeline, inadvertently discarding the fine-grained line art details essential in animations.



Reviewing the Pipeline

Here’s a brief overview for the pipeline that this mode follows:

Inputs (Coloured First Frame & Sketches) -> Frozen 3D VAE Encoder -> Channel Concatenation -> CogVideoX + LoRA Backbone -> 3D VAE Decoder -> Final Output Video



The Inputs

The pipeline begins with two distinct visual inputs:

  • The Coloured First Frame ($I_{start}$): A single RGB frame that establishes the colour palette, lighting and artistic style.

  • The Sketched Video Sequence ($S$): A sequence of black-and-white frames detailing the motion and action of the scene across T frames.



Frozen 3D Vision Auto-Encoder

Before any neural network can process images, the heavy pixel-level data must be compressed into a compact, low-dimensional mathematical format known as latent representation (which also includes the most detailed features captured from the image).

  • Working Mechanism: The system passes both the coloured first frame and the sketched video sequence through a 3D VAE (Variational Auto-encoder), which compresses spatial and temporal information into a compact latent space.

  • Why are we using it though?: The weights in VAE are kept frozen. Even though black-and-white line art exists in a totally different distribution space than full-colour RGB images, the frozen VAE is robust enough to map both into compatible latent structures. Freezing it saves massive amounts of GPU memory and computational power, eliminating the need to train a separate specialised sketch encoder from scratch.



Channel Concatenation

Now this method is ingenious to the architecture and helps in preventing colour bleed while retaining textural info which the other models failed to do.

  • Working Mechanism: The model takes the latent of the coloured first frame, the latent of the scheduled noise ground-truth video, and the latent of the sketched video sequence and stacks them together channel-wise into a single uniform input tensor.

  • Its significance: Channel concatenation forces the network to read the style data, the working canvas and the structural sketches simultaneously within the exact same tensor channels. This ensures strict spatial alignment across frames, effectively resolving the latent-gap problem and preventing colour bleeding during fast motion. Furthermore, the sketch projection weights are initialised with zeros to seamlessly match output magnitude without disrupting the model.



CogVideoX + LoRA Backbone

Once the inputs are stacked, we finally come to use CogVideoX-5B-I2V, a modern Diffusion Transformer (DiT) which serves as the backbone of our model.

  • Working Mechanism: Unlike older U-Net models that downsample images locally, the DiT backbone tokenizes the video latents into spatio-temporal patches and processes them using a global transformer architecture. To guide this process without retraining billions of parameters, LoRA (Low-Rank Adaptation) matrices are injected into the attention QKVO (Query, Key, Value, Output) projections and feed-forward layers.

  • But why DIT?: The DiT architecture provides a global context window, allowing the model to understand how objects move across an entire scene rather than guessing frame by frame. Meanwhile, utilising a rank-192 LoRA keeps training extremely efficient (adding only about 10 million trainable parameters) while maintaining superior temporal consistency.



The 3D VAE Decoder (Final Output)

After the Diffusion Transformer finishes denoising and generating the output latents the data must be converted back into viewable pixels.

  • Working Mechanism: The final generated latent tensor is passed through the corresponding 3D VAE Decoder (sharing the structure of the encoder).

  • The decoder reverses the compression process, translating the mathematical latent representation back into a high-resolution, fully colored RGB video sequence that matches the animator's original sketches and color references.