Attribute Diffusion

Overview

The aim of this project is to develop an attribute based facial image manipulation system using Denoising Diffusion Probabilistic Models (DDPMs) to modify specific facial attributes while maintaining the original identity and visual realism of the face.

Given a source face and a target attribute the model generates a new image of the same face with the selected attribute edited while preserving identity and realism.

Approach

1. Studied the fundamentals of Neural Networks.
2. Implemented MNIST digit classification using Artificial Neural Networks from scratch with NumPy.
3. Learned optimizers and hyperparameter tuning along with CNN fundamentals.
4. Implemented ANN and CNN models on MNIST using PyTorch.
5. Implemented a CNN on CIFAR-10
6. Learned basic Probability Theory.
7. Learned the fundamentals of Generative AI
8. Studied GANs and StyleGANs and generated images from CIFAR-10 using DCGAN.
9. Studied Denoising Autoencoder,Variational Autoencoder and Hierarchical Variational Autoencoder and implemented them on MNIST.
10. Learned DDPMs and implemented DDPM on MNIST.
11. Finally studied Attribute Diffusion and implemented the approach after reading its research paper.

Custom Dataset

Since there is no readily available dataset containing pairs of faces with and without specific attributes we created our own dataset using existing editing tools to generate pairs of faces with and without attributes.

The original and edited images were passed through Encoder4Editing (e4e) for StyleGAN2 to obtain their latent vectors in (W+) space. The difference between these vectors is then normalized to zero mean this gives the final edit direction vector.

FACE WITHOUT ATTRIBUTE	FACE WITH ATTRIBUTE

Model Architecture

Training

1. Load Data: Load a batch of latent edit direction vectors from the custom dataset.
2. Sample Noise: Generate random Gaussian noise and randomly select a timestep for each sample.
3. Forward Diffusion: Add noise to the original edit directions using a noise scheduler according to the sampled timesteps.
4. Noise Prediction: Pass the noisy edit vector and timestep embedding through the MLP denoising network to predict the added noise.
5. Calculate Loss: Compute Mean Squared Error (MSE) between the predicted noise and the actual noise.
6. Update Weights: Backpropagate the loss and update the model parameters using AdamW. Gradient clipping is applied to stabilize training.
7. Save Model: Save the model weights for checkpoints.

Editing Process

A noisy flattened edit vector is sampled and passed through the reverse diffusion process across 1000 timesteps to generate a coarse variation edit direction vector. While going to previous timestep we add noise this makes the path taken by the model to be probabilistic rather than deterministic resulting in different generated direction vectors every run.

this edit direction vector is scaled by a parameter gamma(γ) before adding the mean back this helps control the diversity of the images then the final edit vector is scaled by lambda(λ) to control the strength of the edit and then added to the encoded latent vector of the input image obtained from the e4e encoder. Finally passed through a decoder to generate the edited face.

To generate fine variations the output at some timestep (t) is saved in the initial run to generate coarse variation vector and is later used as the starting point for further reverse diffusion runs to generate fine variation vectors. Repeating this process ‘n’ times produces ‘n’ fine variation latent vectors which are then processed in a similar way as before and passed through the decoder to generate the corresponding attribute edits. The lower the selected timestep the finer the variations produced.

Results

EYEGLASSES

HAIRSTYLES

AGE

SMILE

TEAM MEMBERS

• SYED ASLAM
• SAMARTH GUPTA
• ANAMIKA DASGUPTA

MENTORS

• AMAN KARKI
• SHIVANSH SATYAM
• ROSHAN KUMAR SINGH
• MAYANK SINGH
• RITIKA VARSHNEY