The project aimed to implement the Neural Style Transfer Algorithm, one of the exciting applications of Convolutional Neural Networks.
The technique involves taking two images—a content image(C) and a style reference image(S) (such as an artwork by a famous painter)—and blending them together such that the output image(G) looks like the content image, but “painted” in the style of the style reference image (as shown beside).
Procedure
- A Cost Function (J) is defined that measures how good is our Generated image (G).
- There are two parts to this Cost Function.
- The first part is called the Content Loss function which is a function of the Content image and the Generated image. It measures how similar is the contents of Generated image to the content of the Content image.
- The second part is the Style Loss function which is a function of the Style Reference image and the Generated Image. It measures how similar the style of the generated image is to the style of the Style Reference image.
- Pretrained VGG-19 ConvNet is used to implement the algorithm, owing to its ability to extract more complex features from both content and style images. In the network shown below-
- The feature maps of the Style image extracted from the layers marked in red are used to store the style of the Style image.
- The feature map of the Content image in the layer marked in blue is used to extract the content of the Content image.
- A Gram Matrix is then calculated which mathematically gives us the correlation between the feature maps of an image, which in simple terms, stores the style of an image.
- Content loss is then found by simply finding the Mean Squared Error between the feature maps of the content and generated images.
- Next, Style loss is calculated as the Mean Squared Error between the gram matrices of the style and generated images.
- The pixels of the Generated image G are then trained to minimize the loss, which implies that the pixels of the Generated Image reach a stage whwere the image looks like the Content image in the style of the Style Reference image.
Results
After fine tuning of the parameters, the following results were obtained.
One can infer that if the Content and Style images are chosen appropriately, the result may turn out to be a fantastic and artistically pleasing image.