Optimization

The merging of the identified content and style with a Neural Style Transfer (NST) is posed as an optimization problem. The optimization is performed on the basis of a PerceptualLoss. A distinction is made between two different approaches.

Image optimization

In its basic form, an NST optimizes the pixels of the input_image directly. That means they are iteratively adapted to reduce the perceptual loss. This process is called image optimization and can be performed in pystiche with a image_optimization() .

Model optimization

While the image optimization approach yields the highest quality results, the computation is quite expensive and usually takes multiple minutes to complete for a single image. Model optimization on the other hand trains a model called transformer to perform the stylization. The training is performed with the same perceptual loss as before, but now the transformer weights are used as optimization parameters. The training is even more time consuming but afterwards the stylization is performed in a single forward pass of the input_image through the transformer. The quality however, while still high, is lower than for image optimisation approaches since the transformer cannot finetune the output_image. In pystiche a model optimization can be performed with a model_optimization() .

Note

Due to the differences in execution time image and model optimization approaches are often dubbed slow and fast respectively.