Early experiments

note: this is a work in progress

On the quest to create an infinite 'zoom and enhance' system, a question is: can artificial neural networks serve as suitable image generators? A good image generator for this task should be able to enlarge any given input image to an output image with a higher resolution. While upsampling, the generator should predict and add imaginary details to the new, larger image. If it fails to do so, the output will become more and more blurred, which is undesirable. Conventional image scalers are not up to this task (see the initial-thoughts blog).
If the generator network is able to produce sufficient detail, and offers enough variety, it can be used to feed back into itself, and thus produce virtually endless zoom sequences, as follows :

1x zoom (original)

2x zoom

4x zoom (2x2)

8x zoom (2x2x2)

16x zoom (2x2x2x2)

32x zoom (2x2x2x2x2)

64x zoom (2x2x2x2x2x2)

128x zoom (2x2x2x2x2x2x2)

256x zoom (2x2x2x2x2x2x2x2)

512x zoom (2x2x2x2x2x2x2x2x2)

1024x zoom (2x2x2x2x2x2x2x2x2x2)

2048x zoom (2x2x2x2x2x2x2x2x2x2x2)

4096x zoom (2x2x2x2x2x2x2x2x2x2x2x2)

8192x zoom (2x2x2x2x2x2x2x2x2x2x2x2x2)

16384x zoom (2x2x2x2x2x2x2x2x2x2x2x2x2x2)

32768x zoom (2x2x2x2x2x2x2x2x2x2x2x2x2x2x2)

65536x zoom (2x2x2x2x2x2x2x2x2x2x2x2x2x2x2x2)

and so on

and so on

...

After a few weeks of intense experimentation with neural network architectures and various machine learning algorithms, it is evident that generative systems based on these foundations are promising and justify further research. Almost a hundred trained network instances have been tested, and almost all of them were able to zoom in without ending up as a monochrome.
The above zoom sequences have been created by using a cropped part of the output as the new input for the network, creating a feedback loop that can be repeated indefinitely. For now this is a manual process, quite tedious. Soon the neural networks will be integrated in an application with which one can zoom and steer in real-time.

All of the tested generators encode a different 'pixel growth' process and tend to have a visually distinct style or material expression. Some of the output looks very natural, and could be plausible renderings of unseen worlds. Have a look at these 10 snapshots (click for a larger view) :

v0.7/g80-08

v0.7/g80-13

v0.7/gb36-13

v0.7/gb69-14

v0.7/gb100-09

v0.8/g52-05

v0.8/g93-05

v0.8/g114-14

v0.8/148-15

v0.8/162-13

So what are these 'artificial neural networks', and what do they do?

All the image generators networks designed and tested so far are variants of convolutional networks. Convolutional networks consist of a number of processing layers, configured so that image-like data can progress from layer to layer, undergoing increasing levels of transformation. Low level features that are detected and stored in the first layers are for example points, lines, edges, arcs and gradients, in all kinds of orientations. Each layer creates new features by combining features of the previous layer. When the network is learning, all of its layers learn how to construct filters for detecting features of interest, and also how to combine these features to form higher level features/abstractions, in such a way that the network as a whole performs well. 

The convolutional network in effect transforms an input image into an output image, in this specific case, doubling the resolution. A typical network has millions of parameters (the filter kernels and connection strengths) so the space of possible behaviors is huge. How to search this space?

The actual computations in a convolutional network are simple. Most of them are multiplies and adds, with some conditionals here and there. Simple operations, but in huge quantities. Usually billions of them! And there is feedback involved. A malleable, generic computing machinery that can operate as a universal approximator of any imaginable function. A fertile precondition for emergent properties to arise. How to stimulate this?

A future blog will go deeper into the details of the network architectures and learning algorithms. These are still in heavy flux in the quest for rich, and plausible but unexpected detail.



note: this is a work in progress