Reality Hypothesis

Saturday, December 26, 2020

Renewed interest

After a break of about half a year, i got back to doing 'infinite zoom' experiments using a variety of neural network setups. Here are some snapshots made during my 'travels' in which i iterated through many combinations of network architectures, training methods, hyper-parameter settings and input modes:

v17b g5d2 08000_026


v10b g6d2 079000_011


v10b g8d2 060000_055


v12c g6d2 006000_026

v14b g1d2 065000_026

v14b g1d2 065000_013

v16b g1d2 029000_019

v17b g6d2 019000_052

v10b g8d2 030000_017

v12b g5d2 050000_034

v17b g5d2 013000_039

v14b g1d2 050000_052

v10b g8d2 060000_029

v17b g1d2 025000_053

v14b g1d2 080000_013

v17b g5d2 080000_012

The image titles are formatted so that they encode important details about the setup that generated the image. Lots of experiments have been done and documenting the output in this way makes it easier to spot the most promising setups when going through the ever growing collection of snapshots.

v* : train method variant.
b/c : input sample(s) specifier (b = single, c = multiple).
g*d* : generator and discriminator variant.
*_* : trainstep_zoom-depth at the time of the image capture

Stay tuned! This blog is a work under construction.

Thursday, January 9, 2020

Early experiments

note: this is a work in progress

On the quest to create an infinite 'zoom and enhance' system, a question is: can artificial neural networks serve as suitable image generators? A good image generator for this task should be able to enlarge any given input image to an output image with a higher resolution. While upsampling, the generator should predict and add imaginary details to the new, larger image. If it fails to do so, the output will become more and more blurred, which is undesirable. Conventional image scalers are not up to this task (see the initial-thoughts blog).
If the generator network is able to produce sufficient detail, and offers enough variety, it can be used to feed back into itself, and thus produce virtually endless zoom sequences, as follows :

1x zoom (original)

2x zoom

4x zoom (2x2)

8x zoom (2x2x2)

16x zoom (2x2x2x2)

32x zoom (2x2x2x2x2)

64x zoom (2x2x2x2x2x2)

128x zoom (2x2x2x2x2x2x2)

256x zoom (2x2x2x2x2x2x2x2)

512x zoom (2x2x2x2x2x2x2x2x2)

1024x zoom (2x2x2x2x2x2x2x2x2x2)

2048x zoom (2x2x2x2x2x2x2x2x2x2x2)

4096x zoom (2x2x2x2x2x2x2x2x2x2x2x2)

8192x zoom (2x2x2x2x2x2x2x2x2x2x2x2x2)

16384x zoom (2x2x2x2x2x2x2x2x2x2x2x2x2x2)

32768x zoom (2x2x2x2x2x2x2x2x2x2x2x2x2x2x2)

65536x zoom (2x2x2x2x2x2x2x2x2x2x2x2x2x2x2x2)

and so on

...

After a few weeks of intense experimentation with neural network architectures and various machine learning algorithms, it is evident that generative systems based on these foundations are promising and justify further research. Almost a hundred trained network instances have been tested, and almost all of them were able to zoom in without ending up as a monochrome.
The above zoom sequences have been created by using a cropped part of the output as the new input for the network, creating a feedback loop that can be repeated indefinitely. For now this is a manual process, quite tedious. Soon the neural networks will be integrated in an application with which one can zoom and steer in real-time.

All of the tested generators encode a different 'pixel growth' process and tend to have a visually distinct style or material expression. Some of the output looks very natural, and could be plausible renderings of unseen worlds. Have a look at these 10 snapshots (click for a larger view) :

v0.7/g80-08

v0.7/g80-13

v0.7/gb36-13

v0.7/gb69-14

v0.7/gb100-09

v0.8/g52-05

v0.8/g93-05

v0.8/g114-14

v0.8/148-15

v0.8/162-13

So what are these 'artificial neural networks', and what do they do?

All the image generators networks designed and tested so far are variants of convolutional networks. Convolutional networks consist of a number of processing layers, configured so that image-like data can progress from layer to layer, undergoing increasing levels of transformation. Low level features that are detected and stored in the first layers are for example points, lines, edges, arcs and gradients, in all kinds of orientations. Each layer creates new features by combining features of the previous layer. When the network is learning, all of its layers learn how to construct filters for detecting features of interest, and also how to combine these features to form higher level features/abstractions, in such a way that the network as a whole performs well.

The convolutional network in effect transforms an input image into an output image, in this specific case, doubling the resolution. A typical network has millions of parameters (the filter kernels and connection strengths) so the space of possible behaviors is huge. How to search this space?

The actual computations in a convolutional network are simple. Most of them are multiplies and adds, with some conditionals here and there. Simple operations, but in huge quantities. Usually billions of them! And there is feedback involved. A malleable, generic computing machinery that can operate as a universal approximator of any imaginable function. A fertile precondition for emergent properties to arise. How to stimulate this?

A future blog will go deeper into the details of the network architectures and learning algorithms. These are still in heavy flux in the quest for rich, and plausible but unexpected detail.

note: this is a work in progress

Thursday, November 14, 2019

Initial thoughts

note: this is a work in progress

The devil is in the details

Instead of:

We want:

Can it be done?

If not, can it perhaps be done in smaller steps? Instead of the 8x upsampling depicted above, 4x or 2x upsampling can be attempted. In fact, if 2x upsampling works, it can be used recursively:
2x 2x = 4x, 2x 2x 2x = 8x, and so on.

2x upsampling

2x upsampling doesn't look impossible. For each pixel in the original image, four new pixels have to be generated, as a 2x2 patch that will replace the source pixel. I like to think of this process as pixel splitting: a parent picture-cell is splitting into four child picture-cells. Of course, for optimal results, the context of the parent pixel should be taken into account. Consider the way a human interpolates: local structure is filled in using knowledge of more global structures. In this example, the human will recognize the low resolution image as a fly, and this global concept can then be used to guide the generation of finer, higher resolution features. We have all seen the facet eyes of a fly and will thus be able to imagine and integrate these features into our 'minds eye' view of the fly. Surely, mistakes can and will be made in this refinement process, it is a prediction after all. We guess. We apply probabilities. Comparisons with the 'real thing' are rare, so we don't know if, and how far we are off!

A series of 2x upsampling operations: 2x 2x 2x = 8x upsampling

When starting out with a low resolution stimulus and refining it more and more, differentiation into a variety of 'species' is possible by making slightly different choices along the path. Small differences don't just add up, they multiply! This makes pixel splitting a potentially rich and fascinating image generation process, while still intuitive and close to nature.

Let's have a look at a not-interpolated image (left) and an interpolated image (right). Regular cubic interpolation is applied, which can be found in most image processing applications, nothing fancy. How do the results compare visually? If one squints the eyes, then the upsampled images both look the same. If one squints the eyes very strongly, the original images can no longer be discerned from the upsampled ones.
If one looks carefully though, both forms of upsampling disappoint, they are vague, blurry, they lack detail.

Difference images (white = no error).
Left: no interpolation, right: cubic interpolation.
Top: original (ideal) image, middle: 8x upsampled image, bottom: difference image.

Although the difference images look fairly similar, when one computes the average pixel error, the cubic interpolation wins with 1.86% compared to 2.07% for the non-interpolated image. Cubic upsampling might be favorable with many images, not only with this fly. If there are gradients in the image, then it is likely to perform quite well in those regions.

Let's put cubic interpolation to the test, using a conventional image processing application. Start with the original image of the fly. Scale it up 4x. Then scale the result up 4x again, and so on, until we have a series of 10 images:

1x zoom

4x zoom

16x zoom (4x4)

64x zoom (4x4x4)

256x zoom (4x4x4x4)

1024x zoom (4x4x4x4x4)

4096x zoom (4x4x4x4x4x4)

16384x zoom (4x4x4x4x4x4x4)

65536x zoom (4x4x4x4x4x4x4x4)

262144x zoom (4x4x4x4x4x4x4x4x4)

Clearly, each cubic upsampling step suffers from significant information loss. After only a few zooming steps, we are left with a monochrome image. Blurring did its work...

While it is perhaps scientifically sound to not introduce information we are not sure about, an artist may think differently. We can imagine details! Given a certain state of affairs, we can fantasize how it will develop and differentiate. Can such an imaginative power be implemented in software? With plausible results?
Deep Dive v1 (https://notnot.home.xs4all.nl/expandingimages/deepdive/deepdive.html) is an attempt to accomplish this. v1 uses a generative process that is based on 'shallow learning'. What if we would employ 'deep learning? Deep Dive v2?

After a week of experimentation and about ten artificial neural network architectures later, we are at a point of no return. There is some promise in this approach. It certainly beats conventional cubic interpolation: