Reality Hypothesis: 2019

note: this is a work in progress

The devil is in the details

Instead of:

We want:

Can it be done?

If not, can it perhaps be done in smaller steps? Instead of the 8x upsampling depicted above, 4x or 2x upsampling can be attempted. In fact, if 2x upsampling works, it can be used recursively:
2x 2x = 4x, 2x 2x 2x = 8x, and so on.

2x upsampling

2x upsampling doesn't look impossible. For each pixel in the original image, four new pixels have to be generated, as a 2x2 patch that will replace the source pixel. I like to think of this process as pixel splitting: a parent picture-cell is splitting into four child picture-cells. Of course, for optimal results, the context of the parent pixel should be taken into account. Consider the way a human interpolates: local structure is filled in using knowledge of more global structures. In this example, the human will recognize the low resolution image as a fly, and this global concept can then be used to guide the generation of finer, higher resolution features. We have all seen the facet eyes of a fly and will thus be able to imagine and integrate these features into our 'minds eye' view of the fly. Surely, mistakes can and will be made in this refinement process, it is a prediction after all. We guess. We apply probabilities. Comparisons with the 'real thing' are rare, so we don't know if, and how far we are off!

A series of 2x upsampling operations: 2x 2x 2x = 8x upsampling

When starting out with a low resolution stimulus and refining it more and more, differentiation into a variety of 'species' is possible by making slightly different choices along the path. Small differences don't just add up, they multiply! This makes pixel splitting a potentially rich and fascinating image generation process, while still intuitive and close to nature.

Let's have a look at a not-interpolated image (left) and an interpolated image (right). Regular cubic interpolation is applied, which can be found in most image processing applications, nothing fancy. How do the results compare visually? If one squints the eyes, then the upsampled images both look the same. If one squints the eyes very strongly, the original images can no longer be discerned from the upsampled ones.
If one looks carefully though, both forms of upsampling disappoint, they are vague, blurry, they lack detail.

Difference images (white = no error).
Left: no interpolation, right: cubic interpolation.
Top: original (ideal) image, middle: 8x upsampled image, bottom: difference image.

Although the difference images look fairly similar, when one computes the average pixel error, the cubic interpolation wins with 1.86% compared to 2.07% for the non-interpolated image. Cubic upsampling might be favorable with many images, not only with this fly. If there are gradients in the image, then it is likely to perform quite well in those regions.

Let's put cubic interpolation to the test, using a conventional image processing application. Start with the original image of the fly. Scale it up 4x. Then scale the result up 4x again, and so on, until we have a series of 10 images:

1x zoom

4x zoom

16x zoom (4x4)

64x zoom (4x4x4)

256x zoom (4x4x4x4)

1024x zoom (4x4x4x4x4)

4096x zoom (4x4x4x4x4x4)

16384x zoom (4x4x4x4x4x4x4)

65536x zoom (4x4x4x4x4x4x4x4)

262144x zoom (4x4x4x4x4x4x4x4x4)

Clearly, each cubic upsampling step suffers from significant information loss. After only a few zooming steps, we are left with a monochrome image. Blurring did its work...

While it is perhaps scientifically sound to not introduce information we are not sure about, an artist may think differently. We can imagine details! Given a certain state of affairs, we can fantasize how it will develop and differentiate. Can such an imaginative power be implemented in software? With plausible results?
Deep Dive v1 (https://notnot.home.xs4all.nl/expandingimages/deepdive/deepdive.html) is an attempt to accomplish this. v1 uses a generative process that is based on 'shallow learning'. What if we would employ 'deep learning? Deep Dive v2?

After a week of experimentation and about ten artificial neural network architectures later, we are at a point of no return. There is some promise in this approach. It certainly beats conventional cubic interpolation: