Sky Semantics (2023)

At the origin of Sky Semantics was the desire to see what the machine, in its turn, was capable of perceiving. Although we are now accustomed to the technological feats of AI, this new “operational” competence deployed on the plane of the digital image did not seem trivial to me. During the conception of Postdigital flipbook, as already mentioned, the fact that the computer was able to identify as a human face a face shown to it for the first time had deeply struck me. All the more so, I was very impressed by semantic segmentation networks for images. These artificial intelligences, which have been the object of colossal efforts in the race toward autonomous vehicles — but certainly also in the race toward the automation of war — are capable of detecting, in a moving image, not only the presence of a large number of distinct elements¹, but also of outlining their contours precisely, including when these objects are in motion.

The analytical method having failed in the face of this kind of complex task, a notable improvement emerged from the experimental work of engineers, who developed the deep training of neural networks — deep learning — a process relying on a colossal volume of data². The fact that this kind of technology exists and works efficiently is a concrete expression of the scale of big data, which exceeds our capacity for apprehension³.

If the machine is now capable of “perceiving,” the question arises of the potential and the specificities of this new mechanical competence. We know the human tendency to find shapes in nature, a tendency that rests on a great capacity to idealize perceived forms, to schematize them, in order to derive generic figures from them — for a toddler who can barely speak, a clothespin very quickly becomes a crocodile. What, then, is the machine capable of in this respect? There is, probably, a spectacle of fleeting forms that appear and disappear in the matter of the world without anyone ever having seen them, because of their extremely rapid temporality, or of their scale, too small or too vast.

Questions of this kind motivated this experiment, in resonance with others — more or less scientific, more or less esoteric — undertaken since time immemorial around newly invented technologies of vision, understood as instruments extending the visual capacities with which nature endowed the human. One thinks of the shock of the moment when photography, toward the end of the nineteenth century, became able to freeze movement — a possibility exploited aesthetically by photographers such as Jacques Henri Lartigue and scientifically by Étienne-Jules Marey, who, by means of chronophotography, set out to study the movement of living beings. Some looked askance at this photographic — mechanical — decomposition of space-time. Such was the case of the Bragaglia brothers who, in a more esoteric register — although they criticized the belief in ghosts⁴ — sought to represent the real expression of movement. Thanks to sufficiently long exposure times, the Bragaglias captured staged movements so as to find, on the surface of the photographic print, forms produced by the moving elements, understood as visual manifestations of the Bergsonian élan vital⁵.

The desires to see at the origin of Sky Semantics responded less to a form of esotericism than to the expression of a playful attitude, one that sought to be surprised by what would result from disconnecting the neural network from its natural visual environment — urban views associated with vehicle traffic — and connecting it to a part of reality outside its design environment, namely images drawn from nature.

Having discovered Bachelard — whom I did not know at the time — I can now say that the experiment put to the test the existence — somewhat fantasized — of a form of material imagination within the machine. For it is above all with regard to the human that Bachelard proposes, as a complement to formal imagination, an imagination that grasps images from forms arising from matter. These images “have weight⁶,” they are the result of a “dynamic joy [that] kneads them, lightens them⁷,” and if it is through sight that the human names these matter-images, “the hand knows them⁸” already.

To see whether the machine could come into contact with this dynamic at work in matter, tests were carried out on natural landscapes: the movement of leaves in the wind in a forest, the movement of water, and the movement of clouds. The latter proved the most promising, which suited me perfectly. Indeed, one could not fully describe the germ⁹ at the origin of this project without mentioning the desire to see, quite simply, images of clouds in black and white, in time-lapse, after an accentuated contrast treatment, seeking to provoke a dramatic effect, to activate a form of the sublime in the image.

Concrete work with these materialities — that of the images, that of the clouds, that of the neural network — shifted the initial aim of the work. First of all, it proved necessary to loosen the precision of the semantic segmentation neural network in order to increase the frequency and number of forms identified. The neural network then began to identify and outline a large quantity of forms, associated with various objects, even though their detection remained uncertain. From that moment on, the image began to express itself.

Certain forms were flagged by the neural network frequently, but for extremely short durations. Sometimes the network — now out of tune — identified objects associated with forms that stretched across a good part of the frame, appearing and disappearing very quickly, or else surging up intermittently. The whole contributed to the emergence, over these dramatic, gradually evolving clouds, of more or less extensive flickerings that resembled flashes of lightning.

The frame, at that point, had become the site where various dialectics were expressed. On the one hand, there was a level of image that evolved in resonance with a human stream of consciousness: images of clouds modulated in space-time through sensitive editing work. These images — prepared by the human — were nevertheless jolted by the manifestations of the neural network, which certainly gave information about forms that could not otherwise have been glimpsed, but in so doing suspended the free course of all human — material — imagination. The image seemed to want to signal the impossibility of an imagination-perception (material imagination) in which human and machine would converge, as if a choice were required: either in favor of a completely human perception, or of a completely machinic one.

These two types of perception, whose origins and effects proved very different, marked the image more fundamentally as the site of a confrontation between two temporalities, themselves diametrically opposed: human temporality, on the one hand, continuous and progressive; jolted, on the other, by a machinic temporality, discontinuous and disruptive.

In other words, what emerges from Sky Semantics is the idea that the material imagination evoked by Bachelard — and probably imagination as such, and with it perception — is not a faculty that can be exteriorized from the human to the machine.

The image had also become the expression of an irony. Indeed, on the one hand, there was a machine that had conquered its capacity to “see” only through an enormous expenditure of energy resources, associated with very substantial material infrastructures, mobilized for long periods during the deep training of neural networks. Yet this machinic perceptual capacity was here directed toward the clouds — the cloud — the latter being the object of a misleading metaphor, untenable nowadays, which seeks to associate the digital industry with a form of airy, almost immaterial lightness.

More fundamentally, the work seems to make sense in relation to a form of apprehension of the world that would deserve to be considered less on the basis of contributions coming from an “outside” — transcendent — than in connection with processes taking place here, among us, very concretely, although not always within reach of our perceptual and/or cognitive apparatus. From this immanent perspective, it would be a matter of thinking about the birth of forms out of matter; but these forms would not have as their sole purpose that of triggering the activity of a material imagination, such as the one evoked by Bachelard. It would be a matter, even more, of considering the very concrete birth of forms, of objects, but above all — and against the Platonic tradition in which we are steeped — that of symbols, of codes, whose genesis should be conceived outside any transcendence, on the basis of the generative activity arising from interactions between matter(s) and energy(ies).

Put differently, what may emerge from the work is not the idea of a dematerialization of the world through the digital but, on the contrary, that of a complete and radical rematerialization — including of the symbolic and the spiritual — which would need to be reassessed afresh on the basis of their genesis and their immanent mode of existence.

1. The system used is capable of identifying 80 types of objects in an image: airplane, sports ball, banana, bench, baseball bat, donut, bicycle, bowl, fire hydrant, bottle, broccoli, toothbrush, couch, carrot, kite, chair, horse, cat, dog, keyboard, knife, spoon, elephant, sink, traffic light, oven, fork, frisbee, baseball glove, cake, giraffe, toaster, clock, hot dog, bed, book, microwave, motorcycle, sheep, teddy bear, laptop, orange, bear, bird, parking meter, umbrella, person, pizza, potted plant, surfboard, apple, cell phone, tennis racket, refrigerator, backpack, handbag, sandwich, skateboard, skis, mouse, cup, remote, mobile phone, television, dining table, toilet, train, suitcase, vase, cow, wine glass, car and zebra. See the GitHub page of the implemented neural network: https://github.com/ayoolaolafenwa/PixelLib/blob/master/Tutorials/Pytorch_image_instance_segmentation.md.↩︎

2. For a review of this technology, see: Farhad Mortezapour Shiri Thinagaran Perumal Norwati Mustapha et al., A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU [en ligne]. arXiv, 2023.↩︎

3. In an attempt to make perceptible the volume of images churned by our civilization through machines, the North American artist Evan Roth entirely covered the very high walls of the immense hall of the Jeu de Paume in Paris with his work Since you were born (2019), during the exhibition Le supermarché des images, curated by Peter Szendy in 2020. These were only the images stored in his web browser’s cache, that is, a minuscule fraction of big data.↩︎

4. See Giovanni Lista, «Les Frères Anton Giulio et Arturo Bragaglia: Inventeurs du photo-dynamisme futuriste ». Ligeia [en ligne]. September 2020, vol. N° 181-184, n° 2, p. 22–29.↩︎

5. Ibid.↩︎

6. Gaston Bachelard, L’eau et les rêves, op. cit., p. 8.↩︎

7. Ibid.↩︎

8. Ibid.↩︎

9. We will see, with Simondon, that the intention at the origin of an artistic project can be thought of as an intention-image, which the philosopher problematizes in the context of technical invention, but which remains valid when thinking about artistic creation.↩︎