Thomas Isler: "How does landscape sound?" (2023)

Bild

Our research process consisted of two phases: Image and sound recording in the Swiss mountains and sound postproduction in the sound lab at the ICST of the ZHDK . In the sound postproduction phase, we divided the research team into two groups. The two students (Lily Pelaud and Olekandra Tsapko) created an audiovisual installation from the recorded material with only one requirement: to use a surround five-channel sound system with equipment on loan from the ZHDK for DFA students. Otherwise, they were free in their artistic approach.

The other group (Thomas Isler, Reto Stamm and Eric Larrieux) transferred the sound recordings into the immersive sound system of the ICST studio in order to test the immersiv qualities of this material in relation to the projected image, i.e. to find out how a three-dimensional soundscape behaves in relation to a two-dimensional visual landscape.

Phase 1:

In the sound recording phase, we focused on different multi-channel recording techniques in order to compare them later in the sound lab:

- double MS stereo
- Neumann KU 100 dummy head
- 7 simple omnidirectional microphones from Sennheiser for 5.0.2 spaced array recording technology
- 4 Zylia microphones for 3rd order Ambisonics arranged in spaced array setup

A multichannel array recording technique works with the slight delay of the sound, which occurs when the sound hits the microphones, which are a certain distance apart, at different time intervals. We set up the 7 omnidirectional microphones on a base area of approx. 3 meters in diameter. 4 microphones covered each of the four corners, one microphone was directed forward in the center and two microphones covered the upward direction. We also placed the 4 Zylia microphones in parallel at the four corners. A Zylia microphone itself already has 19 microphones. Actually, one Zylia microphone is enough to cover the Ambisonics 3rd order sound field. However, this technology does not work with the time delay of the sound (array recording technology), as all microphones are practically located at one point on the microphone sphere. So what happens in the Ambisonics sound field when we work with 4 Ambisonics microphones that are positioned at a distance from each other?

We recorded images and sound of the landscape at four locations in the Swiss Alps:

- the landscape of the Gotthard with a wind turbine
- the landscape of the Grimsel reservoir with the dam under construction at night
- the landscape of the Grimsel valley with mountain pass road and Gravel pit
- the landscape of Obergoms with bird sanctuary and waterfall from the Rhone glacier

The four landscapes have different levels of industrial infrastructure and these vary in terms of noise and visual dominance.

Phase 2:

In the sound lab, the first task was to prepare the various recording formats so that they could be played back in the immersive sound system of the ICST of the ZHDK, which required a considerable amount of work. The following phase of the spatial design of the soundscapes was the core of the work. Recording sound in the field is in a certain sense a deconstruction of the soundscape, a breaking down of the acoustic events on site into individual parts. In the sound laboratory, the task is now to reconstruct and reassemble the diverse materials recorded in the field. We tested this reconstruction in various settings presented to us by the recording location, both visually and in terms of sound:

- On the screen, the Gotthard landscape with a wind turbine can be seen at a distance of around 200 meters. In addition to the expected mountain atmosphere, the hissing and whooshing sound of the wind turbine's rotor blades can be heard.
To test how the two-dimensional image relates to the three-dimensional acoustic sound system, we carried out the following test:

At the beginning, the sound direction is defined as mono from the direction of the wind turbine on the screen. Slowly, the acoustic space opens up into a stereo image, which means that the sound takes on a spatiality from the direction of the screen. Slowly, however, the perception of the sound space expands, as if a two-dimensional sound zoom were being expanded into a three-dimensional acoustic space. Suddenly you are no longer sure whether the hissing of the wind turbine is coming from behind you. Now it becomes clear that the noise is not emanating from the wind turbine on screen, but must be coming from a wind turbine behind the camera. At what point does the connection between the sound and the image become implausible? At what point are we tempted to look to the rear, where the sound comes from but there is no image?

- The night shot of the construction site of the dam on the Grimsel is a paradox. In the panoramic image on the screen, this floodlit construction site on the edge of a mountain lake at night does not seem real. It is also difficult to deal with the dimensions: the mountains seem mighty, while the building site seems miniaturized. The soundscape has a clear direction, the sounds of the construction site are the only dominant acoustic events of the night. In this experiment, we worked with the immersive illusion of a large acoustic space. The distances between the sound sources are huge, as is the visible space of the mountains at night.

To what extent can this characteristic be used to magnify the visual space? How large can the perceived sound spaces become before the dam wall visually enlarges or shrinks again?

- The image on the screen of the Grimsel valley by day is a kind of visual overload. There is a lot going on on the meandering pass road. Colonies of cars move uphill, motorcyclists overtake trucks, even yellow postbuses appear. An excavator suddenly starts working in the gravel pit. The acoustic events that are visible in the picture are present and yet far away. Can't we also hear the rushing of a mountain stream? The wind from the mountain tops? The passing of an airplane far up in the sky? How do we construct an acoustically believable soundscape with all these sound events that allows us to immerse ourselves in this bizarre image of busy mountain landscape that doesn't want to be idyllic at all? How strong is the immersive effect of the array recording technique applied here? How do we achieve this immersion effect, which is used for the perception of a large space and produces the feeling of being right in the middle of it?

- We hardly see any infrastructure buildings in Obergoms. An almost idyllic landscape spreads out on the screen. Meadows with flowers, trees with birds, a waterfall cascading down rocks, a mountain pass road is barely visible in the background. However, hardly any of this idyll can be heard in the sound recordings on location. The sounds of birds and even the humming of insects have been recorded on the audio equipment. But there is background noise of motorized traffic masking everything else, as the valley is a thoroughfare to the south and, like most valleys in the Swiss Alps, has lots of roads. How can we create the acoustic illusion of a mountain idyll here? A reconstruction from the recorded material does not work even with the application of filters and effects available in our sound software. We decided to rebuild this soundscape from archive recordings. Natural atmospheres, birdsong, insect humming, waterfall noise: everything comes from the archive. We can only use the hissing and whistling of the steam locomotive from our sound recordings, the rest of the recordings are not from Obergoms. The humming of the insects offers an additional design option. We are testing here how present, i.e. how close we can place this sound in the room so that one gets the feeling the insect is flying around right past our ears.

How far can we take this artificiality before it becomes implausible? How much "sound kitsch" are listeners willing to accept?

The research arrangement led us into technical and aesthetic questions. With the limited resources at hand, we were just able to carry out a preliminary analysis. We underestimated the technical effort required to transfer the sound recordings to the immersive sound system of the ICST studio and make them available for sound design. Nevertheless, we were able to gain some insights and experiences. It was impressive to experience what happens to our acoustic perception when soundscapes are recorded using array recording technology and played back in the immersive sound system of the ICST studio. Array recording technology is complex in the recording process in terms of the amount of equipment, but technically it is actually quite simple. However, the effect in the screening room was a very strong immersive experience, which we want to continue to use in our own practice. The various test installations have given us a few answers to the question of how acoustic landscapes in three-dimensional space work in relation to a two-dimensional image projection. The effect of the acoustic spatial experience in relation to a projected video image is a fragile one. There are tipping points that undermine the credibility of the image-sound relationship. These tipping moments are fragile and are strongly related to qualities of the reconstructed soundscape. The potential to give depth and size to a two-dimensional image through sound alone is astonishing. The possibilities of creating sound directions and sound movements in an immersive sound system are impressive. Sound design as an illusion generator offers huge possibilities.

Where we have not made enough progress in our research project is in translating these technical and aesthetic investigations into an artistic practice. By the time we were ready to do this, our time was already up. The presentation at the end was also too short to present our results to the guests on site and discuss them with them. The half hour allotted for this was not enough. Nevertheless, this presentation gave an insight into our research work, which was very fruitful for us and beneficial for our future practice.

The idea of allowing the two students to make their own artistic use and further processing of the jointly recorded sound and image material in phase 2, independently of the investigations described above, was very successful in our view. They used this freedom to find a low-tech application for the high-tech material and transfer it into their own practice. We really enjoyed their video and sound installation. Working with them was a great enrichment for us. Lily Pelaud and Olekandra Tsapko have written their own report, which is attached here.