Article on repetition

Boingy Boingy Boingy
Why I hate repetition and what can be done about it

I suspect it is the time I have spent in the video game industry that is the main cause of my hatred of repetition in audio. PCs and video game consoles have always had limitations. The amount of storage space on a disk, available memory, seek and read times of disk media (CD, DVD and Blueray) are all examples of some of the many issues developers need to deal with when producing games. Because of the limited resources and the need to share these resources amongst all aspects of the game, usually it is necessary to design a project that can cope with resource limitations. For game audio this often meant less variation than might have been desirable.Ever since I started working on games I have been aware of their limitations and that sometimes some repetition is unavoidable, however far too often I have played games that do not seem to be pushing the resources of the relevant console and yet still have overly repetitive sound design. One of the worst instances I encountered was a PS2 game that literally only had two footstep sounds, and the developers had also implemented them too loudly, so the entire game was accompanied by this clip, clop, clip, clop, clip, clop of the main character. For me it made the game unplayable after only a few hours. I am happy to admit I am pretty fussy where game audio is concerned, but it really spoilt what was otherwise a fun game.

So why am I so distressed by a little repetition? Think of any sound that might repeat itself, and then think carefully about that sound. A dripping tap, bouncing a ball and footsteps are all good examples of sounds that repeat cyclically, but if you really listen and analyse the sounds themselves you realise that they are not exact replicas of the same sound. Bounce a ball sometime and see if you can get exactly the same sound twice in a row, let alone for longer. Each bounce is subtly different, the downward pressure you apply to the ball, the angle on which it might land and the surface on which the ball bounces will all alter the sound, they may not alter it much, but it doesn’t take much variation at all to break up a repetitive cycle. In fact it’s the super subtle differences that really make the difference. If you isolate a single tic toc from the sound of a clock, loop it and listen back to it, it does sound like a clock, but there is something about it that just doesn’t sound quite right.One of the issues with modern video games and CG films is that they are becoming very close to being indistinguishable from real life; close, but not yet completely there, and that’s one of the problems. The closer you get to making something that reflects reality the more the subtle differences stand out. So with audio, a bouncing ball or ticking clock sound that repeats is pretty close to how a ball or clock may really sound, but the subtle difference tells the listener that what they are listening to is not the real thing. In this case and to help create many other types of repetitive sounds, subtle variations can go a long way to convince a listener they are listening to the real thing.

To use the ball analogy lets assume we are creating a basketball game. One of the main audio elements we require is the sound of the ball being bounced. With sound files time equals memory, at 44.1 KHz, 16 bit mono, a ten second long sound is 860Kb, this is almost a megabyte of memory for only ten seconds of sound. So we would not want to record long samples of someone bouncing a ball to add to the game as we would quickly run out of memory. Alternatively if we use a recording of a single bounce it will result in very obvious repetition. 

If we use 5 or 6 different recordings of a single bounce they can be used as a pool of sounds to draw from. Randomly selecting a different recording to play for each bounce event will instantly add more variety than using a single bounce. Adding a very slight variation of pitch each time a ball bounce is played will produce more variation and represent slightly more forceful or less forceful bounces. Finally adding a very slight volume variation to each bounce will add even more variety. In the end these three simple steps can turn a selection of 5 sound files into a sound event that can play endlessly without ever sounding like it is repeating. This effect has been achieved using a small number of files that will not use much memory, this is particularly important for smaller consoles like a Nintendo DS or for iphone apps.

Repeated bounce
A single bounce sound file, looped to repeat four time
No variation in pitch, volume or time.

Non repeated bounce
Five unique bounce sound files played in sequence
Each bounce is different in sound and pitch, volume and time spacing.

When creating slightly more complex sounds there are more ways in which random elements can be added. Footstep sounds are created from the sound of the particular shoe impacting on the ground and the reaction of any materials that are stepped on. So a boot walking on short grass would only have a basic impact sound, but walking on gravel or leaves would produce the sound of the leaves being crushed or scattered, or the stones moving. Because there are essentially two layers the sound event can be created by combing two sound pools. The first would contain a selection of boot impact sounds to randomly draw from, and the second a selection of leaf sounds. If we have five sounds in each pool the number of combinations suddenly increases dramatically. This simple solution can effectively remove repetition almost entirely.

I want to return to why I dislike repetition, and look at it from a different angle. As I mentioned, initially it was from frustration at being presented games that appeared to have poor or lazy sound design. As I looked into solutions to this issue so I would not fall into the same trap I realised that the randomisation of sound might even improve the overall feel of a game. Several years ago I read an article where some film makers were previewing their latest project to a test audience. They showed the audience two versions of the film and asked for their reactions. The vast majority of those surveyed said the second version was much better, they claimed the special effects were better and the pace and editing was improved. The amazing thing was that visually both versions were identical. The only difference was the music and sound effects used. Now obviously this can be seen as a good example of how important sound is for supporting visuals, but more than that it demonstrates that sound can even alter how someone perceives visuals. It has been well documented that altering a musical score can seriously change the feel of a film, adding ominous music to a scene can make even the most angelic character seem disturbing and playing the theme to Benny Hill has the disturbing effect of making the most horrific footage feel like a comedy routine. Music affects human’s emotional state to a level we still do not fully understand, so it is not surprising that it can have this kind of influence when accompanying a movie clip, but what about sound effects?

Recently I was asked to write an article about using FMOD Designer for sound event creation. I have been using FMOD for many years as it allows me to create sounds with a large level of variation. For the purposes of the article I wanted to include a film clip of an explosion. Most people respond to visuals, so adding a film clip to watch helps convey the idea more easily. I chose a clip from a friend I have worked with of a mine exploding and added an appropriate sound effect. The effect was generated in FMOD as it would be in a game situation. I had set up the event to randomize the sound each time it played, so the explosion would always sound different. I recorded a series of this sound so I could choose which version I wanted to add to the clip. When it came to editing the clip I found lots of versions of the sound that worked really well. I added one to the clip and uploaded it for the article.

I realised that this single clip did not demonstrate FMOD’s ability to randomise the sound each time it was triggered, so I made a second clip where I edited the explosion to repeat 4 times. I added a different version of the sound to each explosion. This clearly demonstrated what FMOD was capable of, but when I previewed the clip I realised it also demonstrated something else. The sounds were different enough that they even give the impression that you are viewing four different explosions. Part of this is the way I edited the clip, but most of it is a result of the sound effects. For me this really highlighted the fact that sound and visuals are more closely linked than we give credit for and that one can often influence how the other is perceived. 

For a long time I have worked towards reducing or removing repetition from game projects wherever possible, and I always try to produce high quality audio environments, but the example of the multiple explosions showed me how audio could go even further and be used to not just highlight visuals, but even alter how they are perceived. I would be interested to work on a game project where a level didn’t change visually but went through a series of transformations for the audio environment and see just how far you could alter people's perceptions without changing the visuals.  Games are fantastic things to work on creatively because you can create worlds where almost anything is possible. Maybe you could even make a game where lots of repetition was entertaining, but I am doubtful.