There always comes a time when I’m playing a game and feeling a bit labored, so I put down my gamepad and just watch the game screen for a brief rest. However, one game happened to hit me out of nowhere at that exact moment. When I first stopped in NieR: Automata, I felt a powerful soundstage surround me, not just for its excellent soundtrack but as a combined experience of the different layers of sound in the game. In this blog, I’m not going to talk about Studio Platinum’s excellent action and fight design or the philosophical ideas that Taro Yokoo wanted to discuss. However, instead, I’m just going to talk about its soundtrack and sound effects, how it affects the potential for interactivity in the medium of video games as game development technology continues to evolve, and what considerations are made for sound effects design.
NieR: Automata is an ARPG video game released in 2017, developed by Platinum Studios and published by Square Enix, whose story revolves around a proxy war between robots created by aliens and robots created by humans. The music in Nier was arranged by Keiichi Okabe and his team at Monaca, and Platinum Studios used real-time processing of the audio in the game to achieve unprecedented effects. The following technical discussion comes from Platinum Studios member Shuji Kohata’s technical presentation at the Game Developers Conference 2018, please check out the presentation, which includes a number of compelling live demos.
Spatial-based real-time sound effects
For large-scale games like NieR: Automata, which include open-world elements, players exploring different areas within the game is undoubtedly one of the key gameplay elements. And the Platinum Studios team realized powerful spatial audio effects through two technologies: spatial audio technology independent of the output device (simple 3D) and interactive reverb technology (K-verb).
Among these, simple 3D is a spatial audio processing technique that does not depend on the sound output hardware and is designed to enable the player to be aware of objects behind them using only stereo sound. Audio differentiation is accomplished by adjusting the sound source’s volume based on its position, increasing or decreasing its high and low frequency components, and responding to objects behind it with bandpass filters. Interestingly, Shuji Kohata and his colleagues manually adjusted this set of techniques by comparing the audio of a stereo in front of Kohata and one behind Kohata. K-verb refers to a real-time reverb technique (named by Shuji Kohata himself) based on the position of the game character and the size of the surrounding space. By shooting ranging rays in multiple directions from the character’s position and calculating the size of the space around them, the duration of the audio reverb is controlled, resulting in a spatial audio effect that is dominated by the character rather than the player’s point of view.
Real-time sound effects based on game mechanics
Hacking into enemy robots is an essential part of the gameplay. This is where the player enters a shooting mini-game to simulate the process of hacking into other machines. And based on this gameplay, the Platinum Studios team developed a unique audio filter that doesn’t rely on the 8-bit version of a specific track but instead 8-bitizes the current audio through software to achieve smooth overdubs and adjustability of the 8-bit effect. Secondly, like many games where scarlet begins to color around the screen when the protagonist is injured, NieR: Automata produces a unique lo-fi sound effect when the player is injured, mimicking the state of old-school machinery when it’s damaged without making the noise too jarring to add to the immersion of the game.
In addition to the above two types of audio technology, Shuji Kohata also mentioned other fun designs used in games, such as odd voice effects if the player adjusts the volume of a character’s voice too high or too low, as well as Doppler effect processing of audio produced by rapidly traveling units, and other sound effects designed to complement the game’s graphics. It’s worth noting that all of the above audio effects must be lightweight and not cause the game to lag by acting out the audio. For example, the lo-fi effect reduces the sample rate of the game’s audio before processing it and subsequently resampling it, while the 8-bit sound effect simply uses mono audio.
Whether it’s a K-verb or 8-bit effect, these techniques demonstrated by Shuji Kohata make in-game audio relevant to the player’s actions, with exact feedback when the player presses a button. This is fundamentally different from playing footsteps when the player starts moving and switching to a “battle” soundtrack when they enter a battle. The sound design in NieR: Automata is integrated into the gameplay. There are few video games I’ve seen that focus on the player’s auditory experience as much as NieR: Automata, which seems to be an aspect many creators don’t pay enough attention to. However, gamers who have played the NieR series know that sometimes games can impress you with their sound alone (please try it here).
—Howard
New Media: Storytelling in Literature, Films, and Games 
Howard,
This is a really cool post and a topic I have never really thought about before. As someone who often mutes sound when playing games on my phone or computer and barely plays games on a console, I’m not often thinking about the sound in the game at all, let alone the work that goes into creating it. The only games I can think of playing and noticing sound are in Fortnite and Minecraft, which both have audio that does not go into the level of detail as the game you are discussing. However, I think this audio is a great example of hypermediacy in a game, as the gamemakers worked tirelessly so that the audio could make one feel (on a subconscious level) almost as if they are in the game, with the sound working the way it would if one were to be physically in the game world. Hypermediacy is something that has really amped a lot of more recent games and left players coming back for more, which is why I think a lot of popular games are first-person (like Minecraft). This leaves me with questions of whether or not more or less hypermediacy would be better or worse for the game making community both in terms of popularity and art. While it seems clear that gamers love feeling transported through a game, I also wonder if game makers like Sadie in Tomorrow and Tomorrow and Tomorrow wish for more creativity in conforming to a game world rather than trying to recreate true existence in a fake reality.
-Sarah Beth
Hey Howard,
I really liked this post. The way you deconstructed the auditory techniques used in this game is awesome. This also helps me relate to many times when I’ve been playing my own games and the soundtrack comes to enhance an experience even further. Whether its a heart-wrenching moment with an orchestra playing in the background or booming sounds in the middle of a battleground. What I do think is there should also be an analysis of visual effects and how they serve to further immerse the player into a game. I feel like visual and auditory effects go hand-in-hand in creating an awesome immersive experience for the player.
-Axel Barredo
Howard! This was an incredible post and really features an aspect of gaming not a lot of people focus on, but is vitally important. Thank you for including the list of effects and mechanisms used to create this kind of sound atmosphere. It’s really interesting. I’m thinking about how these kind of technologies are used in more ambient games, such as Journey, Abzu, even Minecraft to induce full immersion. I’m wondering how these kind of soundscapes are adapted to the audio output of games(tv, speakers, or headphones, etc.), and how that connects to visual immersion as well.
-Spencer
Howard, I am so happy that you brought this topic up in your blog!! Sound design is the unsung (pun intended) hero in media as a whole, but especially in video games. Some of my favorite examples to bring up is the Persona series, which you and I have briefly discussed in class. Each game in the series has its own complete separate identity from each other, not just in the game’s main color and visual style, but in the music that is used. Persona 3 has a hip hop vibe, Persona 4 is more rock-esque music, and Persona 5 has a “fusion jazz” musical style. Music and sound design is so important to media to establish identity, and for lack of a better word, the vibes, of what you’re watching or playing.
-Ethan