FinalCutPro 10.4 was finally released last week, the long awaited update includes compelling new features namely the native support of 360 video editing and authoring.
While the editing of visual content has received the usual attention to detail and ease of use, sound has gotten left out. For most users 360 editing is still a enigma alone, before one even can grasp the implications on sound.
Importance of Sound
The general consensus on sound for picture is, that it is of equal importance.
Over the years I have devices a formula to explain the importance of sound in a more easy way to understand.
The experience of a movie, a picture is equal to 100%. To understand the importance of this it needs to be broken down into sub segments.
- Story
- Picture
- Sound
Each one contributing towards the overall experience.
A feature action block buster has a different make-up then a documentary or news story, but the ingredients are the same only with a different ratio of importance of the key elements.
A news story is based heavily on the story narrative, followed by supporting imagery and lastly sound.
A feature action film is driven today more visually, with sound and unfortunately to a lesser extent story.
This devision changes from project to project, but it becomes apparent that each part plays a different role at different moments in a project.
Sound delivers emotional subtext, pacing along with picture. Picture alone is flat, despite todays efforts in virtual reality and 360 video. Regardless how compelling the images are, without sound they are flat and dead. Just imagine the movie Avatar with its ground braking CGI images, without adequate sound design the jungles of Pandora would look like computer images. In fact they are computer images, but they suck us right into the alien world.
Anybody who has edited picture to dialog, that was shot from different points of view, possibly at different times, knows that continuity gets rapidly lost when the ambient tone from voice to voice changes in the background.
The ambient sound needs to be consistent to picture to picture, regardless if we hear a off voice over the phone or if we see the character standing in a particular scene while talking.
To go back to the above formula, if the image can not translate the mood and tone according to the story, then sound needs to come in. Equally the color grading of a image can sub consciously deliver emotional and stylistic cues to tell a story.
Scene to scene we end up redistributing the balance between the three main elements.
Stark score, sweeping images, with airplanes closing in on Pearl Harbor, while our protagonist stares blank into space. Hearing and feeling the low rumble of the approaching airplanes, even when out of frame, suck us into the moment !
A person, telling the camera the horrible story of how their loved ones where killed in the attack, requires little more then good voice and lesser a strong image. Here sound design takes a back seat.
In 360 or VR, sound takes a different position. As the viewer can look around a scene, possibly wander off and look around a corner, sound becomes a invisible thread and point of attraction.
We automatically turn around and look for that alien sound of a create crawling beneath us, or we try to locate the source of the vibrations created by a approaching dinosaur !.
Sound becomes the next camera focal point !
Magically stealing the viewer subconsciously into the right direction.
Yet such features are largely absent today.
360 Sound Fields / Ambisonic
Ambisonics is a full-sphere surround sound technique: in addition to the horizontal plane, it covers sound sources above and below the listener. Unlike other multichannel surround formats, its transmission channels do not carry speaker signals.
A ambisonic microphone setup consists of a configuration of several microphones, in a similar fashion that a stereo microphone setup picks up left and right signals, a surround setup consists of 5 or more microphones plus 1 channel for low frequencies that is derived from the primary mic’s.
The ambisonic setup records a sound field, a 360 degree spherical representation all sounds arriving at the point of recording (the listeners ears).
Unlike stereo or surround, this mic channels are not directly mapped to speakers or headphone channels. Just like a 360 image can only be viewed 120 degrees at a time, or mapped to a flat screen.
Directional information of the viewing angle, steers the picture or the sound field, rotating it automatically to the orientation of the viewer.
Advanced sound rendering algorithms process the information from the sound field and translate this into a directional signal that can be mapped to speakers or headphones, regardless if stereo or surround. The screen (HMD, or display) virtually rotates with the head of the viewer, the sound follows.
Apple for some odd reason has opted out from including this features into their latest FinalCut Pro or Logic Pro X update.
360 Sound Design
In order to support the picture sound design is required much more then ever, as sounds are no longer a representation of left and right.
A ambisonic sound field is augmented with sound effects (SFX) of various elements, the sound of insects buzzing and crawling past the viewer, airplanes approaching and flying overhead.
Advanced sound design requires this sounds to be animated, tracked and rendered acoustically accurate. Much like light is ray traced in CGI images to produce the correct highlights and shadows.
Sound can become occluded by a passing by object, reflected of surfaced, muffled or become subjected to doppler shifts when passing by rapidly.
Wether such objects approach or move away from the listener the frequency shift changes. Just like serene of a ambulance does in real world. In the 3D space of 360, this doppler shifts are no longer two dimensional, they changed according to the viewers orientation / point of view within the image.
With this it should become clear how important this advanced sound features are.
Post Production and Design
There are a number of third party plug-in’s and solutions in the market. I have become aware of two such tools that work within general post production pipelines, in the form of AU/VST/AAX plugin, or standalone applications.
The dearVR plugin operates from within FinalCutPro or LogicPro (or most other NLE’s or DAWs). It allows for spatial positioning of individual sounds, but it does not stop there. In combination with a VR headset and controllers, such sounds can be animated and positioned in real time in 3D space. The automation information required for the X/Y/Z orientation can be recorded and played back in real time. Advanced features for sound occlusion, reflection or room reverberation are present with the plugin.
Soundparticles are a particle system analog to the CGI equivalent, dedicated to sound. Large ‘clouds’ of individual sound elements can be spatially and temporally created and animated.
Because of the computational complexity it operates as a standalone application, just like Maya, Cinema4D or other 3D packages.
Surround sound setups alone can do much, most if not all DAWs today have surround integration, both for production and for export.
Export and Integration
Today I can only speak of Apples native production tools, FinalCutPro X, Logic Pro X, Motion, they all recently got 360 support for picture, yet the required sound features are absent.
Tight integration of picture and sound is required, regardless if the projects are semi static 360 videos or fully interactive productions to the liking of VR games.
The later are a different ball game, as the entire scene is computed in real time, often interactive beyond near point of view control. Yet we all use the same pre/post production tools to generate content used downstream in the according tools.
Event the support of the right export formats is still absent. Standard stereo, quadraphonic or surround formats are no longer adequate. Regardless of the flexibility and features of the plugins, they are of little usage when the data can not be shared and passed along.
FinalCut Pro needs to be able to ‘pan’ sound fields according to the viewing angle, it is possible to use multi track audio natively with FinalCutPro, but there is no link between orientation and ‘panning’ information that comes out of the box.
LogicPro requires more flexibility in custom export formats beyond surround.
Motion requires potentially full feature along the lines of sound particles, backed into the application – but that is a stretch.
SaKiE 