What is virtual production?

What is virtual production?

Neatly defining virtual production is not straightforward. The technologies used, the methods applied and the uses of virtual production approaches are rapidly evolving. 

In broad terms, virtual production is a way of making film and television which harnesses computer generated content allowing real-time visualisation and control of the digital environment in which you are shooting. Importantly, virtual environments are captured ‘in camera’, rather than added in post-production.

Virtual production has seen large investments of time, money and R&D from major production companies and studios. For example, virtual production was used by the BBC for their coverage of the Tokyo Olympics and is regularly used in the production of advertisements and music videos (Kadnar, 2019). It is predicted that from 2022 to 2030, the virtual production industry will have an annual growth rate of 17.8 % globally. In 2021 it was valued as being worth $1.6 billion (Grand View Research, 2022).

What are the key technologies used in virtual production?

Blue or green screen

Many film and TV productions utilise green or blue screen backdrops that allow CGI environments to be added in post-production. For decades, green or blue screen compositing was the go-to solution for creating fantastical environments. The reflection of hues from green screen or blue screen on costumes or props, however, is a costly and time-consuming process to correct in post-production. In virtual production, blue or green screens are used to allow real-time rendering of virtual worlds as the live action occurs in a studio, rather than being added in post-production. 

Although not true virtual production, The Jungle Book (2016) was a massive step forward in testing the technologies. Neel Sethi who played Mowgli was filmed on a blue screen stage and the film was directed as if it were traditional live action, but the jungle was created in computer graphics (CG). The virtual world was combined with this live action, physical sets, motion capture and virtual effects (VFX) to create what we see on screen. Prior to shooting with actors, block animation for Mowgli and other characters was created along with a virtual environment. This allowed the filmmakers to prepare for shooting in the pre-visualised environments. Characters could then be played back in real-time allowing cast and crew to see the combination of virtual worlds and human performers on monitors (Giardina, 2017).

Virtual Reality (VR)

VR is typically experienced using a headset (such as Oculus Rift or Google Cardboard) with headphones, and involves experiencing some degree of virtual realm or ‘reality’. VR technologies enable filmmakers to be able to scout a digital version of a real place or construct a virtual location to visualise a scene to plan the filming process. Doing this in VR also allows real-time collaboration across the world.

VR platforms have also been used to run virtual production systems, with trackers on cameras, props and real spaces used to provide the tracking needed to accurately combine movements and filming angles in the physical world with virtual environments. 

‘The Lion King’ (2019) was shot in a 100 mile-wide virtual world which could be freely explored using VR. The world was created using traditional animation and modelling tools, such as Maya and translated into a video game engine, Unity, to create an environment explorable using headsets. This pre-visualisation allowed the director, cinematographer and camera team to understand where to shoot from. For final shooting, the crew used virtual cameras linked to tracker-enabled physical camera rigs to move it around the virtual set. The 3D path of the camera was tracked and reflected inside the virtual world (Summers, 2019).

LED Volumes

Major advancements in LED panel technology, developed in the live events field and combined with technologies from the aviation industry and military flight simulation systems, have enabled virtual production using large walls of LED screens called volumes. Virtual environments are projected on the screens in camera-ready quality. Their size allows the use of physical props and sets with virtual environments, and the volumes are big enough for cast and crew to operate in. 

LED volumes harness volumetric capture technologies which allow tracking and synchronisation of cameras, lights and the virtual environments displayed on the LED panels. Video games engines (typically Unity or Unreal Engine) are used to control virtual sets and environments, and can be combined with sections of greenscreen for compositing other elements if VFX work is required in post-production. The majority of the VFX work with LED screens is captured in final-pixel in-camera, however, thus reducing the amount of work which needs to be completed in post-production (Unreal Engine, 2019).

‘The Mandalorian’ was shot in an LED volume and has become the poster-child for virtual production.

What’s next?

Jon and Nina have been exploring the geographies of virtual production and interviewing leading figures using and developing the technologies involved. A common view is that virtual production will help reduce carbon emissions in the filmmaking process, but this claim is yet unverified. Jon and Nina will lead a panel about the environmental credentials of virtual production at the BEYOND conference on 19th October.

Last year SIGN funded Richard Southern and Harvey Goodall at Bournemouth University to examine the zero carbon potential of virtual production which will be reporting soon.

If you work in the screen industries and want to understand more about virtual production, SIGN is offering free training.