Green walls meters high, a desk in the middle, and a wood veneer on the green floor: since May 2016, NDR has been broadcasting its regional Schleswig-Holstein programming from a virtual studio in the regional broadcasting station in Kiel. What were the reasons for the change, and how satisfied is NDR with the result?
Even technologically uninitiated television viewers are now familiar with the concept of a virtual studio: Instead of using a real set, news presenters stand in the middle of a large blue or green box. The monochrome background serves as a mask, and by means of a key, it can be removed from the video image. In this way the large green area filmed by the camera can be replaced by a virtual, artificial background, which can include computer-generated scenery, or graphics such as photos.
In 2015, in Kiel, capital of the German state of Schleswig-Holstein, Norddeutscher Rundfunk (NDR, Northern German Broadcasting) converted its television studio 1 to this type of virtual production environment. Here virtual technology from Avid Technology was used. In April 2015, Avid, a producer of video postproduction and audio solutions with headquarters in Massachusetts, USA, took over Orad, a long-standing specialist in virtual studio technology.
For NDR, virtual technology was by no means an end in itself. “The idea of the editorial department,” explains Matthias Rach, Head of Production of NDR radio and television in Kiel, “was to work with large images that appeal to viewers and immediately involve them in the topic.” In a real studio set, it is difficult to show large background images with different shots and perspectives. Thus a large green box was an obvious solution. “We then considered what technology we would need, and very quickly arrived at a virtual approach,” says Rach.
An essential difference between a virtual studio and a conventional green screen is the camera tracking. This makes it possible to combine camera motions exactly with a graphics computer, so that the virtually generated studio set reproduces all of the camera motions. No matter whether zooms, pans or tilting of the camera are involved, the graphics processor obtains position data that are as exact as possible, and can then calculate the graphics segment precisely. But how are the camera motions captured? This is in fact a central element of every virtual studio. For this purpose various technologies are used, often in parallel, in order to increase the precision. NDR in Kiel selected the camera robotics and sensor system of the Japanese manufacturer Shotoko which, like Avid technology, is often used for virtual studios.
The control is similar to what we had before,” explains Christian Graf, camera operator at the studio. “However, previously discrepancies were not as relevant as they are now. Now every camera setting must be linked with the background. Therefore, camera position values for tilt, panning and focal length are specified. Every camera must achieve these values precisely for each presentation position.”
The use of a presenter duo with various studio positions permits a significantly more flexible broadcast design. In the virtual studio there are four Sony HDC-2400 studio cameras, each with three 2/3-inch CCDs. However, one camera serves only as a backup, in the event of unexpected failure of one of the other three cameras. “Since we have a relatively small studio, we must use a fairly wide-angle approach,” explains video engineer Norbert Sieben. The cameras are equipped with Fujinon DigiPower box lenses, 6.5 to 180 mm, 1:1.5.
A special feature of the virtual studio in Kiel is that the cameras are mounted on mobile Vinten pedestals which, however, are locked in place. The camera supports are not moved, and the position of the cameras is changed only with regard to height, panning, tilt and zoom. The cameras and the graphics system must work together precisely, and the camera position data must be transmitted accurately.
“If the values no longer correspond, this can be seen in the images,” explains Sieben. “The background then begins to float behind the foreground.” During the system installation, a base calibration was carried out. “This calibration is very demanding,” says Philip Schwenzer, Project Manager from Avid. “The cameras are aligned with fixed measured points, in approximately ten different positions, to determine the exact location of the cameras.” Sensors from the robotics manufacturer Shotoko capture the position data, which are transmitted to the Avid system. The Avid rendering and graphics platforms constitute another key element of the virtual studio. Finally, four HDVG+ render engines combine the real and virtual image content; each engine is responsible for one of the four cameras. The HDVG+ is based on server components, which Avid equips with special graphics cards and video boards developed in-house. The graphics computers, in a 19-inch housing three units high, work with a Linux operating system. A tracking set processes the position data for the respective camera, transmitted by the Shotoko system via a network. From this, a render engine calculates the image segment and combines the real foreground with the virtual background. For this to succeed, the key or mask described above is also required.
A Challenge for Video Technology
For this purpose, NDR in Kiel uses an Infuse keyer integrated into the HDVG+ system. The parameters must be adjusted precisely to ensure a clean mask – one of the main responsibilities of the video engineer. “The keyer must always be exactly right,” explains Sieben. “For example, shading can be seen here on the image, if the key is not set perfectly.”
The diffused green lighting of the studio background presents a challenge for the video engineer. “Due to the comparatively small distances from the green screen, a relatively large amount of green light from the background is reflected on the news desk and presenters,” explains Sieben.
Painting the extensive walls of the studio set green is far from sufficient to achieve a clean chroma key. For the key, uniform illumination of the entire area is necessary, because the mask is generated for a particular color tone at a defined intensity. “We illuminate the studio background with approximately twenty area lamps, each with three compartments,” explains Maria Lindinger, lighting engineer at the studio. Here NDR uses Integra area lights from the lighting specialist Despar. Smaller area lights from Nesys serve to illuminate the news presenters and the real set, comprised of the studio desk and floor elements. In contrast, Lindinger uses the classic Fresnel lenses from Desisti sparingly: “We use the Fresnel lenses for backlighting and in part as supplementary lighting. However, they cast shadows that are too sharp. Therefore, in the virtual studio we prefer to work with area lighting.” For each of the individual presentation positions in the studio, lighting effects in the studio are illuminated and stored in the lighting control console. The regional studio works with a Grand MA2 lighting console. “We retrieve the lighting cues from the lighting console, according to the editorial workflow,” explains Lindinger.
The opening sequence of the program Schleswig-Holstein Magazin presents a special challenge. It begins with the presenter duo standing in semi-darkness in front of the virtual background. The camera zooms in on the presenters, while the illumination of the presenters is gradually increased. “The presenter lighting consists primarily of area lights,” explains lighting engineer Lindinger. “When we increase the illumination during the zoom, this results in a difference of a good 150 lux in the illumination intensity of the background. Even though the area lights are equipped with doors, the light cannot be directed so precisely that none reaches the background.”
Video engineer Sieben adds: “The beginning sequence is the most difficult for the keyer.” He has therefore stored several presets in the keyer. “For the opening sequence, the key has a somewhat fixed setting. For the other positions we use a softer setting,” explains Sieben.
Although the HDVG+ is a real-time graphics platform, the complex calculations cannot be performed entirely in real time. As a result, there is a several-frame delay. This must be taken into account in the audio signal path, otherwise the sound would no longer not be synchronized with the images. The audio engineers therefore set a delay of four video frames (160 milliseconds) in the Stage Tec Aurus audio console. Meanwhile, the background or studio graphics are fed in via the Avid Maestro Media Engine, controlled by a graphics operator. The complete broadcast workflow is transmitted from the Annova OpenMedia editing system, via a specially configured software interface, to the Maestro graphics system. The editors develop the entire broadcast workflow in the OpenMedia system where, by means of an Avid plugin, they can insert the broadcast graphics. These are then transmitted to the Maestro system, together with their exact position data. The implementation of the required MOS software interface was one of the responsibilities of Avid Project Manager Schwenzer.
The Maestro software runs on a control computer, which controls two other Avid HDVG2 graphics platforms. These each have two playout channels, making a total of four channels. “I have two insert channels and two full-screen channels, which I can record independently,” explains graphics operator Jannis Redmer. The background graphics and text inserts for the studio presentations are combined in the HDVG+ systems.
The text graphics for pre-produced contributions are added in the Sony MVS-7000X video mixer, in the downstream keyer. Thus, with the implementation of virtual production, less has changed for video mixer Ronja Reinke than for other colleagues in the studio team. The crew agrees that for virtual production, a higher level of concentration and even better communication are required. But does the new production environment fulfill the expectations of NDR? Martin Bechert, editorial project coordinator, appears very satisfied with the result. “Everything that we planned journalistically, that we had considered in advance, is possible with this studio. Now we can integrate large images, and we have a presenter duo format.” The use of presenter duos was part of the new broadcast concept. “It enables us to divide different perspectives on a topic between the presenters,” explains Bechert.
NDR Kiel Head of Production Matthias Rach summarizes: “With the large-format images, we are also taking into account changed viewing habits that have developed due to tablets and smartphones. One objective was to improve the visibility of the broadcast on these mobile devices.” This also seems to be reflected in the ratings: “We cannot conclusively evaluate the response yet,” says Bechert, “however, the trend is that in comparison to last year we have an increase in audience share of approximately three percent.”