How Fox Sports Will Use Virtual Fans Created On Unreal Engine To Fill Empty Stadiums On MLB Streams

Baseball is back. But MLB will look a little different in the COVID-19 world, with empty stadiums filled with silent cardboard cutouts instead of thousands of fans screaming in an effort to prevent the virus from spreading. That is unless you are watching Fox Sports. Starting this weekend, the network will not only offer fake audio to fans, but will also fill the stands with digitized fans for its MLB broadcasts, starting with the Yankees-Nationals game this Saturday.

All 30 MLB teams will depend on fake noise from fans in real stadiums (coming from Sony MLB: The Show games), along with the usual announcers in the stadium, music and other aspects of the pomp and circumstances that accompany a Major League Baseball game. But Fox Sports is the only station that has announced plans for virtual fans.

“We think the crowd and people watching in the seats is part of a broadcast, it’s part of the high-level sports broadcast in the major leagues. So we wanted to find a solution for that, “explains Brad Zager, executive producer and head of production and operations at Fox Sports.

“We are not looking to fool anyone, it is still about the game. But shot by shot, when you’re watching a broadcast, it’s not more noticeable that you’re watching what is normally a broadcast with fans in an empty stadium and it feels weird: We want to give people a sense of normalcy, “says Zager. “And we feel that going down this path and trying to use a virtual crowd will hopefully get that mixed up, and you can focus more on the game because you’re not thinking about a major league stadium void during the game.”

The effect is a combination of technologies that you may have seen before. The augmented reality software used to insert crowds is called Pixotope, which has worked on AR graphics for things like the Super Bowl and The Weather Channel terrifying storm warning demonstrations.

It works by taking advantage of graphics (created by Silver Spoon Animation creative agency) created in Epic’s Unreal Engine. Unreal Engine is used here for the same reason that it is popular for creating video games or for creating virtual backgrounds on set for shows like The Mandalorian. Unlike most movie graphics, which must be rendered in post-production after the fact, Unreal can render in real time, making it much more suitable for live television.

Lastly, SportsMedia Technology (SMT), the same company that handles most of the sports-focused computer overlays that you’re familiar with, such as football’s first yellow line, on-screen clock, and scoreboard, and more, it handles camera tracking to insert those graphics into the live camera feed. Four cameras: home high, center, first high and third high cameras, will show virtual fans.

Fox Sports producers will be able to control things like how full virtual “crowds” are for a given game, what weather fans are dressed for, and what percentage of the crowd will be local fans vs. visitors, though the company is still figuring out how will make some of those decisions. “We are solving this right now, and we will continue to evolve that process as we go through this weekend and from game to game, week to week,” says Zager.

Fox Sports broadcasts will also integrate simulated fan audio playing in the stadium in the background of player microphone clips (such as during pitches) and other in-game audio, but will augment that live audio on your own end, too, similar to how the MLS games have fared.

And while interactions between virtual crowds and audio will be basic for now (such as applause, booing, and the wave), the company hopes to better integrate the two as more is practiced throughout the season. “It’s something we hope will become even more collaborative and more on the same page as we do this more and more, being able to combine the audio and the crowd.”

For now, virtual fans will only stream in MLB games on Fox, but it is possible that digitally generated fans may also appear elsewhere. Zager says the company is working with all of its sports sponsors and that “as we get closer to the fall of soccer and other sports, if we feel it is going in the right direction, we will launch it and implement it. other sports “too.