Changing the Flow of Content from Inception to Consumption

Content Insider #598 – Video Production AI

By Miles Weston

“It’s not so much staying alive, it’s staying human that’s important. What counts is that we don’t betray each other.” – Winston Smith, “1984,” MGM, 1984

While SMPTE attendance provides ample time for comradery and networking, most of the event’s activities are designed to provide new M&E opportunities and answer key questions: 

• How do we make the development/production process more efficient, faster?
• How do we deliver the right content to the viewer that will grab him/her?
• How do we ensure every video project will be a profitable (ultra-profitable) investment?

Addressing the various stages of the video storytelling process, the answer was always the same … AI.

With the rapid growth of OTT and streaming content, the biggest challenges the industry faces are accelerating content creation cycles, producing and delivering content for multiple platforms and creating personalized user experiences for the non-linear services.

All of these areas require more efficient processes of creating, producing, delivering and managing recorded video material. AI can be trained effectively to streamline the most time- consuming and repetitive processes.

As a result, AI is being used in several parts of the content production and delivery workflows as well as in the consumer services to enhance the viewing experience.  For industry executives at the conference, the goal was to help them develop new monetization opportunities.

Still in its infancy in the M&E industry, AI has the potential to streamline, clean-up and materially improve many phases of content production/distribution process.

AI can be powerful and effective assistants for the industry. 

Tedious Work – Surprisingly, much of the work done in the production of a film, TV show or video short is boring, mundane, routine stuff like tracking people, equipment, locations, digital assets and costs.  Timing and execution are everything and machines and MAM (media asset management) tools can handle these tasks to help ensure the final product is really viewer ready. 

Right now, it can only base its decisions and recommendations on the information available to it and creative product data is far from perfect.

Whether you’re a producer, show runner, director, cameraperson, DIT, editor, audio engineer or other member of the production team; it’s impossible to inventory every aspect of the workflow.

Gut Decisions

There are still those experienced, intuitive moments when you know that doing something will ruin a good video project or put it over the top.

Sometimes it requires a gut feeling or a quiet voice in the recesses of your mind that says, “Now!”

Many at this year’s SMPTE conference noted that the technology is in daily use for M&E production and distribution.

Jason Perr, media workflow specialist, at Workflow Intelligence Nexus, told the audience to wade through the marketing noise and find the right AI tools for each media workflow project. 

Focusing on MAM (media asset management), he said that today, AI provides better accessibility and visibility across large libraries of content to extract metadata and relieve people of the dull, time-consuming task of doing the work manually.

“It can not only do the job more quickly,” he noted, “but it can also do it more accurately.”

According to Perr, MAM AI tools are increasingly being used for facial recognition, automated metadata tagging, effective searches across transcriptions and even in uncovering and monetizing long-forgotten, older content. 

He also said AI tools can be of considerable assistance during the production process by carrying out phonetic searches, keyword spotting, object recognition and speaker audio fingerprinting. 

Because the MAM AI tools are just beginning to be widely used, he urged the audience to test them in their own organization before buying them.

Editor’s Toolkit – Norman Hollyn, a respected editor and professor at USC, explained how AI can help producers, DITs and post production people in keeping track of video assets to deliver content people will want to view. In the future, he sees AI assisting in localizing and personalizing content in realtime. 

A long-time film, television and music editor as well as professor at USC; Norman Hollyn focused his presentation on editing suite AI tools.

Hollyn said that AI’s visual recognition capabilities can be invaluable in hyper-localization of content as organizations begin streaming content to specific countries and audiences. 

He explained that AI can identify and replace objects in a scene to incorporate different sponsors or change signage and labels to local languages. 

Editing

Hollyn noted that visual recognition tools like these are presently used only in high-end projects but that he believes they will quickly be affordable for low-budget, Indie and web-based video.  

AI script analysis tools are also being used for shot and scene identification as well as to identify characters in a film. He added that task AI tools are becoming available to identify sentiment, gender and other scene attributes.

“This will become extremely important as content providers broaden their global content distribution and need to quickly and economically localize the content for the various audiences,” he explained. 

AI development is also underway to provide content developers and producers with improved insight into what increases a story’s emotional pull in the form of musical scores or visual images.

When tools like these are available, they will be of considerable boost for video storytellers to meet audiences’ demands around the globe.

During one of the coffee breaks, we discussed the impact of AI on content distribution and delivery with Allan McLennan, Chief Executive of PADEM Media Group.

“For newsroom, sports, TV/film production, AI isn’t coming, it’s here!” he emphasized.

“It helps speed up and enhance the creative process by helping project managers, creators and delivery people make decisions that speed up the production process and enhance the presentation to viewers.

“It won’t replace the human aspect of the industry,” he noted, “but it will enable professionals to enhance and smartly streamline the content creative and management tasks.  In addition, it will enable content streaming services to efficiently and effectively deliver very personalized content to each individual viewer.”

Streaming Flow – If you get the feeling that everyone is producing and sending video content over the air for consumers, you’re probably not too far off.  More than 75 percent of the Internet’s bandwidth is being used to distribute video stories; and it’s only going to increase.

According to McLennan, the industry’s primary goal is to deliver content to today’s hungry consumers that they want, when they want it and on the screen they’re looking at at that particular moment. 

The challenge facing everyone in the content creation, production and distribution industries is to do it more efficiently, more effectively and ultimately; more profitably than the next guy. 

“AI can be of tremendous assistance, providing more options in the creative process and ultimately, an accelerated way to deliver,” McLennan noted. 

While Netflix’s subscriber recommendation has issues, McLennan said the company – and several others – are refining robust personal assistants that will make effective personalized viewing recommendations that will be extremely accurate rather than just “pretty close.”   

Speakers and attendees agreed that the technology will benefit the entire creative spectrum from scriptwriting to script selection and production as well as content acquisition and marketing. 

It’s impossible to have humans manage these complexities without the help of an AI-powered, cognitive support system and tools.

Moving Target

Today, there are greater demands on each stage of the content lifecycle, from story development/selection, scriptwriting, cast/equipment/location schedules and increasingly complex post production–all against increased cost and budget constraints. 

In addition, content creators need to be aware of and take advantage of multi-platform opportunities which require overhauling projects from conception to delivery.   

To understand if video storytellers agreed that AI will be a valuable assistant rather than a “competitor,” we had a breakfast meeting with several area filmmakers.

Andy Cochrane – who is working on a broad range of 4K, VR and volumetric projects, started off the discussion by saying, “Alarmists have created a lot of fog around AI.”

He and Andrew Shulkind, one of last year’s SMPTE keynote speakers and an immersive content pioneer, said researchers and technicians are already working to put AI in much of the technology filmmakers use already, which will make all of the project workflow more intuitive than was previously possible.    

AI learning techniques will help filmmakers make intelligent decisions and ultimately make the video assets more valuable to the viewer and the filmmaker.

Recommendation Engine – Netflix is one of the first companies to make consistent use of technology and creativity to determine what projects would be produced for their target audiences around the globe.  They are fine-tuning their recommendation engine to be more in tune with the viewer’s entertainment wants and needs–including when and where they view material and other factors to tailor recommendations just for him/her. 

“We’re about to be at the point where AI allows us to get back to the reason creatives are in this business,” Cochrane explained. “It will relieve us of the mundane and time consuming tasks and let us focus on being creators.”

According to a recent Adobe/MIT study, a creative AI assistant reduces much of drudgery out of projects by addressing the most tedious and time-consuming tasks such as script breakdowns, storyboarding, shot list generation, schedule optimization and creating budgets. 

“Professional filmmakers are doing increasingly complex 4K, 8K, HDR, immersive AR/VR/MR projects that require original and disruptive creative work,” Cochrane noted. “It will improve things immensely when we have AI assistants to handle the boring but very necessary detail tasks.

“There will still be plenty of room for human error,” he assured us, “but with AI assistant tools, we’ll be much more certain that we’ll create something that will elicit the emotions and response we want from the viewer.”  

Shulkind has unique experience with where AI meets narrative storytelling, dating back to 2000 when he was a visual technician on the Steven Spielberg film, A.I. Artificial Intelligence.

That film used technology to explore the Pinocchio-like story of a robot boy, brought to life by artificial intelligence to replace a couple’s dead son. 

“One of my favorite examples of where AI currently meets storytelling is End Cue’s Sunspring,  a sci-fi short film written by an AI algorithm trained on existing produced scripts.  Its bizarre and terrifying and wonderful – and almost totally incoherent.” Shulkind said, “Putting the efficiency of artificial intelligence to work in creative workflows must be managed by humans.”

For the scripting of Sunspring, the AI identified patterns and trends based on past films from the 1980s and 1990s as well as identified patterns, trends and segments that would be emotional for the viewer.  However, it didn’t really know what the emotion was going to be and why, because emotion is a human response to a specific scene or the entire film. 

“We’ll see more experiments like Sunspring and they’ll probably get better over time,” Cochrane commented. “But AI can only augment the expertise of the video storyteller; it will not replace him or her…at least not for a long, long time to come.”

“It still requires creative talent, common sense and the willingness to risk doing something different and better to connect with the viewer,” he added.

We’ll See.

Just remember that back in the early 1900s renowned economist, John Maynard Keynes, predicted that by 2030, we’d have widespread technological unemployment and folks would only work 15 hours a week.

Most of the folks we know in the industry still count themselves lucky if they only work 15 hours a day.

And even with the new AI tools that are available (and being developed), none of the SMPTE speakers bothered to end their discussions with disclaimers…garbage in, garbage out and what you don’t know can hurt you!

Recalling a comment from an AI researcher at a recent creative digital conference in France, Cochrane added, “Thinking Outside The Box Is Impossible When You ARE A Box.”

When it comes to the use of AI tools, Winston Smith said, “Freedom is the freedom to say two plus two equals four. If that is granted, all else will follow.”