In the past, acquiring and running software for desktop and laptop computers was a slow, thoughtful process. We would stand in stores (back when they existed) and stare at software boxes (back when they existed). Turning a box around, we’d comb through the specs, making informed decisions based on our intimate knowledge of the computers we owned.
That was for simple “home stuff” like Quicken or Doom but what about “work stuff”? As professional craft editors responsible for large projects it was even more critical to understand the specs and the tech behind the whole process.
At the time Avid Media Composer wasn’t just software in a box. It was acquired as part of an expensive “turnkey” system – a machine designed from the ground up with the sole purpose of running it as best as possible. It’s the etymology behind referring to an edit system as “an Avid”.
Today we download and install everything we see to our phones, tablets and even desktops. We’re app-happy. It’s how we test and consume new workflows, and it’s all thanks to this massive D.I.Y. culture. The result is editors and assistants building and supporting their own machines. Unless we’re responsible for outfitting a massive facility, turnkey systems are largely gone. Today all we need to do is click, download and pay a monthly subscription fee to get a working Media Composer system. But what kind of computer are we downloading it to? Will it handle Media Composer? Or perhaps the more appropriate question: How will Media Composer handle the computer?
We’ve all asked these questions over the years, especially when we were students learning the trade. In fact today, places like the Assistant Editors Bootcamp are great examples of how we bring new women and men into the industry. But are they learning these basics? When Noah at the AE Bootcamp reached out to me with these questions, they were at the request of his Lead AE and Independent Editor class. I was eager to help. But to truly get answers, we need to get them from Avid’s engineers directly.
Conveniently, Avid is in a wonderful mindset of transparency right now.
As a Vice Chair of Avid’s ACA, as well as a volunteer Moderator of Avid’s Pro Video Community, I was able to have conversations with a number of senior level peeps at Avid. Reponses came in from many of them – from Avid offices in Massachusetts, Québec and California.
It was the response from Shailendra Mathur, VP and Chief Architect that kicked everything into gear. Here’s what I asked:
For the purpose of assisting editors and AE’s, I’m hoping to create a 1-page guide that explains what parts of a computer are used by Media Composer – broken down by processes like rendering, AVX Plug-ins, real time playback, processor-intensive codecs and so on. Would you or one of your staff be willing to assist me in its creation?
The response from Shailendra Mathur:
Hi Chris, sure we can help. We have quite a lot of info on this since it has been a popular question through the ages, and it will be a wonder to fit it into a single page :-).
Shortly thereafter I got an email from a number of Avid’s engineers and away we went. The first question: How simple would this one-pager be?
I opted for a simple concept we could start with – listing only the “intentions” of the app, meaning answers to questions like, “For video effects, does MC use the CPU or the GPU?” and, “How many cores are used and for what tasks?” From there we started diving into the details behind those simple explanations.
It took a while, which wasn’t their fault but rather mine. In addition to my craft-editing schedule, there were a lot of emails and phone conversations with Avid’s engineers to help me understand things. I’m not an engineer, so I’ve been very thankful for their patience. I still can’t say I comprehend it all, yet the goal from the beginning was to have this written by an editor for editors.
Here is the result:
So… Are you begging for more details? Of course! We’re craft editors and that’s what we do.
First… What is this whole thing and how long has it been here?
The Avid Intelligent Compute Architecture as it’s called was initially developed when Avid Media Composer v3.0. It evaluates the whole system – the OS, hardware, GPU capabilities, availability of processors and number of cores. It dynamically distributes the processing to the device best suited to the specific task for different segments of the timeline. Rather than targeting just the GPU, just the CPU, or just the FPGA based cards, the philosophy changed to use them all in a holistic fashion. Thus the whole system is turned into an accelerator. The Intelligent media player in the application acts as an orchestra conductor, keeping as many of the resources playing to provide the performance required. Keeping a holistic view of the whole system in mind, particular attention is paid to the cost of transferring heavy video data across the system bus when deciding which compute hardware should be used for a particular process.
OK let’s get into it. Below are cutouts from the above 1-pager, followed by the notes I jotted-down during my conversations with Avid’s Architecture team.
RAM and Cache
1. Everything works better with more RAM. Filling a computer with the maximum it can handle is now a standard recommendation. Having less RAM constricts Media Composer’s abilities.
Media Composer works best when it encounters the least restrictions possible, especially when it comes to RAM. For example, take a 2014 MacPro with 16GB of RAM. Hit play and watch the RAM usage in the macOS Activity Monitor. (This is how we monitor apps and their efficiency.) Media Composer may hover generally around the 8GB area. But if you take that same system and change the RAM to 32GB, Media Composer may hover generally around the 14-16GB area. This isn’t Media Composer “hogging” more resources, but rather the smaller RAM is constricting the system’s ability to use any available resources. This is a prime argument for increasing a system’s RAM to max capacity.
Maxing-out the RAM “lifts the computers ceiling” as high as possible. Thus all functions have the possibility to operate at their own maximums on that computer, without constraints placed on them by lesser amounts of RAM. If a new iMac can physically hold a maximum of 64GB of RAM, then that’s what is recommended by Avid.
Since RAM can be the easiest and least expensive way a user can upgrade a computer, Avid has been architecting the 64-bit playback engine to take advantage of RAM first.
Minimum requirements will, of course, still exist. For example, a laptop with only 8GB of RAM and a 5400RPM external drive holding a project’s DNxHD media will work as a minimally qualified machine.
2. Larger raster sizes (UHD, 4K etc.) use more RAM than smaller ones.
Larger raster sizes means more pixels, which require more processing power to play in real time.
3. The Interactive Frame Cache within Composer often assists playback. Since cache from Composer is stored in RAM, increasing the Media Cache -> Video Memory can improve stream counts. This allows more streams to be played in real time without rendering. Users should learn to use this setting as needed.
Leaving the Desired Video Memory cranked-up all the time may negatively affect other processes and apps.
In Media Composer’s processing algorithms for playback, video is actually looked at as individual frames. During processing, Composer determines how those frames get played in streams. For example, a single video layer of DNxHD media qualifies as one stream. More layers and effects add more streams.
The term processing refers to the heavy lifting of playback. There may be pre-processing involved at some stage (where things are transformed – partial renders would be one form), but getting everything from the drives put forth into the output, all of that is referred to as processing.
The term cache in a timeline refers to a way that processing distributes the handling of playback. When processing gets really dense and complicated, here is where CPU cache can assist.
The cache might be thought of as a sort of invisible Video Mixdown that the system uses to reduce strain and help playback. That’s a pretty narrow and somewhat incorrect definition, but it hopefully gets the basic point across. A better way of explaining it would be: Instead of doing a complex evaluation over and over again (processing), Composer keeps the result in RAM, saving the pain of fully reprocessing over and over. Cache is downstream of processing, and will remember the results of processing.
Cache does eventually fill up. At a certain point, when there is no longer room in the cache, the oldest used frame is thrown away in favor of the new frame. Want more frames saved in cache? Get more RAM.
Note: On this particular system, which is loaded with 32GB of RAM, Media Composer is operating at around 8 GB. Setting the Media Cache’s Video Memory to 22 GB adds to that memory pressure, reaching a grand total of around 30GB. This means only 2 GB of RAM is left, which can cause memory issues if any other apps are launched or if background processes like Dropbox or iTunes begin to sync.
When Media Composer is closed, the Memory Cache setting is not saved, thus allowing for fast, easy re-launching later.
Note: Increasing this Video Memory setting by large amounts may negatively affect other processes that require RAM, so do not to leave this cranked-up all of the time. Adjust it up/down as needed.
4. The Playback Video Frame Cache improves single frame play responsiveness.
While the Media Cache -> Video Memory increases the number of frames saved, this setting increases the responsiveness of those frames during playback. It gets better results with Media Cache -> Video Memory set higher.
5. Currently the CPU handles encoding/decoding of codecs.
RED Camera files (R3D) are the exception, which encode/decode with help from the GPU, and certainly more so when a Red Rocket card is assigned the workload.
Most codecs are structured in a way that one might call “GPU averse”. But this could (and likely will) change in the future. If codecs become more GPU-friendly as bitstream formats, then Avid will no doubt chase that philosophy.
6. Playing codecs smoothly in a timeline requires processing, which benefits from more CPU cores. Codecs with large raster sizes also benefit from more cores.
Playing a timeline that contains Linked (AMA) clips plus many effects results in a higher stream count, especially with more complex codecs. With more cores there are more opportunities to distribute the processing of those streams effectively.
7.All codecs benefit from more RAM, but some codecs (LongGOP, AVC/H.264) need much more to work effectively.
Some codecs are easy for computers to play and edit (like DNxHD). Other codecs are more complex and require a great deal more processing power to get minimally acceptable results. There are certainly situations where computers that are considered minimally qualified to run MC are not powerful enough to run MC plus those codecs. More RAM will have to be added.
Files accessed through the Source Browser can also be transcoded in to Avid codecs (Clip > Consolidate/Transcode) which use less streams and less system resources to play. Workflows using Avid HD-sized codecs do not require a high number of cores to work effectively.
Note: The H.264 codecs (.mov and .mp4) are no longer being handled by the legacy 32-bit QuickTime engine. As of Media Composer 8.9.1, the 64-bit playback engine handles them. This applies to XAVC-S (.mp4) as well.
Video Quality Menu
8. The Video Quality Menu changes the raster size of the viewed output to allow weaker computers to play complex codecs smoother. Green/Green mode plays the codec in full raster. Yellow/Green mode reduces that raster to 25%. Full Yellow mode reduces that raster to 6.25% (1/16th size). Currently the CPU, not the GPU, handles this raster resizing.
The CPU is doing this resizing from its original raster size. Operating in these lower modes on HD-sized projects usually allows for a weaker computer to play back a timeline smoother.
For larger-than-HD raster sizes however, this can add a CPU-based bottleneck. The computer is using more resources to do a real time conversion of the raster.
9. If a codec is greater than 8-bit, switching the Video Quality Menu to Green/Green/10-bit mode playback can sometimes be a more effective use of overall processing. This depends on the amount of effects on a clip.
Video Effects, Timeline & Playback
10. GPU is front loaded/preferred by Composer when playing from a timeline. Playback looks at the topmost layer in a timeline first, which is seen as one stream.
The term “front loading” means that the top-most-layer in a Media Composer timeline will target the GPU first. When playing a sequence, Media Composer looks at a timeline’s play head (blue vertical bar) from above, and not from the side like we users do. (Imagine the play head as the light bar in a photocopier or a scanner, hovering over a sequence.)
Before playback of a sequence with effects begins, Composer’s algorithms allocate as much of the timeline as possible to the GPU. It also allocates some CPU power, but only after it identifies specifics within the timeline that need CPU-only processing and/or multithreading over a number of cores.
The primary reason for all of this is we want to read back only one stream from the GPU, and that starts with the topmost layer. So loading a computer with the hottest qualified GPU and highest amount of GPU RAM possible helps with this. The better and more RAM-heavy the graphics card, the less data needs to be sent to the CPU and the cores.
Render and Expert Render can relieve system stress by collapsing multiple streams.
11. Some effects are processed only using the CPU. They were engineered to do so. More recent effects have been engineered towards GPU usage.
12. Some Color Adapters (source effects for example) are processing-intensive, so a more powerful GPU will handle them more effectively.
That’s it on my notes in-context with the basic document, but here are a few other tidbits of information I picked up in conversation.
– As of Media Composer v8.8.3, the QuickTime AMA plugin relies a lot less on the QuickTime engine (operating in the background outside of Media Composer), and more on Composer’s own playback engine.
– Higher frame rate clips benefit from more cores as well as bandwidth. This is because higher frame rates can be handled by something called stream-based parallelism (the processing of sub-sequent frames in parallel). This eats up a lot of buffer memory. Thus another argument for more RAM. Note: this is only for I-frame codecs. Hence why GOP codecs are so much more difficult. Their processing of frame rates cannot be handled in any sort of method of divide-and-conquer. By design, GOP codecs have a lot of interdependencies between the video’s frames. If processing of a later frame can only proceed once the dependent earlier frame has been decoded, then you have a result that needs to be addressed much more practically and linearly.
If you’re really into learning the architecture behind all of this, Avid has public access granted to many documents created over the last few years by Avid’s architecture team.
Here are the links:
1) A blog discussing the architecture itself – the Avid Intelligent Compute Architecture: http://community.avid.com/blogs/mediacomposer/archive/2013/03/12/how-intelligent-computing-powers-our-editorial-architecture.aspx
2) And here is my personal favorite – the public file ion Google of the patent filed which includes the performance architecture: https://www.google.com/patents/US8358313
Any questions? I’m sure there are. If this info doesn’t prompt more questions, I’d be surprised. For the sake of being heard as best as possible, let’s please post all questions and comments at one place, on the Avid Community.
Here is the link: http://community.avid.com/forums/p/182676/849425.aspx#849425
Ask as many questions as you’d like. If it’s something I don’t know (which is plenty), then I’ll pass it along to others. Or perhaps other users here can step up? The goal is a universal understanding, in order to make us better at our craft. Hopefully this is one good step in the right direction.
Thank you for you time!
(AKA “Pixel Monkey” on the Avid Community)