Yes — many codecs can be optimized for decoding at the expense of encoding. This is appropriate for any sort of broadcast (YouTube, television, etc).

Also, in many applications, it’s suitable to exchange time for memory / compute. You can spend an hour of compute time optimally encoding a 20-minute YouTube video, with no real downside.

Neither of these approaches are suitable for things like video conferencing, where there is a small number of receivers for each encoded stream and latency is critical. At 60fps, you have less than 17ms to encode each frame.

Interestingly, for a while, real-time encoders were going in a massively parallel direction, in which an ASIC chopped up a frame and encoded different regions in parallel. This was a useful optimization for a while, but now, common GPUs can handle encoding an entire 1080p frame (and sometimes even 4K) within that 17ms budget. Encoding the whole frame at once is way simpler from an engineering standpoint, and you can get better compression and / or fewer artifacts since the algorithm can take into account all the frame data rather than just chopped up bits.

Surely videoconferencing doesn’t actually use 60 FPS...

Why not? It's not full-motion video, it's literally talking heads. Talking heads are easy to push to 60fps on a relatively low bitrate.

Some web conferencing would want to do 60fps. There's also realtime streaming like Twitch, PS Now, and Google's Stadia.

Twitch isn't real time.

Yes it is. The delay on a Twitch stream doesn't mean they don't have to deal with encoding and transmitting frames at full speed. If Twitch wasn't real time, you'd only be able to watch live streams slowed down!

That's a different definition than most people mean when they say "real-time".

Not me. All realtime systems have some latency, but what makes them realtime is that they must maintain throughput, processing data as quickly as it comes in. You can subdivide to hard-realtime and soft-realtime depending on how strict your latency requirements are, but it is still realtime.

really? I think not. Lets use speech synthesis as an example: I would call speech synthesis real time if it takes less than one second to produce one second of synthesized speech. I think you're probably thinking of the word "live". There's always going to be a small delay when re-encoding. Real time doesn't mean 0ms delay, that's impossible. Twitch has a small delay, but it's still re-encoded in real time (encoding 1 second takes ≤ 1 second for twitch).

The delay (configurable) isn't due to the encoder. And the encoder has to process everything in real time, otherwise you start skipping frames or fall behind.

Pretty much all of them. Encode complexity for most codecs is way higher than decode complexity (on purpose).

This has been an issue with AV1, it's got relatively high decode complexity and there's not a lot of hardware acceleration available. The encode complexity is fantastic though and is very slow even on very powerful hardware, less than 1fps so ~30 hours to encode a one hour video. Even Intel's highly optimized AV1 encoder can't break 10fps (three hours to encode an hour of video) while their h.265 encoder can hit 300fps on the same hardware.

A lot of video codecs are NP hard to encode optimally, so rely on heuristics. So you could certainly say that some approaches take a lot of compute power to encode, but are much more easily decodable.

The codecs aren't NP hard. Rather, the "perfect" encode is. That's where the heuristics are coming into play. The codec just specifies what the stream can look like, the encoders have to pick out how to write that language and the decoders how to read it.

Decoders are relatively simple book keepers/transformers. Encoders are complex systems with tons of heuristics.

This is also why hardware decoders tend to be in everything and are relatively cheap with equal quality to software counterparts. On the flip side, hardware encoders are almost always worse than their software counterparts when it comes to the quality of the output (while being significantly faster).

> The codecs aren't NP hard. Rather, the "perfect" encode is.

That's what I meant by my first sentence.

And I'll throw out there that the vast majority of 'hardware codecs' are in fact software codecs running on a pretty general purpose DSP. You could absolutely reach the same quality as a high quality encoder given the right organizational impetus of the manufacturer; they simply are focused on reaching a specific real time bitrate for resolution rather than overall quality. By the time they've hit that, there's a new SoC with it's own DSPs and it's own Jira cards that needs attention. If these cores were more open, I'm sure you'd see less real time focused encoder software targeting them as well.