On the 30C3 there was a talk about a C Compiler (actually an llvm optimization plugin) that can eliminate nearly every memory management related vulnerability by adding memory checks. The penalty is only a 100% increase in runtime.

> If you're a C programmer and somebody says like: I've that optimization that makes your program 3% faster you go: wow! Then you come and say: well, now I make it half as fast you go: w00t?! But then you remember that people actually use ruby to serve web pages.

I'd say this would be actually worth it for a far more secure FFmpeg.

https://www.youtube.com/watch?v=2ybcByjNlq8

The difference is that FFMpeg is very, very CPU-bound, taking four hours instead of two to encode a movie is a big hassle. On the other hand, not many people use a world-facing FFmpeg instance to submit jobs to. I don't see why security should trump speed here.

You mean, not many people use VLC?

https://www.videolan.org/developers/vlc.html#thirdparty

For realtime playback locally sure. I don't think there are many/any public-facing systems backed by VLC. Presumably VLC prioritizes speed over security.

Oh, that was the point being made. My view is that VLC essentially is a public facing system, in the same sense that my browser is a public facing system. Sure, it is not as easy to do a targeted attack against my VLC as against my mail client (or against the nginx on my server). But it is fed a lot of untrusted data. The same goes for Chrome, which also uses FFmpeg according to the article.

or plex, which i believe is ffmpeg based for real-time transcoding

There's also AT&T Labs' memory-safe dialect of C called Cyclone: http://cyclone.thelanguage.org/

It's been around since 2006, but I haven't really heard of anyone using it. I wonder why.

Cyclone has been dead since years (the last release is dated 5 May 2006)

It never compiled on 64bit architectures, and to get it to compile with 4.0 or 4.1 (newer versions of gcc won't work) you need to fetch it from SVN (last commit is dated May 2009)... and obviously: no distribution still ship such an old version of the build tools... you'd need something like debian lenny, which does not even receive security updates anymore

"that can eliminate nearly every memory management related vulnerability"

This is like saying "my bulletproof vest will stop nearly every round fired at it". Too bad it only takes ONE...

So I suppose you'd rather not wear any bulletproof vest?

Before C escaped UNIX, there used to be safer system programming languages like Modula-2, just to cite an example.

But Modula-2 had Lilith to offer, while C had UNIX.

There are other things to consider as well.

Compiler support across various platforms: are there compilers available and do they generate good code?

Mindshare: how big is the intersection of people who know the language and have the domain knowledge to contribute to the project?

When C was UNIX only, the generated code quality also had a lot to be desired.

Sorry, I don't get what you're arguing here. Are you saying the C-compilers generated poor code when the FFmpeg project was started back in 2000? Are you arguing that if the FFmpeg project had used Modula-2, this would have sparked more work on Modula-2 compilers?

Or are you perhaps just wishing that history had taken a different turn?

The initial release of FFmpeg was in 2000, well after that stage in C's life.

> What should FFmpeg have been written in?

There'll be a HN topic A FFmpeg reimplementation in JavaScript soon enough ...

Process videos in browser using FFmpeg compiled to JavaScript (56 days ago) https://news.ycombinator.com/item?id=6739582

The only options I could think of that're mature enough are OCaml and Haskell. I'm aware of video transcoders in both, though nothing with anything like the format support of ffmpeg.

OCaml I know doesn't support SIMD acceleration. Not sure about Haskell. You need SIMD support to get anything decently fast in the field of video/audio manipulation.

Haskell's GHC is working on it [1]. It's not primetime (not even close, really), but it does work with modified libraries and compiling to llvm.

There is also Intel Lab's HRC (Haskell Research Compiler) that does have SIMD capabilities [2]. Unfortunately, there isn't a public release of HRC yet.

---

[1]: https://ghc.haskell.org/trac/ghc/wiki/SIMD

[2]: http://dorchard.wordpress.com/2013/10/14/automatic-simd-vect...

That is a compiler issue, not language.

ISO/ANSI C also don't support SIMD, you have to go down to Assembly when writing portable code across C compilers.

if we're willing to posit the existence of unwritten software that makes simd work from ocaml or haskell, we could also just assume the existence of C that checks memory accesses for zero cost

also, I think you may be able to do reasonably well with icc and not much assembly

Ya, it's definitely a language issue; barring super-intelligent compilers that can vectorize things on their own (which GCC does for certain cases), you need a means to express vectorizable operations.

You don't need to work at the assembly level for this; you just need a language that can express vector operations in a way that doesn't require a compiler to solve the halting problem, and you need a compiler that knows how to compile such operations to SIMD.

Yes, ANSI C doesn't have such a construct, but GNU C (which I'd bet 95% of open-source code uses anyway) does.

Nearly all SIMD in ffmpeg/libav/x264 is just written in assembly. Back in the day, gcc's code generation for even x86 SIMD intrinsic functions was too poor to consider using it.

(Plus x86's intrinsics are just plain ugly. It's easier to read asm.)

As for the architecture-independent SIMD in GNU C, it's something, but not quite flexible enough. Even C is really not a good enough language to consider implementing this in, because it has an overly-conservative memory model when it comes to char *, which aliases all other memory. For a library dedicated to parsing things which come in the form of bytestreams and 8-bit pixels, this means that most compiler optimizations are defeated right in the inner loops where you'd need them.

Isn't that what the restrict keyword is for?

gcc mostly ignores restrict, though it's got a little better in recent versions. icc uses it a little more.

But restrict is only good in specific situations - if you have three pointers, and #2 can alias #1 but not #3, there's no way to express that.

It also tends to be used on function arguments, not at some higher up point of declaration. So when compilers inline the function and can see more global info about it, the restrict gets lost and optimization gets worse.

I think some kind of stronger 'typedef' would be better.

Yes, but how long does this exist? My guess would be that it isn't really used (yet) in ffmpeg.

Since C99 so 15 years, which is older than ffmpeg.

What I meant is how long is it supported by compilers. I know its by GCC, but I think it's not by MSCV. Is it possible to compile ffmpeg with MSVC? But ok, one could use it and just #define it away for crappy compilers.

According to MSDN, Visual C++ has supported the equivalent __restrict keyword since at least 2005 [1], and according to the ffmpeg documentation [2], "FFmpeg can be built with MSVC 2012 or earlier using a C99-to-C89 conversion utility and wrapper, or with MSVC 2013 and ICL natively."

[1] http://msdn.microsoft.com/en-us/library/5ft82fed(v=vs.80).as...

[2] http://www.ffmpeg.org/platform.html

Well, expressions like these look to me very clearly like a pattern that can be implemented using SIMD instructions:

    map (uncurry (+)) (zip [1,2,3] [4,5,6])

I guess because of these standard functions and the fact that everything in Haskell is immutable and side effect free it should be comparably easy to do SIMD optimizations. It's only a guess, though.

Not if [1,2,3] and [4,5,6] are actually lists, or aren't aligned properly, or aren't the correct bitwidth, or the expression can't saturate/wrap, etc.

The compiler needs to be able to see that the data conforms to all those restrictions (possibly more), and that's not a trivial thing to do.

Shouldn't the compiler be able to see this? After all, Haskell is statically typed. Ok, maybe the compiler (optimizer) needs to look across function boundaries, but it should be possible, shouldn't it?

In the simplest cases, sure. But inferring that amount of information in general programs is very much an open problem. Even with GCC, which incorporates what is roughly (at least within an order of magnitude) state-of-the-art autovectorization, you have to be careful how you write your loops and structure your arrays or the compiler won't pick up on it.

Here's an example in Haskell: consider the case that the list of numbers was generated from a list of some other structure; something like "map x coordinates" to get all the X values from a list of X-Y-Z coordinates. For vectorization to work, those values need to be packed together in memory; however the structure isn't organized as such (at best, you'd have XYZXYZXYZ). That means a nontrivial memory copy, which will cost much more than the vectorization will gain you. So the structure needs to be reorganized, but then that might cause other pessimizations.

I see.

Clearly the whole project should be scrapped and rewritten in Ruby ;)

Ada would have been a good one, and mature enough. :)

A) Ada isn't sexy and unproven B) Most people who do mad algorithmz don't give a fuck if they introduce security vulns (even if they do, they don't know that other languages exist other than C), nor do they care about other languages (probably are put off by the facts that 1. other mainstream languages don't support mad algorithmz (Ada isn't mainstream) 2. if you look up any mainstream programming language you get nothing but dogma)