News for hackers. Stuff that matters.
Re:Realtime encoding - over 150x faster by felixge on Monday December 20, 2010 @02:22PM (#2025407)
As you can probably tell, we are super excited about this, feel free to ask me anything : ).
Re:Realtime encoding - over 150x faster by JonnieCache on Monday December 20, 2010 @05:52PM (#2026090)
What have you done about the security implications of pushing user-provided data into ffmpeg straight off the wire? How worried about this are you? Obviously ffmpeg was not built with untrusted input in mind. I have only paranoia as a justification, but I bet I could make it crash pretty hard, and with a bit more work I bet I could design an input that caused ffmpeg to just fill your disk forever.

How much work have you had to do around these issues? Did they prove to be bigger or smaller than you anticipated?

Re:Realtime encoding - over 150x faster by nitrogen on Tuesday December 21, 2010 @02:36AM (#2027246)
I'd bet that ulimit, quota, and chroot would go a long way to solving those issues.
Re:Realtime encoding - over 150x faster by zdw on Monday December 20, 2010 @02:41PM (#2025477)
This is really neat, but since you said "anything"... I have a legal question that I haven't been able to find the answer for:

How do you deal with the licensing issues regarding open source encoding software? Do you pay the MPEG-LA a fee directly for use of the software? Is it per file/per minute/flat fee?

Just wondering about the mechanics of this, mainly for an in-house streaming application.

Re:Realtime encoding - over 150x faster by jon_dahl on Monday December 20, 2010 @03:39PM (#2025693)
IANAL, but generally, you need a license if you're using open-source encoders or decoders and you're the one who actually compiles them. (Again, not a lawyer, but that changes ffmpeg from "a source code description of an encoder" to "an encoder.")

MPEG video stuff is generally pretty cheap or free for low volume. Audio codecs are fairly expensive, on the other hand. AAC + MP3 + AMR is >$20K in minimum fees.

Re:Realtime encoding - over 150x faster by felixge on Monday December 20, 2010 @03:00PM (#2025552)
No, we don't pay the MPEG-LA (yet). The information they provide has given us reason to believe that we don't need a license at this point.
Re:Realtime encoding - over 150x faster by maxklein on Monday December 20, 2010 @02:56PM (#2025536)
How many ffmpeg instances are you holding in memory? You have a single ffmpeg process for each upload that is happening, do you? Additionally, if I am uploading files with B-frames and GOPs of 300 (which I assume will lead to a lot of frames being kept in memory, not sure about this), is your RAM requirement not going to be massive if you have several uploads happening at the same time? And considering the length of time a single video takes, it's reasonably likely you'll have a lot running in parallel, is it not?
Re:Realtime encoding - over 150x faster by felixge on Monday December 20, 2010 @03:05PM (#2025577)
We have one ffmpeg process for each upload, yes. From our tests with the kind of videos we mostly deal with (user generated content), this seem to work quite well while only requiring reasonable memory allocations.

So sure, there could be problems we don't know about yet, but that's the price for doing something nobody has done before. We will deal with these problems as they come up. In the worst case that might mean falling back to normal encoding if a ffmpeg child process starts to misbehave.

Re:Realtime encoding - over 150x faster by maxklein on Monday December 20, 2010 @03:24PM (#2025642)
So you're using plain command line ffmpeg or are you feeding the frames yourself? How do you throttle the speed of ffmpeg?
Re:Realtime encoding - over 150x faster by felixge on Monday December 20, 2010 @03:30PM (#2025668)
We are feeding raw bytes into a command line ffmpeg process via standard input.

Ffmpeg only reads as much from stdin as it can process. Node.js gives us a 'drain' event when this happend, which means we can pump more data into it. So we are no throttling the speed, just the flow of data.

Re:Realtime encoding - over 150x faster by badmash69 on Monday December 20, 2010 @06:15PM (#2026167)
As a curious outsider, I have to ask why are you giving away the recipe to your secret sauce ? I mean, there are several of your competitors such as zencoder , etc. that could easily copy this idea and implement it . So why would you blog about implementation know-how that made your product cooler and enabled you to command premium pricing ?
Re:Realtime encoding - over 150x faster by siculars on Monday December 20, 2010 @09:14PM (#2026621)
I don't really think it matters for a number of reasons. Chief among them being motivation. If you are motivated enough to copy this idea then you would reverse engineer it as soon as you heard what they are doing without knowing exactly how, which is what I did knowing that they were based on nodejs and I know how nodejs handles buffers.

Ultimately, if you are a provider of services, these or any other, you will add functionality or run the risk of being eclipsed.

Re:Realtime encoding - over 150x faster by felixge on Tuesday December 21, 2010 @10:57AM (#2028136)
Well, anybody in the video encoding business will know how we did it. It's kind of painfully obvious. So it's not so much spilling the secret sauce.

Also, we don't believe individual features themselves will give us a long term advantage. Consistently being a leader of innovation however, will.

Re:Realtime encoding - over 150x faster by dantheman on Monday December 20, 2010 @02:41PM (#2025481)
Looks great. Does this do encoding of streaming video or only files?
Re:Realtime encoding - over 150x faster by felixge on Monday December 20, 2010 @02:49PM (#2025504)
It only encodes uploaded files.
Re:Realtime encoding - over 150x faster by Kaizyn on Monday December 20, 2010 @05:52PM (#2026089)
How will this cool feature help your startup make money? How much cost in time and effort did it take to add the feature?
Re:Realtime encoding - over 150x faster by felixge on Tuesday December 21, 2010 @09:16AM (#2027764)
1. We are infinitely [1] faster at converting video uploads than our competitors now. Speed matters. This is a strong USP.

2. Publicity. There are a bunch of features that would certainly be equally useful to our customers. However, none of them would get much interest from hacker news. We got a lot of signups and page views from this.

3. It took me probably 4 full days [2] to implement this, including the screencast and blog post. That's reasonable for the value added by this feature.

--fg

[1] We are really infinitely faster if the upload speed is slower than our encoding speed. This is probably the case for 99% of all consumer internet connections at this point. In any case we reduced the time required for an encoding from SUM(Upload, encoding) to MAX(Upload, Encoding).

[2] We already made a lot of technology decisions that lowered the cost of implementing this. A competitor would certainly need much more resources to pull this off.

Re:Realtime encoding - over 150x faster by app on Monday December 20, 2010 @03:47PM (#2025712)
Is this limited to 1-pass encoding only?
Re:Realtime encoding - over 150x faster by felixge on Monday December 20, 2010 @03:50PM (#2025722)
yes, 2-pass encoding will not be doable in realtime.