Despite all of that, JPEG has now become computationally tractable. I remember the days where it took tens of seconds to encode a JPEG on a commodity machine. Now, with the help of SIMD, we can encode a high quality image in msec on a mobile device.
Fortunately you can choose the quantization matrix that determines the amount of loss. Even if you were to choose a unitary matrix, no human, not even superman with his laser eyes, can "detect" the quantization noise.
For SnappyCam, I chose to invest in JPEG a little more because it's a ubiquitous standard for still image compression.... and with the right hardware and algorithms, quite tractable.
I'll consider adding a JPEG "quality setting" so you can choose the amount of loss introduced... sounds like a great idea to me.
The idea behind SnappyCam was also to code each picture independently, and not rely on motion prediction or video codecs. If you try and pull a single frame from a HD video you might be disappointed: they compress the YUV dynamic range (studio swing) and it looks washed out, even if you land on an i-frame.
Lastly, as far as I can tell, the image sensor is yielding complete scans with each frame. I'd hazard a guess to say that any motion prediction or frame deltas might actually slow the whole chain down.
With so many recent requests for AGIF, I'm absolutely going to add it to the app. (It's been on my list for a while, but lower priority than getting up a solid core product and the social sharing that is in development at the moment.)
I don't recommend the same path for everyone, but my ugrad at RMIT University, dual bachelors of EE and CS. I then went on to do a Ph.D at the University of Melbourne in EE. My dissertation was on mathematical optimization of wireless and wireline DSL. (Prof Jamie Evans is an awesome guy if you're on the lookout for an advisor!)
I've been in SFO for just over 5.5 years, and started SnappyLabs after winning the "greencard lottery".
SnappyCam, you might say, is the embodiment of both a very practical ugrad and somewhat applied but very theoretical pgrad.
Guy behind me said "Hey, this is pretty cool. Have you seen... oh, you're looking at it already."
The other SnappyCam purchaser I mentioned is an iOS/Android developer who is a sub-tenant in my office. We actually have a cheap, spare desk in this room at the moment if you wanted to visit and work for a while. Email me if you want to ask any questions.
Another fellow Australian and Melburnian.
Well done on the app John!
I first tried using the GPU, using old school GPGPU textures and OpenGL ES 2.0 shaders, but unfortunately the performance wasn't there for a variety of reasons given in [1].
SnappyCam has since been making extensive use of ARM NEON for the JPEG codec and a bunch of other image signal processing operations, like digital zoom. It's a great instruction set!
[1] http://www.snappylabs.com/blog/snappycam/2013/07/31/iphone-k...
It's like MMX / SSE of x86 world, a set of extra instructions to process many small integers in parallel in one instruction. Since image data are usually independent 3-byte pixels (or 3 planes of 1-byte subpixels, one per color channel), NEON is great for many image-processing tasks.
See e.g. https://en.wikipedia.org/wiki/ARM_architecture#Advanced_SIMD...
I think you definitely made the right choice though - it's interesting that the obvious delta-coding and motion compensation tricks to reduce bandwidth are rarely used for video acquisition apart from the most limited devices like phones, stills cameras and the GoPro. Everything that can afford to uses per-frame coding like ProRes, REDCODE, AVC-Intra, DNxHD, Cineform... being able to seek quickly is important!
In fact Canon's 1DC 4k camera uses (dun dun duhhhhh...) motion JPEG :)
The only reason would be that you are pressed for memory or bandwidth, but certainly you have the resources to store one full frame and produce deltas, or just apply part of the JPEG chain, enough to remedy memory pressure. You can always encode it to an actual JPEG after the process.
And yes, pulling single frames from a completely encoded video isn't helpful, because they can get away with more compression. But there are very sophisticated algorithms for eliminating the redundancy between frames, which would have been my first avenue in attempting to do something like this.
Have you attempted to do something like this? Because he not only has attempted, he's done it. Therefore I think you should stop talking down to him ("completely pointless", "it's just bizarre", "my first avenue"). It comes across as wanting to prove how smart you are instead of seeking to learn from someone who has done incredible work and—lucky for us—is bursting at the seams with enthusiasm to share it.
Oh and congratulations jpap on what's looking like the most successful and technically solidest HN launch in quite some time! I hope your hard work pays off.
I buffer as much as possible, while also encoding on any other cores that are available.
There's a reason why a "simple camera app" can total some 80 kLOC. :)
It's not quite that simple. They have different strengths and weaknesses, so you can't say one is categorically better than the other. DCT has uniform frequency resolution, which sounds desirable but isn't once you start quantizing due to ringing, etc. See slide 25 or so of [1]. Wavelets overcome this problem by adapting the resolution in opposite ways at the extremes. This works fantastic for medium signal rates, but can have severe low passing at low rates. For low rate coding DCT with the bells and whistles (deblocking, etc) will typically win.
[1]: http://people.xiph.org/~tterribe/pubs/lca2012/auckland/intro...
That said, if their emphasis is on producing pictures with minimal time delta at highest resolution, algorithms used for still pictures are out of place. Video compression algorithms still use DCT and wavelets, but they do so only after they have reduced redundancies between series of pictures, a process that tends to work significantly better than anything you can get out of these lossy transformations when you want to preserve quality.
Of course, eliminating redundancy in a series of pictures might have tipped them off to the fact that the image sensor isn't actually producing fresh pictures at the rate they want.
1: as used in JPEG. The transformation itself is perfectly invertible, assuming infinite precision arithmetic.