For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG.
If someone can do this over video files that will be PiedPiper comes into real life.
The reason this can be done with JPEGs is the compression hasn't been updated. There are folks that have used h264 for compressing images and webp uses vp8 compression, both with much better results than JPEG.
Lepton is cool because it helps make existing technology a whole lot better, but what we actually need is a better image format.
You wouldn't see the same leap for videos because people have been working hard to make those great for a while while JPEG has been left to rot.
All that is true, but it misses out a key point: Lepton is risk-free.
I could re-encode all of my JPEG photos with a better codec, but the problem is that then they have gone through a lossy compression scheme twice: once with JPEG, and once with the new codec. This could ruin the image quality on some photos, giving them nasty artefacts.
The newer codecs might even be hindered further by the fact that they are being asked to compress an image that's been JPEG'd. I don't think any of the codecs you mentioned are tuned to work their best with image data that already has been butchered by JPEG. They expect to be given the 'pure' original image. This may well cause the codec to perform worse than expected.
Using Lepton gives none of these risks, since jpeg<->lepton is lossless. I could throw away all the JPEGs afterwards, safe in the knowledge that if I want to go back, I can.
Also, it's not really true that JPEG has been left to rot. There are lots of programs available that greatly improve upon the basic JPEG compression, while still outputting a JPEG file that any compliant decoder can read. And there's an even simpler way to shrink your photo file sizes - just drop the JPEG quality level slightly. Most cameras and programs default to a very high quality value. Dropping the default by even a tiny amount can produce remarkably smaller files, and you'll probably never notice the miniscule image quality loss.
I suspect that JPEG is so popular because it is 'good enough'. Most people and programs don't care so much about squeezing out smaller files, so they'll keep using JPEG regardless.
Indeed. In one project where I converted jpegs to webp I saw around 70% savings at similar quality. This was on Android where it's fully supported. On the web pretty much only Chrome supports it, I'm not sure why Firefox, IE and Safari are hesitant.
Thanks for testing this! For archival purposes, be sure to check the exit code of the lepton binary after compressing each JPEG: the default parameters only support a subset of JPEGs (you need to pass -allowprogressive and -memory=2048M -threadmemory=256M to support a wider variety of large or progressive JPEG files). Lepton will not write images that it is unable to compress using the settings provided. In those cases be sure to keep the original JPEG.
Hi Daniel, very cool project! I am interested in hearing your thoughts on alternative image formats / compression methods.
WebP is very promising (also based on VP8) for lossless and lossy compression. Have you considered using it to compress PNGs in the same way Lepton is compressing JPGs? Odds are it wouldn't be bit-perfect though (despite being pixel-perfect).
Also interested in hearing about the tradeoffs between server-side decoding and client-side. Not to keep focusing on it, but WebP has native support in Chrome and javascript decoders for everything else.
I think it is a very exciting time for image formats with several promising new ones on the way (WebP, FLIF, maybe BPG but possible legal issues).
I'm running over several terabytes and on the 5k images I've done so far (3.6GB), the compression has been 0.78x, which is damn good for lossless compression!
EDIT: 17GB now (my server is kinda slow) and it's still holding at 0.78x.
Yes and no. It's not a lossless way to encode an image, but the Lepton step itself doesn't add any loss that wasn't already present in the JPEG to begin with, and it generates a smaller file overall.
Someone just had to mention that _stupid_ show. Seriously. By empowering that show, you're just making fun of yourself and all the other programmers on here.
I really admire Dropox for open sourcing this, it shows their commitment.
Saving almost a quarter of space for most images stored is something that truly gives a competitive edge. (I say most because people probably primarily have JPEG images).
Especially considering how many images are probably stored on services like Dropbox.
IMO, it shows how they don't plan to make any money from this, don't consider it a large competitive advantage, and think that the largest advantage of open sourcing it are the 'free' open source volunteers will improve the software.
It's interesting that they don't consider it a large enough competitive advantage.
Either that or they are using this to attract engineers.
> Either that or they are using this to attract engineers.
On a related note (I can't speak for Dropbox specifically) there are many engineers who desire their work to be open source for their own motivations, and when it's not a significant business risk to do so, nice companies will allow it.
So it works on two fronts, as a "hey, we'll let you open source your stuff" and a "hey, we've got people here who care about and contribute to open source."
You also open source if it's in your best interest for something to become the 'standard'.
In google there is a regret that they didn't open source a lot of stuff, because it ends up being reproduced outside of google in some form. The open source companies have an advantage in hiring and overall advancement of their product by using the open source version whatever their internal version was.
You also see it in facebook's open source initiatives, like react, buck, haxe and so on.
Then why do you admire them? Would you also admire them if you were their investor? Dropbox management is obligated by law to act in the best interests of their shareholders, i.e. to make them as much profit as possible.
It's more likely that they have released it because of some profit-seeking interest. They are not charity.
> Dropbox management is obligated by law to act in the best interests of their shareholders, i.e. to make them as much profit as possible.
Can you cite the law which you believe makes that obligation?
The reason you can't is because it's not actually a legal requirement and the reason is obvious: it's hard to say what the best interest is over all but the shortest time frame:
Dropbox's management might argue that they benefit more from open-source than they're giving away, that this kind of favorable attention will help them hire the top engineers who make far larger contributions to their bottom-line, that the pricing models are complex enough that this just doesn't matter very much, etc. Absent evidence that they're acting in bad faith, it's almost impossible to say in advance whether those arguments are right or wrong.
"Dropbox management is obligated by law to act in the best interests of their shareholders, i.e. to make them as much profit as possible." - no they are not.
Even if that was their obligation (which it's not, as others have pointed out), there's no reason to believe that keeping the algorithm proprietary would be more profitable. They're not handing a critical capability to their competitors (competing with Dropbox isn't fundamentally about cheap storage). They don't have the infrastructure to licence the algorithm (who should they licence it to? For how much? Under which terms? Those are not simple questions to answer.)
Releasing it freely allows it to work its way into browsers, allowing Dropbox to serve up the smaller images directly to users, saving money on bandwidth. It allows other users to contribute improvements. It markets Dropbox as a desirable place to work for engineers.
I admire them because they certainly considered the pros and cons of doing so, and decided that they feel secure enough in their market position and prefer to give this back to the open source community
Pretty much every modern company safes tremendous amounts of money from open source (Linux and upwards in the stack), and so the OSS community should rightly be considered a STAKEholder.
The share vs stakeholder obsession in the space of large companies and corporations represents a lot that's wrong with our current markets.
--
Also, the only upside to open sourcing this is getting other involved in development. I just tested on 10k images, and the promise both on compression rate and bit parity after decompression holds true.
Seems to be a pretty stable product, so the that motivation is probably only miniscule.
A similar approach has also been developed for use with zpaq, a compression format which stores the decompression algorithm as bytecode in the archive:
[The configuration] "jpg_test2" by Jan Ondrus compresses JPEG images (which are already compressed) by an additional 15%. It uses a preprocessor that expands Huffman codes to whole bytes, followed by context modeling.
http://mattmahoney.net/dc/zpaqutil.html
Backblaze storage is $0.005/GB/Month = $5k/PB/Month.
The GitHub repo has 7 authors, perhaps costing Dropbox $200k/year each and taking most of a year ~ $1M to develop this system.
So this might pay for itself after 200PB*Months, assuming Dropbox's storage costs are the same as Backblaze's prices, and assuming CPU time is free. (TODO: estimate CPU costs...)
Of course, advancing the state of the art has intrinsic advantages, but again, it's interesting to look at the purely financial point.
I think you may be crunching the numbers the wrong way...
Dropbox is valued at $10b (regardless of what you think of that number, someone was willing to buy at that price). But they only have 1800 employees. If by some miracle 80% of them are engineers that is $7M per engineer. All this for a "pure software" business. No inventory, no manufacturing, they only recently started doing their own operations.
Investors would be telling them "more engineers" => "more software" => "better Dropbox". So now you have to go out and snap up engineers in Silicon Valley, which is very very hard. But you don't offer the perk of a Google/Apple line on a resume nor the compensation of e.g. Microsoft. What do you do?
Answer: you offer people the chance to work on cutting edge breakthrough technology. Not only that but it's all open source! This is a dream job for some people. They will turn down every other gig for this one.
It doesn't matter that PB/mo is chump change because an investor doesn't know that, they only know the engineering headcount. If by some miracle an investor does know this is a waste of time, then you just point to this HN post and observe how many developers are interested in this technology and how it is attracting developer mindshare that can be exploited for additional hires down the road.
I'm not saying it's rational–it's not. But I think there were strong incentives to greenlight a project like this, even if the actual cost savings were zero.
They will once Silicon Valley takes them to task for it. Just like season 1 pretty much wiped "making the changes world a better place" from the lingo of SV companies.
> To encode an AC coefficient, first Lepton writes how long that coefficient is in binary representation, by using unary. [...] Next Lepton writes a 1 if the coefficient is positive or 0 if it is negative. Finally, Lepton writes the the absolute value of the coefficient in standard binary. Lepton saves a bit of space by omitting the leading 1, since any number greater than zero doesn’t start with zero.
The wording almost implies that this is novel, but it is actually Gamma coding [1], which in the signal compression community is often called Exp-Golomb coding [2]. I wonder why this is not acknowledged, considering that they mention the VP8 arithcoder instead.
So, am I to read this as to mean that when I send Dropbox a JPEG, they are behind the scenes further compressing it using Lepton? Then when I request it back, they are re-converting it back to JPEG?
That's the idea behind the algorithm, yes. And since it's lossless, every original bit is preserved. The same idea could be applied on the Desktop client instead of on the server, which would save 22% of the bandwidth as well and make syncing faster.
This is amazing, we've been struggling with JPG storage and fast delivery at my lab (terabytes and petabytes of microscopy images). We'll be running tests and giving this a shot!
jpg is a weird format to be storing microscopy images, no? Usually end up in some sort of bitmap TIFF (or their Zeiss/etc. proprietary format) from what I've seen.
So the weird thing is that we're not only a lab, but also a web tool. We have the files backed up in a standard format in one place, but delivering 1024x1024x128 cubes of images over the internet has been tricky. We don't need people to always view them at full fidelity, just good enough.
We tried JPEG2000, which was better quality per a file size, but the web worker decoder was slower than the JPEG one adding seconds to the total download/decode time.
EDIT: We're currently doing 256x256x256 (equivalent to a 4k image) on eyewire.org. We're speeding things up to handle bigger 3D images.
EDIT2: If you check out Eyewire right now, you might notice some slowdown when you load cubes, that's because we're decoding on the main thread. We'll be changing that up next week.
The size difference between "high quality" jpeg and lossless compressed TIFFs can be quite significant - so I it might not be feasible to use eg. TIFF for archiving. And jpegs might very well be "good enough".
Eg. just tested on a small 10MP Sony ARW raw image - the raw file is 7MB, the camera jpeg is 2.3MB, and a tiff compressed with LZW is 20MB (uncompressed 29MB). The raw tiff run through lzma is 9.3MB. But either way, if ~7MB is the likely lossless size, if the JPEG is good enough, at ~2.3MB it's a pretty big difference, if we're talking petabytes, not megabytes.
(I'll get around to testing lepton on the jpegs shortly)
Some TIFF-derived container is the most common representation. But note even these could use JPEG/J2K if desired.
Most microscopy images are stored uncompressed or with lossless compression. But unfortunately this doesn't scale with newer imaging modalities. Here's two examples:
Digital histopathology. Whole-slide scanners can create huge images e.g. 200000x200000 and larger. These are stored using e.g. JPEG or J2K in a tiled BigTIFF container, with multiple resolution levels. Or JPEG-XR. When each image is multiple gigabytes, lossless compression doesn't scale.
SPIM involves imaging a 3D volume by rotating the sample and imaging it from multiple angles and directions. The raw data can be multiple terabytes per image and is both sparse and full of redundant information. The viewable post-processed 3D image volume is vastly smaller, but also still sparse.
For more standard images such as confocal or brightfield or epifluorescence CCD, lossless storage is certainly the norm. You don't really want to perform precise quantitative measurements with poor quality data.
Readit News
For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG.
If someone can do this over video files that will be PiedPiper comes into real life.
Lepton is cool because it helps make existing technology a whole lot better, but what we actually need is a better image format.
You wouldn't see the same leap for videos because people have been working hard to make those great for a while while JPEG has been left to rot.
I could re-encode all of my JPEG photos with a better codec, but the problem is that then they have gone through a lossy compression scheme twice: once with JPEG, and once with the new codec. This could ruin the image quality on some photos, giving them nasty artefacts.
The newer codecs might even be hindered further by the fact that they are being asked to compress an image that's been JPEG'd. I don't think any of the codecs you mentioned are tuned to work their best with image data that already has been butchered by JPEG. They expect to be given the 'pure' original image. This may well cause the codec to perform worse than expected.
Using Lepton gives none of these risks, since jpeg<->lepton is lossless. I could throw away all the JPEGs afterwards, safe in the knowledge that if I want to go back, I can.
Also, it's not really true that JPEG has been left to rot. There are lots of programs available that greatly improve upon the basic JPEG compression, while still outputting a JPEG file that any compliant decoder can read. And there's an even simpler way to shrink your photo file sizes - just drop the JPEG quality level slightly. Most cameras and programs default to a very high quality value. Dropping the default by even a tiny amount can produce remarkably smaller files, and you'll probably never notice the miniscule image quality loss.
I suspect that JPEG is so popular because it is 'good enough'. Most people and programs don't care so much about squeezing out smaller files, so they'll keep using JPEG regardless.
http://flif.info/
https://people.xiph.org/~jm/daala/revisiting/subset1_psnrhvs...
so when you use a better encoder like mozjpeg, JPEG is actually competitive with the newest formats, given enough quality
it gets slightly grainier results with blocking artifacts, but BPG and WebM look like they had a painting algorithm blur out all of the details
http://bellard.org/bpg/
WebP is very promising (also based on VP8) for lossless and lossy compression. Have you considered using it to compress PNGs in the same way Lepton is compressing JPGs? Odds are it wouldn't be bit-perfect though (despite being pixel-perfect).
Also interested in hearing about the tradeoffs between server-side decoding and client-side. Not to keep focusing on it, but WebP has native support in Chrome and javascript decoders for everything else.
I think it is a very exciting time for image formats with several promising new ones on the way (WebP, FLIF, maybe BPG but possible legal issues).
EDIT: 17GB now (my server is kinda slow) and it's still holding at 0.78x.
Saving almost a quarter of space for most images stored is something that truly gives a competitive edge. (I say most because people probably primarily have JPEG images).
Especially considering how many images are probably stored on services like Dropbox.
And they just gave it away to their competitors.
It's interesting that they don't consider it a large enough competitive advantage.
Either that or they are using this to attract engineers.
On a related note (I can't speak for Dropbox specifically) there are many engineers who desire their work to be open source for their own motivations, and when it's not a significant business risk to do so, nice companies will allow it.
So it works on two fronts, as a "hey, we'll let you open source your stuff" and a "hey, we've got people here who care about and contribute to open source."
In google there is a regret that they didn't open source a lot of stuff, because it ends up being reproduced outside of google in some form. The open source companies have an advantage in hiring and overall advancement of their product by using the open source version whatever their internal version was.
You also see it in facebook's open source initiatives, like react, buck, haxe and so on.
It's more likely that they have released it because of some profit-seeking interest. They are not charity.
Can you cite the law which you believe makes that obligation?
The reason you can't is because it's not actually a legal requirement and the reason is obvious: it's hard to say what the best interest is over all but the shortest time frame:
https://en.wikipedia.org/wiki/Business_judgment_rule
Dropbox's management might argue that they benefit more from open-source than they're giving away, that this kind of favorable attention will help them hire the top engineers who make far larger contributions to their bottom-line, that the pricing models are complex enough that this just doesn't matter very much, etc. Absent evidence that they're acting in bad faith, it's almost impossible to say in advance whether those arguments are right or wrong.
http://www.nytimes.com/roomfordebate/2015/04/16/what-are-cor...
Releasing it freely allows it to work its way into browsers, allowing Dropbox to serve up the smaller images directly to users, saving money on bandwidth. It allows other users to contribute improvements. It markets Dropbox as a desirable place to work for engineers.
Not obviously less profitable that the opposite.
Pretty much every modern company safes tremendous amounts of money from open source (Linux and upwards in the stack), and so the OSS community should rightly be considered a STAKEholder.
The share vs stakeholder obsession in the space of large companies and corporations represents a lot that's wrong with our current markets.
--
Also, the only upside to open sourcing this is getting other involved in development. I just tested on 10k images, and the promise both on compression rate and bit parity after decompression holds true.
Seems to be a pretty stable product, so the that motivation is probably only miniscule.
[The configuration] "jpg_test2" by Jan Ondrus compresses JPEG images (which are already compressed) by an additional 15%. It uses a preprocessor that expands Huffman codes to whole bytes, followed by context modeling. http://mattmahoney.net/dc/zpaqutil.html
It's saved "multiple petabytes" of space.
Backblaze storage is $0.005/GB/Month = $5k/PB/Month.
The GitHub repo has 7 authors, perhaps costing Dropbox $200k/year each and taking most of a year ~ $1M to develop this system.
So this might pay for itself after 200PB*Months, assuming Dropbox's storage costs are the same as Backblaze's prices, and assuming CPU time is free. (TODO: estimate CPU costs...)
Of course, advancing the state of the art has intrinsic advantages, but again, it's interesting to look at the purely financial point.
https://www.backblaze.com/b2/cloud-storage-pricing.html
Dropbox is valued at $10b (regardless of what you think of that number, someone was willing to buy at that price). But they only have 1800 employees. If by some miracle 80% of them are engineers that is $7M per engineer. All this for a "pure software" business. No inventory, no manufacturing, they only recently started doing their own operations.
Investors would be telling them "more engineers" => "more software" => "better Dropbox". So now you have to go out and snap up engineers in Silicon Valley, which is very very hard. But you don't offer the perk of a Google/Apple line on a resume nor the compensation of e.g. Microsoft. What do you do?
Answer: you offer people the chance to work on cutting edge breakthrough technology. Not only that but it's all open source! This is a dream job for some people. They will turn down every other gig for this one.
It doesn't matter that PB/mo is chump change because an investor doesn't know that, they only know the engineering headcount. If by some miracle an investor does know this is a waste of time, then you just point to this HN post and observe how many developers are interested in this technology and how it is attracting developer mindshare that can be exploited for additional hires down the road.
I'm not saying it's rational–it's not. But I think there were strong incentives to greenlight a project like this, even if the actual cost savings were zero.
You're correct ofc, download costs = 10 months of storage.
> For those familiar with Season 1 of Silicon Valley, this is essentially a “middle-out” algorithm.
I really wish that every single compression-related blog post would stop referencing Silicon Valley.
Did it really?
The wording almost implies that this is novel, but it is actually Gamma coding [1], which in the signal compression community is often called Exp-Golomb coding [2]. I wonder why this is not acknowledged, considering that they mention the VP8 arithcoder instead.
[1] https://en.wikipedia.org/wiki/Elias_gamma_coding
[2] https://en.wikipedia.org/wiki/Exponential-Golomb_coding
We tried JPEG2000, which was better quality per a file size, but the web worker decoder was slower than the JPEG one adding seconds to the total download/decode time.
EDIT: We're currently doing 256x256x256 (equivalent to a 4k image) on eyewire.org. We're speeding things up to handle bigger 3D images.
EDIT2: If you check out Eyewire right now, you might notice some slowdown when you load cubes, that's because we're decoding on the main thread. We'll be changing that up next week.
Eg. just tested on a small 10MP Sony ARW raw image - the raw file is 7MB, the camera jpeg is 2.3MB, and a tiff compressed with LZW is 20MB (uncompressed 29MB). The raw tiff run through lzma is 9.3MB. But either way, if ~7MB is the likely lossless size, if the JPEG is good enough, at ~2.3MB it's a pretty big difference, if we're talking petabytes, not megabytes.
(I'll get around to testing lepton on the jpegs shortly)
Most microscopy images are stored uncompressed or with lossless compression. But unfortunately this doesn't scale with newer imaging modalities. Here's two examples:
Digital histopathology. Whole-slide scanners can create huge images e.g. 200000x200000 and larger. These are stored using e.g. JPEG or J2K in a tiled BigTIFF container, with multiple resolution levels. Or JPEG-XR. When each image is multiple gigabytes, lossless compression doesn't scale.
SPIM involves imaging a 3D volume by rotating the sample and imaging it from multiple angles and directions. The raw data can be multiple terabytes per image and is both sparse and full of redundant information. The viewable post-processed 3D image volume is vastly smaller, but also still sparse.
For more standard images such as confocal or brightfield or epifluorescence CCD, lossless storage is certainly the norm. You don't really want to perform precise quantitative measurements with poor quality data.