As a designer, I've built variants of this several times throughout my career.
The author's approach is really good, and he hits on pretty much all the problems that arise from more naive approaches. In particular, using a perceptual colorspace, and how the most representative colour may not be the one that appears the most.
However, image processing makes my neck tingle because there are a lot of footguns. PNG bombs, anyone? I feel like any library needs to either be defensively programmed or explicit in its documentation.
The README says "Finding main colors of a reasonably sized image takes about 100ms" -- that's way too slow. I bet the operation takes a few hundred MB of RAM too.
For anyone that uses this, scale down your images substantially first, or only sample every N pixels. Avoid loading the whole thing into memory if possible, unless this handled serially by a job queue of some sort.
You can operate this kind of algorithm much faster and with less RAM usage on a small thumbnail than you would on a large input image. This makes performance concerns less of an issue. And prevents a whole class of OOM DoS vulnerabilities!
Author here: the library just accepts RGB8 bitmaps, probably coming either from Rust's image crate [1] or Python's Pillow [2], which are both mature and widely used. Dealing with codecs is way out of scope.
As for loading into memory at once: I suppose I could integrate with something like libvips and stream strips out of the decoded image without holding the entire bitmap, but that'd require substantially more glue and complexity. The current approach works fine for extracting dominant colours once to save in a database.
You're right that pre-resizing the images makes everything faster, but keep in mind that k-means still requires a pretty nontrivial amount of computation.
That'll stream the source image and make a RGBRGBRGB memory buffer of single precision floats. You could perhaps use kernel="linear" and avoid any ringing from lanczos3.
I think I would downsample in a linear light space, like scRGB. Averaging there means averaging photons, which will surely be better than OKLab. Maybe switch to OKLab for clustering. Though of course I've not tested it.
If you ever did want to wrap this in code processing untrusted images there's a library called "glycin" designed for that purpose (it's used by Loupe, the default Gnome image viewer).
Somehow I missed that, oops. I see that the library samples a maximum of 250K pixels from the input buffer (I jumped over to the project readme)
That being said, this is sampling the fixed-size input buffer for the purposes of determining the right colour. You still have to load the bitmap into memory, with all the associated footguns that arise there. The library just isn't making it worse :) I suppose you could memmap it.
Makes me wonder if the sub-sampling is actually a bit of a red herring, as ideally you'd want to be operating on a small input buffer anyway. Or some sort of interface on top of the raw pixel data, so you can load what's needed on-demand.
That's 500x500, I'm sure you can get good results at 32x32 or 64x64 but then part of your color choice is also getting done by the downsampling algorithm. I wonder if you could get away with just using a downsampling algorithm into a 1x1 and just use that as the main color.
Back in the late 1980s people thought about color quantization a lot because a lot of computers of the time had 16 or 256 colors you could choose out of a larger palette and if you chose well you could do pretty well with photographic images.
OKPalette by David Aerne is my favorite tool for this, it chooses points sensibly but then also lets you drag around or change the number of colors you want: https://okpalette.color.pizza/
I thought somewhere online there was a relatively-old (Material v1?) breakdown of how Android selects "main" colors (because it does a fairly good job imo), for the Palette¹ class... but I'm having no luck finding it at all. I can (and will) just read its source code, but it doesn't carry along any qualitative justification or comparisons or etc and it'll take a fair bit of time to re-research that knowledge.
Does anyone know how this strategy differs? I've been wanting to build a product-color-selection-thing for stuff like fabric, where finding something that has similar tones is important, but I'm struggling to find much with concrete details about the strategy like this article has.
I've been doing something similar! I've got a Home Assistant dashboard on my desk and wanted the media controls to match the current album art. I need three colors: background, foreground, and something vibrant to set my desk lamp to [1].
The SpotifyPlus HA integration [2] was near at hand and does a reasonably good job clustering with a version of ColorThief [3] under the hood. It has the same two problems you started with though: muddying when there's lots of gradation, even within a cluster; and no semantic understanding when the cover has something resembling a frame. A bit swapped from okmain's goal, but I can invert with the best of them and will give it a shot next time I fiddle. Thanks for posting!
I’m surprised the baseline to compare against is shrinking the image to one pixel, that seems extremely hacky and very dependent on what your image editor happens to do (and also seems quite wasteful… the rescaling operation must be doing a lot of extra pointless work keeping track of the position of pixels that are all ultimately going to be collapsed to one point).
So, making a library that provides an alternative is a great service to the world, haha.
An additional feature that might be nice: the most prominent colors seem like they might be a bad pick in some cases, if you want the important part of the image to stand out. Maybe a color that is the close (in the color space) to the edges of your image, but far away (in the color space) from the center of your image could be interesting?
Tbh shrinking the image is probably the cheapest operation you can do that still lets every pixel influence the result. It’s just the average of all pixels, after suitable color conversion.
The author of the article seems to assume there is no color conversion (e.g., the resizing of the image is done with sRGB-encoded values rather than converting them to linear first). Which is a stupid way to do it but I'd believe most handwritten routines are just that.
It reminds me a bit of this post from the Facebook engineering blog (2015) [1] where they discuss embedding a very tiny preview of images into the html itself so they show immediately while loading the page, especially with very slow connections.
I've wanted something like this for level of detail processing.
This is a render from Second Life, in which all the texture images were shrunk down to one pixel, the lowest possible level of detail, producing a monocolor image.
For distant objects, or for objects where the texture is still coming in from the net, there needs to be some default color. The existing system used grey for everything. I tried using an average of all the pixels, and, as the original poster points out, the result looks murky.[1] This new approach has real promise for big-world rendering.
Readit News
The author's approach is really good, and he hits on pretty much all the problems that arise from more naive approaches. In particular, using a perceptual colorspace, and how the most representative colour may not be the one that appears the most.
However, image processing makes my neck tingle because there are a lot of footguns. PNG bombs, anyone? I feel like any library needs to either be defensively programmed or explicit in its documentation.
The README says "Finding main colors of a reasonably sized image takes about 100ms" -- that's way too slow. I bet the operation takes a few hundred MB of RAM too.
For anyone that uses this, scale down your images substantially first, or only sample every N pixels. Avoid loading the whole thing into memory if possible, unless this handled serially by a job queue of some sort.
You can operate this kind of algorithm much faster and with less RAM usage on a small thumbnail than you would on a large input image. This makes performance concerns less of an issue. And prevents a whole class of OOM DoS vulnerabilities!
As a defensive step, I'd add something like this https://github.com/iamcalledrob/saferimg/blob/master/asset/p... to your test suite and see what happens.
As for loading into memory at once: I suppose I could integrate with something like libvips and stream strips out of the decoded image without holding the entire bitmap, but that'd require substantially more glue and complexity. The current approach works fine for extracting dominant colours once to save in a database.
You're right that pre-resizing the images makes everything faster, but keep in mind that k-means still requires a pretty nontrivial amount of computation.
[1]: https://crates.io/crates/image
[2]: https://pypi.org/project/pillow/
image = pyvips.Image.new_from_file(filename, access="sequential")
scale = min(200 / image.width, 200 / image.height)
thumbnail = image.colourspace("oklab").resize(scale).extract_bands(0, n=3)
rgbf = thumbnail.write_to_memory()
That'll stream the source image and make a RGBRGBRGB memory buffer of single precision floats. You could perhaps use kernel="linear" and avoid any ringing from lanczos3.
I think I would downsample in a linear light space, like scRGB. Averaging there means averaging photons, which will surely be better than OKLab. Maybe switch to OKLab for clustering. Though of course I've not tested it.
https://gnome.pages.gitlab.gnome.org/glycin/
> Okmain downsamples the image by a power of two until the total number of pixels is below 250,000.
That being said, this is sampling the fixed-size input buffer for the purposes of determining the right colour. You still have to load the bitmap into memory, with all the associated footguns that arise there. The library just isn't making it worse :) I suppose you could memmap it.
Makes me wonder if the sub-sampling is actually a bit of a red herring, as ideally you'd want to be operating on a small input buffer anyway. Or some sort of interface on top of the raw pixel data, so you can load what's needed on-demand.
EDIT: then (when url refreshed) triggered a redir loop culminating in a different error ("problem occurred repeatedly")...
ah, ofc, your intent was to demonstrate a problematic asset.
Got any to share? A self-contained command-line tool to get a good palette from an image is something I’d have a use for.
https://www.fmwconcepts.com/imagemagick/dominantcolor/index....
Does anyone know how this strategy differs? I've been wanting to build a product-color-selection-thing for stuff like fabric, where finding something that has similar tones is important, but I'm struggling to find much with concrete details about the strategy like this article has.
1: https://developer.android.com/reference/androidx/palette/gra... with most of the color selection logic here: https://cs.android.com/androidx/platform/frameworks/support/...
The SpotifyPlus HA integration [2] was near at hand and does a reasonably good job clustering with a version of ColorThief [3] under the hood. It has the same two problems you started with though: muddying when there's lots of gradation, even within a cluster; and no semantic understanding when the cover has something resembling a frame. A bit swapped from okmain's goal, but I can invert with the best of them and will give it a shot next time I fiddle. Thanks for posting!
[1] https://gist.github.com/kristjan/b305b83b0eb4455ee8455be108a... [2] https://github.com/thlucas1/homeassistantcomponent_spotifypl... [3] https://github.com/thlucas1/SpotifyWebApiPython/blob/master/...
So, making a library that provides an alternative is a great service to the world, haha.
An additional feature that might be nice: the most prominent colors seem like they might be a bad pick in some cases, if you want the important part of the image to stand out. Maybe a color that is the close (in the color space) to the edges of your image, but far away (in the color space) from the center of your image could be interesting?
[1] https://engineering.fb.com/2015/08/06/android/the-technology...
This is a render from Second Life, in which all the texture images were shrunk down to one pixel, the lowest possible level of detail, producing a monocolor image. For distant objects, or for objects where the texture is still coming in from the net, there needs to be some default color. The existing system used grey for everything. I tried using an average of all the pixels, and, as the original poster points out, the result looks murky.[1] This new approach has real promise for big-world rendering.
[1] https://media.invisioncic.com/Mseclife/monthly_2023_05/monoc...