First, why would you want lightfields? Computational photography -> variable boken (e.g. fix the bokeh for different depths, make it smoother, etc), depth info (extremely valuable for VFX), having some variability in your camera position - so being able to add camera shake more realistically. There's a bunch of other cool things you could do.
I've only taken a look at existing lightfield solutions briefly, but this looks cool. It's novel because it seems to only use the bare possible minimum number of ray samples to be called a lightfield, and makes up for it with heavy compute (even heavier than a typical lightfield).
This approach has interesting tradeoffs. A typical lightfield camera typically uses, say, 25 ray samples. This number of ray samples isn't enough to avoid compute, but cuts the resolution by a lot (25x in our case - instead of having 50 megapixels, you're at 2 megapixels). A typical implementation uses a micro-lens array in front of the camera sensor, and that means that there's wasted overhead due to tiling circles leaving a bunch of area empty.
Their proposed approach "only" leaves you with 9x less resolution (which is cool), which they try to reconstruct / upscale back to 2x less resolution (which is also cool and should be workable).
3-4 years ago we already had Lytro / the Light camera. Wasn't great apparently[1], but a much more practical idea than this massive lens.
Judging from the samples, the iPhone with ML-based depth sensing and Lidar also seems to do a much better job. As will most of the latest ML models available.
Light fields will be extremely conducive for building ML models around. It'll be possible to generate incredibly rich and realistic 3D scenes with the additional information light fields convey. Possibly some downsampled number of rays to make compute easier.
Multiple view scene constructions will make for incredible filmmaking that can be tweaked in post. No shot, reverse-shot. No cheating the camera. (That's assuming we don't switch from optics to mocap + rendering outright.)
Very cool. I do always wonder with projects like this that rely heavily on proprietary software, what happens if a few years down the road the company folds? Now do I have a fancy brick to mount on my camera? I was hoping to see a more explicit statement about whether the raw data was still reasonably usable or at least readable outside the proprietary software. That may be the case and I just missed it, but as a potential customer of this the possibility of the lens being useless without their software can be a bit spooky. That having been said the tech looks great and I hope they do really well!
I had one of the first Lytro light field cameras [0], and thats exactly what happened. The camera was essentially bricked and all my photos were deleted from the only cloud platform that could render them.
Luckily, or maybe unluckily, for me the camera ended up taking terrible pictures and no images of any consequence were taken with it.
Don’t worry too much. It looks like this is Bring Your Own Camera so the files are whatever format you’re used to working with.
The algorithm for processing plenoptic images is also well-known and this company is not the first to do it. Someone will come up with a free tool for dealing with these images.
Good point! I’ve been excited to see where light field tech leads on the practical side of things. If indeed they’re able to overcome the resolution limitations using AI assist then this is a great step forward!
Sample images at https://hidrive.ionos.com/share/ayvcprk57q show the resulting 5k*5k depth map. It has lower depth resolution than what I would expect from a $4k lens. It would be interesting to see a comparison of this depth map compared to an RGBZ image generated from Google or Apple's Portrait Mode AI filters. The primary benefit of this is probably the ability for RGBZ video, but that can also be generated from 2 or more DSLR cameras + software.
It is kind of disappointing that it doesn't seem to map any of the folds in the garment, the individual fingers, or other details. It also seems to get the depth around the knob by the right elbow quite wrong. All-in-all, no apparent order-of-magnitude (if any?) improvement over single image "AI" algorithms that have been shown before.
Im not sure how good our instinctive expectations are. The folds in the shirt, for example, are very prominent in the normal photo because of the shadows. But the difference in depth really isn’t that large.
Say you have the 255 shades of gray in RBG, and you want to spread them evenly over the distances of 1-5m. That would give you a 1-step increase in brightness for every 1.6cm or so, which happens to be pretty close to what I believe these folds‘ magnitude might be. I’m not entirely sure how prominent the difference would be to the naked eye. IIRC, the MPAA considers even 50 to be plenty.
I‘m leaving out lots of details (pun not intended, but appreciated): you’d spread your resolution logarithmically, for example, not linear. And, of course, you could work with more than the resolution of 255. But it’s a different domain and I believe some intuitions are off if you compare it with the x and y dimensions.
This solution uses the same sensor for the depth and RGB, which is a win. When you use separate depth and RGB sensor like most smartphones the images do not align exactly which causes problems at close range. You also get weird artifacts when the depth sensor saturates but the RGB does not.
I think this design will require a lot more cpu for post processing though.
That's super neat but that is quite the price tag for a KickStarter lens. They're backed by a solid lens manufacturer but this is going to need specialized software for post image processing which is a risk of its own. I would need a really compelling reason to spend $2-4k on this lens instead of others I can guarantee I'll use.
Recently my son was enthusiastic about anaglyph images viewed with red-cyan glasses. These are much better than what they had when I was a kid because the cyan filter passes both the green and blue channels. You can use whatever color you like as long as it isn't red or cyan.
I got enthusiastic too.
We found out that you can't (in general) make a good anaglyph image with a 2-d image and a height map because the left eye and the right eye can see around objects so the 2-d image usually is missing pixels that the left and right eye would see.
You could reproject an anime image if you had the cels; you could reproject the image for the left and right eyes by simulating the multiplane camera.
I am just starting to take 3-d photographs by moving my camera and that way I get all the pixels for the left and right eye. That lens might be better for taking portraits at very close range but I don't know if it would beat my current lenses. Certainly I could get a really good telephoto lens for that price today whereas I don't even know if I'll be taking 3-d photos a year from now when that lens would arrive.
as crazy as it sounds, look for a cheap used nintendo 3ds. they have a stereo camera for the lenticular display and some photoshop filtering could turn them to anaglyph
I'm a big fan of Loreo stereo lenses. They're well made and inexpensive.
You can use stereo pairs to calculate depth, and in this case, you only lose half your horizontal resolution (vs. losing significant horizontal and vertical resolution on the K lens)
Apple has both an SDK for turning a series of photos into 3D models and scenes, as well as LIDAR sensors in their high-end phones and iPads. Notably, though, last I checked these are separate initiatives and the SDK doesn’t use the LIDAR data, but depth information the camera gathers by using two of its lenses.
With lightfield cameras, you do actually get several slightly different perspectives. You can imagine it as if there were several pinhole cameras around the (large) front lens element.
It looks like this one splits the field 3 ways in 2 different directions for a total of 9 images. I guess I could pull out the middle left and middle right and make an anaglyph from that.
To me, this looks like a rip-off of https://raytrix.de/ who have been granted a patent for pretty much the same thing in 2013. And they have been selling pretty much the same thing, except that for Raytrix, the lens is pre-installed by them whereas with K-Lens, you install it onto your own camera.
Also, they effectively split the resolution by 3.x on each axis, because they are multiplexing a 3x3 image grid onto the same sensor. That means a $50k RED MONSTRO isn't "good enough" anymore to get 4K output.
There have been several commercial manufacturers of lightfield cameras (Lytro being another one). Which part is patented?
The hardware tech is different too, Raytrix seems to be using a microlens setup (like most other lightfield cameras). A microlens setup also reduces resolution fwiw.
Readit News
First, why would you want lightfields? Computational photography -> variable boken (e.g. fix the bokeh for different depths, make it smoother, etc), depth info (extremely valuable for VFX), having some variability in your camera position - so being able to add camera shake more realistically. There's a bunch of other cool things you could do.
I've only taken a look at existing lightfield solutions briefly, but this looks cool. It's novel because it seems to only use the bare possible minimum number of ray samples to be called a lightfield, and makes up for it with heavy compute (even heavier than a typical lightfield).
This approach has interesting tradeoffs. A typical lightfield camera typically uses, say, 25 ray samples. This number of ray samples isn't enough to avoid compute, but cuts the resolution by a lot (25x in our case - instead of having 50 megapixels, you're at 2 megapixels). A typical implementation uses a micro-lens array in front of the camera sensor, and that means that there's wasted overhead due to tiling circles leaving a bunch of area empty.
Their proposed approach "only" leaves you with 9x less resolution (which is cool), which they try to reconstruct / upscale back to 2x less resolution (which is also cool and should be workable).
Judging from the samples, the iPhone with ML-based depth sensing and Lidar also seems to do a much better job. As will most of the latest ML models available.
[1] https://www.theverge.com/circuitbreaker/2018/4/10/17218758/l...
Light fields will be extremely conducive for building ML models around. It'll be possible to generate incredibly rich and realistic 3D scenes with the additional information light fields convey. Possibly some downsampled number of rays to make compute easier.
Multiple view scene constructions will make for incredible filmmaking that can be tweaked in post. No shot, reverse-shot. No cheating the camera. (That's assuming we don't switch from optics to mocap + rendering outright.)
Deleted Comment
Luckily, or maybe unluckily, for me the camera ended up taking terrible pictures and no images of any consequence were taken with it.
[0] https://en.wikipedia.org/wiki/Lytro#Original_Lytro_Light_Fie...
The algorithm for processing plenoptic images is also well-known and this company is not the first to do it. Someone will come up with a free tool for dealing with these images.
Say you have the 255 shades of gray in RBG, and you want to spread them evenly over the distances of 1-5m. That would give you a 1-step increase in brightness for every 1.6cm or so, which happens to be pretty close to what I believe these folds‘ magnitude might be. I’m not entirely sure how prominent the difference would be to the naked eye. IIRC, the MPAA considers even 50 to be plenty.
I‘m leaving out lots of details (pun not intended, but appreciated): you’d spread your resolution logarithmically, for example, not linear. And, of course, you could work with more than the resolution of 255. But it’s a different domain and I believe some intuitions are off if you compare it with the x and y dimensions.
I think this design will require a lot more cpu for post processing though.
Recently my son was enthusiastic about anaglyph images viewed with red-cyan glasses. These are much better than what they had when I was a kid because the cyan filter passes both the green and blue channels. You can use whatever color you like as long as it isn't red or cyan.
I got enthusiastic too.
We found out that you can't (in general) make a good anaglyph image with a 2-d image and a height map because the left eye and the right eye can see around objects so the 2-d image usually is missing pixels that the left and right eye would see.
You could reproject an anime image if you had the cels; you could reproject the image for the left and right eyes by simulating the multiplane camera.
I am just starting to take 3-d photographs by moving my camera and that way I get all the pixels for the left and right eye. That lens might be better for taking portraits at very close range but I don't know if it would beat my current lenses. Certainly I could get a really good telephoto lens for that price today whereas I don't even know if I'll be taking 3-d photos a year from now when that lens would arrive.
You can use stereo pairs to calculate depth, and in this case, you only lose half your horizontal resolution (vs. losing significant horizontal and vertical resolution on the K lens)
http://www.loreo.com/pages/products/loreo_3dcap.html
Also, they effectively split the resolution by 3.x on each axis, because they are multiplexing a 3x3 image grid onto the same sensor. That means a $50k RED MONSTRO isn't "good enough" anymore to get 4K output.
The hardware tech is different too, Raytrix seems to be using a microlens setup (like most other lightfield cameras). A microlens setup also reduces resolution fwiw.
If I remember correctly, Lytro went bankrupt in the period between when Raytrix filed their patent and were granted it, so they "got lucky".