Nice! I peeked at the code and thought I’d share a few tips for improving the low frame rate:
Base64 encoding the JPEG bytes will increase payload size up to ~30% and burns CPU cycles on both client and server. This is unnecessary, as Websocket protocol can send binary payloads (doesn’t need to be text).
Consider removing lossy jpg compression as well, ie just send the raw RGB bytes over the network. Then on the server side you can simply call Image.frombuffer(…).
StreamDiffusion can achieve high frame rates because of extensive batching in the pipeline. You’re not benefiting from that here as the client is only sending one frame at a time and then waiting for a response. See this example for an idea of how to queue input frames and consume them in batches https://github.com/cumulo-autumn/StreamDiffusion/blob/main/e... .
Alternatively you could take a look at the SDXL Turbo and Lightning models. They are very fast at img2img but have limited resolution of 512² or 1024² pixels respectively. Which appears a bit lower than what you’re aiming for here, but they can be run locally in real time on a high end consumer grade GPU. For reference I have some code demonstrating this here https://github.com/GradientSurfer/Draw2Img/tree/main
Ok, but I wonder if it really needs to be real time like this? Wouldn't it make more sense to have some kind of button: somebody makes a pose, takes a picture, the picture is run through some kind of transformation and comes back as a painting that stays there until someone takes another picture? Wouldn't the illusion of art be better that way? (It would not be a "mirror" anymore though.)
yeah yeah yeah, do all these things, and afterwards, look at 2d interpolation methods that don't require AI for your inbetweens. There's some real fast kernel math that can lerp from one blob to another at 8 billion fps.
I think you’re getting downvoted because “yeah yeah yeah” is normally a sign that someone is sarcastically dismissing an idea, but the rest of your comment suggests you’re not at all - linerp is a great idea!
> Making art is hard. But art is mostly about surfacing the inner world, and only in part about skill. It’s unfortunate that art selects so strongly for skill.
Not to sound like a luddite, but I do question the idea that the skill gap is merely an inconvenience. I suspect learning how to paint or make music changes something in yourself which teaches you some deeper life lessons.
I've heard the phrase (paraphrased): No great work of art was made by a genius, genius comes to you unexpectedly like a gust of wind. It seems that cultivating these opportunities is the most an artist can do, and removing the skill gap seems to be removing the cultivation, the thing that changes you, the essence.
There seems to be a few of these inherent deep workings that we as a people keep coming back to, without knowing what they are or how to discuss them (personally at least!). Not to rain on your parade OP, the project looks fun and super useful to a lot people! Just something I ponder on at times.
I think he's on the right lines with "surfacing the inner world", but that's why I see this as more wallpaper than art. He's not doing a deep dive into his own psychology and hauling up pearls; none of these images have that property.
Does the totality of the project have that property? That would be less clear, but IMO, no. I see it as technically driven, not psychologically driven, although I can see how you could write an artist's statement that claims it was (it's about mirrors, after all, which are hugely symbolic).
To be clear I still like it, and if I'd done it I'd be proud of it. But it's more artifice than art.
(If I was him, I'd slow the frame rate down, not speed it up - work with the technical limitation, not against it. Have the system only display "good" images, and not update the display until another "good" image is generated. The code that decides if an image is "good" or not would be the most interesting part of the system, and could fairly be said to embody the artist's intent, and so cross (my own personal) threshold into capital-A Art.
I'd also experiment with buffering the image stream à la _Light of Other Days_ by Bob Shaw.
Oh, and as Halloween is nearly with us, the temptation to occasionally inpaint a figure standing behind the viewer would be massive.
Idle thought: to get some stability in the image, would it be possible to have an LLM generate random video filter code, instead of random images? "Write me a video filter that makes the input video look cubist". "...like an oil painting" "...with a Flash aesthetic". etc etc. Every time a filter gets generated that doesn't actively crash, swap to it. No idea if that's feasible or not.)
Talk like "surfacing the inner whatever" can impress the masses, for a very short while, but good art requires more. All art has a language, standards that are learned though an artist's development as they learn the needed skills. The mona lisa isn't just a good picture of a person. It is full of details and meaning only understandable to people who have studied paintings. AI can generate a good or interesting picture but it cannot speak the language of painting. That requires actual graft to learn and appreciate. Injecting the paintbrush skills into someone's brain, or into an AI tool, isn't going to make them an artist.
> I suspect learning how to paint or make music changes something in yourself which teaches you some deeper life lessons.
I enthusiastically support this notion. A simpler, than painting, example would be writing. Sure everyone has a story to tell, and everyone can write, but to make it worth other people's time would take you days if not years of perfecting the craft, as you inevitably learn things about yourself and crystallize your perspectives on this world.
Literally 95% of the replies in that thread are irrelevant bullshit from "experts" that have no idea how redundant server PSUs work. I replied to some under the same username. Meanwhile another guy successfully wired two 100W USB-C ports in parallel to power an entire PC. He had no idea that the resistance of his crappy wires kept the two smps control loops stable and divided the current evenly between the two ports ensuring that neither one trips OCP: https://www.reddit.com/r/UsbCHardware/comments/1g8pser/let_m...
? Have you ever read musician interviews: Its like a competition on how much clichee, naivete and reality denying drivel one can compress into 5 minutes.
Idealism is not a victimless crime, millions suffer every day because some artist threw a buggy,idealized world model over the fence and the idiocy stuck hypnotizing millions into permanently damaging themselves.
There something compelling inside musicians that comes out in their music. Everyone feels it but few can define it. There's a reason they did not choose conversation as their medium.
Also, the screen he uses (HMTECH Raspberry Pi Screen 10.1) is pretty hard to find. Do people have other good recommendations for screens with similar quality and specs?
Any idea why this uses infrared light and an infrared camera?
Every time I've looked into building something like this, when I price it out, just using an old Android tablet in kiosk mode and making it a web app ends up being much cheaper.
The screen is available on Amazon? But it's a touch screen, which seems overkill; any screen would probably work? One can always build a custom frame around it.
Touch screen I think is cool because then you could interact with the art.
I did a little research and these Waveshare displays look really beautiful. For a particular size display, they have models in different resolutions (from lower to higher res). But they're only for rpis and windows, not like mac or ipad. The QLED ones are the best.
> But art is mostly about surfacing the inner world, and only in part about skill.
I always thought that art is all about emotions, both preserving and creating them. That’s why I find a banana on the wall art, popular music is still art.
(Might be school influence but I also tend to ask myself „why”)
This invention is definitely work of art, but its output - in my eyes - not. It’s like a cloud. It can make different shapes, and some are funny, some might remind me of my close ones, but it’s still averaged randomness.
However the idea to reflect reality to this digital randomness I find without doubt - art. And, well: while not visual or audible, there is software, hardware, code and design which are also art (and difficult to make so counterpoint to thesis in first paragraph ;)).
> I always thought that art is all about emotions, both preserving and creating them.
But this leaves the role of intention ambiguous. If I double-park a BMW across two handicap spaces because I'm angry and entitled, is that art? It certainly evokes emotion, and it's also produced by emotion.
I would dissect it though since it’s not pure anger - frustration maybe, a complex one and caused by additional input. E.g. anxiety (you took what could be mine) or internal discord (after observing act of injustice).
I’d argue that it’s an act of random (again, like a cloud) and wouldn’t treat it as art.
But, if you’d park it sideways in front of the mall entrance, blocking it considerably, I’d consider it art (in my imagination I can see the headlines about artist making a statement against the overconsumption).
Yet art is art, so everyone has their own definition. I’d prefer “positive” emotions, so awe, nostalgia, etc., but I don’t see reason why anger should be excluded.
<<proceeds to print “THIS IS AN ART INSTALLATION” message to put behind the windshield>>
Art is something that grabs you and is also unique in the way it presents the subject. Also, there has to be skill involved, otherwise the whole thing goes up in smoke if it feels cheap or superficial.
A banana on the wall is not art, because it involves an everyday object in an everyday setting.
You might feel it's art, but if that's the case, then you haven't exposed yourself to true art enough to have a pool of experience or only seen so far superficial examples.
I think this would be cooler if the camera was somewhere different than the frame. Looking at an artistic mirror seems a little boring. Maybe make a 2nd one, and put it in someone else's house, and then feed the camera from one into the other. So you can look at the 'reflection' of someone else and have these little moments where you're both looking at the picture at the same time. Heck, make many so you never know who you're looking at. It'll be the Omegle of picture frames.
that's a good idea but something completely different than OP wanted to accomplish.
i remember that this has been done already as an art installation in public places so people could see - and interact - with others from around the world.
> an art installation in public places so people could see - and interact - with others from around the world
Sure, and it wasn't long before this happened:
> It took less than a week for people eager to share their ‘assets’ with the world to shut down the visual portals set up between the cities of Dublin and New York on 8 May. Although the project was aimed at bringing people together and connecting cultures, a few visitors to the locations have decided to take the invitation of getting to know others to a whole other level.
(One wonders why flashing is such a big problem that the whole installation needs to be shut down; but it seemed quite obvious from the start that people would try to do this...)
Really neat idea, I'd also love to have on my bookshelf.
> The main issue with the current setup is the low frame rate.
I'd call it a feature rather a limitation, it's not bad that I have second to process the image for a bit.
I would even increase the update frequency to 5 - 15 minutes and let it capture and generate a new image whenever it detects something changing / moving.
That’s the technique I used on a piece that does inpainting across an image at a rate of about 1 image every 8 seconds - I ‘melt’ the results in for the duration until the next patch is ready:
https://hardwork.party/aws-epoch-optimizer/
I actually think the high frame rate will look worse unless the styling changes less in between frames. Otherwise, it's going to look erratic and there will be a ton of "popping" if each frame changes as much as the frames in the current video. Or maybe op wants it to be a chaotic feeling.
Many years ago I did a couple of hundred variations on the mona lisa in photoshop, and stacked them as a gif. As the eyes remained unchanged in each frame, they gave a fixed point you could focus on while the rest of the face strobed chaotically. It was a nice effect.
Readit News
Base64 encoding the JPEG bytes will increase payload size up to ~30% and burns CPU cycles on both client and server. This is unnecessary, as Websocket protocol can send binary payloads (doesn’t need to be text).
Consider removing lossy jpg compression as well, ie just send the raw RGB bytes over the network. Then on the server side you can simply call Image.frombuffer(…).
StreamDiffusion can achieve high frame rates because of extensive batching in the pipeline. You’re not benefiting from that here as the client is only sending one frame at a time and then waiting for a response. See this example for an idea of how to queue input frames and consume them in batches https://github.com/cumulo-autumn/StreamDiffusion/blob/main/e... .
Alternatively you could take a look at the SDXL Turbo and Lightning models. They are very fast at img2img but have limited resolution of 512² or 1024² pixels respectively. Which appears a bit lower than what you’re aiming for here, but they can be run locally in real time on a high end consumer grade GPU. For reference I have some code demonstrating this here https://github.com/GradientSurfer/Draw2Img/tree/main
Not to sound like a luddite, but I do question the idea that the skill gap is merely an inconvenience. I suspect learning how to paint or make music changes something in yourself which teaches you some deeper life lessons.
I've heard the phrase (paraphrased): No great work of art was made by a genius, genius comes to you unexpectedly like a gust of wind. It seems that cultivating these opportunities is the most an artist can do, and removing the skill gap seems to be removing the cultivation, the thing that changes you, the essence.
There seems to be a few of these inherent deep workings that we as a people keep coming back to, without knowing what they are or how to discuss them (personally at least!). Not to rain on your parade OP, the project looks fun and super useful to a lot people! Just something I ponder on at times.
Does the totality of the project have that property? That would be less clear, but IMO, no. I see it as technically driven, not psychologically driven, although I can see how you could write an artist's statement that claims it was (it's about mirrors, after all, which are hugely symbolic).
To be clear I still like it, and if I'd done it I'd be proud of it. But it's more artifice than art.
(If I was him, I'd slow the frame rate down, not speed it up - work with the technical limitation, not against it. Have the system only display "good" images, and not update the display until another "good" image is generated. The code that decides if an image is "good" or not would be the most interesting part of the system, and could fairly be said to embody the artist's intent, and so cross (my own personal) threshold into capital-A Art.
I'd also experiment with buffering the image stream à la _Light of Other Days_ by Bob Shaw.
Oh, and as Halloween is nearly with us, the temptation to occasionally inpaint a figure standing behind the viewer would be massive.
Idle thought: to get some stability in the image, would it be possible to have an LLM generate random video filter code, instead of random images? "Write me a video filter that makes the input video look cubist". "...like an oil painting" "...with a Flash aesthetic". etc etc. Every time a filter gets generated that doesn't actively crash, swap to it. No idea if that's feasible or not.)
I enthusiastically support this notion. A simpler, than painting, example would be writing. Sure everyone has a story to tell, and everyone can write, but to make it worth other people's time would take you days if not years of perfecting the craft, as you inevitably learn things about yourself and crystallize your perspectives on this world.
Sometimes it's good to have someone with fresh eyes looking and something and not be shaped by decades of prior history.
I love this because I seem to encounter situations like that every day. Who came up with this saying?
Recent example: This guy asked a very simple question about something that's commonly done in industry (wiring two power supplies in parallel and balancing the current between them): https://www.reddit.com/r/AskElectronics/comments/1g84zd7/usi...
Literally 95% of the replies in that thread are irrelevant bullshit from "experts" that have no idea how redundant server PSUs work. I replied to some under the same username. Meanwhile another guy successfully wired two 100W USB-C ports in parallel to power an entire PC. He had no idea that the resistance of his crappy wires kept the two smps control loops stable and divided the current evenly between the two ports ensuring that neither one trips OCP: https://www.reddit.com/r/UsbCHardware/comments/1g8pser/let_m...
Idealism is not a victimless crime, millions suffer every day because some artist threw a buggy,idealized world model over the fence and the idiocy stuck hypnotizing millions into permanently damaging themselves.
https://www.leroymerlin.fr/produits/decoration-eclairage/dec...
For those curious, it's a MILO 21 x 29.7 cm black frame. These links work for me:
https://www.leroymerlin.pt/produtos/decoracao-e-tapetes/mold...
Also, the screen he uses (HMTECH Raspberry Pi Screen 10.1) is pretty hard to find. Do people have other good recommendations for screens with similar quality and specs?
Any idea why this uses infrared light and an infrared camera?
I'd guess it's to make it work in the dark as well.
I did a little research and these Waveshare displays look really beautiful. For a particular size display, they have models in different resolutions (from lower to higher res). But they're only for rpis and windows, not like mac or ipad. The QLED ones are the best.
https://www.waveshare.com/product/ai/displays/10.1hp-capqled...
I always thought that art is all about emotions, both preserving and creating them. That’s why I find a banana on the wall art, popular music is still art.
(Might be school influence but I also tend to ask myself „why”)
This invention is definitely work of art, but its output - in my eyes - not. It’s like a cloud. It can make different shapes, and some are funny, some might remind me of my close ones, but it’s still averaged randomness.
However the idea to reflect reality to this digital randomness I find without doubt - art. And, well: while not visual or audible, there is software, hardware, code and design which are also art (and difficult to make so counterpoint to thesis in first paragraph ;)).
But this leaves the role of intention ambiguous. If I double-park a BMW across two handicap spaces because I'm angry and entitled, is that art? It certainly evokes emotion, and it's also produced by emotion.
I would dissect it though since it’s not pure anger - frustration maybe, a complex one and caused by additional input. E.g. anxiety (you took what could be mine) or internal discord (after observing act of injustice).
I’d argue that it’s an act of random (again, like a cloud) and wouldn’t treat it as art.
But, if you’d park it sideways in front of the mall entrance, blocking it considerably, I’d consider it art (in my imagination I can see the headlines about artist making a statement against the overconsumption).
Yet art is art, so everyone has their own definition. I’d prefer “positive” emotions, so awe, nostalgia, etc., but I don’t see reason why anger should be excluded.
<<proceeds to print “THIS IS AN ART INSTALLATION” message to put behind the windshield>>
But the best part about art is that other people don't have to consider what you do as art.
I would find it hard to believe you are presenting it as art though, as you'll want your BMW back after you've bought whatever crap at the store.
I don't have to believe your AI generated slop is art because it doesn't actually convey any emotion.
This generation of the picture itself is art, similar to other "experience" art pieces. Its referred to as immersive art.
A banana on the wall is not art, because it involves an everyday object in an everyday setting.
You might feel it's art, but if that's the case, then you haven't exposed yourself to true art enough to have a pool of experience or only seen so far superficial examples.
Is that not what was meant by "surfacing the inner world?"
i remember that this has been done already as an art installation in public places so people could see - and interact - with others from around the world.
edit: like this here - https://www.inavateonthenet.net/news/article/vc-art-installa...
Sure, and it wasn't long before this happened:
> It took less than a week for people eager to share their ‘assets’ with the world to shut down the visual portals set up between the cities of Dublin and New York on 8 May. Although the project was aimed at bringing people together and connecting cultures, a few visitors to the locations have decided to take the invitation of getting to know others to a whole other level.
https://traveltomorrow.com/new-york-dublin-portals-shut-down...
(One wonders why flashing is such a big problem that the whole installation needs to be shut down; but it seemed quite obvious from the start that people would try to do this...)
> The main issue with the current setup is the low frame rate.
I'd call it a feature rather a limitation, it's not bad that I have second to process the image for a bit.
I would even increase the update frequency to 5 - 15 minutes and let it capture and generate a new image whenever it detects something changing / moving.