This reminds a bit of a project I coded the audio for an art exhibition around 20 years ago, which was for multiple simultaneous players. As the game got more intense it became apparent that the ball/wall sounds were playing music, and that the three players were all actually playing one musical piece.
It was based around 3 arcade cabinets pointing together, so the players couldn't see what was on each others screens.
This was achieved by modifying the ball speed/direction slightly so that it arrived at the bat/wall at a musically relevant point and triggered the correct sound.
Somewhere in between. It was a static composition (as in, it had a start and an end) but there were multiple loops per instrument, so more competent players would reach a more frenetic musical performance of the piece if they kept more of the balls in play.
The audio engine (the bit I worked on) was in effect a stem based mixer, but with a state transition diagram per stem, with multiple loops available. Depending on the route through the transitions (triggered by events from the game play, for example, how many balls the player is keeping in play) it was possible to reach more complex or simpler performances of the same musical piece, so the players decisions and ability would affect how the performance of the music, not the music itself if that makes sense?
Very cool! As a further variation on this idea, I'm imagining training a reinforcement learning agent on atari games / super mario, but with an additional music-based reward/input, to try to get a "musical" looking playthrough... (Not sure how good it would look / whether it would be worth it though...)
I'm a novice at machine learning, but Open AI made a python library for reinforcement learning in video games, and a fork of it is still actively maintained [1]. It's been a few years, but I remember being able to get it up and running in a day or two (maybe a weekend). It used the Retroarch emulator, which is compatible with a huge number of emulators and consoles.
I encourage you to give it a try! I feel that video games are a bit underrated by current AI buzz, I think there's lots of potential for a machine to learn a skill through playing a game. And lots of potential for a game being selected or even created with the goal of teaching a specific skill. However at that point maybe it's better to forego the audio and visuals and speak to the machine in text or with pure data.
On the other hand, I have seen a video about convolutional neural networks that feed on each pixel of an image. So perhaps training with sound data, or with the pixels of a spectrogram, could have some positive results. And certainly I would be amused to see a game played in time with music, or possibly even dancing with a songs melody, harmony, and story as well.
Anything that's ever been created by humans, existed first in the imagination of a human brain. And you've got one of those. A mental vision pursued and brought from the mind into physical reality is a beautiful thing, a gift for all of humanity in my eyes. I think it's quite worthwhile. But that's just my perspective. Thank you for sharing your imagination. Have a nice day
Not very good. Mario doesn’t have enough rhythm to evoke a lot of musicality.
The original game’s sound was tied to the frame rate so this vaguely happened by default. Later ports to PAL broke this because it ran at a slower frame rate.
Probably almost impossible to adapt written works 'backwards' into a visualization but it might be fun to have different bars represent different notes and have the balls split for chords.
Having N paddles would be cool as well, perhaps with a cost for paddles getting too close (to avoid trivialities), and optimizing which note each paddle represents/beats, maybe imposing a cost on drastic pitch changes so that each specializes in a pitch range.
This is so freaking cool! I was mesmerized watching the paddles move as the beat progressed. There are certain things that just look right which makes it beautiful.This project is one of them!
Bit different though! In your example, the video is made from manually syncing with the song bpm, as the beep is at a constant rate. It's basically just a hand-made visualization of (every other) kick drum.
While the submission has the notes not at a basic 1/4 tempo, and is automatically "animated" based on the constrained optimization. Also leads to a much more interesting visualization :)
While technically okay, there are multiple cases where a paddle and the ball move at the almost identical speed and it looks like the paddle is pushing the ball all the time. (By the way, p[i] = 0 should be disallowed for this reason.) This is inevitable when a large d[i] is immediately followed by a very small d[i+1], but it might be able to avoid them whenever feasible.
Imagining an `installation` in my space, using both my MT-80S and a display. Can I even reason about this, the timing? I'm not smart here, just interested
Readit News
It was based around 3 arcade cabinets pointing together, so the players couldn't see what was on each others screens.
This was achieved by modifying the ball speed/direction slightly so that it arrived at the bat/wall at a musically relevant point and triggered the correct sound.
Ah, here you go, Josh as a reference to it on his site: https://www.autogena.org/work/ping
The audio engine (the bit I worked on) was in effect a stem based mixer, but with a state transition diagram per stem, with multiple loops available. Depending on the route through the transitions (triggered by events from the game play, for example, how many balls the player is keeping in play) it was possible to reach more complex or simpler performances of the same musical piece, so the players decisions and ability would affect how the performance of the music, not the music itself if that makes sense?
https://github.com/Farama-Foundation/Gymnasium
There also SethBling's excellent video on YouTube about machine learning specifically with Super Mario World:
https://www.youtube.com/watch?v=qv6UVOQ0F44
I encourage you to give it a try! I feel that video games are a bit underrated by current AI buzz, I think there's lots of potential for a machine to learn a skill through playing a game. And lots of potential for a game being selected or even created with the goal of teaching a specific skill. However at that point maybe it's better to forego the audio and visuals and speak to the machine in text or with pure data.
On the other hand, I have seen a video about convolutional neural networks that feed on each pixel of an image. So perhaps training with sound data, or with the pixels of a spectrogram, could have some positive results. And certainly I would be amused to see a game played in time with music, or possibly even dancing with a songs melody, harmony, and story as well.
Anything that's ever been created by humans, existed first in the imagination of a human brain. And you've got one of those. A mental vision pursued and brought from the mind into physical reality is a beautiful thing, a gift for all of humanity in my eyes. I think it's quite worthwhile. But that's just my perspective. Thank you for sharing your imagination. Have a nice day
Deleted Comment
The original game’s sound was tied to the frame rate so this vaguely happened by default. Later ports to PAL broke this because it ran at a slower frame rate.
https://www.youtube.com/@LucidRhythms
Probably almost impossible to adapt written works 'backwards' into a visualization but it might be fun to have different bars represent different notes and have the balls split for chords.
While the submission has the notes not at a basic 1/4 tempo, and is automatically "animated" based on the constrained optimization. Also leads to a much more interesting visualization :)
https://www.matrixsynth.com/2014/07/roland-mt-80s-midi-playe...