Any new "defense" that claims to use adversarial perturbations to undermine GenAI training should have to explain why this paper does not apply to their technique: https://arxiv.org/pdf/2406.12027
The answer is, almost unfailingly, "this paper applies perfectly to our technique because we are just rehashing the same ideas on new modalities". If you believe it's unethical for GenAI models to train on people's music, isn't is also unethical to trick those people into posting their music online with a fake "defense" that won't actually protect them?
You are assuming input-transformation based defenses in the image domain transfer to the music recognition domain, when we know they don't automatically even transfer to the speech recognition domain.
But 'protection' of any one song isn't the entire point. It only takes less than a fraction of a percent of corpus data to have persistent long term effects in the final model, or increase costs and review requirements to those stealing their content.
As most training is unsupervised, because the cost and limited access to quality, human labeled data, it wouldn't take much if even some obscure, limited market, older genres which still have active fan bases, like Noise rock to start filtering into recommendation engines and impact user satisfaction.
Most of the speech protections, just force attacks to be in the perceptible audio range, with lo-fi portions like those of TripHop, that would be non-detectable without the false positive rate going way up. With bands like Arab On Radar, Shellac, or The Oxes, it wouldn't be detectable.
But it is also like WAFs/AV software/IDS. The fact that it can't help with future threats today is immaterial. Any win of these leaches has some value.
Obviously any company intentionally applying even the methods in your linked paper to harvest protected images would be showing willful intent to circumvent copyright protections and I am guessing most companies will just toss any file that it thinks has active protections just because how sensitive training is.
Most musicians also know that copyright only protects the rich.
I am ignorant here, this is a genuine question - is there any reason to assume that a paper solely about image mimicry can be blanket-applied, as OP is doing, to audio mimicry?
To add, all the new audio models (partially) use diffusion methods that are exactly the same methods as used on images - the audio generation can be thought of as an image generation of a spectrogram of an audio file.
For early experiments people literally took Stable Diffusion and fine tuned it on labelled spectrograms of music snippets, then used the fine tuned model to generate new images of spectrograms guided by text, and then took those images and turned them back into audio via re-synthesis of that spectral image to a .wav.
The more advanced music generators out now I believe have more of a 'stems' approach and a larger processing pipeline to increase fidelity and add tracking vocal capability but the underlying idea is the same.
Any adversarial attack to hide information in the spectrograph to fool the model into categorizing the track as something it is not isn't different than the image adversarial attacks which have been found to have ways to be mitigated.
Various forms of filtering for inaudible spectral information coupled with methods that destroy and re-synthesize/randomize phase information would likely break this poisoning attack.
The short answer is that they are applying the same defense to audio as to images, and so we should expect that the same attacks will work as well.
More specifically, there are a few moving parts here - the GenAI model they're trying to defeat, the defense applied to data items, and the data cleaning process that a GenAI company may use to remove the defense. So we can look at each and see if there's any reason to expect things to turn out differently than they did in the image domain. The GenAI models follow the same type of training, and while they of course have slightly different architectures to ingest audio instead of images, they still use the same basic operations. The defenses are exactly the same - find small perturbations that are undetectable to humans but produce a large change in model behavior. The cleaning processes are not particularly image-specific, and translate very naturally to audio. It's stuff like "add some noise and then run denoising".
Given all of this, it would be very surprising if the dynamics turned out to be fundamentally different just because we moved from images to audio, and the onus should be on the defense developers to justify why we should expect that to be the case.
Some of the sibling comments had questions around purposefully releasing defenses which don’t work. I think Carlini’s (one of the paper authors) post can add some important context: https://nicholas.carlini.com/writing/2024/why-i-attack.html.
TLDR: Once these defenses are broken, all previously protected work is perpetually unprotected, so they are flawed at a foundational level.
Ignoring these arguments and pretending they don’t exist is pretty unethical.
>o it seems a logical leap to say they know it doesn't and are doing this as a scheme?
In some of the earlier image protection articles the people involved seemed rather shady about the capabilities. Would have to do some HN searching for those articles.
But everything at the end of the day will be a scheme if the end result is for humans to listen to it. You cannot make a subset of music that can be heard by humans (and actually sounds good) that cannot be prefiltered to be learned by AI. I've said the same thing about images, the same thing will be true about audio, movies, actions in real leave, et al.
These schemes will likely work for a few of the existing models, then fall apart quickly the moment a new model arrives. What is worse for defense is audio quality for humans is remaining the same while GPU speeds and algorithms increase in speeds over time meaning the time until a model beats the new defense will trend to unity.
I like Benn Jordan because he’s clearly got a grasp on a functional understanding of machine learning, but that’s not his primary background. He comes from a music production background, so his focus is more practical and results-oriented.
It will be really interesting as this knowledge percolates into more and more fields, what domain experts do with it. I see ML as more of a bag of tricks that can be applied to many fields.
>He comes from a music production background, so his focus is more practical and results-oriented
It's his art and his livelihood too, so it's also personal. These people want to steal his art and create a world full of soulless cheap muzak, while simultaneously putting him out of work.
They’ve been doing that since the recording studio process developed the model in the 1920s or so. They would hire songwriters to make generic pop music with generic lyrics, and keep it in the vault until you have some attractive young singer you want to use for marketing then you give them an album of these songs to sing. And they are sure to sell because you’ve been priming the american ear for these chord progressions for a long time, and you fill all the air in the room with your marketing for this singer leaving people little option but to hear the latest carefully crafted earworm. Still happens today maybe even more perfected with psychological studies intersecting with music and marketing. The best musicians have never and will never be a product of that machine. Seek out live music.
Are you sure you mean "stealing"? As in deprive him of his own recordings?
I am curious if anyone read Harry Potter in bootleg form from a LLM. I mean, LLMs are the worst tools for infringing - they are approximate, expensive and slow, while copying is instant, perfect and free. You can apply the same logic for other modalities.
Moreover, who's got the time to see someone else's AI shit when they can generate their own, perfectly customized to their liking? I personally generated a song about my cat and kid. It had zero commercial value but was fun for 2-3 people to listen.
Benn is one of my fave subscriptions on YouTube--both for the (now more occasional) music gear stuff and for the in-depth music industry education. The fact that he has been hacking away at IP and AI stuff for ages is just icing on the cake.
All this stuff is snake oil, either already, or eventually.
There's new models showing up regularly. Civitai recognizes 33 image models at this point, and audio will also see multiple developments. Any successful attack on a model isn't guaranteed to apply to another one, not even yet invented. There's also a multitude of possible pre-processing methods and their combinations for any piece of media.
There's also the difficulty of attacking a system that's not well documented. Not every model out there is open source and available for deep analysis.
And it's hard to attack something that doesn't yet exist, which means countermeasures will come up only after a model was already successfully created. This is I'm sure of some academic interest, but the practical benefits seem approximately none.
Since information is trivially stored, anyone having any trouble could just download the file today and sit on it for a year or two not doing anything at all, just waiting for a new model to show up.
It seems some models ignore it and produce mostly clean images on the output (looking like the last image), while others just interpret is as a texture, the character is just wearing a funny patterned shirt. This is while the intended result is fooling the model to generate something other than the intended character.
The problem is that copyright is the law of the land, and it demands our participation.
Because of that reality, every artist who wants to make money must either participate in it, or completely isolate themselves from it.
These models have become an incredible opportunity for giant corporations to circumvent the law. By training a model on a copyrighted work, you can launder that work into your own new work, and make money from it without sharing that money with the original artists. Obviously, this is an incredibly immoral end to copyright as we know it.
So what are we going to do about this situation? Are we really going to keep pretending that copyright can work? It wasn't even working before all the AI hype! Ever heard the words "starving artist"? Of course you have!
We need a better system than copyright. I'm convinced that no system at all (anarchy) would be a superior option at this point. If not now, then when?
> By training a model on a copyrighted work, you can launder that work into your own new work, and make money from it without sharing that money with the original artists.
Not sure if "you" refers to model developers, hosting company or end users. But let's see each one of them in turn
- model development is a cost center, there is no profit yet
- model deployment brings little profit, they make cents per million tokens
- applying the model to your own needs - that is where the benefit goes.
So my theory is that benefits follow the problem, it is in the application layer. Have a need, you can benefit from AI, don't need it, no benefit. Like Linux. You got to use it for something. And that usage, that problem - is personal. You can't sell your problems, they remain yours. It is hard to quantify how people benefit from AI, it could be for fun, for learning, professional use, or for therapy.
Most gen-AI usage is seen by one person exactly once. Think about that. It's not commercial, it's more like augmented imagination. Who's gonna pay for AI generated stuff when it is so easy to make your own.
My point is that this entire situation has to be framed in the narrative that copyright demands it be framed in. It's "you" the participant of copyright.
When someone creates art, copyright says that there is a countable result we can refer to as their "work". Copyright also says that that artist has a monopoly over the distribution and sale of that work. The implication is that the way for an artist to get paid for their labor is for them to leverage the monopoly they have been granted, and negotiate a distribution scheme that involves paying them.
When an artist chooses to work outside the copyright model, that means they must predetermine part of their distribution negotiation. That might be the libertarian option (gratis distribution with no demands), or it might be the copyleft option, where the price is demanded, but also set to 0. The artist may find payment for their labor by other means, but that's challenging to do in an economy where copyright participants dominate.
Indeed! I've definitely been a fan of his for a while, and I laud him for trying to make things work in a space where all the cards are stacked against him.
I do wish, though, that he would have introduced that perspective of the situation in this particular video. Leaving it out feels like making a video about learning to swim, set in the middle of the ocean.
adversarial noise is very popular in the media but imo is a complete dead end for the desired goals - representations do not transfer between different models this easily
Readit News
The answer is, almost unfailingly, "this paper applies perfectly to our technique because we are just rehashing the same ideas on new modalities". If you believe it's unethical for GenAI models to train on people's music, isn't is also unethical to trick those people into posting their music online with a fake "defense" that won't actually protect them?
But 'protection' of any one song isn't the entire point. It only takes less than a fraction of a percent of corpus data to have persistent long term effects in the final model, or increase costs and review requirements to those stealing their content.
As most training is unsupervised, because the cost and limited access to quality, human labeled data, it wouldn't take much if even some obscure, limited market, older genres which still have active fan bases, like Noise rock to start filtering into recommendation engines and impact user satisfaction.
Most of the speech protections, just force attacks to be in the perceptible audio range, with lo-fi portions like those of TripHop, that would be non-detectable without the false positive rate going way up. With bands like Arab On Radar, Shellac, or The Oxes, it wouldn't be detectable.
But it is also like WAFs/AV software/IDS. The fact that it can't help with future threats today is immaterial. Any win of these leaches has some value.
Obviously any company intentionally applying even the methods in your linked paper to harvest protected images would be showing willful intent to circumvent copyright protections and I am guessing most companies will just toss any file that it thinks has active protections just because how sensitive training is.
Most musicians also know that copyright only protects the rich.
https://securitycryptographywhatever.com/2025/01/28/cryptana...
For early experiments people literally took Stable Diffusion and fine tuned it on labelled spectrograms of music snippets, then used the fine tuned model to generate new images of spectrograms guided by text, and then took those images and turned them back into audio via re-synthesis of that spectral image to a .wav.
Riffusion was one of the first to experiment with this, 2 years ago now: https://github.com/riffusion/riffusion-hobby
The more advanced music generators out now I believe have more of a 'stems' approach and a larger processing pipeline to increase fidelity and add tracking vocal capability but the underlying idea is the same.
Any adversarial attack to hide information in the spectrograph to fool the model into categorizing the track as something it is not isn't different than the image adversarial attacks which have been found to have ways to be mitigated.
Various forms of filtering for inaudible spectral information coupled with methods that destroy and re-synthesize/randomize phase information would likely break this poisoning attack.
More specifically, there are a few moving parts here - the GenAI model they're trying to defeat, the defense applied to data items, and the data cleaning process that a GenAI company may use to remove the defense. So we can look at each and see if there's any reason to expect things to turn out differently than they did in the image domain. The GenAI models follow the same type of training, and while they of course have slightly different architectures to ingest audio instead of images, they still use the same basic operations. The defenses are exactly the same - find small perturbations that are undetectable to humans but produce a large change in model behavior. The cleaning processes are not particularly image-specific, and translate very naturally to audio. It's stuff like "add some noise and then run denoising".
Given all of this, it would be very surprising if the dynamics turned out to be fundamentally different just because we moved from images to audio, and the onus should be on the defense developers to justify why we should expect that to be the case.
TLDR: Once these defenses are broken, all previously protected work is perpetually unprotected, so they are flawed at a foundational level.
Ignoring these arguments and pretending they don’t exist is pretty unethical.
Deleted Comment
In some of the earlier image protection articles the people involved seemed rather shady about the capabilities. Would have to do some HN searching for those articles.
But everything at the end of the day will be a scheme if the end result is for humans to listen to it. You cannot make a subset of music that can be heard by humans (and actually sounds good) that cannot be prefiltered to be learned by AI. I've said the same thing about images, the same thing will be true about audio, movies, actions in real leave, et al.
These schemes will likely work for a few of the existing models, then fall apart quickly the moment a new model arrives. What is worse for defense is audio quality for humans is remaining the same while GPU speeds and algorithms increase in speeds over time meaning the time until a model beats the new defense will trend to unity.
Deleted Comment
It will be really interesting as this knowledge percolates into more and more fields, what domain experts do with it. I see ML as more of a bag of tricks that can be applied to many fields.
It's his art and his livelihood too, so it's also personal. These people want to steal his art and create a world full of soulless cheap muzak, while simultaneously putting him out of work.
Get 'em, Benn! I should go buy one of his albums.
I am curious if anyone read Harry Potter in bootleg form from a LLM. I mean, LLMs are the worst tools for infringing - they are approximate, expensive and slow, while copying is instant, perfect and free. You can apply the same logic for other modalities.
Moreover, who's got the time to see someone else's AI shit when they can generate their own, perfectly customized to their liking? I personally generated a song about my cat and kid. It had zero commercial value but was fun for 2-3 people to listen.
Any musician these days that thinks there is money in music by selling songs is delusional. Sad but true.
The Flashbulb - Parkways: https://youtu.be/C6pzg7I61FI
There's new models showing up regularly. Civitai recognizes 33 image models at this point, and audio will also see multiple developments. Any successful attack on a model isn't guaranteed to apply to another one, not even yet invented. There's also a multitude of possible pre-processing methods and their combinations for any piece of media.
There's also the difficulty of attacking a system that's not well documented. Not every model out there is open source and available for deep analysis.
And it's hard to attack something that doesn't yet exist, which means countermeasures will come up only after a model was already successfully created. This is I'm sure of some academic interest, but the practical benefits seem approximately none.
Since information is trivially stored, anyone having any trouble could just download the file today and sit on it for a year or two not doing anything at all, just waiting for a new model to show up.
Seems like an awful risk to deliberately strip such markings. It's a kind of DRM, and breaking DRM is illegal in many countries.
For instance, I've seen somebody experiment with Glaze (the image AI version of this). Glaze at high levels produces visible artifacts (see middle image: https://pbs.twimg.com/media/FrbJ9ZTacAAWQQn.jpg:large ).
It seems some models ignore it and produce mostly clean images on the output (looking like the last image), while others just interpret is as a texture, the character is just wearing a funny patterned shirt. This is while the intended result is fooling the model to generate something other than the intended character.
Because of that reality, every artist who wants to make money must either participate in it, or completely isolate themselves from it.
These models have become an incredible opportunity for giant corporations to circumvent the law. By training a model on a copyrighted work, you can launder that work into your own new work, and make money from it without sharing that money with the original artists. Obviously, this is an incredibly immoral end to copyright as we know it.
So what are we going to do about this situation? Are we really going to keep pretending that copyright can work? It wasn't even working before all the AI hype! Ever heard the words "starving artist"? Of course you have!
We need a better system than copyright. I'm convinced that no system at all (anarchy) would be a superior option at this point. If not now, then when?
Not sure if "you" refers to model developers, hosting company or end users. But let's see each one of them in turn
- model development is a cost center, there is no profit yet
- model deployment brings little profit, they make cents per million tokens
- applying the model to your own needs - that is where the benefit goes.
So my theory is that benefits follow the problem, it is in the application layer. Have a need, you can benefit from AI, don't need it, no benefit. Like Linux. You got to use it for something. And that usage, that problem - is personal. You can't sell your problems, they remain yours. It is hard to quantify how people benefit from AI, it could be for fun, for learning, professional use, or for therapy.
Most gen-AI usage is seen by one person exactly once. Think about that. It's not commercial, it's more like augmented imagination. Who's gonna pay for AI generated stuff when it is so easy to make your own.
When someone creates art, copyright says that there is a countable result we can refer to as their "work". Copyright also says that that artist has a monopoly over the distribution and sale of that work. The implication is that the way for an artist to get paid for their labor is for them to leverage the monopoly they have been granted, and negotiate a distribution scheme that involves paying them.
When an artist chooses to work outside the copyright model, that means they must predetermine part of their distribution negotiation. That might be the libertarian option (gratis distribution with no demands), or it might be the copyleft option, where the price is demanded, but also set to 0. The artist may find payment for their labor by other means, but that's challenging to do in an economy where copyright participants dominate.
https://www.youtube.com/watch?v=PJSTFzhs1O4
I do wish, though, that he would have introduced that perspective of the situation in this particular video. Leaving it out feels like making a video about learning to swim, set in the middle of the ocean.
https://www.youtube.com/watch?v=xMYm2d9bmEA
the [transferability] rates just drop off significantly for audio (always felt it was a similar vibe to RNN ‘vanishing gradients’)
edit — specifically mention transferability