So, the lack of outrage here (I mean in hacker news) is unsurprising, and would of course be greater if there were more black people "in tech" and by extension here.
And even as I write this, I'm not trying to judge -- just more open the door to understanding, like, y'all understand this fact, right? I find myself searching for metaphorical equivalents. I'm thinking something like "what if it labeled Jewish people as rats" or something similar.
Even if that doesn't work -- I suppose the broader point I'd try to make here is: A lot of y'all (and me, often) assume the following very wrong idea, subconsciously -- "Because I'm working with a computer I can assume that what we are doing is infused with objectivity, like math"
Important to remember nooooooooope. This is dead wrong. We're ALWAYS bringing our subconscious whatevers to this and there is very little, if ANYTHING at all, that can be correctly considered "coldly and obviously correct and object like math is." (perhaps even what we imagine math to be)
I also happen to be black but that actually has very little bearing on what I have to say and am merely mentioning it to prevent anyone from suggesting that my opinion on the matter is related to my skin color.
The problem with this is no one actually trained the AI to have a racially bias but rather did not train it on recognizing the difference. Why? Probably because no one knew the computer was going to identify this resemblance and immediately label it as such (I have heard babies learning to talk and name shapes/colors do the exact same thing with no outside influence) Computer vision operates on matrices of color coordinates that share resemblances with other similar matrices and attempts to match patterns between multiple sets to establish resemblance. It is not the fault of the computer or the programmer that there are coincidental resemblances that exceed it’s learned threshold where it decides that the two sets represents objects of the same type.
I personally am ok with admitting that my skin color visually is more similar to images of certain non human primates and can understand why this problem is not a programmer bias but actually a complicated computer vision problem as it’s easy to spot the resemblance but often difficult to train a neural net on why the images resemble eachother but are not related.
That first sentence, man. Like, I feel bad second guessing people, in some ways it's not my right -- but it so feels like the kind of "respectability" talk I used to say out of what I thought was "the truth," but now realize it was a sort of habitual way to defuse the room.
So I will state clearly -- yes, I am aware my opinions are absolutely related to my skin color and that's precisely why you all need to pay special attention to them. Not because of some sort of silly inherent genetic thing, but because I have experiences that many of you don't.
I find it baffling and dispiriting that you, as a Black person, believe this is more likely to be an "error" than an act of deliberate racism. It's getting silly, now. A white racist programmer has done this on purpose because he finds it hilarious. The chances of a computer equating monkey and man, given the correct datasets, are close to nil. There's a blindness - a wilful ignorance - to look away and show good faith. Why is that?
Glad you posted. A lot of AI code simply cannot be trusted. In particularly, it's pretty obvious that the teams using it either don't do a lot of QA or don't understand how to test their systems in the first place. I find it mind-boggling that this technology is so widely used with such limited understanding.
The interesting question is how the hell do we fix this?
One example doesn't support your claim that QA was not sufficient. You will always get misclassifications, most won't be offensive to anyone. This is not a medical device or a self driving car, this is Facebook trying to squeeze more engagement..
I find a lot of the discussion here baffling and tone deaf. There are a lot of comments saying "but humans are primates". It's a dismissal of systemic racism issues coming from a privileged position that's akin to responding the black lives matter movement with "all lives matter".
Some of the commenters may exactly think that the algorithm is objective, forgetting that it's built by ppl, for ppl, tested by ppl. If the algorithm kept labelling bearded white dudes as vaginas, it wouldn't be pushed to production, because it would found as a bug and dealt with somehow. But labelling people of color as monkeys is apparently something they're unaware of, which doesn't necessarily point to deliberate racism, but a systemic racism issue.
In a lot of discussion here, everyone, on all sides of the argument, is getting voted down. Perhaps this shows this community is not ready for these types of discussions.
Now imagine the care taken when applying this technology to dating sites, and other social services; for clustering or recommendation engines, or prediction services. This whole field is a joke, that refuses any introspection.
At this point, if a web scale service like this do this it should be grounds for significant sanctions: increasing fines (think GDPR scale) and closer scrutiny (think mandatory inspections and quality controls every month, paid for by the company) should do I think.
But, while I'm really annoyed by this and support you I'm also sure there will be enough outrage against it in time from.
Neural nets, in particular, aren't objective... They're bias-learning engines. Their whole raison d'etre is to take (in this case) a bunch of images and go "Okay... Humans believe these random patterns of pixels have semantically different meanings. I'm going to try to explore a space of what those differences could be until they tell me I got it right." If there are things we as humans care about classifying differently that we fail to tell the computer, it won't know to separate them.
The fact we keep making this mistake is a real problem. And it's a human problem, not an algorithm problem.
On a technical level, yes, all humans are primates, but if the algorithm were following this rule, it would tag all humans in all photos as primates, but it didn’t do that. It only tagged black people.
That takes us to a social meaning level. In this context, you have to consider the long, racists history of dehumanizing black people. They have often been called “monkey” and “ape” with the intent to degrade and to segregate them from “people”.
In that light, and the fact that the label was not applied in all cases where appropriate, it is hard to see it as anything but racist. Probably not deliberately but if the training material contained racist material, it could bias the result. Even if the training material was inadvertently unbalanced and contained very few black faces, the algorithm would have less experience with black faces and would not know how to categorize them as well as white faces.
This is analogous to people who grow up and live largely isolated from close associates with those of other races. Their experiences are filled with varied samples of interactions with people of their own race, but are poor in samples with other races. In the case where we have less experience, we tend to fall back on broad stereotypes and our reactions show strong racial bias.
I've dealt with a lot of badly labelled datasets, but I can't say I ever saw one maliciously labelled by racists.
There are other more subtle problems that may contribute. As an amateur photographer I've been told that most cameras are calibrated for light skin. If you aren't careful with your setup you'll get bad definition in shadows and relatively very dark areas. If you have badly calibrated/exposed photos then very dark faces, whether of sub-Saharan people or gorillas, may be poorly distinguished. I bet there are loads of unfair, hard-to-catch issues like that.
By the way I don't mean to say that you're wrong, but just to offer a Hanlon's Razor-type counterpoint as it occurred to me.
This is usually a good example of societal bias compounding.
It's possible that nobody directly involved in this was actively racist, but we're building on centuries of racism in every facet. So a little bit of bias in each technology and data set involved, not to mention the engineers and their testing, just adds up
That was my point. If the training data contained mostly white faces and few black faces, the AI would have fewer cases to base it’s weightings when trying to analyze a photo with black faces.
I'm sorry but no. I don't believe this at all. What are the chances of say, you finding a dataset maliciously labelled by racists and how would you uncover that? Would you ask every software engineer how he felt about Black Lives Matter as he tagged images? It's more likely that there is malicious intent in a company where 2% of software engineers are African-American than not.
OK, I'll try to tread carefully here, but I think it's really helpful to separate the technical issues from the societal issues.
In the Google photo case, I've seen the picture in question, and it's not difficult an all to see how a rudimentary statistical algorithm would mistake the photo for a gorilla. No, no human would make this mistake, but due to the lighting in the photo and the woman's hair it's really not hard to see how that mistake would be made, similar to how Tesla's AI can mistake the sun, low on the horizon, as a yellow traffic light.
The algorithm was not "racist", it just didn't have the social context that describing black people as apes and monkeys has a long racist history. And that is really the fundamental problem with most AI these days - it can get very good at statistical inference, but it doesn't have the background knowledge and logic to be able to "think" in the same way humans do.
In a similar vein, I recall someone lamenting a couple years ago how Google's street directions would never mispronounce "Malcolm X Boulevard" as "Malcolm 10 Boulevard" if they had any black programmers. Again, given 99% of the time when you see "X" in an address you'd pronounce it as 10, it's not hard to see how this could happen. The problem is that some errors are much more offensive than others, and AI can't really reason about that.
Google's street directions also fail miserably with international road names.
Words are fine, but person names or abbreviations will often use the English pronunciation which can be inexact or even incomprehensible.
For example: In Brazil, the major roadways are named with the initial code of the state, or BR if it's a federal road, and a bunch of numbers. So BR-101, for example. These are never named in full, so someone reading "BR-101" would not say "Brasil 101", they would say "B R 101".
There is a state called Santa Catarina, with initials SC, but Google decides that we somehow get transported to the United States somehow, since it reads "SC-406" as "South Carolina 406".
A human would at the very least have the context that SC definitely is not a common abbreviation for South Carolina in Brazil, even if they did not know any state names.
If you know in advance that Primate vs. Human confusions are going to be a potential issue that upset a lot of people (say, because it already happened with a similar system at another company) there are certainly ways of trying to ensure that same confusion doesn't happen.
Ways to actually fix the problem:
1. Probe the decision boundary between these two classes in your training and test sets. I.E. look for the humans closest to being misclassified as primates. You could probably quickly get a sense if you are near/at risk of making this misclassification. You may also possibly find 1 or 2 mislabeled examples that are throwing things off and can be corrected.
2. Boost your training set by labeling more unlabeled images that are near this decision boundary.
So my point was just: this was an issue that could have been anticipated because it has already happened before. It is unfortunate that this issue came up when there are things you can do (such as the steps above), which I believe could have totally squashed this issue with enough iteration.
I can't know for sure that this is not what the Facebook team did. If they did take care to try and avoid this harmful model confusion, then I would be very surprised given my experience with deep learning and computer vision.
At a minimum, it would have been pretty trivial and a low impact to users to remove the primate class if they didn't have the time/bandwidth to really investigate this more thoroughly as I've described.
You can try to argue "hey no one should get upset about this confusion because computer vision is hard and mistakes happen", but I don't think that's a very solid argument either given the long and relatively recent history of racists misclassifying a particular subset of humans as primates.
The problem is the training set, which apparently skews too much towards white ppl, and doesn't include checks for the primates/non-white humans differentiation. One could argue this is due to white people building things for white ppl, which does indeed point towards systemic racism in the overall development process.
Separating technical issues from social ones is the source of the problem. I don't think it'll be the source of the solution.
One of the big issues with relying on AI is training data, and the responsibility for an inadequately trained AI model falls on the people who judged it ready for use, but that doesn't mean the outcome of the AI isn't racist. An AI doesn't need to know the sociopolitical history of race to be doing racist things.
Also, I've never heard of a location with Roman numerals in the name. I agree that programming in the context of knowing who Malcolm X is into a map program might be asking a bit much, but I think an "easier" implementation (reading them as letters) would have also done the right thing.
For Asians, there is a history of cameras saying "Someone Blinked!" I think it is, or at least should be, part of the conversation. My partner is Asian, and we had a camera with this problem. We thought it was funny, but it demonstrates weaknesses in computer image processing vs. human visual processing that probably has nothing to do with racism.
To me it is just a reminder that AI based picture recognition is not a perfect science and that I should fight any and all attempts to include it in places where it could cause most harm ( policing comes to mind ).
Not really wanting to delve into the touchy issue of this particular ai fuckup, but it got me thinking about the issue of using ai to compartmentalize and categorize things in general.
This is a very obvious example of how bad it can be. Regular complaints from people about ai recommendation engines and most other similar ai driven tagging systems exist.
What i wonder though is on a more subtle level, how these ai categorizers are quietly altering our perceptions of things and the world?
We notice when it's something obvious like the story in the OP, but how many quiet little mistakes do we just ignore or not notice that on a subconcious level are changing the way we think about or classify things.
More and more it seems like society and all things are being neatly placed in little boxes, where they're filed, stamped, indexed, and numbered by fancy algorithms operating without much human oversight, until something like this happens.
How much of this is subconciously affecting human perception of the world?
> The video, dated June 27, 2020, was by The Daily Mail and featured clips of Black men in altercations with white civilians and police officers. It had no connection to monkeys or primates.
Huh. All the other incidents the NY Times article mentions come with links, but this one (that the article is ostensibly about) doesn't. Odd.
> The Daily Mail and featured clips of Black men in altercations with white civilians and police officers. It had no connection to monkeys or primates.
I'm guessing it ingested the Daily Mail's comment sections.
There’s an amazing number of “but people are primates” comments. Correctness here includes specificity. If person and primate are both candidates, then primate isn’t a correct answer to what this is trying to do. You wouldn’t react to the phrase ‘treating women as objects’ by saying people are objects, right?
You make a good point, and I'd agree that it's implicit that they mean "non-human primates". Otherwise they could solve this by labelling lots of humans as "primates" in the dataset.
On the other hand, I think there's a semantic difference between your two cases. I'd argue that treating a woman as an "object" is a different sense of the word to how women (and men and teapots) are literally "objects".
Readit News
So, the lack of outrage here (I mean in hacker news) is unsurprising, and would of course be greater if there were more black people "in tech" and by extension here.
And even as I write this, I'm not trying to judge -- just more open the door to understanding, like, y'all understand this fact, right? I find myself searching for metaphorical equivalents. I'm thinking something like "what if it labeled Jewish people as rats" or something similar.
Even if that doesn't work -- I suppose the broader point I'd try to make here is: A lot of y'all (and me, often) assume the following very wrong idea, subconsciously -- "Because I'm working with a computer I can assume that what we are doing is infused with objectivity, like math"
Important to remember nooooooooope. This is dead wrong. We're ALWAYS bringing our subconscious whatevers to this and there is very little, if ANYTHING at all, that can be correctly considered "coldly and obviously correct and object like math is." (perhaps even what we imagine math to be)
The problem with this is no one actually trained the AI to have a racially bias but rather did not train it on recognizing the difference. Why? Probably because no one knew the computer was going to identify this resemblance and immediately label it as such (I have heard babies learning to talk and name shapes/colors do the exact same thing with no outside influence) Computer vision operates on matrices of color coordinates that share resemblances with other similar matrices and attempts to match patterns between multiple sets to establish resemblance. It is not the fault of the computer or the programmer that there are coincidental resemblances that exceed it’s learned threshold where it decides that the two sets represents objects of the same type.
I personally am ok with admitting that my skin color visually is more similar to images of certain non human primates and can understand why this problem is not a programmer bias but actually a complicated computer vision problem as it’s easy to spot the resemblance but often difficult to train a neural net on why the images resemble eachother but are not related.
So I will state clearly -- yes, I am aware my opinions are absolutely related to my skin color and that's precisely why you all need to pay special attention to them. Not because of some sort of silly inherent genetic thing, but because I have experiences that many of you don't.
The interesting question is how the hell do we fix this?
Some of the commenters may exactly think that the algorithm is objective, forgetting that it's built by ppl, for ppl, tested by ppl. If the algorithm kept labelling bearded white dudes as vaginas, it wouldn't be pushed to production, because it would found as a bug and dealt with somehow. But labelling people of color as monkeys is apparently something they're unaware of, which doesn't necessarily point to deliberate racism, but a systemic racism issue.
In a lot of discussion here, everyone, on all sides of the argument, is getting voted down. Perhaps this shows this community is not ready for these types of discussions.
Dead Comment
Dead Comment
Dead Comment
Dead Comment
At this point, if a web scale service like this do this it should be grounds for significant sanctions: increasing fines (think GDPR scale) and closer scrutiny (think mandatory inspections and quality controls every month, paid for by the company) should do I think.
But, while I'm really annoyed by this and support you I'm also sure there will be enough outrage against it in time from.
I'm genuinely curious what else should have happened.
Neural nets, in particular, aren't objective... They're bias-learning engines. Their whole raison d'etre is to take (in this case) a bunch of images and go "Okay... Humans believe these random patterns of pixels have semantically different meanings. I'm going to try to explore a space of what those differences could be until they tell me I got it right." If there are things we as humans care about classifying differently that we fail to tell the computer, it won't know to separate them.
The fact we keep making this mistake is a real problem. And it's a human problem, not an algorithm problem.
Words have meaning on multiple levels.
On a technical level, yes, all humans are primates, but if the algorithm were following this rule, it would tag all humans in all photos as primates, but it didn’t do that. It only tagged black people.
That takes us to a social meaning level. In this context, you have to consider the long, racists history of dehumanizing black people. They have often been called “monkey” and “ape” with the intent to degrade and to segregate them from “people”.
In that light, and the fact that the label was not applied in all cases where appropriate, it is hard to see it as anything but racist. Probably not deliberately but if the training material contained racist material, it could bias the result. Even if the training material was inadvertently unbalanced and contained very few black faces, the algorithm would have less experience with black faces and would not know how to categorize them as well as white faces.
This is analogous to people who grow up and live largely isolated from close associates with those of other races. Their experiences are filled with varied samples of interactions with people of their own race, but are poor in samples with other races. In the case where we have less experience, we tend to fall back on broad stereotypes and our reactions show strong racial bias.
There are other more subtle problems that may contribute. As an amateur photographer I've been told that most cameras are calibrated for light skin. If you aren't careful with your setup you'll get bad definition in shadows and relatively very dark areas. If you have badly calibrated/exposed photos then very dark faces, whether of sub-Saharan people or gorillas, may be poorly distinguished. I bet there are loads of unfair, hard-to-catch issues like that.
By the way I don't mean to say that you're wrong, but just to offer a Hanlon's Razor-type counterpoint as it occurred to me.
It's possible that nobody directly involved in this was actively racist, but we're building on centuries of racism in every facet. So a little bit of bias in each technology and data set involved, not to mention the engineers and their testing, just adds up
https://www.theverge.com/2018/1/12/16882408/google-racist-go...
In the Google photo case, I've seen the picture in question, and it's not difficult an all to see how a rudimentary statistical algorithm would mistake the photo for a gorilla. No, no human would make this mistake, but due to the lighting in the photo and the woman's hair it's really not hard to see how that mistake would be made, similar to how Tesla's AI can mistake the sun, low on the horizon, as a yellow traffic light.
The algorithm was not "racist", it just didn't have the social context that describing black people as apes and monkeys has a long racist history. And that is really the fundamental problem with most AI these days - it can get very good at statistical inference, but it doesn't have the background knowledge and logic to be able to "think" in the same way humans do.
In a similar vein, I recall someone lamenting a couple years ago how Google's street directions would never mispronounce "Malcolm X Boulevard" as "Malcolm 10 Boulevard" if they had any black programmers. Again, given 99% of the time when you see "X" in an address you'd pronounce it as 10, it's not hard to see how this could happen. The problem is that some errors are much more offensive than others, and AI can't really reason about that.
Words are fine, but person names or abbreviations will often use the English pronunciation which can be inexact or even incomprehensible.
For example: In Brazil, the major roadways are named with the initial code of the state, or BR if it's a federal road, and a bunch of numbers. So BR-101, for example. These are never named in full, so someone reading "BR-101" would not say "Brasil 101", they would say "B R 101".
There is a state called Santa Catarina, with initials SC, but Google decides that we somehow get transported to the United States somehow, since it reads "SC-406" as "South Carolina 406".
A human would at the very least have the context that SC definitely is not a common abbreviation for South Carolina in Brazil, even if they did not know any state names.
Ways to actually fix the problem:
1. Probe the decision boundary between these two classes in your training and test sets. I.E. look for the humans closest to being misclassified as primates. You could probably quickly get a sense if you are near/at risk of making this misclassification. You may also possibly find 1 or 2 mislabeled examples that are throwing things off and can be corrected.
2. Boost your training set by labeling more unlabeled images that are near this decision boundary.
So my point was just: this was an issue that could have been anticipated because it has already happened before. It is unfortunate that this issue came up when there are things you can do (such as the steps above), which I believe could have totally squashed this issue with enough iteration.
I can't know for sure that this is not what the Facebook team did. If they did take care to try and avoid this harmful model confusion, then I would be very surprised given my experience with deep learning and computer vision.
At a minimum, it would have been pretty trivial and a low impact to users to remove the primate class if they didn't have the time/bandwidth to really investigate this more thoroughly as I've described.
You can try to argue "hey no one should get upset about this confusion because computer vision is hard and mistakes happen", but I don't think that's a very solid argument either given the long and relatively recent history of racists misclassifying a particular subset of humans as primates.
One of the big issues with relying on AI is training data, and the responsibility for an inadequately trained AI model falls on the people who judged it ready for use, but that doesn't mean the outcome of the AI isn't racist. An AI doesn't need to know the sociopolitical history of race to be doing racist things.
Also, I've never heard of a location with Roman numerals in the name. I agree that programming in the context of knowing who Malcolm X is into a map program might be asking a bit much, but I think an "easier" implementation (reading them as letters) would have also done the right thing.
They were careful, otherwise they wouldn't have labeled it "primates", but e.g. "monkeys" instead.
Dead Comment
https://www.ecupatria.org/2020/11/24/the-primate-of-church-o...
https://en.wikipedia.org/wiki/Primate_(bishop)
This is a very obvious example of how bad it can be. Regular complaints from people about ai recommendation engines and most other similar ai driven tagging systems exist.
What i wonder though is on a more subtle level, how these ai categorizers are quietly altering our perceptions of things and the world?
We notice when it's something obvious like the story in the OP, but how many quiet little mistakes do we just ignore or not notice that on a subconcious level are changing the way we think about or classify things.
More and more it seems like society and all things are being neatly placed in little boxes, where they're filed, stamped, indexed, and numbered by fancy algorithms operating without much human oversight, until something like this happens.
How much of this is subconciously affecting human perception of the world?
Huh. All the other incidents the NY Times article mentions come with links, but this one (that the article is ostensibly about) doesn't. Odd.
I'm guessing it ingested the Daily Mail's comment sections.
On the other hand, I think there's a semantic difference between your two cases. I'd argue that treating a woman as an "object" is a different sense of the word to how women (and men and teapots) are literally "objects".
Dead Comment