> While the study did not use the exact software companies like Tesla use to power self-driving cars because they are confidential, the software systems used for the study are based on the same open-source AI those companies use, according to Zhang.
I was under the impression that commercial self-driving software was deeply proprietary and confidential, and there is no way to know that this study will generalize if run on state of the art detectors. Tesla and Cruise are name-checked in the article - how do we know this isn’t a problem they have worked extensively on and made great improvements to, relative to the open source components?
The BI article is definitely outrage for clicks. I wouldn't be surprised if the actual journal article was more measured in its conclusions and this is just typical bad science reporting.
Presumably these companies are free to provide their software for research. The onus is on them to demonstrate it works in the first place....
> how do we know this isn’t a problem they have worked extensively on and made great improvements to, relative to the open source components?
They are a private, for-profit entity with a strong incentive to mislead people about their products. I see no reason to assume they've addressed this issue.
The point of the article, to me, is computer vision will never be enough. These are machines and need to be augmented with radar and other object detection methods.
Radar etc. almost certainly make it easier, but for it to be "never" this would also have to be provably impossible for humans.
Which isn't too say it's not never, as I remember studies in my own childhood that said human drivers were also bad at recognising how far away children were, and I've never heard of human perception of skin colour being tested in this way so it might just turn out that melanin is unfortunately good camouflage against tarmac…
…but unless and until that suggestion turns out to be correct of all humans, I default to assuming we're an existence proof of the capability to do without, and that means I still wouldn't say "never" to sufficiently advanced AI doing at least as well.
It’s intended to replace human drivers who have not yet evolved radar. I agree that radar could make it easier and/or more reliable, but there’s a pretty strong argument that building a system to equal/exceed humans using vision alone is possible.
Weird, you articulated the exact point of bias in AI but the tone you used is dismissing.
Yes obviously AI is not a moral agent and it isn't racist per se.
But if it's input is biased and the test is biased then the application will be biased. That's a problem, if you go and deploy these models where their training data is lacking.
Let's say, self driving car using the AI you described deployed in Uttar Pradesh is less safe because of bias.
What is wrong with this statement in your opinion?
Human driver eyes (and I suspect any other optical systems working in the visible color range) are also less likely to detect people of color. Five years ago I avoided running over a pedestrian at night only by luck: he was black, wearing a black jacket, black pants, walking across a badly-lit suburban street; I think that either my visual system did not perceive him at all until the last fraction of a second, or perhaps perceived him as a shadow. I managed to swerve. But a fraction of a second later? I am afraid to think about it...
I am a big fan of Scandinavian style pedestrian safety reflectors. Attach one to your bag or jacket if you are walking late at night; it might save your life. But if you don't have a reflector, wear at least one piece of bright, light-colored clothing; this is particularly important it your skin color is dark!
Regardless of race, it shouldnt be on the road if it cannot detect jaywalkers at night wearing all black. Where I live POC are more likely to be the pedestrians because it's an immigrant-heavy community. I've seen a white guy jaywalking at night wearing all black in between street lights but I couldn't tell he was white until he got across the street and I could see him head-on.
> The detection systems were 19.67% more likely to detect adults than children, and 7.52% more likely to detect people with lighter skin tones than people with darker skin tones, according to the study.
while they all had a harder time with adults vs children, that 7.52% is gotten by averaging 2 algorithms that performed abysmally, with 6 that had no statistically significant differences
The two with significantly worse performance were RetinaNet and YOLOX. I don't really know anything about the field, but it's interesting they're both single stage performant models, while the slower but lower miss-rate RCNN variants are two-stage. It's interesting that the pedestrian-specific models are all worse than the general models at detecting people!
The conclusion is kind of weird: apparently their "findings
reveal significant bias in the current pedestrian detectors" despite the bias being almost entirely within the single-pass general object detectors. And where it's statistically significant in the other models, the miss rate is low in both cases, and the effect is reversed! (Dry-weather Cascade-RCNN does better on dark-skin than light-skin, among others.)
I think you misunderstood table 6. All algorithms show significant differences in miss rate for children, two show significant differences based on gender, and four others based on skin color. The four that showed no statistically significant difference between light and dark skin had very high miss rates overall. Of the other four, two are much worse for dark skin, and two are slightly better. Those last two are also best at detecting children, but 28% miss rate is still a bit too high for my taste.
Yeah I missed the two statistically significant algorithms that favor darker skin since they have smaller percentage differences than the ones they didn't mark as statistically significant (but I guess that's because of how it relates to the overall miss rate)
RE: 28% miss rate, I think this is meaningless as it's looking at single images/data points, while self driving cars get a continuous stream of data
Are these pedestrian detection models in use in any widely-deployed commercial self-driving car? Is there a limitation since these are images rather than videos? I would've expected these to be addressed in the "Threats to Validity." There is also no control comparison to humans, beyond the two annotators. Are these detectors significantly worse than humans?
There is telling whether these results are valid or applicable at all, but they purport that there are statistically significant unfairness based on gender and skin color. At best, this feels misleading.
How do they work in winter then? You can't see much skin if someone is wearing a winter coat.
Right - self driving cars are a solution for Silicon Valley only only so they don't even bother testing those cars elsewhere.
The skin color is going to be the least of your issues in the winter time. How are the cars going to "see" the road under 10cm of snow? Granted humans shouldn't really be driving in those conditions either, but we do and mostly successfully, to avoid sleeping on the side of the road in -10C.
What is it that makes it so hard for all these algorithms to work on people with darker skin? This has been an issue for more than ten years, surely someone has started adding various skin colors into the training data. Is it a case of lack of training material, or is it just faster to focus on one skin type?
My employer makes a multi-band synthetic aperture radar that penetrate snow and is high-resolution enough to "see" painted road markings and reflectors beneath a layer of snow.
It is nowhere near small or cheap enough for self-driving car applications, but will be one day.
Another challenge is affordable real-time processing of the data. Churning through 3,200MB/s of phase-history data is expensive but again that will solve itself given time.
> Surely someone has started adding various skin colors into the training data.
What has to occur for this to happen:
* Someone has to take the time and effort to measure things, to identify that there is a problem.
* They have to get that message out so that it's heard.
* That message needs to:
* hit the public hard enough that people demand intervention from their elected representatives
* or, alert the company directly, and hope that the incentives align. (Will the company make more money by fixing this?)
There's plenty of easier alternatives:
* Call the problem too hard to solve
* Call it bad science
* Call it ragebait
* Call it woke
* Make up a bunch of equivalences and channel it into inertia:
* If people are wearing winter coats, then they won't show enough skin for the cars to be racist. And if the cars aren't racist in cold places, then it isn't a problem in warm places.
* People don't have radar/lidar either, and they're allowed to drive
Yup - as someone from a country where -10 degrees is considered a normal winter day I think that self driving cars should not rely on road markings at all. Even road signs can be unreadable after a particularly nasty blizzard.
I've yet to see self driving cars successfully navigating during bad winter conditions. They can't even avoid killing pedestrians in California.
> A team of researchers in the UK and China tested how well eight popular pedestrian detectors worked depending on a person's race, gender, and age.
- edit -
Sorry, I read the article too quickly and assumed it was talking about the countries UK and China. Perhaps they only bothered testing the cards in UK, Silicon Valley and China, Silicon Valley.
"A new technology reduces mortality risk for all people, but has slightly better outcomes for white adults."
Conclusion - we call on lawmakers to make this technology illegal. We prefer more people die at equal rates more than we prefer less people to die at unequal rates.
I am not sure I agree with the ethics that underlies this way of seeing the world.
What if the technology was only/primarily tested with white people, rather than inherently having better outcomes for white adults? It's not really as clear cut as you make it out to be, technologies at this level of complexity aren't just derived from physical/biological principles. Perhaps there was a better variant of the technology that was scrapped as a cost-cutting measure, because it performed the same for white adults, but better for other classes of people (alluding to radar here, though I'm not sure it really performed the same, but I'm trying to make a larger point than any specific technology anyway).
Fair point. Cost benefit analysis for adoption of safety features in automobiles is inconsistent.
Drunk drivers kill more Americans each few months than terrorists on planes have in the last 25 years. Yet every airline passenger must prove they aren't a terrorist but no one driving a car has a default presumption they are drunk. Unless you've been convicted previously, then maybe sometimes.
I wouldn't be surprised if a better model exists for object detection and we aren't using it to save pennies. Politics and ethics in automobile safety is asinine. Fair point.
> What if the technology was only/primarily tested with white people, rather than inherently having better outcomes for white adults?
I think we have a precedence for that in testing of drugs. The majority of drugs are primarily tested on white men, meaning that their effect and dosages may be problematic for women or people of color.
There's also the issue of the majority of tools being designed for right handed people and any left handed either needs to spend more on tools or accept a certain risk when operating a chainsaw.
It should be illegal, the current systems will never deliver without additional object detection methods (radar, lidar, etc). Get this shit recalled and back to the drawing board.
I wouldn't say 'never'. But the right thing to do is to keep the training wheels (radar, lidar) until it's proven that the system has learned to drive without them when there's good visibility and to reduce speed (or stop, if necessary) when the visibility is reduced. This also includes learning to recognize almost every possible situation, including zero shot events.
In other news, it turns out that detecting smaller and lower contrast objects is harding with optical sensors. Almost, you know, like how it is with real people.
It won't be long till they won't just be detecting people, by identifying specifically which people they are. Think of the data! All those cars logging time/location of all those people. (which, of course, will only be used for good and the occasional targeted ad)
Hey Buddy, you realize things improve, right? Especially novel software that has currently improved tremendously in the past 10 years. Are you actually suggesting that it will suddenly stop improving today? Weird.
I think what you're trying to say is cars will start identifying folks and logging their location. Cell tracking/triangulation data is already available from cell companies. Self driving cars aren't going to make things worse.
Readit News
I was under the impression that commercial self-driving software was deeply proprietary and confidential, and there is no way to know that this study will generalize if run on state of the art detectors. Tesla and Cruise are name-checked in the article - how do we know this isn’t a problem they have worked extensively on and made great improvements to, relative to the open source components?
Feels like a case of outrage-for-clicks.
The BI article is definitely outrage for clicks. I wouldn't be surprised if the actual journal article was more measured in its conclusions and this is just typical bad science reporting.
> how do we know this isn’t a problem they have worked extensively on and made great improvements to, relative to the open source components?
They are a private, for-profit entity with a strong incentive to mislead people about their products. I see no reason to assume they've addressed this issue.
Which isn't too say it's not never, as I remember studies in my own childhood that said human drivers were also bad at recognising how far away children were, and I've never heard of human perception of skin colour being tested in this way so it might just turn out that melanin is unfortunately good camouflage against tarmac…
…but unless and until that suggestion turns out to be correct of all humans, I default to assuming we're an existence proof of the capability to do without, and that means I still wouldn't say "never" to sufficiently advanced AI doing at least as well.
Like 99% of these “AI discrimination” articles.
>human-detecting AI is developed in a western country with ~60% white population. Most of the training data is collected there
>the AI performed slightly worse in Uttar Pradesh, where the people and everything else in the background look different
>AI is prejudiced! Get outraged!
Every time.
What is wrong with this statement in your opinion?
I am a big fan of Scandinavian style pedestrian safety reflectors. Attach one to your bag or jacket if you are walking late at night; it might save your life. But if you don't have a reflector, wear at least one piece of bright, light-colored clothing; this is particularly important it your skin color is dark!
Some saying: racists are people that are thinking of race, talking about race, and acting based upon race.
> The detection systems were 19.67% more likely to detect adults than children, and 7.52% more likely to detect people with lighter skin tones than people with darker skin tones, according to the study.
while they all had a harder time with adults vs children, that 7.52% is gotten by averaging 2 algorithms that performed abysmally, with 6 that had no statistically significant differences
https://arxiv.org/pdf/2308.02935.pdf table 6
The conclusion is kind of weird: apparently their "findings reveal significant bias in the current pedestrian detectors" despite the bias being almost entirely within the single-pass general object detectors. And where it's statistically significant in the other models, the miss rate is low in both cases, and the effect is reversed! (Dry-weather Cascade-RCNN does better on dark-skin than light-skin, among others.)
RE: 28% miss rate, I think this is meaningless as it's looking at single images/data points, while self driving cars get a continuous stream of data
There is telling whether these results are valid or applicable at all, but they purport that there are statistically significant unfairness based on gender and skin color. At best, this feels misleading.
What is it that makes it so hard for all these algorithms to work on people with darker skin? This has been an issue for more than ten years, surely someone has started adding various skin colors into the training data. Is it a case of lack of training material, or is it just faster to focus on one skin type?
It is nowhere near small or cheap enough for self-driving car applications, but will be one day.
Another challenge is affordable real-time processing of the data. Churning through 3,200MB/s of phase-history data is expensive but again that will solve itself given time.
Yes
> Surely someone has started adding various skin colors into the training data.
What has to occur for this to happen:
There's plenty of easier alternatives:I've yet to see self driving cars successfully navigating during bad winter conditions. They can't even avoid killing pedestrians in California.
- edit -
Sorry, I read the article too quickly and assumed it was talking about the countries UK and China. Perhaps they only bothered testing the cards in UK, Silicon Valley and China, Silicon Valley.
From what I can see, a couple of the detectors used really seem shit overall, making the combined data of questionable value.
Conclusion - we call on lawmakers to make this technology illegal. We prefer more people die at equal rates more than we prefer less people to die at unequal rates.
I am not sure I agree with the ethics that underlies this way of seeing the world.
Drunk drivers kill more Americans each few months than terrorists on planes have in the last 25 years. Yet every airline passenger must prove they aren't a terrorist but no one driving a car has a default presumption they are drunk. Unless you've been convicted previously, then maybe sometimes.
I wouldn't be surprised if a better model exists for object detection and we aren't using it to save pennies. Politics and ethics in automobile safety is asinine. Fair point.
I think we have a precedence for that in testing of drugs. The majority of drugs are primarily tested on white men, meaning that their effect and dosages may be problematic for women or people of color.
There's also the issue of the majority of tools being designed for right handed people and any left handed either needs to spend more on tools or accept a certain risk when operating a chainsaw.
Dead Comment