I wish car engines had mics built in, as well as one close to each suspension.
It would record for the first 5 minutes, then at certain intervals or at certain conditions.
The car would give you FFT images which you could then look at and see how it changes over the years. No need for online-stuff, just an USB port where you stick in a big stick and software for your computer/tablet to evaluate/visualize it.
If you'd then see some problems, you could ask for the audio to be recorded with it, which you could send to a friend who knows about cars or to your car workshop.
At Heading On we had exactly this idea, plus using Canbus, to detect when cars would need maintenance. We ended up rethinking the idea because most of the people we spoke with weren't interested.
Ah damn I need this! My Skoda has squeaky brakes, except they only squeak when you aren't pressing the brake. Clearly something is rubbing but I can't see what, and it's intermittent and also impossible to Google.
I'm surprised after this many replies no one mentioned that cars do have something pretty close to this across virtually every modern ICE: knock sensors.
Knock sensors are just ruggedized piezoelectric microphones that bolt onto your engine: when they detect a knock it's by transmitting the sound a knocking cylinder makes as an electrical signal to the ECU.
When we designed our DARPA Grand Challenge vehicle back in 2003 I considered using a guitar pickup for that, to pick up vibration and get a measure of how rough the road was.
I would love something that could use audio to diagnose problems in bikes. Issues can be hard to replicate on a repair stand when the bike is not under load.
The concept you outlined is technically very feasible but economically prohibitive. There are high value situations that incorporate these techniques as standard practice. For example, the diagnosis of rolling mill fault conditions with FFT vibration monitoring has been in practice for at least 50 years. Of course when a cold strip mill goes down, the damage can be measured in megabucks.
Most people don’t care about maintenance. They do it only when they must. So I doubt they would pay a penny extra for accurate forecasts of maintenance needs. Insurers and other types of dealers might though!
But wouldn’t constantly changing road types and weather conditions create so much variable noise that trending would be sloppy?
True, but people who buy expensive cars care, which is exactly the target for such a system I imagine.
I am not a mechanic but can at least hazard a guess at what's wrong with a car from the way it sounds with reasonable confidence. I imagine a purpose build AI design by the car's manufacturer with multiple microphones in the engine compartment and around the car could gain quite a lot of insight, perhaps even outside of identifying maintenance problems simply to tune the performance of the engine, especially when combined with other sensors.
Having said that, we seem to be near the end of the line for ICE vehicles, so I wouldn't hold my breath for such a system to be developed.
One of the manifestations of "I don't care about maintenance" is taking a car to the dealer/whoever on a regular schedule without understanding what they need to actually do to the car -- which is not caring about maintenance by having someone else care.
Having a report that's quickly spit out to the mechanic about issues over the past X,000 miles so they can quickly quote to the customer before it becomes a bigger issue would be nice.
It's a though market to enter, someone I know suggested doing this for a company that had a fleet of 1000+ vehicles and the representative from Bosch told them they wouldn't provide them with diagnostic tools anymore if they did.
As darling of data science Andrew Ng said, "90% of machine learning is feature engineering". So you should probably run the STFT spectrograms through machine learning models rather than the raw audio signal. And anyway raw audio is over 41k samples per second so quite low information density compared to some simple and quite common transformations like STFT anyway.
I guess nowadays SMART is pretty good, though, and you can hear if your fans are dying :). Maybe for computers that don't have RPM monitoring and aren't being listened to? It would at least be interesting data to correlate with other metrics.
People might be interested in BirdWeather (https://app.birdweather.com/) which collates data from people's bird listening stations (raspberry pi's with a mic and a modest NN to classify what is heard). Basically, the technology for this has already arrived as consumer tech.
As someone who is a fairly serious volunteer birdwatcher, we could absolutely do with more of this. There are not enough bird surveyors (paid or volunteer) and audio moths hooked up to AI capable of filling in the gaps are badly needed to support conservation and re-wilding efforts.
I thought about setting up such a device outside of my window (there's a lot of birds nearby), and the obvious concern of setting up a mic in your house and uploading recordings is privacy.
I found two privacy policies related to Birdweather:
> You may choose to keep your audio recordings private, so that other Bird Weather users cannot access or listen to those recordings. This is configured on the Settings page.
Except Birdweather explicitly mentions they're based on Cornell's BirdNET, whose privacy policy (#2) says:
> BirdNET is an artificial neural network that identifies bird species by sound. Our servers process small audio snippets recorded with the BirdNET app to detect and recognize bird sounds. We store all submitted recordings on our servers. Therefore, we advise users not to submit any audio recordings that they might consider private. The collection of audio data helps us to improve BirdNET and we will use those recordings for research purposes only.
Seems like Birdweather would benefit from clarifying how their privacy policy plays together with BirdNET's.
> Third-Party Research: We share the bird detections and audio as well as labeled data with the Cornell University Laboratory of Ornithology, so that they may use such data for scientific research and to help improve the accuracy of the bird sound detection.
You can set the audio recordings to be inaccessible for other Bird Weather users. Ie. by default audio is shared on the website, but you can turn that off. Independently, they also share the data with Cornell.
> BirdNET is an artificial neural network that identifies bird species by sound.
A while ago there was an HN thread raving about how Seek by iNaturalist was fun (true!) and did such a good job of identifying whatever it was you were looking at (maybe!) and not overstepping the bounds of what it could know for sure. (See below!)
So I set up an account and I tried it out.
It's pretty clear that Seek places far more weight on giving you a full species-level identification than it does on whether it can be confident that that identification is correct. I guess it's possible that the stand of groundcover I found on the shore of an artificial lake in a public park is a different species from the stand of groundcover with identical coloration and shape about 12 inches away... but I doubt it. Even if they were different species, I'm pretty sure my low-resolution images of two stands of plants with no flowers or seeds wouldn't be enough for an expert to tell them apart.
In a parallel occurrence, I took a photo of a local beetle that Seek identified as "Strawberry seed beetle", harpalus rufipes. I uploaded that to iNaturalist proper (labeled "beetles"; I already didn't trust the identification) and checked out some nearby photos. A very similar-looking beetle had been photographed in my area and identified as harpalus sinicus. So I left a comment asking how the tagger could tell that this was sinicus and not some other kind of harpalus. And I got a response, saying "I'm not trained in entomology and I couldn't explain what the difference is. But me and my friend think this is sinicus."
I harbor a sneeking suspicion that the "friend" was Seek.
They advertise that Seek is trained on labels from iNaturalist. But those labels appear to be generated in large part by Seek. Something needs to change.
I've been using BirdNet-Pi for a while and it's great fun. I'm a little confused about the commercial offering from BirdWeather - the PUC looks like a neat product, but I understood most of the BirdNet stuff to be Creative Commons NonCommercial. Maybe there's some nuance about how they're using it, or maybe some pieces have a more lenient open source license?
Thank you for posting this. I'm going to see about setting up one of these for my mom, who is a VERY avid birder (I already have the Pi and outdoor enclosure). It looks like one slightly difficult issue is finding a good weatherproof microphone that works with the Pi.
I volunteer with Birdlife Australia. If you are in the USA I think it’s called the Audubon Society. They will have a number of surveying programs running at any given time.
It’s terrible, but this technology is likely coming to a bar/pub/cafe/store/park/trail near you… diarization, voice recognition… I’m concerned that in the near future, humans can no longer assume that their words are not being captured and potentially used for who-knows-what purposes.
It's not like recording technology is new. It's only because most countries have laws providing that tools like these are not widely used by government. And I don't believe AI is going to change the laws significantly. I don't believe most of the 1984 stuff will happen at least in near future.
Easily solved with more tech! Subaudible recording with mic on throat —> encrypted transmission to peers —> peers’ earbuds. Conversation 2.0, or something. Sarcasm, but wouldn’t be surprising if it happens eventually
Freedom and privacy are rapidly going to become a thing of the past. You won't be able to escape even if you attempt to. The world is becoming a disgusting place and I want none of it anymore.
I think it more likely that it becomes something you pay for. Ads only pay so much, and richer people (upper-middle-class and above) can afford to pay more out of pocket than ads targeted to them are worth. Or, at the very least, it's worth enough for third-party non-ad companies such as LifeLock to run interference on other companies privacy invasion tools.
There is some future where AI, deepfakes, and what not become so indistinguishable from real people, that those industries (news media, social-media, etc) that make sport out of hearing "wrong speak" and performing shaming rituals will become obsolete.
Same with the security state, if all the data is suspect, does it matter?
Last time I was sitting in the garden, I was wondering how far we are with actualy understanding the bird chirp and tweets. I felt like there were certain patterns I could hear repeatedly in certain situations (danger approach warning, feeding communication, always the same sound before one bird flew out and collected food kind of like "make us breakfast" :P).
Unfortunately some quick searches on Google scholar didn't help much, I feel like I'm lacking the right search terms here. Most ML in this area seems to be related to species classification. If there's any bird experts reading, what is the state of the art of "understadning" bird language? Any paper recommendations? I'm curious about basic things like...is there a universal bird language or does each species use their own language (can a crow understand a sparrow), are there some sounds that are identified with meaning etc.
Edit: just checked my bookmarks and "Bird-DB: A database for annotated bird song sequences" was the most interesting find, along with some interesting datasets (NIPS, Cornell Birdcall).
I was envisioning a ML powered device that could translate bird chirps to human readable text but my guess is that we're still in the fundamental research phase. One idea I had was recording stuff in my garden and running it through some unsupervised algorithms to identify some patterns than match it with video and maybe tag context. Would be really neat to find a pattern and see it correspond to the same situation (cat approaching). Seems like a neat summer project but I'm not even sure how to tag and label things :)
A first version that identifies the individual birds and their species would already be cool and I suppose feasible. Bird A (sparrow) speaking...bird B speaking etc. would already be fun...might give them random names, too instead of A,B,C. That would also help with the next stage for tagging etc.
I recently saw Nathan Pieplow's talk The Language of Birds. It was a pretty good intro to the kind of things we know about bird communication (a fair amount) and what we don't know (a lot).
I use the Merlin app which is pretty accurate at identifying birds by sound. Once it hits on a bird it will give you a list of different calls that bird makes regarding what it might be trying to do (aka mating, predator near) so you can match it further but that part is not automated yet.
I've been living in southern Mexico in the jungle for the last year or so and this app has been a blast. Whenever we hear something new we all whip our our phones and learn something.
Not necessarily true for (b): quiet inhabitants may be ratted out by other inhabitants. Birds will literally call out "snake" (in their bird language) or "puma" to alert others.
Readit News
It would record for the first 5 minutes, then at certain intervals or at certain conditions.
The car would give you FFT images which you could then look at and see how it changes over the years. No need for online-stuff, just an USB port where you stick in a big stick and software for your computer/tablet to evaluate/visualize it.
If you'd then see some problems, you could ask for the audio to be recorded with it, which you could send to a friend who knows about cars or to your car workshop.
Acoustic sensors in the car: https://v2minc.com/#solution
Skoda made s sound analyzer mobile app 3 years ago: https://beebom.com/this-ai-app-detects-internal-issues-in-ca...
I'm skeptical it would actually work well though.
Knock sensors are just ruggedized piezoelectric microphones that bolt onto your engine: when they detect a knock it's by transmitting the sound a knocking cylinder makes as an electrical signal to the ECU.
..
But wouldn’t constantly changing road types and weather conditions create so much variable noise that trending would be sloppy?
I am not a mechanic but can at least hazard a guess at what's wrong with a car from the way it sounds with reasonable confidence. I imagine a purpose build AI design by the car's manufacturer with multiple microphones in the engine compartment and around the car could gain quite a lot of insight, perhaps even outside of identifying maintenance problems simply to tune the performance of the engine, especially when combined with other sensors.
Having said that, we seem to be near the end of the line for ICE vehicles, so I wouldn't hold my breath for such a system to be developed.
Having a report that's quickly spit out to the mechanic about issues over the past X,000 miles so they can quickly quote to the customer before it becomes a bigger issue would be nice.
Good idea, though AI is much better at doing that in 90% of situations. So filter it thru AI first
I guess nowadays SMART is pretty good, though, and you can hear if your fans are dying :). Maybe for computers that don't have RPM monitoring and aren't being listened to? It would at least be interesting data to correlate with other metrics.
As someone who is a fairly serious volunteer birdwatcher, we could absolutely do with more of this. There are not enough bird surveyors (paid or volunteer) and audio moths hooked up to AI capable of filling in the gaps are badly needed to support conservation and re-wilding efforts.
I found two privacy policies related to Birdweather:
1. https://www.birdweather.com/privacy
2. https://birdnet.cornell.edu/privacy-policy/
The first one mentions the following:
> You may choose to keep your audio recordings private, so that other Bird Weather users cannot access or listen to those recordings. This is configured on the Settings page.
Except Birdweather explicitly mentions they're based on Cornell's BirdNET, whose privacy policy (#2) says:
> BirdNET is an artificial neural network that identifies bird species by sound. Our servers process small audio snippets recorded with the BirdNET app to detect and recognize bird sounds. We store all submitted recordings on our servers. Therefore, we advise users not to submit any audio recordings that they might consider private. The collection of audio data helps us to improve BirdNET and we will use those recordings for research purposes only.
Seems like Birdweather would benefit from clarifying how their privacy policy plays together with BirdNET's.
> Third-Party Research: We share the bird detections and audio as well as labeled data with the Cornell University Laboratory of Ornithology, so that they may use such data for scientific research and to help improve the accuracy of the bird sound detection.
You can set the audio recordings to be inaccessible for other Bird Weather users. Ie. by default audio is shared on the website, but you can turn that off. Independently, they also share the data with Cornell.
A while ago there was an HN thread raving about how Seek by iNaturalist was fun (true!) and did such a good job of identifying whatever it was you were looking at (maybe!) and not overstepping the bounds of what it could know for sure. (See below!)
So I set up an account and I tried it out.
It's pretty clear that Seek places far more weight on giving you a full species-level identification than it does on whether it can be confident that that identification is correct. I guess it's possible that the stand of groundcover I found on the shore of an artificial lake in a public park is a different species from the stand of groundcover with identical coloration and shape about 12 inches away... but I doubt it. Even if they were different species, I'm pretty sure my low-resolution images of two stands of plants with no flowers or seeds wouldn't be enough for an expert to tell them apart.
In a parallel occurrence, I took a photo of a local beetle that Seek identified as "Strawberry seed beetle", harpalus rufipes. I uploaded that to iNaturalist proper (labeled "beetles"; I already didn't trust the identification) and checked out some nearby photos. A very similar-looking beetle had been photographed in my area and identified as harpalus sinicus. So I left a comment asking how the tagger could tell that this was sinicus and not some other kind of harpalus. And I got a response, saying "I'm not trained in entomology and I couldn't explain what the difference is. But me and my friend think this is sinicus."
I harbor a sneeking suspicion that the "friend" was Seek.
They advertise that Seek is trained on labels from iNaturalist. But those labels appear to be generated in large part by Seek. Something needs to change.
Dead Comment
The commons, Alexafied.
Deleted Comment
Warning: Potential dissident behavior. Monitoring level upgraded to TIER-3. Recommendation: Dispatch surveillance drones for real-time observation.
Same with the security state, if all the data is suspect, does it matter?
I was envisioning a ML powered device that could translate bird chirps to human readable text but my guess is that we're still in the fundamental research phase. One idea I had was recording stuff in my garden and running it through some unsupervised algorithms to identify some patterns than match it with video and maybe tag context. Would be really neat to find a pattern and see it correspond to the same situation (cat approaching). Seems like a neat summer project but I'm not even sure how to tag and label things :)
A first version that identifies the individual birds and their species would already be cool and I suppose feasible. Bird A (sparrow) speaking...bird B speaking etc. would already be fun...might give them random names, too instead of A,B,C. That would also help with the next stage for tagging etc.
https://youtu.be/3tUXbbbMhvk?si=6ESL-zuPIHXFULbC
https://earbirding.com/blog/talks
Maybe you can find a recording somewhere or further references.
The whole concept of birding by ear is pretty cool. We always have the Merlin bird sound AI app at the ready whenever we're hiking.
https://youtu.be/IO04p2qM5Y0?si=WY6MzH2c9QGcOHsf
https://www.youtube.com/watch?v=By_rSYI0Ovs
"We're going to use this robot to find people inside of urban rubble... uhh.... after earthquakes of course.."
Listening means that:
(a) coverage is limited to receptor area, so proximity is required (proximity bias)
(b) the inhabitants have sufficient volume to be heard (frequency bias)
Stuff You Should Know covered it on an episode "How We're Learning to Talk to Animals": https://www.iheart.com/podcast/105-stuff-you-should-know-269...