I had this idea nearly 10 years ago when my wife and I started dating and she introduced me to (very amateur) birding. I even talked about contacting Cornell for their database. But it was always going to be a hobby project.
So when she read about this a few weeks ago, she literally smacked me for not building it. Now if I tell her it's on the front page of Hacker News AND everyone here loves the idea even more, I'm going to get another, harder smack because she knows how HN is full of others like me!
(yes I'm aware that this community more than others will agree that ideas by themselves are a dime a dozen, but nonetheless, it would've been a really fun project)
I had this idea as well, but as you start to build the app you quickly realize that each bird doesn't just have one sound, but many sounds and trying to do this accurately takes much more effort than you're probably expecting.
Download any of the existing bird apps that help you recognize birds by their sound and you'll see that each bird often has 3-4 distinct sounds, each of them different, and these are just partial examples.
You'd have to be extremely dedicated (and very good at machine learning) to see this idea through to completion.
For the 400+ sounds that Merlin (the user-friendly, offline-functional sister app to BirdNET) can now identify with 80-90% accuracy, it took a team of dozens of bird sound ID experts several years to annotate tens of thousands of individual audio spectrograms. On average, they needed about 1,000 recordings per species). Here is a bit more about how they did it: https://www.macaulaylibrary.org/2021/06/22/behind-the-scenes...
As I understand it there are even sub-dialects for each of these bird calls/songs. The example shared with me was compared to the English language in the U.S. where you would expect to hear stark differences in the southern states vs the New England area with a greeting like: "hey ya'll".
Unfortunately/fortunately I can't get the visual out of my head of a southern speaking crow looking for trash near my house now.....
Plus, so many birds will improvise their songs, pulling parts of other birds' songs into their songs. And many will even outright imitate other birds (not only parrots, but corvids and some others). Many amateur birders like me often cannot tell imitations from the real ones.
This is actually a pretty difficult problem. I have been trying to learn bird songs for a very long time and it is pretty difficult. Saying that a bird has 3-4 bird sounds is an extreme simplification.
> I had this idea as well, but as you start to build the app you quickly realize that each bird doesn't just have one sound, but many sounds and trying to do this accurately takes much more effort than you're probably expecting.
Effort? Not really so much (stand on the shoulders of giants, etc.)
You can build the next generation bird app. This one requires uploading to the cloud to recognize bird calls.
What if you used the AI hardware in the phone to do the audio recognition?
For efficiency you could even use geolocation and figure out which species are found in a location and download a model just for those. Anything not matched could be uploaded as before.
I work at the Cornell Lab, and we also have an app that is more consumer-oriented, and which DOESN'T require uploading the recordings to the cloud -- see Merlin Bird ID app https://merlin.allaboutbirds.org/
When the Lab’s researchers conceived of BirdNET, there were no reliable bird sound identification tools. BirdNET was built as a rapid prototype, engaging computer science students to build an app for that users to test the machine learning algorithms. BirdNET proved to be a research breakthrough and by 2020 was performing with far better accuracy than five other apps tested.
That success opened the way to apply computer vision to sound identification in the Lab’s outreach and education app, Merlin.
Merlin offers OFFLINE functionality, and multiple ways to help identify birds, including through a user describing the bird, taking a photo of the bird, and now recording a bird song or call. Merlin Bird ID is integrated with the Lab’s systems and resources, including updated taxonomy, bird information from eBird and Birds of the World, rich media from the Macaulay Library, life list building tools integrated with eBird, and more.
I asked a veterinarian friend who specializes in birds (and is a long time bird watcher), and she knew the app and said that it works pretty well (coming from her, that's definitely high praise)
The only thing she'd improve was exactly what you mentioned: offline recognition! (and also keeping the bird sound recordings to export them later)
How about doing the reverse? Listening to birds from different parts of the world on-demand?
Listening to bird sounds is now recognized to have positive impact on mental health[1], So how about selecting a particular region on the earth and listing to high quality bird sounds? There some good YT playlists[2] but a separate service could be more functional, Tie up with bird zoos to do it live, share a piece of revenue for conservation and you'll have my subscription.
My mom is a master naturalist and has listened to (hours of) frog field recordings to determine which types of frogs are at specific locations for our department of natural resources (IIRC). There is a paper that describes how to calculate minimum adult population from audial surveys. If you still need to scratch that itch, I'm sure there are still some interesting applications along these lines.
I also met a fellow student who was working on this too years ago. Not sure how far e got. to be honest I think he was just to reluctant to graduate and get a real job so kept finding ways to stay at uni.
That's also what I would say "there's no money to be made here – it would be a fun hobby project and I could learn some machine learning but that's about it"
This is exactly the kind of thing that keeps me coming back to HN year after year. 99% may be a quick, mildly entertaining read, but that 1% tends to be empowering or life changing for me. I've had a continuously growing interest in plant and bird identification as a hobby (animals are a bit easier). I've gone so far as to research apps, put Audobon society books on my wishlist, and try to look up some specimens I see in my area. Unfortunately it's, frankly, a steep learning curve and not a habit yet for me to take pictures, remember to look at them later, search their characteristics, etc. This will be the perfect tool to help me jumpstart my newfound interest and get more familiar with the flora and fauna around my home.
I know it's weird to mention when you are talking about all these fancy tools, but I was surprised how often I get decent results when simply taking a photo and using Yandex image-search. First time I did it I didn't even expect anything but random pictures of grass, but it's actually good and it is even sort of helpful that I can see similar, but different plants in the same search (which is how I found out that apparently I was mistaking for its close relative a plant that I'm used to putting into my tea from the very childhood: I just called it what my grand-dad thought it was). For well-known decorative plants it often straight up shows the exact plant variety I'm looking for.
iNaturalist is another good option for plant identification (and other forms of life as well). It has decent machine learning for suggesting IDs and is also backed by its community providing IDs. I've been using it a lot this year and have found it pretty helpful for IDs (though some regions and life forms are more likely to have good identifications than others), as well as for showing me what people are seeing in the area, and feeling like I'm contributing useful data.
You can find some interesting starting points at kaggle.com if you want e.g. a large set of photos of agricultural plants labeled as healthy or diseased (allowing you to immediately start in on building a classification model without all the upfront grunt work).
The original version [1] is Theano based, but the newest one is TF-Lite based [2], probably for supporting mobile.
Unfortunately they don't publish the code of TF version and only a TF-Lite model is available. Probably that doesn't matter for the exports though since the paper and original version are both there.
More interesting thing is that they've been making the dataset available [3] for $20 (even before BirdNet). This can be great source for training your own bird-net like.
BirdNet is great and has helped me to identify some birdsong. If you are also interested in identifying plants, PlantNet is good too. https://identify.plantnet.org/
Any comparison of PlantNet to iNaturalist[0] (regarding quality of the product / size of the community)? I use iNat frequently for identifying native plants and animals around the yard and it's been extremely helpful and active (I typically get at least one verification on each item I post, oftentimes two or three).
The local app here for native plants (Obsidentify) beats PlantNet for me, but I have the impression there might be some 'user experience' into play: I know enough about plants to know what the distinctive features are and I know enough about the app/AI to know that it wants properly cropped pictures with those features, and other users whos observations have the most chance of being validated and accepted do so as well. So the thing is probably very well trained for that, which is less the case for PlantNet. Again, that is just a theory, but when talking with other people the story is similar: the people saying it doesn't work for them are typically uploading non-cropped and/or non-identifying pictures.
Try talking to BirdNet, it'll tell you species = homo sapiens. Most plant recognition apps will stil try to match a plant when feeding it pictures of humans.
These new ML image identifiers are neat and they're sure to become the default method, but I feel like a "20 questions" style of tool could have worked very well for a long time with little technology required.
Just ask a series of questions:
- Is it a plant or animal?
- How large is it?
- Where are you (location permission)
- Does it have green leaves?
- Does it have woody stems?
- Does it have whorled or alternate leaves? (Show images)
- Does it have yellow flowers?
This is how a lot of field guide books work, but they require a lot of flipping back and forth, scanning tables of contents, and memorizing terminology. An app could just be tap-tap-tap-tap-tap, drilling down very quickly and showing images at every step.
The "only" difficulty is getting the database of attributes - maybe it already exists. Maybe an app like this already exists and I haven't been able to find it?
Having used such books, I don't think it is that simple. I mean you could make an app just like the books but it might not add any functionality and as such not be usable for the wide public.
Thing is, the series your questions you show are easy to answer but they don't get you any further than halfway. Problem is in the questions you do not show: there it starts going into details, terminolgy starts to matter (seriously, if you've never read those words it's like a foreign language) and differences become hard to spot. Part of this could perhaps be alleviated with pictures but I doubt it; I've seen websites attempt it but none were really good and they all did a subset of plants. Probably because it's quite the amount of work to do them all, even for a region.
Yep! I just left a comment about having this idea 10 years ago and my wife being annoyed that I didn't build it. But hard to explain "I would've literally received a PhD working on this" when my interest was mostly on the hobby side.
I am very proud to say that I was successfully able to get "Northern Cardinal: Cardinalis cardinalis - Almost certain" by whistle-copying the cardinal that lives in my backyard.
I've been able to spot over 50 types of birds (including Great Hornbill, Rufous Necked Hornbill, Greater Flameback) in Goa this monsoon with the help of this app (it discovered over 120 unique species, but they were hard to spot), even though it was not made for South Asia
Apart from the usefulness of the app, interface and usability is great too
I have the app and do like it, but my phone's camera quality doesn't seem good enough for this to work properly (iPhone 6). I have to get pretty lucky and be able to be super close to the bird so that very little zooming or image enlargement is needed.
Readit News
So when she read about this a few weeks ago, she literally smacked me for not building it. Now if I tell her it's on the front page of Hacker News AND everyone here loves the idea even more, I'm going to get another, harder smack because she knows how HN is full of others like me!
(yes I'm aware that this community more than others will agree that ideas by themselves are a dime a dozen, but nonetheless, it would've been a really fun project)
Download any of the existing bird apps that help you recognize birds by their sound and you'll see that each bird often has 3-4 distinct sounds, each of them different, and these are just partial examples.
You'd have to be extremely dedicated (and very good at machine learning) to see this idea through to completion.
Because in nature, it's almost never just one bird you hear. You hear it in the context of all the other sounds.
Unfortunately/fortunately I can't get the visual out of my head of a southern speaking crow looking for trash near my house now.....
Effort? Not really so much (stand on the shoulders of giants, etc.)
Data? Yes. Lots and lots of data.
Deleted Comment
Deleted Comment
https://www.appbrain.com/app/birdnet-bird-sound-identificati...
I think I had someone in my class (about 10 years ago) attempting this as a project.
What if you used the AI hardware in the phone to do the audio recognition?
For efficiency you could even use geolocation and figure out which species are found in a location and download a model just for those. Anything not matched could be uploaded as before.
When the Lab’s researchers conceived of BirdNET, there were no reliable bird sound identification tools. BirdNET was built as a rapid prototype, engaging computer science students to build an app for that users to test the machine learning algorithms. BirdNET proved to be a research breakthrough and by 2020 was performing with far better accuracy than five other apps tested.
That success opened the way to apply computer vision to sound identification in the Lab’s outreach and education app, Merlin.
Merlin offers OFFLINE functionality, and multiple ways to help identify birds, including through a user describing the bird, taking a photo of the bird, and now recording a bird song or call. Merlin Bird ID is integrated with the Lab’s systems and resources, including updated taxonomy, bird information from eBird and Birds of the World, rich media from the Macaulay Library, life list building tools integrated with eBird, and more.
The only thing she'd improve was exactly what you mentioned: offline recognition! (and also keeping the bird sound recordings to export them later)
Listening to bird sounds is now recognized to have positive impact on mental health[1], So how about selecting a particular region on the earth and listing to high quality bird sounds? There some good YT playlists[2] but a separate service could be more functional, Tie up with bird zoos to do it live, share a piece of revenue for conservation and you'll have my subscription.
[1] https://www.nhm.ac.uk/discover/how-listening-to-bird-song-ca...
[2] https://www.youtube.com/watch?v=_EaSSgqsQs4
Deleted Comment
Unfortunately they don't publish the code of TF version and only a TF-Lite model is available. Probably that doesn't matter for the exports though since the paper and original version are both there.
More interesting thing is that they've been making the dataset available [3] for $20 (even before BirdNet). This can be great source for training your own bird-net like.
- [1] https://github.com/kahst/BirdNET
- [2] https://github.com/kahst/BirdNET-Lite
- [3] https://www.macaulaylibrary.org/product/the-cornell-guide-to...
[0] https://www.inaturalist.org/
Just ask a series of questions:
- Is it a plant or animal?
- How large is it?
- Where are you (location permission)
- Does it have green leaves?
- Does it have woody stems?
- Does it have whorled or alternate leaves? (Show images)
- Does it have yellow flowers?
This is how a lot of field guide books work, but they require a lot of flipping back and forth, scanning tables of contents, and memorizing terminology. An app could just be tap-tap-tap-tap-tap, drilling down very quickly and showing images at every step.
The "only" difficulty is getting the database of attributes - maybe it already exists. Maybe an app like this already exists and I haven't been able to find it?
Thing is, the series your questions you show are easy to answer but they don't get you any further than halfway. Problem is in the questions you do not show: there it starts going into details, terminolgy starts to matter (seriously, if you've never read those words it's like a foreign language) and differences become hard to spot. Part of this could perhaps be alleviated with pictures but I doubt it; I've seen websites attempt it but none were really good and they all did a subset of plants. Probably because it's quite the amount of work to do them all, even for a region.
Apart from the usefulness of the app, interface and usability is great too
https://apps.apple.com/us/app/merlin-bird-id-by-cornell-lab/...
Still, nice app as a field guide.