Mick Gordon did some fun hidden spectrogram imagery in the Doom 2016 soundtrack.
He talks about that and plenty of other cool stuff in his talk at the 2017 GDC conference. One of my favorite conference talks ever, he did so much cool experimentation to get the sounds he used on the soundtrack, and watching his talk is one of those moments where you really get to see a master of his craft let loose and explain his process.
Author here. This is a basic spectrogram visualizer that's mobile friendly. It allows to select regions on the spectrogram and play them separately. There is no grand plan behind this web app: it's just a handy basic tool to capture sounds on your phone and see what they look like.
Your spectrogram looks elongated horizontally because the FFT window size is too large. I use window size 1024 with sample rate 48000 Hz, so one window covers 1024/48000=0.02 sec. This window size looks optimal in most cases: if you change it in my web app, you'll see that all other window sizes get the spectrogram blurry in different ways, but at 1024 it gets into focus.
Of course, don't forget the window function (Hann, or raised cosine), but it looks like you've got that covered because your spectrogram looks smooth.
The color palette looks good in your case. FWIW, my color function is like this: pow(fft_amp, 1.5) * rgb(9, 3, 1). The pow() part brightens the low/quiet amplitudes, and the (9,3,1) multiplier displays 10x wider amp range by mapping it to a visually long black->orange->yellow->white range of colors. Note, that I don't do log10 mapping of the amplitudes.
It uses "audio/webm;codecs=opus" to record mic. Now it's possible to change it in the config menu in the top right. Safari probably needs audio/mp3. Edit: also consider "audio/foo;codecs=pcm" where "foo" is something compatible with Safari.
Very neat! May I suggest adding a button to switch to log scale for frequency? I love the ability to select and play back just a particular set of frequencies. But voice uses only about ~15% of the screen height [1], so it's hard to play with.
The WebAudio API has an anlayser node that can create spectrogramms in real-time. The ones I've created in the past were nowhere near as detailed as this one though.
Can I ask what kind of use cases would a spectrogram have for radar data? I've been messing around with making my own spectrogram app as well (linux desktop app and not web app though) and would be stoked to know if there's any potentially easy to reach use cases for it
When I read about ultrasonic cross-device trackers in advertising [1], I installed "org.woheller69.audio_analyzer_for_android" and "hans.b.skewy1_0" (automatic ultrasonic detection) and started scanning through TV channels after running some test tones. Suffice to say I didn't find any, but the entire process was quite fun. There's also "org.billthefarmer.scope" which is an oscilloscope with a spectrum (not spectrogram).
Web apps like this that accesses user's data should provide samples for users to experiment and explore before they have to give access to their actual data.
Brilliant work - I "get" how this works, I've just spent about half-an-hour playing with this (Chrome browser on my kitchen ChromeBook), singing into it and letting it "listen" to the ambient background noise here (old cooker clock ticking, fridge compressor rumbling occasionally). Useful, educational, and fun also - thanks for publishing/hosting this so others can enjoy it!
I usually use Audacity to inspect the spectrogram of FLAC files and see if they really are 44100Hz or if someone packaged a constant rate 320kbps mp3 encode into a FLAC file.
One place I used these was on a toy AI assistant. I recorded myself saying a trigger word thousands of times, cut the audio in pieces and converted each to a spectrogram image. I then feed those to a training model to help recognize the trigger word.
Before the spectrogram, i was feeding the wav file directly, it was incredibly intensive on my laptop. But the image files were easier to process in real time. This tool can be used for debugging.
How would this work with AI? Don’t you need to train the model to discriminate between the trigger word and other words? If all that’s seen during training is the trigger word, the model will just learn to say “yes” to everything, if you get what I mean.
I do have a WebGL-based implementation of FFT, but here I used good old JS. When properly written, it gets translated into really fast machine code, which is even faster than WebAssembly (I tried!). WebGL's problem is the high toll on the CPU--GPU bridge. When you need to transfer a block of audio data from CPU to GPU to perform calculations, you wait. When you need to transfer the FFT data back, you wait. These waits quickly outweight everything else. However on wavelet transforms GPU comes first because you can store some pre-computed FFTs on GPU and reuse them in multiple runs.
Readit News
https://imgur.com/sRe6Ypv
Aphex twin did something similar, but this is more playful in my opinion.
He talks about that and plenty of other cool stuff in his talk at the 2017 GDC conference. One of my favorite conference talks ever, he did so much cool experimentation to get the sounds he used on the soundtrack, and watching his talk is one of those moments where you really get to see a master of his craft let loose and explain his process.
https://youtu.be/U4FNBMZsqrY
Warning - this music freaked my dog out!
Unfortunately it's in Matlab so I can not run it any more.
[1] https://jo-m.ch/posts/2015/01/hack-the-spectrum-hide-images-...
https://octave.org/
[1] https://github.com/DanielAllepuz/ImageToSound
https://www.reddit.com/r/Damnthatsinteresting/comments/kvjil...
Can we hire you to help us improve the (broken) spectral visualizations on our app?
Example: https://fakeyou.com/tts/result/TR:9jy3vew9w0s3ew4keay9m330rd...
I would so love to hire you to help us. This is freaking cool.
Even if you're not interested, mad props. I really love this.
Of course, don't forget the window function (Hann, or raised cosine), but it looks like you've got that covered because your spectrogram looks smooth.
The color palette looks good in your case. FWIW, my color function is like this: pow(fft_amp, 1.5) * rgb(9, 3, 1). The pow() part brightens the low/quiet amplitudes, and the (9,3,1) multiplier displays 10x wider amp range by mapping it to a visually long black->orange->yellow->white range of colors. Note, that I don't do log10 mapping of the amplitudes.
- Allow playback via Space button. Show a play marker to let the user know where in the sample they are, even without having selected a part.
- Choose a sample that is easier on the ears than high-pitched bird song. I was really shocked when the first loud part came.
[1] https://en.wikipedia.org/wiki/Voice_frequency
Is there any way to make this display in real time, or is that not (currently?) possible with audio APIs?
https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_A...
1. https://arstechnica.com/tech-policy/2015/11/beware-of-ads-th...
I usually use Audacity to inspect the spectrogram of FLAC files and see if they really are 44100Hz or if someone packaged a constant rate 320kbps mp3 encode into a FLAC file.
Now I can just check it in my browser :D
One place I used these was on a toy AI assistant. I recorded myself saying a trigger word thousands of times, cut the audio in pieces and converted each to a spectrogram image. I then feed those to a training model to help recognize the trigger word.
Before the spectrogram, i was feeding the wav file directly, it was incredibly intensive on my laptop. But the image files were easier to process in real time. This tool can be used for debugging.
I like the interesting ability to play a "rectangular" (time + frequency limited) section of the audio.