Oh man, this is one of my favorite lines of all time:
> "Anyway, I asked one of the geostatisticians to look into it--"
> "Geostatisticians..."
> "--yes, and she's produced a map showing the radius within which we can send email to be slightly more than 500 miles. There are a number of destinations within that radius that we can't reach, either, or reach sporadically, but we can never email farther than this radius."
I adore when experts use their expertise to analyze real-world things like this and provide ridiculously thorough explanations :-D
My favorite story kinda of this nature, of an expert as alien intelligence, was Feynmann's calculations about computer architecture of the Connection Machine:
It's a few paragraphs, maybe too much to quote, but the bulk of it starts with:
> By the end of that summer of 1983, Richard had completed his analysis of the behavior of the router, and much to our surprise and amusement, he presented his answer in the form of a set of partial differential equations. To a physicist this may seem natural, but to a computer designer, treating a set of boolean circuits as a continuous, differentiable system is a bit strange. [...] Our discrete analysis said we needed seven buffers per chip; Feynman's equations suggested that we only needed five. We decided to play it safe and ignore Feynman.
Guess who was right.
The whole essay is worth reading, if you haven't yet.
My recent version: I was playing a pinball game in an arcade. One particular ramp shot was registering earlier in the day and then stopped working.
Eventually I realized that the sensor is an optical beam, and the receiver happened to be in direct sunlight coming in through a window! So it was continuously receiving infrared and would never report the beam being blocked by a pinball. Sure enough, it started working again once the sun angle changed by a few more degrees.
I have an optical smoke detector that will give (very loud) false alarms if a sun beam can bounce off a windowsill onto it. It works great if the curtain is closed. Debugging that took a few early sunrises.
The 500 mile email story is one of my favorite reminders that, fundamentally, we're still governed by the laws of physics. It's funny, but it's also a reminder that, while networks might be very fast, the latency is still going to be governed by the speed of light.
Well laws of physics is what gave us radio in the first place.
Some of my favorite video documentaries are on how it was theorized and then slowly developed over years and decades until they finally got to spark-gap transmitters.
But just imagine listening to spark-gap morse code radio broadcasts for years as amateur and then suddenly someone does a broadcast test of actual voice (violin!) That must have been incredible to hear wirelessly.
24 December 1906 Reginald Fessenden, that was the leap that eventually gave us wifi
I actually like thinking about the exchange of physical information as a network propagation delay, and entanglement/coherence as a distributed consensus algorithm. They're kinda samey from a conceptual point of view (in my amateur opinion)
I had a customer who used a line of sight system for extending their network across part of a city.
I had a shortcut on my desktop with the weather for that town ready when they would inevitably call and blame our unrelated equipment for some problem.
I worked at a small, local ISP in the 90:ies that had a point to point link across the river, handling the dial up traffic from the telecom company we partnered with.
Every few days, always at roughly the same time, all incoming dial up traffic would drop. A minute later, the customers could reconnect.
It took a while before we realized that one of the huge passenger ferries that docked a short distance upstream was the cause. When it arrived and departed, its chimneys and possibly bridge and highest deck blocked LOS across the river.
I used to work in high-frequency trading. I had several tabs permanently open to the live weather radar feed for regions where we had microwave towers: the NE USA, the South of England, the Alps...
There was a site with stories like these somewhere, I sadly can't remember the URL any more.
I think the one that stuck out to me was the Soviet mainframe computer that would get weird bit flips almost every day, always at the exact same time. Somebody compared what was different about the days it didn't get bit flips on, it turns out those were the days on which a particular train didn't run, the computer was very close to a railway station. What train was it, you ask? The one transporting the (definitely perfectly safe to eat, definitely not filled to the brim with nuclear radiation) cow meat from Chernobyl. The radiation was intense enough to cause bit flips, I'm sure the quality of soviet components didn't help here either.
As someone with very limited electrical experience, the more magic switch story instantly went "the second terminal of the switch is probably grounded to the switch casing" when they explained it only had one connected terminal.
This is a very common thing in older automotive electronics, for example.
A user was having a really bizarre problem: They could log in when they were sitting down in a seat in front of the keyboard, but when they were standing in front of the keyboard, their password didn't work! The problem happened every time, so they called for support, who finally figured it out after watching them demonstrate the problem many times:
It turned out that some joker had rearranged the numbers keys on the keyboard, so they were ordered "0123456789" instead of "1234567890". And the user's password had a digit in it. When the user was sitting down comfortably in front of the keyboard, they looked at the screen while they touch-typed their password, and were able to log in. But when they were standing in front of the computer, they looked at the keyboard and pressed the numbers they saw, which were wrong!
I did tech support via phone for a popular consumer computer brand. One particular call, a woman reported that her computer was restarting every time someone in the house flushed the toilet.
Long story short, her home was in the back-back woods with the home powered by a generator. In addition to powering the computer, the generator was also the source of power for a water pump which would kick on to refill the toilet bowl whenever it emptied. And wouldn't you know that that water pump had a beefy coil around its motor and would brownout the entire house every time it started?
My personal example: VoIP phones stopping after the Asterisk server was up for 3 days.
Reason: the server had IPv6 turned on, and it steadily accumulated privacy IPv6 addresses. These addresses were all sent in a packet describing the supported media endpoints, using UDP.
And yep, eventually it overflowed the MTU and the phones couldn't handle the fragment reassembly.
I remember one (might have been a hn-er's comment, dunno) about the computer restarting when the toilet was flushed. Turns out it was due to voltage drop when a compressor turned on to refill the reservoir of the toilet.
DNS responses sent over UDP are often truncated if the response is too large. This manifests itself as "machine unreachable if name > x characters" sort of errors when you have really long FQDNs.
I don't know if this is fake, but it could be true. I had a similar situation working for a WISP around 2010.
Every night, for about 10 minutes, the connections from our HQ to a relay tower became flaky. At the time we were using two Mikrotik 5GHz cards and some large antennas.
You could sit in front your computer and wait, a few minutes after the sunset, for the monitoring alerts start arriving.
After a week trying everything, including changing hardware (to the same specs), I was very disappointed with the thing and got out to smoke around the sunset.
Then some huge lamps we had around the building switched on, based on a light sensor. Immediately I received the SMS alerts on my phone. I ran into the building, turned off the external lights and bingo: 0% packet loss.
It turns out that the building management had changed all of external lamps the week before, with new sodium-vapor bulbs. And for some reason, on the first 5 to 10 minutes with these lights on, it caused very high interference on the 5GHz band.
I am just surprised that line of sight issues weren't the first thing checked when you have a bespoke line of sight networking setup, especially when there was no local packet loss.
Reminds me of an extremely similar case with a long distance microwave link at a mobile telecom provider in Australia that I worked for. They relied quite heavily on microwave link chains and this particular one was in northern Queensland where fixed lines were hard to find and no local engineers were locally present/aware of the changing environment.
Every week day + Saturday, from 7-3 the link would keep cutting out intermittently. Then work fine and the rest of the day and on Sunday… a crane, building a new residential building would operate during those hours right in the middle of the microwave path. Many weeks of theories and time wasted until someone had a chance to visit. :)
Amazing. Reminds me of the fact that militaries really don't want wind turbines in areas where good radar coverage is important (case in point: the Finnish Defense Forces anywhere near the Russian border); even though the blades aren't metal, they're still a source of noise and radar shadow.
I'm not at all knowledgable about this, but: is it feasible (and if so, hard?) or impossible to have some sort of live reporting from the turbines about the speed/position of their blades that connects to the radar system allowing it to ignore what it knows to be turbine noise/shadow and therefore be able to have turbines there and still get good radar?
I've done a ton of low-budget analog hardware debugging, and the major problem with hardware debugging is each attempt to fix the problem takes a long time. If I had wanted to test every idea I had I could easily waste a week. Not to mention that I can't just run some automated test suite after the fact. For hardware, approaching debugging methodically is a necessity, not just best practice.
We don't typically have log files for hardware, but I'm always surprised when otherwise extremely intelligent people first try to debug by applying "fixes" that shouldn't have any causal effect on any weird observations we've gotten. I have no problem with people coming up with theories because each modification takes time, but each theory should ideally explain the data...
Reading log files with really obscure error messages might as well be reading a magical grimoire. Especially when the solution turned out to have nothing to do with the error message.
The title might've been a Fleetwood (the other kind of) Mac reference.
o/~ Wi-Fi's only working when it's rainin'
Players only stutter when they're buff'rin'
Websites, they will page load oh so slooooww
When the rain falls down, you can download
The unusual internet setup is pretty important information to bury a few paragraphs in. Once that was explained it seemed like they should have started by checking nothing was blocking the antenna before tediously running around plugging the laptop into things and following cables and checking power supplies on the networking equipment?
Hindsight is 20/20 but I correctly guessed the ending as soon as that information was added.
I thought the same, although I probably would not have immediately gone to the Wifi transmitter.
After a couple decades of debugging various internet issues the first thing I now do is check the 'source' works (i.e. plug a laptop into the modem directly, but with a different cable). If that works I go down 'the line' until something does not work. That usually finds the culprit quite quickly (and also stops me from messing with my router config when it's an ISP issue).
In OPs scenario, the moment you realise that the office internet is working fine and it's only the home internet that is having issues, the connection between the two would have been the obvious place to look next.
That being said, it's still a fun story, and still quite 'unexpected' that rain could be the determinating factor on whether you'll have working internet or not.
I felt the same, but to be fair to the author, the information was probably buried in the back of their mind too - when you've had a set up like this for years, many of its details become invisible to yourself, part of "obvious" background information that your mind doesn't bother bringing to the forefront.
Sure, but the title has you thinking "Why would this external weather event effect my in-home wifi?" when the answer is "because I have a weird thing outside of my house that makes the wifi work". I feel used.
Once upon a time, I owned a 1998 Volkswagen Wolfsburg Edition, a sleek and vibrant red car that turned heads wherever I went. As a city worker, I found it convenient to park my car at the train station and commute to work.
One particularly exhausting day, I trudged back from the train to the parking lot, eager to get home and unwind. As I approached my car, I noticed something peculiar—all the windows were missing. Panic gripped me, and I initially thought someone had vandalized my beloved vehicle. However, as I walked around the car, I couldn't find a single shard of glass on the ground. Upon closer inspection, I realized that the windows had simply been rolled down. Relief washed over me as I rolled them back up and drove home, putting the strange experience out of my mind.
Weeks passed, and the incident faded from my memory. Then, on a lazy Saturday morning, I sat on my back porch, sipping a hot cup of coffee and enjoying the tranquility of the day. Suddenly, the sky darkened, and a light rain began to fall. As the raindrops pattered against the roof, I heard an unexpected sound—the distinct whirring of car windows rolling down.
Perplexed, I set my coffee aside and hurried to the front of the house. To my astonishment, I found my Volkswagen's windows had mysteriously lowered themselves, allowing the rain to pour into the car's interior. It dawned on me that the windows' odd behavior must have been caused by a short circuit in the electrical system.
From that day on, I knew my 1998 Volkswagen Wolfsburg Edition was more than just a cool, bright red car—it had a quirky personality of its own, keeping me on my toes with its unexpected window antics.
Haha, I have a 2015 Opel Astra (I think they are sold as Vauxhall as well in some countries) and I noticed a similar thing one day: it suddenly had all windows lowered by itself, without me doing anything to cause it.
The first time it happened was on a music festival after hauling a lot of camping gear from the car. I locked it using the remote key fob, put the keys into my pocket and hauled the last bunch of stuff to our camp. An hour or so later someone told me that my car had completely open windows, asking whether that was intentional. Of course it wasn't.
The next time it happened was at home, after shopping for groceries. I locked the doors, carried the box with the groceries into my flat, and when I finished unpacking them I looked out of the window and saw my car in the parking lot - with fully lowered windows. I thought it was a glitch in the firmware or whatever.
A few days later the same thing happened again - I shopped groceries, carried them in, looked out of the window - car had lowered the windows entirely by itself. The same glitch twice within a few days? In almost the same situation? How big are the chances for that?
Then it suddenly dawned on me.
I have quite a lot of stuff in my pockets. Including the car key with the remote buttons. Whenever carrying heavy stuff, boxes and such, there is quite a good chance of me accidentally pressing the "unlock" button not shortly, but for a few seconds. So I took my key fob, stood in front of the car, held down the button...and after five seconds of waiting, all of the windows lowered for just a little bit, and after waiting a few additional seconds they lowered completely.
Since then I know about an interesting feature of my car: it can remotely lower the windows for ventilation in summer.
I recall reading a variant of that where a terminal had a "print screen" button, and the claim was it would work when standing but not when sitting down (or was it vice versa).
In the end there were two print screen buttons, only one of them functional, and one of them more obvious when standing (or sitting).
These kind of stories are classic debugging parables that teach you to step back and consider what you may be assuming incorrectly when something absolutely doesn't make sense or seems impossible.
I had experienced this one and felt like I was going mad, and then I guess the humidity changed and I gave up/stopped thinking about it. It wasn't until a couple years later I saw a post here about the gas cylinders causing monitors to blank.
There's a Mister Bean episode in which his TV will only work when he's sitting next to it (where he obviously cannot see it). I think he manages to watch TV by creating a copy of himself with his clothes next to the TV, while he's sitting naked in front of it
Some years ago I put in a point to point wifi link for a family member, from house to garage "block". I specified a pair of Ubiquity Nanostations which are tiny, PoE powered and have a decent range.
The house end is inside a UK standard tiled roof - dense 3/4", allow for slat, so 1"+ thick and dense material.
The other end is 20m away (LoS) and external mounting was forbidden. The garage block has foil lined Kingspan style insulation. I managed to mount that end near enough to a skylight window to work OK. I then daisy-chained an access point off it.
All was fine until the sky light was replaced with a metalicised one. The signal just about worked until it rained which was enough to nobble it.
When it got annoying enough, me and said family member plotted and I rocked up when someone was absent for the weekend. I moved the garage station to the outside. It now looks like a bird box. I put up a real bird box at the other end too. The fake box would get baked in the sun but the real one is always shaded.
If you are renting then you might not be allowed to install an aerial on the building. There are situations where you need council permission to install an aerial or it may be forbidden e.g. listed historic buildings.
Readit News
http://www.ibiblio.org/harris/500milemail.html
Or the Magic/More Magic switch
http://www.catb.org/jargon/html/magic-story.html
It's fun when physical reality meets the abstract models that we have built in our heads of these machines.
> "Anyway, I asked one of the geostatisticians to look into it--"
> "Geostatisticians..."
> "--yes, and she's produced a map showing the radius within which we can send email to be slightly more than 500 miles. There are a number of destinations within that radius that we can't reach, either, or reach sporadically, but we can never email farther than this radius."
I adore when experts use their expertise to analyze real-world things like this and provide ridiculously thorough explanations :-D
https://longnow.org/essays/richard-feynman-connection-machin...
It's a few paragraphs, maybe too much to quote, but the bulk of it starts with:
> By the end of that summer of 1983, Richard had completed his analysis of the behavior of the router, and much to our surprise and amusement, he presented his answer in the form of a set of partial differential equations. To a physicist this may seem natural, but to a computer designer, treating a set of boolean circuits as a continuous, differentiable system is a bit strange. [...] Our discrete analysis said we needed seven buffers per chip; Feynman's equations suggested that we only needed five. We decided to play it safe and ignore Feynman.
Guess who was right.
The whole essay is worth reading, if you haven't yet.
Eventually I realized that the sensor is an optical beam, and the receiver happened to be in direct sunlight coming in through a window! So it was continuously receiving infrared and would never report the beam being blocked by a pinball. Sure enough, it started working again once the sun angle changed by a few more degrees.
[1] https://news.ycombinator.com/item?id=37584399
https://www.youtube.com/watch?v=9eyFDBPk4Yw
https://americanhistory.si.edu/collections/nmah_692464
Some of my favorite video documentaries are on how it was theorized and then slowly developed over years and decades until they finally got to spark-gap transmitters.
https://en.wikipedia.org/wiki/Timeline_of_radio
But just imagine listening to spark-gap morse code radio broadcasts for years as amateur and then suddenly someone does a broadcast test of actual voice (violin!) That must have been incredible to hear wirelessly.
24 December 1906 Reginald Fessenden, that was the leap that eventually gave us wifi
https://en.wikipedia.org/wiki/Reginald_Fessenden
> If the problem had had to do with the geography of the human recipient and not his mail server, I think I would have broken down in tears.
I had a shortcut on my desktop with the weather for that town ready when they would inevitably call and blame our unrelated equipment for some problem.
Every few days, always at roughly the same time, all incoming dial up traffic would drop. A minute later, the customers could reconnect.
It took a while before we realized that one of the huge passenger ferries that docked a short distance upstream was the cause. When it arrived and departed, its chimneys and possibly bridge and highest deck blocked LOS across the river.
I think the one that stuck out to me was the Soviet mainframe computer that would get weird bit flips almost every day, always at the exact same time. Somebody compared what was different about the days it didn't get bit flips on, it turns out those were the days on which a particular train didn't run, the computer was very close to a railway station. What train was it, you ask? The one transporting the (definitely perfectly safe to eat, definitely not filled to the brim with nuclear radiation) cow meat from Chernobyl. The radiation was intense enough to cause bit flips, I'm sure the quality of soviet components didn't help here either.
https://beza1e1.tuxen.de/lore/index.html
Also posted on HN awhile back:
https://news.ycombinator.com/item?id=23005140
Edit: yep, here are your Crash Cows: https://beza1e1.tuxen.de/lore/crash_cows.html
This is a very common thing in older automotive electronics, for example.
https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161...
It turned out that some joker had rearranged the numbers keys on the keyboard, so they were ordered "0123456789" instead of "1234567890". And the user's password had a digit in it. When the user was sitting down comfortably in front of the keyboard, they looked at the screen while they touch-typed their password, and were able to log in. But when they were standing in front of the computer, they looked at the keyboard and pressed the numbers they saw, which were wrong!
I did tech support via phone for a popular consumer computer brand. One particular call, a woman reported that her computer was restarting every time someone in the house flushed the toilet.
Long story short, her home was in the back-back woods with the home powered by a generator. In addition to powering the computer, the generator was also the source of power for a water pump which would kick on to refill the toilet bowl whenever it emptied. And wouldn't you know that that water pump had a beefy coil around its motor and would brownout the entire house every time it started?
Reason: the server had IPv6 turned on, and it steadily accumulated privacy IPv6 addresses. These addresses were all sent in a packet describing the supported media endpoints, using UDP.
And yep, eventually it overflowed the MTU and the phones couldn't handle the fragment reassembly.
https://github.com/danluu/debugging-stories
https://thedailywtf.com
https://www.snopes.com/fact-check/cone-of-silence/
Discussed many times here in HN:
1. https://news.ycombinator.com/item?id=37576633
2. https://news.ycombinator.com/item?id=23775404
3. https://news.ycombinator.com/item?id=9338708
[0] magic-story
Every night, for about 10 minutes, the connections from our HQ to a relay tower became flaky. At the time we were using two Mikrotik 5GHz cards and some large antennas.
You could sit in front your computer and wait, a few minutes after the sunset, for the monitoring alerts start arriving. After a week trying everything, including changing hardware (to the same specs), I was very disappointed with the thing and got out to smoke around the sunset.
Then some huge lamps we had around the building switched on, based on a light sensor. Immediately I received the SMS alerts on my phone. I ran into the building, turned off the external lights and bingo: 0% packet loss.
It turns out that the building management had changed all of external lamps the week before, with new sodium-vapor bulbs. And for some reason, on the first 5 to 10 minutes with these lights on, it caused very high interference on the 5GHz band.
Changed the lamps, problem solved.
The ballast and the bulb itself are quite noisy in RF and then they are heating up they even more noisier.
Every week day + Saturday, from 7-3 the link would keep cutting out intermittently. Then work fine and the rest of the day and on Sunday… a crane, building a new residential building would operate during those hours right in the middle of the microwave path. Many weeks of theories and time wasted until someone had a chance to visit. :)
For me it reminds me most debugging I see at work. People coming up with theories and doing some magic incantations on the interface.
Instead of reading the log files or reading error description which makes usually error and fix obvious in 10 seconds.
We don't typically have log files for hardware, but I'm always surprised when otherwise extremely intelligent people first try to debug by applying "fixes" that shouldn't have any causal effect on any weird observations we've gotten. I have no problem with people coming up with theories because each modification takes time, but each theory should ideally explain the data...
Hindsight is 20/20 but I correctly guessed the ending as soon as that information was added.
After a couple decades of debugging various internet issues the first thing I now do is check the 'source' works (i.e. plug a laptop into the modem directly, but with a different cable). If that works I go down 'the line' until something does not work. That usually finds the culprit quite quickly (and also stops me from messing with my router config when it's an ISP issue).
In OPs scenario, the moment you realise that the office internet is working fine and it's only the home internet that is having issues, the connection between the two would have been the obvious place to look next.
That being said, it's still a fun story, and still quite 'unexpected' that rain could be the determinating factor on whether you'll have working internet or not.
At first (without the full context) I thought it was because rain blocked the external signal interference that was making the AP channel look busy.
Dead Comment
One particularly exhausting day, I trudged back from the train to the parking lot, eager to get home and unwind. As I approached my car, I noticed something peculiar—all the windows were missing. Panic gripped me, and I initially thought someone had vandalized my beloved vehicle. However, as I walked around the car, I couldn't find a single shard of glass on the ground. Upon closer inspection, I realized that the windows had simply been rolled down. Relief washed over me as I rolled them back up and drove home, putting the strange experience out of my mind.
Weeks passed, and the incident faded from my memory. Then, on a lazy Saturday morning, I sat on my back porch, sipping a hot cup of coffee and enjoying the tranquility of the day. Suddenly, the sky darkened, and a light rain began to fall. As the raindrops pattered against the roof, I heard an unexpected sound—the distinct whirring of car windows rolling down.
Perplexed, I set my coffee aside and hurried to the front of the house. To my astonishment, I found my Volkswagen's windows had mysteriously lowered themselves, allowing the rain to pour into the car's interior. It dawned on me that the windows' odd behavior must have been caused by a short circuit in the electrical system.
From that day on, I knew my 1998 Volkswagen Wolfsburg Edition was more than just a cool, bright red car—it had a quirky personality of its own, keeping me on my toes with its unexpected window antics.
The first time it happened was on a music festival after hauling a lot of camping gear from the car. I locked it using the remote key fob, put the keys into my pocket and hauled the last bunch of stuff to our camp. An hour or so later someone told me that my car had completely open windows, asking whether that was intentional. Of course it wasn't.
The next time it happened was at home, after shopping for groceries. I locked the doors, carried the box with the groceries into my flat, and when I finished unpacking them I looked out of the window and saw my car in the parking lot - with fully lowered windows. I thought it was a glitch in the firmware or whatever.
A few days later the same thing happened again - I shopped groceries, carried them in, looked out of the window - car had lowered the windows entirely by itself. The same glitch twice within a few days? In almost the same situation? How big are the chances for that?
Then it suddenly dawned on me.
I have quite a lot of stuff in my pockets. Including the car key with the remote buttons. Whenever carrying heavy stuff, boxes and such, there is quite a good chance of me accidentally pressing the "unlock" button not shortly, but for a few seconds. So I took my key fob, stood in front of the car, held down the button...and after five seconds of waiting, all of the windows lowered for just a little bit, and after waiting a few additional seconds they lowered completely.
Since then I know about an interesting feature of my car: it can remotely lower the windows for ventilation in summer.
Can log in while sitting down, can't log in when standing up.
I need to find the reference ...
Edit: OK, here's one version:
https://www.reddit.com/r/talesfromtechsupport/comments/3v52p...
In the end there were two print screen buttons, only one of them functional, and one of them more obvious when standing (or sitting).
These kind of stories are classic debugging parables that teach you to step back and consider what you may be assuming incorrectly when something absolutely doesn't make sense or seems impossible.
https://news.ycombinator.com/item?id=21978004
[Edit: I see this one has also been mentioned a few times already in the thread]
The house end is inside a UK standard tiled roof - dense 3/4", allow for slat, so 1"+ thick and dense material.
The other end is 20m away (LoS) and external mounting was forbidden. The garage block has foil lined Kingspan style insulation. I managed to mount that end near enough to a skylight window to work OK. I then daisy-chained an access point off it.
All was fine until the sky light was replaced with a metalicised one. The signal just about worked until it rained which was enough to nobble it.
When it got annoying enough, me and said family member plotted and I rocked up when someone was absent for the weekend. I moved the garage station to the outside. It now looks like a bird box. I put up a real bird box at the other end too. The fake box would get baked in the sun but the real one is always shaded.
Also see: https://www.townplanning.info/permitted-development/househol...
Just what I found doing a quick search.
Deleted Comment
...
(Wife Acceptance Factor - far more important than Web App Firewall)