Woah that’s a big deal if true. I remember Nate Silver had a good chapter on this in his book on statistics. He went through the history of failed attempts in Italy and elsewhere. It’s a famously hard problem that has a long history of false promises and good efforts that simply didn’t work out, where experts just generally concluded it’s an unsolvable problem with current tech and the general random nature of it (in terms of usefully specific accuracy).
Yes, this also made me immediately think of that chapter in Nate Silver's book ("The Signal and the Noise"). Considering what I remember Nate's musings on the topic to be, I highly doubt that this is a particularly significant improvement in earthquake prediction. Sure, with the ability to crunch more numbers than ever before, AI can no doubt up the accuracy a bit. However, same as with the non-AI techniques that it's built upon, only a big-ish improvement in our fundamental understanding of tectonics, and/or a big-ish improvement in the physical sensors that feed data into the models, is really going to move the needle in this space.
> the AI algorithm correctly predicted 70% of earthquakes a week before they happened during a seven-month trial
> The outcome was a weekly forecast in which the AI successfully predicted 14 earthquakes within about 200 miles of where it estimated they would happen
> It missed one earthquake and gave eight false warnings.
So about 36% are false positives (which aren't terrible in this particular field) and about 6% false negatives.
This doesn't sound bad, but I'd love to know the precision of previous techniques / prior art.
It may be better than previous models, but that "70%" number is misleading:
> The outcome was a weekly forecast in which the AI successfully predicted 14 earthquakes within about 200 miles of where it estimated they would happen and at almost exactly the calculated strength. It missed one earthquake and gave eight false warnings.
So 14 true positives and 8 false positives, which means the positive predictive value is not great, less than 65%. And I didn't read the paper for the details of the timing, but the article said "a week out", so I'm assuming it means the predictions meant the earthquake could strike in a minute or in 7 days.
What would governments/policy makers actually be able to do with any of this data? Not to denigrate it as a step forward, but I'm having trouble seeing much practical impact at all.
As someone who lives in an earthquake-prone area, I don't think those stats are high enough that they would lead officials to evacuate an area as a precaution. But it would be a useful indicator to encourage people to make sure they have their earthquake supplies topped up and ready to go.
> What would governments/policy makers actually be able to do with any of this data? Not to denigrate it as a step forward, but I'm having trouble seeing much practical impact at all.
You can look at how beneficial hurricane forecasting has been in saving lives, which has been increasing in accuracy for longer lead times. It's very useful to know something bad is likely going to happen somewhere so you can move resources and evacuate people.
> So 14 true positives and 8 false positives, which means the positive predictive value is not great, less than 65%.
This actually seems huge to me unless we already are hitting close to 65%. I'm not sure how this wouldn't be a big deal compared to what I understand the status quo to be (unpredictable).
Of course, if the forecasts are just a couple minutes out at best, then that's way less useful. But at the very least an emergency alert could be sent out so people can get to safety which could help.
If you could forecast earthquakes with principal component analysis and random forests don't you think someone would have done it by now? This paper is trash and will have zero impact.
> The outcome was a weekly forecast in which the AI successfully predicted 14 earthquakes within about 200 miles of where it estimated they would happen and at almost exactly the calculated strength. It missed one earthquake and gave eight false warnings
There were 600 entries into the competition, I wonder how chance played into this solution winning the contest.
I run 200 blogs, each of them predicting a moderate earthquake at a particular place but at different times. When one of them finally “hits”, I need monetization ideas.
Given the map of the predictions versus the actual locations of the earthquakes, I'm not prepared to celebrate the accomplishment here. It looks as if in the essential details, the prediction is just flat out wrong--it's consistently predicting earthquakes in the wrong basin.
Actually, digging in a little more, I'm even more suspicious. The article says
> The outcome was a weekly forecast in which the AI successfully predicted 14 earthquakes within about 200 miles of where it estimated they would happen and at almost exactly the calculated strength. It missed one earthquake and gave eight false warnings.
A "weekly forecast" isn't terribly descriptive, but it sounds to me like a prediction "Will an earthquake occur this week? If so, where and at what strength?" Given that it's 14 earthquakes over a 7 month period (i.e., about 30 weeks), that means you're looking mostly if not entirely at small, probably unnoticeable earthquakes. It also means that there's basically a coin flip of whether or not an earthquake will occur--and if you score it on the accuracy of predicting such, it comes out to 30% wrong (so the p-value, if I'm doing it right is 0.02, which I guess is significant, although if another commenter is right and this is the best of 600 competitive entrants, it should be expected that one would look this good).
Given that both the timing and the location accuracy look less than impressive, the next question is how good a job it did at predicting the magnitude. There's no details on the accuracy here, but given the location accuracy is hailed as impressive despite being clearly visually less than so, it wouldn't surprise me that the magnitude predictions are similarly garbage.
In short, this feels like merely continued evolution in the history of earthquake prediction techniques rather than a revolution, which is to say something that is loudly hailed as being a good start yet turns out to go absolutely nowhere.
They dumped a bunch of data on pca and random forests then only talked about when it happened to be right. Don't get your hopes up that this paper will go anywhere.
I wish the article would explain the criteria for winning the competition. If it's simply percentage of actual earthquakes that were predicted I could easily write an algorithm that would score 100% by always returning the value `true`.
For me, its quite clear from the article, that always returning `true` is not what they did:
"[the ai] predicted 14 earthquakes within about 200 miles of where it estimated they would happen and at almost exactly the calculated strength. It missed one earthquake and gave eight false warnings."
So its:
14 Positives,
8 False Positives and
1 False Negative
Precision was 64% and recall was 93%. In this context recall is a lot more meaningful (how many real earthquakes were predicted) than precision (how many predictions were earthquakes) as long as there aren’t too many false alarms.
> the AI successfully predicted 14 earthquakes within about 200 miles of where it estimated they would happen and at almost exactly the calculated strength. It missed one earthquake and gave eight false warnings.
It gave slightly more than 1 in 3 false alarms unfortunately.
This might go into hands of insurance companies who will decide to not insure areas where Earth Quake is likely to hit and could also do some more forecasting to avoid further regions.
But that's kind of ok? I mean, insurance is ultimately about amortizing the cost of unexpected adverse events. It's not a way of fobbing off the cost of expected damage onto someone else, even though many people intuitively think of insurance this way.
Readit News
> the AI algorithm correctly predicted 70% of earthquakes a week before they happened during a seven-month trial
> The outcome was a weekly forecast in which the AI successfully predicted 14 earthquakes within about 200 miles of where it estimated they would happen
> It missed one earthquake and gave eight false warnings.
So about 36% are false positives (which aren't terrible in this particular field) and about 6% false negatives.
This doesn't sound bad, but I'd love to know the precision of previous techniques / prior art.
that's probably the most important number, and quite good
Very very impressive. The impact of this is enormous.
> The outcome was a weekly forecast in which the AI successfully predicted 14 earthquakes within about 200 miles of where it estimated they would happen and at almost exactly the calculated strength. It missed one earthquake and gave eight false warnings.
So 14 true positives and 8 false positives, which means the positive predictive value is not great, less than 65%. And I didn't read the paper for the details of the timing, but the article said "a week out", so I'm assuming it means the predictions meant the earthquake could strike in a minute or in 7 days.
What would governments/policy makers actually be able to do with any of this data? Not to denigrate it as a step forward, but I'm having trouble seeing much practical impact at all.
You can look at how beneficial hurricane forecasting has been in saving lives, which has been increasing in accuracy for longer lead times. It's very useful to know something bad is likely going to happen somewhere so you can move resources and evacuate people.
> So 14 true positives and 8 false positives, which means the positive predictive value is not great, less than 65%.
This actually seems huge to me unless we already are hitting close to 65%. I'm not sure how this wouldn't be a big deal compared to what I understand the status quo to be (unpredictable).
Of course, if the forecasts are just a couple minutes out at best, then that's way less useful. But at the very least an emergency alert could be sent out so people can get to safety which could help.
There were 600 entries into the competition, I wonder how chance played into this solution winning the contest.
Actually, digging in a little more, I'm even more suspicious. The article says
> The outcome was a weekly forecast in which the AI successfully predicted 14 earthquakes within about 200 miles of where it estimated they would happen and at almost exactly the calculated strength. It missed one earthquake and gave eight false warnings.
A "weekly forecast" isn't terribly descriptive, but it sounds to me like a prediction "Will an earthquake occur this week? If so, where and at what strength?" Given that it's 14 earthquakes over a 7 month period (i.e., about 30 weeks), that means you're looking mostly if not entirely at small, probably unnoticeable earthquakes. It also means that there's basically a coin flip of whether or not an earthquake will occur--and if you score it on the accuracy of predicting such, it comes out to 30% wrong (so the p-value, if I'm doing it right is 0.02, which I guess is significant, although if another commenter is right and this is the best of 600 competitive entrants, it should be expected that one would look this good).
Given that both the timing and the location accuracy look less than impressive, the next question is how good a job it did at predicting the magnitude. There's no details on the accuracy here, but given the location accuracy is hailed as impressive despite being clearly visually less than so, it wouldn't surprise me that the magnitude predictions are similarly garbage.
In short, this feels like merely continued evolution in the history of earthquake prediction techniques rather than a revolution, which is to say something that is loudly hailed as being a good start yet turns out to go absolutely nowhere.
So its: 14 Positives, 8 False Positives and 1 False Negative
Precision was 64% and recall was 93%. In this context recall is a lot more meaningful (how many real earthquakes were predicted) than precision (how many predictions were earthquakes) as long as there aren’t too many false alarms.
It gave slightly more than 1 in 3 false alarms unfortunately.