I know Schmidhuber is famously miffed for missing out on the AI revolution limelight, and despite that he runs a pretty famous and well-resourced group. So with a paper like this demonstrating a new fundamental technique, you'd think they would eat the labor and compute costs of getting this up and running on a full gauntlet of high-profile benchmarks, in comparison with existing SOTA methods, vs the sort of half-hearted benchmarking that happens in this paper. It's a hassle, but all it would take for something like this to catch the community's attention would be a clear demonstration of viability in line with what groups at any of the other large research institutions do.
The failure to put something like that front and center makes me wonder how strong the method is, because you have to assume that someone on the team has tried more benchmarks. Still, the idea of learning a better update rule than gradient descent is intriguing, so maybe something cool will come from this :)
If it’s really that new and different, maybe it’d be a little premature and even misleading to present the sort of full sweep you suggest. People are much better at pooh-poohing new ideas than at accurately assessing their potential.
First, congratulations! It's a paper worth HN attention. Very cool.
Second: do Hacker News posts form a small-world network? I don't know. I don't even know if my question is well posed (it might be a meaningless question). Does the set of Hacker News articles change over time in ways that resemble annealing or self-training matrices? (likewise, I question this question, but I wonder.)
Start with the assumption that someone has already done it... Do a thorough literature survey... Ask experts working on the most similar thing. Don't be disheartened if you weren't the first; ideas don't have to be original to have value; some ideas need reviving from time to time, or were ahead of their time when first discovered.
Sounds like we are in very similar positions and have a very similar question :). My only real plan so far is to try and beat or match SOTA on a recent benchmark from a large corporate / research lab, give them an email and hope they are willing to talk to you.
Demo speaks louder than words. If you don't want to go into the details of how it works, it would still be interesting to just see where it over and under performs compared to existing systems.
Don't burn the capabilities commons. You probably don't have anything, in which case, why bother people? If you do have something, that advances AI capabilities and shortens the time before AGI; and while nobody actually has anything resembling a viable plan for surviving that, the fake plans tend to rely on having more time rather than less time.
I haven't really absorbed this paper yet, but first thoughts were Hopfield Networks we used in the 1980s.
For unsupervised learning algorithms like masked models (BERT and some other Transformers), it makes sense to train in parallel with prediction. Why not?
My imagination can't wrap around using this for supervised (labeled data) learning.
Readit News
The failure to put something like that front and center makes me wonder how strong the method is, because you have to assume that someone on the team has tried more benchmarks. Still, the idea of learning a better update rule than gradient descent is intriguing, so maybe something cool will come from this :)
Second: do Hacker News posts form a small-world network? I don't know. I don't even know if my question is well posed (it might be a meaningless question). Does the set of Hacker News articles change over time in ways that resemble annealing or self-training matrices? (likewise, I question this question, but I wonder.)
https://en.m.wikipedia.org/wiki/Small_world_network
I am never sure if it is a waste of time or has some value.
If you guys had some unique ML technology that is different to what all the others do, what would you do with it?
Deleted Comment
Use the "proof is in the pudding" method:
Do something with it - preferably useful - that no one else can.
For unsupervised learning algorithms like masked models (BERT and some other Transformers), it makes sense to train in parallel with prediction. Why not?
My imagination can't wrap around using this for supervised (labeled data) learning.
Deleted Comment