So, you oppose their desire to diversify their funding?
NPR gets 10% of their funding from tax dollars.
Who you are harming is small rural public radio stations in red areas.
Readit Newstyronehed commented on Clawback of $1.1B for PBS and NPR puts rural stations at risk theconversation.com/clawb... · Posted by u/rntn
archive org.
Look for the audiobook "Mawson's Will: The Greatest Survival Story Ever Told"
tyronehed commented on All AI models might be the same blog.jxmo.io/p/there-is-o... · Posted by u/jxmorris12
Especially if they are all me-too copies of a Transformer.
When we arrive at AGI, you can be certain it will not contain a Transformer.
tyronehed commented on Ask HN: Any insider takes on Yann LeCun's push against current architectures? · Posted by u/vessenes
I've always felt like the argument is super flimsy because "of course we can _in theory_ do error correction". I've never seen even a semi-rigorous argument that error correction is _theoretically_ impossible. Do you have a link to somewhere where such an argument is made?
As soon as you need to start leaning heavily on error correction, that is an indication that your architecture and solution is not correct. The final solution will need to be elegant and very close to a perfect solution immediately.
You must always keep close to the only known example we have of an intelligence which is the human brain. As soon as you start to wander away from the way the human brain does it, you are on your own and you are not relying on known examples of intelligence. Certainly that might be possible, but since there's only one known example in this universe of intelligence, it seems ridiculous to do anything but stick close to that example, which is the human brain.
tyronehed commented on Ask HN: Any insider takes on Yann LeCun's push against current architectures? · Posted by u/vessenes
> Each time there is an algorithmic advancement, people quickly rush to apply it to both existing and new problems, demonstrating quick advancements. Then we tend to hit a kind of plateau for a number of years until the next algorithmic solution is found.
That seems to be how science works as a whole. Long periods of little progress between productive paradigm shifts.
This is actually a lazy approach as you describe it. Instead, what is needed is an elegant and simple approach that is 99% of the way there out of the gate. Soon as you start doing statistical tweaking and overfitting models, you are not approaching a solution.
tyronehed commented on Ask HN: Any insider takes on Yann LeCun's push against current architectures? · Posted by u/vessenes
I believe he’s talking about some sort of ‘energy as measured by distance from the models understanding of the world’ as in quite literally a world model. But again I’m ignorant, hence the post!
When an architecture is based around world model building, then it is a casual outcome that similar concepts and things end up being stored in similar places. They overlap.
As soon as your solution starts to get mathematically complex, you are departing from what the human brain does. Not saying that in some universe it might be possible to make a statistical intelligence, but when you go that direction you are straying away from the only existing intelligences that we know about. The human brain. So the best solutions will closely echo neuroscience.
tyronehed commented on Ask HN: Any insider takes on Yann LeCun's push against current architectures? · Posted by u/vessenes
Sligtly related: Energy Based Models (EBMs) are better in theory and yet too resource intensive. I tried to sell using EBMs to my org, but the price for even a small use case was prohibitive.
I learned it from: https://youtube.com/playlist?list=PLLHTzKZzVU9eaEyErdV26ikyo...
Yann LeCun, and Michael Bronstein and his colleagues have some similarities in trying to properly Sciencify Deep Learning.
Yann LeCun's approach, as least for Vision has one core tenet- energy minimization, just like in Physics. In his course, he also shows some current arch/algos to be special cases for EBMs.
Yann believes that understanding the Whys of the behavior of DL algorithms are going to be beneficial in the long term rather than playing around with hyper-params.
There is also a case for language being too low-dimensional to lead to AGI even if it is solved. Like, in a recent video, he said that the total amount of data existing on all digitized books and internet are the same as what a human children takes in in the first 4/5 years. He considers this low.
There are also epistemological arguments against language not being able to lead to AGI, but I haven't heard him talk about them.
He also believes that Vision is a more important aspect of intellgence. One reason being it being very high-dim. (Edit) Consider an example. Take 4 monochrome pixels. All pixels can range from 0 to 255. 4 pixels can create 256^4 = 2^32 combinations. 4 words can create 4! = 24 combinations. Solving language is easier and therefore low-stakes. Remember the monkey producing a Shakespeare play by randomly punching typewriter keys? If that was an astronomically big number, think how obscenely long it would take a monkey to paint Mona Lisa by randomly assigning pixel values. Left as an exercise to the reader.
Juergen Schmidhuber has gone a lot queit now. But he also told that a world-model, explicitly included in training is reasoning is better, rather than only text or image or whatever. He has a good paper with Lucas Beyer.
Since this exposes the answer, the new architecture has to be based on world model building.
tyronehed commented on Ask HN: Any insider takes on Yann LeCun's push against current architectures? · Posted by u/vessenes
The alternative architectures must learn from streaming data, must be error tolerant and must have the characteristic that similar objects or concepts much naturally come near to each other. They must naturally overlap.
tyronehed commented on Ask HN: Any insider takes on Yann LeCun's push against current architectures? · Posted by u/vessenes
Any transformer based LLM will never achieve AGI because it's only trying to pick the next word. You need a larger amount of planning to achieve AGI.
Also, the characteristics of LLMs do not resemble any existing intelligence that we know of. Does a baby require 2 years of statistical analysis to become useful? No. Transformer architectures are parlor tricks. They are glorified Google but they're not doing anything or planning.
If you want that, then you have to base your architecture on the known examples of intelligence that we are aware of in the universe. And that's not a transformer. In fact, whatever AGI emerges will absolutely not contain a transformer.
Only Transformer based architectures are over.
It amazes me that everyone so fetishizes Transformer architectures that they cannot imagine alternative--when the alternative is obvious.