Readit NewsAtheists in 1999 might have to go Usenet comp.lang.scheme to find Scheme experts.
(Scheme polo shirt at church in 1999? My first guess is around Rice University. Second guess is Indiana.)
And a false positive screening result is not innocuous: it incurs costs in a variety of different ways, including human health.
The trial followed 25,000 adults from the US and Canada over a year, with nearly one in 100 getting a positive result. For 62% of these cases, cancer was later confirmed.
(It also had a false negative rate of 1%:)
The test correctly ruled out cancer in over 99% of those who tested negative.
There is a testable prediction in your article. Unfortunately, I don't think the mechanism is quite restricted enough. TFA says that repairing the BBB helps amyloid plaque clearance. Would the author of your blog post claim that as a win, or admit that the plaques are downstream of the problem, and that BBB integrity is closer to the root cause of the disease process?
The author of that blog post, for whom I am in an excellent position to speak, would point to the "sole intended mechanism" clause in the testable prediction. That is, if the therapeutic's developers do not claim any other intended pathway for clinical benefit from improved BBB integrity other than amyloid−β clearance, then it would count. If not, then it would not count, even if it's plausible or even likely that that's the main pathway by which the benefits are accruing.
However, because this is early preclinical research, it's not likely to reach a late-stage clinical trial within the 12-year window of the author's prediction. Furthermore, in every year there are about a dozen of these preclinical studies that go viral for some reason or other, often having little correlation with how promising the science is. I haven't had a chance to look into this one in detail, so this isn't a negative comment about it, but the base rate of this stuff panning out is low, even if it's good research.
The author of that article would also point out that the concept of "the root cause" isn't terribly well-defined, but that strong evidence points to amyloid pathology as the common entrypoint in all cases of Alzheimer's disease, even if multiple upstream factors (some possibly relating to the BBB) can feed into that, depending on the specific case. Similarly, calorie surplus causes obesity in nearly all cases, but the specific cause of calorie surplus may vary from person to person.
I can't guess what point you're trying to make with a long article that acknowledges the fraud in the beginning, and then rehashes the initial reasons for looking into the amyloid hypothesis. No one is claiming it was stupid to look into the amyloid hypothesis. They are complaining that it hasn't been the most promising theory in quite a long time, and it was fraudulently held as the most promising. Other theories, arguably more promising, are listed throughout your article.
A correction: the article does discuss other hypotheses, in pointing out that they can't account for crucial evidence, whereas there isn't any major evidence the amyloid hypothesis seems to have trouble accounting for, and it thus remains very strong.
Has progress stalled in this area? I don't know, but surely there are people working on it. In fact I recently saw an interesting post on HN about a new technique that among other things enables faster estimation of Lyapunov exponents: https://news.ycombinator.com/item?id=45374706 (search for "Lyapunov" on the github page).
Just because we haven't seen much progress, doesn't mean we won't see more. Progress never happens on a predictable schedule.
Have you seen the research in phase-targeted auditory stimulation, memory, amyloid, and sleep? Do you have thoughts on that?
Acoustic stimulation during sleep predicts long-lasting increases in memory performance and beneficial amyloid response in older adults - https://doi.org/10.1093/ageing/afad228
Acoustic Stimulation to Improve Slow-Wave Sleep in Alzheimer's Disease: A Multiple Night At-Home Intervention https://doi.org/10.1016/j.jagp.2024.07.002
I hadn’t seen that research, thanks for passing it along. It seems like an interesting approach to improve slow wave sleep, which is known to help with amyloid clearance.
Meanwhile: the big challenge for screening tests is base rate confounding: the test needs to be drastically more specific the lower the percentage of the cohort that truly has the condition is. Relatively low rates of false positives can pile up quickly against true positives for conditions that are rare in the population.
The bad thing here is: you get a test suggestive of early-onset Alzheimers. It could realistically be the case that the test positive indicates in reality a coin-flip chance you have it. But that doesn't matter, because it will take years for the diagnosis to settle, and your mental health is materially injured in the meantime.
One correction here: the amyloid antibodies that successfully clear out a large amount of plaque have yet to report data from intervention trials prior to symptom onset, so we can’t say this with confidence and in fact we have good reason to suspect they would be more effective at this disease stage.
I wrote about this and related topics here: https://www.astralcodexten.com/p/in-defense-of-the-amyloid-h...
Edited to add: the sort of test discussed in the OP wouldn’t be relevant to presymptomatic treatment, however, since it’s a test of symptoms rather than biomarkers for preclinical disease.
Maybe there was an implicit hope of a better/larger language model leading to new intelligent capabilities, but I've never seen the Transformer designers say they were targeting this or expecting any significant new capabilities even (to their credit) after it was already apparent how capable it was. Neither Google's initial fumbling of the tech or Shazeer's entertainment chatbot foray seem to indicate that they had been targeting, and/or realized they had achieved, a more significant advance than the more efficient seq-2-seq model which had been their proximate goal.
To me it seems that the Transformer is really one of industry/science's great accidental discoveries. I don't think it's just the ability to scale that made it so powerful, but more the specifics of the architecture, including the emergent ability to learn "induction heads" which seem core to a lot of what they can do.
The Transformer precursors I had in mind were recent ones, in particular Sutskever et als "Sequence to Sequence learning with Neural Networks [LSTM]" from 2014, and Bahdanau et als "Jointly learning to align & translate" from 2016, then followed by the "Attention is all you need" Transformer paper in 2017.
I think there has been a strong case that the "stochastic parrot" model sells language models short, but to what extent still seems to me an open question.
Perhaps in 1999 it seemed reasonable to think that passing the Turing Test, or maximally compressing/predicting human text makes for a good AI/AGI test, but I'd say we now know better, and more to the point that does not appear to have been the motivation for designing the Transformer, or the other language models that preceded it.
The recent history leading to the Transformer was the development of first RNN then LSTM-based language models, then the addition of attention, with the primary practical application being for machine translation (but more generally for any sequence-to-sequence mapping task). The motivation for the Transformer was to build a more efficient and scalable language model by using parallel processing, not sequential (RNN/LSTM), to take advantage of GPU/TPU acceleration.
The conceptual design of what would become the Transformer came from Google employee Jakob Uzkoreit who has been interviewed about this - we don't need to guess the motivation. There were two key ideas, originating from the way linguists use syntax trees to represent the hierarchical/grammatical structure of a sentence.
1) Language is as much parallel as sequential, as can be seen by multiple independent branches of the syntax tree, which only join together at the next level up the tree
2) Language is hierarchical, as indicated by the multiple levels of a branching sytntax tree
Put together these two considerations suggests processing the entire sentence in parallel, taking advantage of GPU parallelism (not sequentially like an LSTM), and having multiple layers of such parallel processing to hierarchically process the sentence. This eventually lead to the stack of parallel-processing Transformer layers design, which did retain the successful idea of attention (thus the paper name "Attention is all you need [not RNNs/LSTMs]").
As far as the functional capability of this new architecture, the initial goal was just to be as good as the LSTM + attention language models it aimed to replace (but be more efficient to train & scale). The first realization of the "parallel + hierarchical" ideas by Uzkoreit was actually less capable than its predecesssors, but then another Google employee, Noam Shazeer, got involved and eventually (after a process of experimentation and ablation) arrived at the Transformer design which did perform well on the language modelling task.
Even at this stage, nobody was saying "if we scale this up it'll be AGI-like". It took multiple steps of scaling, from early Google's early Muppet-themed BERT (following their LSTM-based ELMo), to OpenAI's GPT-1, GPT-2 and GPT-3 for there to be a growing realization of how good a next-word predictor, with corresponding capabilities, this architecture was when scaled up. You can read the early GPT papers and see the growing level of realization - they were not expecting it to be this capable.
Note also that when Shazeer left Google, disappointed that they were not making better use of his Transformer baby, he did not go off and form an AGI company - he went and created Character.ai making fantasy-themed ChatBots (similar to Google having experimented with ChatBot use, then abandoning it, since without OpenAI's innovation of RLHF Transformer-based ChatBots were unpredictable and a corporate liability).
I was just responding to this claim:
> An LLM was only every meant to be a linguistics model, not a brain or cognitive architecture.
Plenty of people did in fact see a language model as a potential path towards intelligence, whatever might be said about the beliefs of Mr. Uszkoreit specifically.
There's some ambiguity as to whether you're talking about the transformer specifically, or language models generally. The "recent history" of RNNs and LSTMs you refer to dates back to before the paper I linked. I won't speak to the motivations or views of the specific authors of Vaswani et al, but there's a long history, both distant and recent, of drawing connections between information theory, compression, prediction, and intelligence, including in the context of language modeling.
The redacted words are also redacted in the word index, but the alphabetically preceding and succeeding words are visible, as is the number of index lines taken up by the redacted word's entry, which correlates with the number of appearances of that word.
This seems like rather useful information to constrain a search by such a tool.