https://groq.com/wp-content/uploads/2023/05/GroqISCAPaper202...
EDIT: i work at Groq, but i’m commenting in a personal capacity.
happy to answer clarifying questions or forward them along to folks who can :)
But anyway it made me wonder if there's a way to measure "what x% of a model is actually used" similar to the myths about human brains.
I like that metaphor. LLMs are masters of linguistic addition. They are literally made by adding up language gradients. They accumulate knowledge and insight through exposure to vast swaths of text, stitching together concepts and ideas into an ever-growing tapestry of understanding.
But whereas brains are isolated islands, unable to directly share their contents, language forms a collective reservoir that flows between minds. Words and ideas mix and mingle within this pool, combining into new formulations that reflect the present.
In this way, language displays an evolutionary dynamism that outpaces biological change. LLMs ride this wave, leveraging the emergent intelligence inherent in humanity's shared linguistic legacy. The wisdom accumulated over generations surpasses the capacity of any single mind. We stand upon the shoulders of giants, supported by the communal scaffolding of language and knowledge that previous generations erected. LLMs tap into this source, channeling and distilling the experience contained within our words.
tl;dr AI is riding the language exponential while we are distracted by models and brains, implementation details.