An aside: please use proper capitalization. With this article I found myself backtracking thinking I’d missed a word, which was very annoying. Not sure what the authors intention was with that decision but please reconsider.
I'm all for Graham's pyramid of disagreement: we should focus on the core argument, rather than superfluous things like tone, or character, or capitalisation.
But this is too much for me personally. I just realised I consider the complete lack of capitalisation on a piece of public intellectual work to be obnoxious. Sorry, it's impractical, distracting and generates unnecessary cognitive load for everyone else.
You're the top comment right now, and it's not about the content of the article at all, which is a real shame. All the wasted thought cycles across so many people :(
Capitalization and punctuation are to written language what pronunciation and stress are to spoken language. If someone was mispronouncing every word, using incorrect vowels, stressing the wrong syllables, etc., you'd have a really hard time understanding anything they're saying. Writing with incorrect punctuation and capitalization impedes comprehension in the same way.
This looks like a personal blog post, in a blog where the author have avoided capitalization fairly consistently. The blog post was likely not meant to be a research paper, and reading it as a research paper is probably setting the wrong expectations.
If people wanted to read formal-looking formatted text, the author has linked to one in the second paragraph:
This is the norm for Gen Z. We don’t see it because children don’t set social norms where adults are present too, but with the oldest of Gen Z about to turn 30, you and I should expect to see this more and more, and get used to it. If every kid can handle it, I think we can, too.
It doesn't change the point of your comment necessarily, but as far as TFA goes, the author was teaching a University course in 2015, so is highly unlikely to be Gen Z.
call me old-fasahioned, but two spaces after a period will solve this problem if people insist on all-lower-case. this also helps distinguish between abbreviations such as st. martin's and the ends of sentences.
i'll bet that the linguistics experimentalists have metrics that quantify reading speed measurements as determined by eye tracking experiments, and can verify this.
> [I]'ll bet that the linguistics experimentalists have metrics that quantify reading speed measurements as determined by eye tracking experiments, and can verify this.
You appear to be trolling for the sake of trolling, but for reference: reading speed is determined by familiarity with the style of the text. Diverging from whatever people are used to will make them slower.
There is no such thing as "two spaces" in HTML, so good luck with that.
( do away with both capitalization and periods ( use tabs to separate sentences ( problem solved
[( i'm only kind of joking here ( i actually think that would work pretty well ))] )))
( or alternatively use nested sexp to delineate paragraphs, square brackets for parentheticals [( this turned out to be an utterly cursed idea, for the record )] )
Yeah, things like these make me glad that humans don't live forever. By the time you are 30 you already hate the way so many things work around you. If you argue about it you are called a philistine luddite who can't stomach change. There's no right or wrong, but it's good you don't have to deal with stuff you find annoying indefinitely. You just... die eventually.
It's a better equilibrium this way and one of the main reasons I don't care much for transhumanism.
Language evolves. Capitalization is an artifact of a period where capitalizing the first letter made a lot of sense for the medium (parchment/paper). Modern culture is abandoning it for speed efficiency on keyboards or digital keyboards. A purist would say that we should still be using all capitals like they did in Greek/Latin which again was related to the medium.
I'll likely continue using Capitalization as a preference and that we use it to express conventions in programming, but I totally understand the movement to drop it and frankly its logical enough.
It's slower for sure, but capitalization does impart information: beginning of sentences, proper nouns, acronyms, and such. Sure, you could re-read the sentence until you figured all that out, but you are creating unnecessary hitches in the reading process. Capitalization is an optimization for the reader, and lack of capitalization is optimization for the writer.
As much as I dislike it sometimes, language absolutely does evolve. Proper capitalization does not fit into this, though. It can completely change the meaning of something if it is not capitalized. It's not just at the beginning of sentences, it's proper nouns within a sentence. Unfortunately I don't have an example of this handy but it's happened to me several times in my life where I've been completely confused by this (mostly on Slack).
This is a merely showing off your personal style which, when writing a technical article, I don't care about.
For people interested in the softmax, log sum exp and energy models, have a look at "Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One" [1]
There are many useful tricks - like cosine distance.
In contrast, softmax has a very deep grounding in statistical physics - where it is called the Boltzmann distribution. In fact, this connection between statistical physics and machine learning was so fundamental that it was a key part of the 2024 Nobel Prize in Physics awarded to Hopfield and Hinton.
Study of thermodynamics gave rise to many concepts in information theory and statistics, but I wouldn't say that there is any direct connection per se between thermodynamics and any field where statistics or information theory are applicable. And the reasoning behind the 2024 Nobel Prize in Physics was... quite innovative.
> I wouldn't say that there is any direct connection per se between thermodynamics and any field where statistics or information theory are applicable.
Thermodynamics can absolutely be studied through both a statistical mechanics and an information theory lens, and many physicists have found this to be quite productive and enlightening. Especially when it gets to tricky cases involving entropy, like Maxwell's Demon and Landauer's Eraser, one struggles not to do so.
Because the domain is a Korean name, I half-expected this to be about an old Korean game company[1] with the same name. They made some banger RPGs at the time and had really great art books.
Softmax’s exponential comes from counting occupation states. Maximize the ways to arrange things with logits as energies, and you get exp(logits) over a partition function, pure Boltzmann style. It’s optimal because it’s how probability naturally piles up.
I personally don’t think much of the maximum entropy principle. If you look at the axioms that inform it, they don’t really seem obviously correct. Further, the usual qualitative argument is only right in a certain lens: namely they say choosing anything else would require you to make more assumptions about your distribution than is required. Yet it’s easy to find examples where the max entropy solution suppresses some states more than is necessary etc., which to me contradicts that qualitative argument.
No need to introduce the concept of energy. It's a "natural" probability measure on any space where the outcomes have some weight. In particular, it's the measure that maximizes entropy while fixing the average weight. Of course it's contentious if this is really "natural," and what that even means. Some hardcore proponents like Jaynes argue along the lines of epistemic humility but for applications it really just boils down to it being a simple and effective choice.
The connection isn't immediately obvious, but it's simply because solving for the maximum entry distribution that achieves a given expectation value produces the Botlzmann distribution. In stat mech, our "classifier" over (micro-)states is energy; in A.I. the classifier is labels.
For details, the keyword is Lagrange multiplier [0]. The specific application here is maximizing f as the entropy with the constraint g the expectation value.
If you're like me at all, the above will be a nice short rabbit hole to go down!
The way that energy comes in is that you have a fixed (conserved) amount of it and you have to portion it out among your states. There's nothing inherently energy-related about, it just happens that we often want to look energy distributions and lots of physical systems distribute energy this way (because it's the energy distribution with maximal entropy given the constraints).
(After I wrote this I saw the sibling comment from xelxebar which is a better way of saying the same thing.)
i think that log-sum-exp should actually be the function that gets the name "softmax" because its actually a soft maximum over a set of values. And what we call "softmax" should be called "grad softmax" (since grad of logsumexp is softmax).
Off topic: Unlike many out there I'm not usually bothered by lack of capitalization in comments or tweets, but for an essay like this, it makes the paragraphs so hard to read!
If someone can't even put in a minimal amount of effort for basic punctuation and grammar, I'm not going to read their article on something more sophisticated. If you go for the lowercase i's because you want a childish or slob aesthetic, that can be funny in context. But in math or computing, I'm not going to care what someone thinks if they don't know or don't care that 'I' should be capitalized. Grammarly has a free tier. ChatGPT has a free tier. Paste your word slop into one of those and it will fix the basics for you.
We just had a similar discussion at work the other day when one of our junior engineers noticed that a senior engineer was reflexively tapping the space bar twice after each sentence. That, too, was good style back when we were writing on typewriters or using monospace fonts with no typesetting. Only a child or a slob would fail to provide an extra gap between sentences, it would be distracting to readers and difficult to locate full stops without that!
But it's 2025, and HTML and Word and the APA and MLA and basically everyone agree that times and style guides have changed.
I agree that not capitalizing the first letter in a sentence is a step too far.
For a counter-example, I personally don't care whether they use the proper em-dash, en-dash, or hyphen--I don't even know when or how to insert the right one with my keyboard. I'm sure there are enthusiasts who care very deeply about using the right ones, and feel that my lack of concern for using the right dash is lazy and unrefined. Culture is changing as more and more communication happens on phone touchscreens, and I have to ask myself - am I out of touch? No, it's the children who are wrong. /s
But I strongly disagree that the author should pass everything they write through Grammarly or worse, through ChatGPT.
Readit News
I'm all for Graham's pyramid of disagreement: we should focus on the core argument, rather than superfluous things like tone, or character, or capitalisation.
But this is too much for me personally. I just realised I consider the complete lack of capitalisation on a piece of public intellectual work to be obnoxious. Sorry, it's impractical, distracting and generates unnecessary cognitive load for everyone else.
You're the top comment right now, and it's not about the content of the article at all, which is a real shame. All the wasted thought cycles across so many people :(
It's the new black turtleneck that everyone is wearing, but will swear upon their mother's life isn't because they're copying Steve Jobs.
If people wanted to read formal-looking formatted text, the author has linked to one in the second paragraph:
https://arxiv.org/abs/1511.07916 - Natural Language Understanding with Distributed Representation
Deleted Comment
https://kyunghyuncho.me/bye-felix/
call me old-fasahioned, but two spaces after a period will solve this problem if people insist on all-lower-case. this also helps distinguish between abbreviations such as st. martin's and the ends of sentences.
i'll bet that the linguistics experimentalists have metrics that quantify reading speed measurements as determined by eye tracking experiments, and can verify this.
You appear to be trolling for the sake of trolling, but for reference: reading speed is determined by familiarity with the style of the text. Diverging from whatever people are used to will make them slower.
There is no such thing as "two spaces" in HTML, so good luck with that.
( or alternatively use nested sexp to delineate paragraphs, square brackets for parentheticals [( this turned out to be an utterly cursed idea, for the record )] )
It's a better equilibrium this way and one of the main reasons I don't care much for transhumanism.
I'll likely continue using Capitalization as a preference and that we use it to express conventions in programming, but I totally understand the movement to drop it and frankly its logical enough.
This is a merely showing off your personal style which, when writing a technical article, I don't care about.
Interestingly programming is the one place where I ditch it almost entirely (at least in my personal code bases).
[1]: https://arxiv.org/abs/1912.03263
In contrast, softmax has a very deep grounding in statistical physics - where it is called the Boltzmann distribution. In fact, this connection between statistical physics and machine learning was so fundamental that it was a key part of the 2024 Nobel Prize in Physics awarded to Hopfield and Hinton.
Thermodynamics can absolutely be studied through both a statistical mechanics and an information theory lens, and many physicists have found this to be quite productive and enlightening. Especially when it gets to tricky cases involving entropy, like Maxwell's Demon and Landauer's Eraser, one struggles not to do so.
[1] https://en.m.wikipedia.org/wiki/ESA_(company)
For details, the keyword is Lagrange multiplier [0]. The specific application here is maximizing f as the entropy with the constraint g the expectation value.
If you're like me at all, the above will be a nice short rabbit hole to go down!
[0]:https://tutorial.math.lamar.edu/classes/calciii/lagrangemult...
(After I wrote this I saw the sibling comment from xelxebar which is a better way of saying the same thing.)
Note: I am the author
But it's 2025, and HTML and Word and the APA and MLA and basically everyone agree that times and style guides have changed.
I agree that not capitalizing the first letter in a sentence is a step too far.
For a counter-example, I personally don't care whether they use the proper em-dash, en-dash, or hyphen--I don't even know when or how to insert the right one with my keyboard. I'm sure there are enthusiasts who care very deeply about using the right ones, and feel that my lack of concern for using the right dash is lazy and unrefined. Culture is changing as more and more communication happens on phone touchscreens, and I have to ask myself - am I out of touch? No, it's the children who are wrong. /s
But I strongly disagree that the author should pass everything they write through Grammarly or worse, through ChatGPT.