Very cool project. If you haven't already, for JAX and PyTorch support take a look at the Python Array API Standard, https://data-apis.org/array-api/latest/, and see https://data-apis.org/array-api-compat/ for how to use it. If you have or can write everything in terms of the subset of NumPy supported in the Array API Standard, you can get support for alternative array libraries almost for free.
Readit Newssteppi commented on Show HN: Fast Thermodynamic Calculations in Python dlr-institute-of-future-f... · Posted by u/Saloc
steppi commented on Deep Learning Is Applied Topology theahura.substack.com/p/d... · Posted by u/theahura
I really liked this article, though I don't know why the author is calling the idea of finding a separating surface between two classes of points "topology." For instance, they write
"If you are trying to learn a translation task — say, English to Spanish, or Images to Text — your model will learn a topology where bread is close to pan, or where that picture of a cat is close to the word cat."
This is everything that topology is not about: a notion of points being "close" or "far." If we have some topological space in which two points are "close," we can stretch the space so as to get the same topological space, but with the two points now "far". That's the whole point of the joke that the coffee cup and the donut are the same thing.
Instead, the entire thing seems to be a real-world application of something like algebraic geometry. We want to look for something like an algebraic variety the points are near. It's all about geometry and all about metrics between points. That's what it seems like to me, anyway.
If I had to give a loose definition of topology, I would say that it is actually about studying spaces which have some notion of what is close and far, even if no metric exists. The core idea of neighborhoods in point set topology captures the idea of points being nearby another point, and allows defining things like continuity and sequence convergence which require a notion of closeness. From Wikipedia [0] for example
The terms 'nearby', 'arbitrarily small', and 'far apart' can all be made precise by using the concept of open sets. If we change the definition of 'open set', we change what continuous functions, compact sets, and connected sets are. Each choice of definition for 'open set' is called a topology. A set with a topology is called a topological space.
Metric spaces are an important class of topological spaces where a real, non-negative distance, also called a metric, can be defined on pairs of points in the set. Having a metric simplifies many proofs, and many of the most common topological spaces are metric spaces.
That's not to say that topology is necessarily the best lens for understanding neural networks, and the article's author has shown up in the comments to state he's moved on in his thinking. I'm just trying to clear up a misconception.
steppi commented on Perfect Random Floating-Point Numbers specbranch.com/posts/fp-r... · Posted by u/pclmulqdq
Why is that surprising? No one ever uses real numbers for anything.
There are many situations where you have something you want to compute to within low number of units in the last place, that seem fairly involved, but there are very often clever methods that let you do it without having to go to extended or arbitrary precision. Maybe not that surprising, but it's something I find interesting.
steppi commented on Perfect Random Floating-Point Numbers specbranch.com/posts/fp-r... · Posted by u/pclmulqdq
The floats are a surprisingly bad approximation of the reals. They're barely better than the integers, while their shortcomings are much harder to understand, so I'd say it's like if you want approximately a spaceship and you're choosing between an F-14 Tomcat and a Fiat Punto.
Neither of these things is a spaceship, but, it will be obvious to almost anybody why the Punto isn't a spaceship whereas you are less likely to know enough about the F-14 to see why it's not "good enough".
I think the truly surprising thing is just how well floating point numbers work in many practical applications despite how different they are from the real numbers. One could call it the "unreasonable effectiveness of floating point mathematics".
steppi commented on Probability-generating functions entropicthoughts.com/prob... · Posted by u/todsacerdoti
but isn't a characteristic function just "the" way to bridge the gap between sets, functions, and logic(? ...a 3way bridge!?)
I mean, it was useful for me to think about like a translation between sets and logic (this variable x is in the set xor not) into functions (a function f(x) that returns 1 or true whenever x is in set S)
how the heck is that a fourier transform!??
> I'm not sure how much you know about Wittgenstein.
Not much. But I don't have to know much to make my case here. All I have to do is point to the Tractatus, which really is manifest nonsense, and point out that publishing this bit of manifest nonsense didn't seem to hurt Wittgenstein's career much. Tractatus is not the only example of this sort of thing, just the one that sticks out in my mind as the most blatant.
And this is not to say that Wittgenstein, or philosophy in general, never produced anything of value. But the problem is that to find the value you have to wade through all this horse shit, so it's so much more effort than it needs to be. A stopped clock is right twice a day. That doesn't mean there is any value in consulting it to find out what time it is.
Wittgenstein himself states that the Tractatus is nonsense in its closing pages.
My propositions are elucidatory in this way: he who understands me finally recognizes them as senseless, when he has climbed out through them, on them, over them. (He must so to speak throw away the ladder, after he has climbed up on it.)
I think you may agree with the Wittgenstein of the Tractatus more than you realize. My understanding is that his main goal at that time was to show that many of the classic problems of metaphysics which plagued philosophers for centuries or more are literally just nonsense. He didn't write the Tractatus to convince regular people though, but to convince academic philosophers of his time. He earned his fame by being somewhat successful. Rather than making a logical argument for his point, I understand his aim as stimulating his audience to think things out for themselves by offering them carefully crafted nonsense that gave a fresh perspective.
I think you just have no use for the Tractatus because you're not preoccupied with metaphysical questions.
Fair enough, this is not meant to be some endorsement of the standard Fortran BLAS implementations over the optimized versions cited above. Only that the mainstream compilers cited above appear capable of applying these optimizations to the standard BLAS Fortran without any additional effort.
I am basing these comments on quick inspection of the assembly output. Timings would be equally interesting to compare at each stage, but I'm only willing to go so far for a Hacker News comment. So all I will say is perhaps let's keep an open mind about the capability of simple Fortran code.
Check out The Science of Programming Matrix Computations by Robert A. van de Geijn and Enrique S. Quintana-Ort. Chapter 5 walks through how to write an optimized GEMM. It involves clever use of block multiplication, choosing block sizes for optimal cache behavior for specific chips. Modern compilers just aren't able to do such things now. I've spent a little time debugging things in scipy.linalg by swapping out OpenBLAS with reference BLAS and have found the slowdown from using reference BLAS is typically at least an order of magnitude.
There is an implication here that the Fortran implementation of `SGEMM` is somehow inadequate. But any modern Fortran compiler will quite easily apply the AVX and FMA optimizations presented here without any additional changes. Both GNU and Intel make these substitutions with the correct flags.
The unrolling optimization is also just another flag away (`-funroll-all-loops`). The Intel Compiler will even do this without prompting. In fact, it appears to only do a modest 2x unroll on my machine, suggesting that the extreme unroll in this article would have been overkill.
Parallelization certainly a lot to ask of Fortran 77 source, but there there is little stopping you from adding OpenMP statements to the `SGEMM` function. In fact, modern Fortran even offers its own parallelization constructs if you're willing to go there.
Which is to say: Let's not belittle this old Fortran 77 function. Yes it is old, and does not even resemble modern Fortran. But the whole point of Fortran is to free the developer from these platform-specific details, and hand the job off to the compiler. If you don't like that approach, then you're welcome to go to C or C++. But this little block of Fortran code is already capable of doing just about everything in this article.
The Fortran implementation is just a reference implementation. The goal of reference BLAS [0] is to provide relatively simple and easy to understand implementations which demonstrate the interface and are intended to give correct results to test against. Perhaps an exceptional Fortran compiler which doesn't yet exist could generate code which rivals hand (or automatically) tuned optimized BLAS libraries like OpenBLAS [1], MKL [2], ATLAS [3], and those based on BLIS [4], but in practice this is not observed.
Justine observed that the threading model for LLaMA makes it impractical to integrate one of these optimized BLAS libraries, so she wrote her own hand-tuned implementations following the same principles they use.
[0] https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprogra...
[1] https://github.com/OpenMathLib/OpenBLAS
[2] https://www.intel.com/content/www/us/en/developer/tools/onea...
[3] https://en.wikipedia.org/wiki/Automatically_Tuned_Linear_Alg...
steppi commented on The complete story of Gödel incompleteness billwadge.com/2024/03/11/... · Posted by u/herodotus
Would anyone happen to have a recommendation for someone hoping to make Gödel's incompleteness theorem "click"? It feels like every time I reapproach it, I have to start the intuition building all over again.
There’s a wonderful book on the subject by Raymond Smullyan, of knights and knaves recreational math puzzle fame which helped make it click for me.
GUIs and proprietary protocols have killed Emacs.
Emacs isn't an editor, as the post implies, it's an environment. "You can do anything from Emacs" made sense back when everything was open and text based. Nowadays everyone is using Slack, or using an IMAP web client for mail that requires bespoke authentication, or organizer apps that automatically sync across all your devices via the cloud. All of the verticals have been slurped up by corporations who did it better in GUI, have better syncing, and have locked down the protocols needed to bridge, and now the brave Emacs user of 2024 is forced to spend lots of time not in Emacs, thus defeating the point of using Emacs.
As an editor alone I don't think Emacs is worth it. There's the old adage "They added everything to Emacs but a good editor" and I think that makes sense. If you're not going to live in Emacs for the above reasons then Vim/Neovim is a better editor with a larger community, and VSCode/Jetbrains are better IDEs that are already adopting AI, which will essentially kill off Emacs.
Emacs still has a lot of important lessons, but I feel bad for new programmers today who will never get the full experience of a text based digital life. Everything has been dumbed down for our own good, and Emacs is now nothing more than a glorified Org editor that forces you to find your own cloud syncing.
I’ve found code generation tools a boon to my Emacs-ing because I can use them to help write Emacs lisp code for any customization I think I might need in the moment without breaking the flow of concentration towards my primary task.