Generalized Additive Models: An Introduction with R by Simon N. Wood
This book is very strong on the fundamentals, while the R code is minimal and easy to follow.
Readit Newsblackbear_ commented on · Posted by u/hereme888
And so fact checking is back in vogue. Seems pretty biased.
blackbear_ commented on Backpropagation is a leaky abstraction (2016) karpathy.medium.com/yes-y... · Posted by u/swatson741
I have a naive question about backprop and optimizers.
I understand how SGD is just taking a step proportional to the gradient and how backprop computes the partial derivative of the loss function with respect to each model weight.
But with more advanced optimizers the gradient is not really used directly. It gets per weight normalization, fudged with momentum, clipped, etc.
So really, how important is computing the exact gradient using calculus, vs just knowing the general direction to step? Would that be cheaper to calculate than full derivatives?
Two thoughts:
> how important is computing the exact gradient using calculus
Normally the gradient is computed with a small "minibatch" of examples, meaning that on average over many steps the true gradient is followed, but each step individually never moves exacty along the true gradient. This noisy walk is actually quite beneficial for the final performance of the network https://arxiv.org/abs/2006.15081 , https://arxiv.org/abs/1609.04836 so much so that people started wondering what is the best way to "corrupt" this approximate gradient even more to improve performance https://arxiv.org/abs/2202.02831 (and many other works relating to SGD noise)
> vs just knowing the general direction to step
I can't find relevant papers now, but I seem to recall that the Hessian eigenvalues of the loss function decay rather quickly, which means that taking a step in most directions will not change the loss very much. That is to say, you have to know which direction to go quite precisely for an SGD-like method to work. People have been trying to visualize the loss and trajectory taken during optimization https://arxiv.org/pdf/1712.09913 , https://losslandscape.com/
blackbear_ commented on Language models are injective and hence invertible arxiv.org/abs/2510.15511... · Posted by u/mazsa
Depends on your definition of the model. Most people would be pretty upset with the usual LLM providers if they drastically changed the sampling strategy for the worse and claimed to not have changed the model at all.
Tailoring the message to the audience is really a fundamental principle of good communication.
Scientists and academics demand an entirely different level of rigor compared to customers of LLM providers.
I really don't know... They made a model to predict the number of views of a video from its metadata, then assumed that model errors correspond to intentional manipulation rather than, you know, metadata not being fully predictive.
blackbear_ commented on Android's sideloading limits are its most anti-consumer move makeuseof.com/androids-si... · Posted by u/josephcsible
> What part of cheaper
The iPhone 17 is the same price as the Pixel 10
> better
But the iPhone 17 has better hardware features, like UWB, better cameras, and a _far_ faster CPU.
> open source
Only if you install Graphene, and then never install anything that requires Google Play Services, which is basically every commercial app.
GOS allows you to install and use apps from the Play Store and the vast majority of them works flawlessly.
blackbear_ commented on Matrices can be your friends (2002) sjbaker.org/steve/omniv/m... · Posted by u/todsacerdoti
Spatial transformations? Take a look at the complex matrices in Fourier transforms with nth roots of unity as its elements. The values are cyclic, and do not represent points in an n-D space of Euclidean coordinates.
Yes; I wrote linear transformation on purpose not to remain constrained on spatial or geometric interpretations.
The (discrete) Fourier transform is also a linear transformation, which is why the initial effort of thinking abstractly in terms of vector spaces and transformations between them pays lots of dividends when it's time to understand more advanced topics such as the DFT, which is "just" a change of basis.
blackbear_ commented on Matrices can be your friends (2002) sjbaker.org/steve/omniv/m... · Posted by u/todsacerdoti
There are a lot more ways to look at and understand these mysterious beasts called matrices. They seem to represent a more fundamental primordial truth. I'm not sure what it is. Determinant of a matrix indicate the area of or volume spanned by its component vectors. Complex matrices used in Fourier transform are beautiful. Quantum mechanics and AI seem to be built on matrices. There is hardly any area of mathematics that doesn't utilize matrices as tools. What exactly is a matrix? Just a grid of numbers? don't think so.
The fundamental truth is that matrices represent linear transformations, and all of linear algebra is developed in terms of linear transformations rather than just grid of numbers. It all becomes much clearer when you let go of the tabular representation and study the original intentions that motivated the operations you do on matrices.
My appreciation for the subject grew considerably after working through the book "Linear Algebra done right" by Axler https://linear.axler.net
blackbear_ commented on Why Wikipedia cannot claim the Earth is not flat en.wikipedia.org/wiki/Wik... · Posted by u/duncanjbrown
Why are fringe beliefs something that need protecting against?
All progress starts out as a fringe belief.
But not all fringe beliefs lead to progress, in fact most of them don't.
https://en.wikipedia.org/wiki/Pharmacovigilance#Adverse_even...