I recently picked up my fountain pens after a 5 year hiatus. I used to collect them in my college years and continued using them in grad school. I loved the feeling of writing formulas on blank sheets of paper and pretending to look like a Physicist. But it all stopped after I began working with a computer. I started to collect HHKBs instead. I find it much easier to gather my thoughts and think logically and remember stuff when I’m writing with my hands.
There’s a running theme in my life that I prefer manual things. I enjoy practicing Olympic weightlifting, driving my 14 year old manual transmission and inking paper and tying with my HHKBs in Vim. To me, tools are the best when they feel like an extension of my physical body.
I think the literature is clear on that?
"LoRA vs Full Fine-tuning: An Illusion of Equivalence" -- https://arxiv.org/abs/2410.21228v1
Quoting from the conclusions:
> The paper describes the finding that LoRA and full fine-tuning, with equal performance on the fine-tuning task, can have solutions with very different generalization behaviors outside the fine-tuning task distribution. We found that LoRA and full fine-tuning yield models with significant differences spectral properties of their weight matrices: LoRA models often containing “intruder dimensions”, high-ranking singular vectors approximately orthogonal to the singular vectors of pre-trained weight matrices. The existence of intruder dimensions correlates with the fine-tuned model forgetting more of the pre-training distribution as well as forgetting more when trained on tasks sequentially in a continual learning setup.
I'm surprised they didn't cite this; it's a well known paper.