Watching geohot code a general matrix multiply algorithm from 0.9 GFLOPS and optimising it to 100 glops by only tinkering with cache locality, it makes me wonder how much effort should be put into single threaded performance before ever thinking about multi threading
Readit Newsremontoire commented on Is parallel programming hard, and, if so, what can you do about it? mirrors.edge.kernel.org/p... · Posted by u/nequo
remontoire commented on What are transformer models and how do they work? txt.cohere.ai/what-are-tr... · Posted by u/tomcam
There's one aspect I never saw explained. Why is masking used instead of a sliding window? Why even bother with masking when future tokens can be easily hidden by simply positioning the context window before the current token? Isn't sliding window optimal for maximizing context available to the model? Is masking done because moving the window would impact computational cost or output stability/quality? Can anyone shed light on this?
I think it's because you want to be able to predict the next token using only 1 token or the whole context window (and any size inbetween). So, you end up getting n different losses for each text snippet (where n is the size of the context window).
If i'm wrong, can someone correct here, would be useful to know.
So the reason he thinks this is because GM and ford have announced more "models".
At no point has the limiting factor been the number of models. The limiting factor is and will be battery cell supply.
This article makes no mention of legacy OEMs making any attempt at solving their supply constraints.
remontoire commented on Differences with other Lisps clojure.org/reference/lis... · Posted by u/susam
> once you start using those you no longer are able to use libraries
This isn't true: if the libraries you use are designed to use generic functions for their internals, rather than normal functions. I've written code using custom data structures heavily, with little or no impact on which libraries I could use.
> The common lisp community has rejected reader macros
cl21 is just confusing matters here. Plenty of code uses named-readtables to use arbitrary reader macros.
I don't understand the "interop reasons" you're talking about: as long as you use named-readtables, interop basically Just Works.
> It's also not possible to put those on the wire (easily) for communication between common lisp processes. Breaking the whole point of homoiconicity.
Using the reader on untrusted input is a bad idea in either Common Lisp or Clojure, because both readers can execute arbitrary code. However, there are packages like safe-read ( https://github.com/phoe/safe-read ) and Eclector ( https://github.com/s-expressionists/eclector ) that let you solve this problem pretty trivially in a safe way. Not to mention projects like my cl-edn ( https://github.com/fiddlerwoaroof/cl-edn ).
It looks like you're right for the most part. Literals are supported if you use https://github.com/vseloved/rutils. If your cl-edn actually works then a combo of rutils and cl-edn could bring most of the value I found in clojure to common lisp.
Deleted Comment
remontoire commented on Differences with other Lisps clojure.org/reference/lis... · Posted by u/susam
This isn’t really a thing, though: Common Lisp has built-in literal syntax for n-dimensional arrays and structs and libraries like fset provide literal syntax for Clojure-style immutable data structures.
I've used fset and cl21. The problem is that once you start using those you no longer are able to use libraries. Data structures have to be built into core.
The common lisp community has rejected reader macros for interop reasons the last time I checked (in the case of cl21)
Even the docs for fset doesn't show the use of data literals. it does things like (set) and (isetq s2 (set 'c 1 "foo" 'a 'c))
This pretty much proves my point. it's not fun typing (set 'c 1 "foo" 'a 'c) instead of #{'c 1 "foo" 'a 'c}. It's also not possible to put those on the wire (easily) for communication between common lisp processes. Breaking the whole point of homoiconicity.
Deleted Comment
remontoire commented on Differences with other Lisps clojure.org/reference/lis... · Posted by u/susam
> There is more to collections than lists. You can have instances of empty collections, some of which have literal support ([], {}, and ()). Thus there can be no sentinel empty collection value.
Probably my favorite feature over common lisp. Combined with comma (,) being treated as whitespace (ie, use comma if you want to but you don't have to), makes typing out data structures in clojure a much more enjoyable process than other languages.
This ease of typing out data structures also extends to edn (clojure's version of json)
remontoire commented on A solution to Newcomb's paradox fakenous.net/?p=2578... · Posted by u/erwald
This doesn't seem like an actual solution to the problem as normally stated. The point of Newcomb's problem is that if someone makes decisions on the basis of accurate prediction of your choices, you should commit to making the choice that gives you the maximum payout.
In other words, the problem as stated is that if you choose both boxes there will be nothing in the opaque box, and if you choose one box there will be a large payout in the opaque box.
Redefining the problem to say "they've already made the decision and filled the box or boxes, so your choice won't change the outcome" is sidestepping the point of the problem.
This is definitely not an actual solution. It's a first order solution to a non-first order problem.
Can someone elaborate on what a grammar means in the context of time series forecasting?