Readit News logoReadit News
ulber commented on Want to get started with LLMs? Here's what you need to know   flyte.org/blog/getting-st... · Posted by u/samhita-alla
behnamoh · 2 years ago
I'm not sure it's even helpful to talk about transformer at all[0]. Modern LLMs are different than the pure transformer architecture in many detailed aspects. The elephant in the room being that they're not even full transformer—only the decoder part.

[0]: One reason: Never once did I need to know the transformer architecture in order to be able to use these models (prompt engineering, chaining, working with local models, etc.).

I argue that the knowledge of concepts such as ROPE, Mirostat, monkeypatching, etc. is much more crucial than knowing how transformer models work.

> I tend to stick with the higher level explanation that they can predict the next word (or next sentence) based on their training text,

I think the same way, but I think it reduces LLMs into "black boxes"—many other models can also predict next tokens based on probabilities. I think we need something that at least captures the general mechanism by which LLMs predict the next token.

ulber · 2 years ago
>I think we need something that at least captures the general mechanism by which LLMs predict the next token.

This bit varies a lot since the capabilities involved in prediction depend on the data. If the text is a math book and the prompt is "... three plus five apples is a total of " the crucial capability is arithmetic (plus of course NL capabilities). On the other hand, if you're completing a post from /r/relationship_advice the capabilities involved will be (vaguely) maintaining literary tone, theory of mind, psychology, etc. Within a text the capabilities needed will also vary a lot, where you might need theory of mind at some crucial inflection points, but most of the time its already clear what is going to be said and the model just has to get the wording right.

So, my take would be to really think hard about the data to understand how predictions might be made.

ulber commented on People had to be convinced of the usefulness of electricity   smithsonianmag.com/smart-... · Posted by u/olalonde
0x0 · 3 years ago
"...a brand new power generation facility that could generate 770,000 kilowatt-hours" - what does this even mean? Did the facility produce a certain amount of electricity and then it had to be shut down, after it had produced 770MWh ? Can't produce any more kWh so just fire everyone and demolish the facility?
ulber · 3 years ago
I bet they forgot to include that it can generate that 770MWh every hour.
ulber commented on The vertical farming bubble is finally popping   fastcompany.com/90824702/... · Posted by u/jeffbee
Ekaros · 3 years ago
How do you supply enough power to vertical farms without those same externalities? And then how green is it to build and run them? Considering all the equipment, infra and inputs. It is not like that stuff magically comes from thin air with vertical farming.
ulber · 3 years ago
Maybe we don't, but we should let market forces sort it out by doing our best to tax the externalities of power generation too.
ulber commented on Why Is Tipping Everywhere You Checkout?   maxinomics.com/blog/why-i... · Posted by u/isthispermanent
lenzm · 3 years ago
> Deadweight loss happens when a person who is willing to pay more is charged less than the amount they're willing to pay.

> Tom is willing to pay $6 for coffee and the shop only charges him $4, the deadweight loss is $2.

That is a consumer surplus, not deadweight loss.

ulber · 3 years ago
The argument the article makes is indeed a bit blunt.

The idea here is that tipping enables better price discrimination, which in turn allows a higher "quantity" of coffee to be produced. With coffee shops that "quantity" will translate to things like 1) higher density or 2) more attractive placement of coffee shops, both of which increase coffee consumption by making it easier to pick one up, or 3) higher quality (think small batch) coffee being produced, which increases the quantity of labor being sold in a cup.

I still don't enjoy this tipping culture, but the argument being made makes sense when you fill in the details.

ulber commented on Gamedevs not baking in monetization are “fucking idiots”, says Unity CEO   pocketgamer.biz/interview... · Posted by u/OskarS
munificent · 3 years ago
He's the Larry Ellison of games.

Less a functioning human and more like a mid-sized neural net trained only on making game company short-term stock evaluations increase.

He's the game dev equivalent of Nick Bostrom's paperclip maximizer thought experiment.

ulber · 3 years ago
As an aside, Universal Paperclips is the best clicker game I've played and has zero monetization.

https://www.decisionproblem.com/paperclips/

ulber commented on Heathrow tells airlines to stop selling additional flights until September   headforpoints.com/2022/07... · Posted by u/edward
prvit · 3 years ago
People making more than $300k/yr TC aren’t exactly a small minority on HN. While the VIP service at Heathrow is unusually expensive compared to other airports, it’s really not a crazy price to avoid some very stressful hours.
ulber · 3 years ago
As a crude point of comparison, you'd be buying this service at about 917 euros/hour. With a pretty lax 1600 hours/year you'd have to make almost 1.5M per year to match the rate.
ulber commented on When U.S. Air Force discovered the flaw of averages (2016)   thestar.com/news/insight/... · Posted by u/skanderbm
ulber · 3 years ago
This was a great read. I kept expecting it to reference the "Curse of Dimensionality" [1] as an explanation for why we see no close-to-average samples in high-dimensional spaces. I'm guessing the book this is an excerpt from might. In any case, I enjoyed this as a relatable instance of the concept!

[1]: https://en.m.wikipedia.org/wiki/Curse_of_dimensionality

ulber commented on 2XL – ‘70s toy that faked AI with an 8-track [video]   youtube.com/watch?v=amuRI... · Posted by u/fortran77
Izkata · 3 years ago
AI is an unattainable goal, not because we'll never create it, but because "what is AI" changes over time as we figure out how to do things, and give them their own name. Decision trees, machine learning, deep learning - each of these was "AI" at some point, but now that we can do them the magic is gone and "AI" has become more distant than it was before.

(Heavily paraphrased from a longer post I read years and years ago, no idea where)

ulber · 3 years ago
I get the point but also fully disagree. At the point where computer cognition is good enough for humans to truly fall in love, build friendships or come to hate as an enemy, I personally think not recognizing it as AI will become a fringe belief.
ulber commented on Working on the weekends – an academic necessity?   thegradient.pub/working-o... · Posted by u/andreyk
j7ake · 4 years ago
Although I agree that working weekends is common in academia, I believe that academia has more freedom in how to organise your time.

Are there any other jobs where someone can take a month off at a time to travel to conferences, meet with international colleagues, and go on long visits to other institutions? Compare this with any tech or high powered job that has much more sensitive deadlines.

The only real deadlines for academia are: PhD graduation (4-6 years), postdoc fellowship ends (2-4 years), and tenure (5-8 years).

Between those selection cutoffs you are absolutely free to structure your life however you want.

If it doesn’t work out, oh well, you go into industry and collect double pay.

ulber · 4 years ago
I feel this description overstates the freedom. Academia is full of rather hard deadlines, as the conference you're targeting a paper for might be the only fitting one for many months. Missing these deadlines can delay the whole "pipeline" of research, as you'd like to build on your existing work and properly cite it, but publishing subsequent work might make the original result harder to publish. Grant application deadlines are similarly hard.

In contrast, deadlines in engineering are often even not expected to be hit.

The "freedom" to travel for conferences is an integral part of your job as a researcher: either you network and sell your ideas, or you stay in obscurity. Of course travelling on someone else's dime can be fun, but the same is true of all business travel. It stops being fun the moment it becomes a chore and you'd rather be home putting time into your hobbies or family. Then it's just more work.

If writing grant applications is your hobby and you're married to your research, academia can be great, but the freedom doesn't include a balance with all the other parts of life. I know I'm not fully contradicting you, there is indeed a lot of freedom to choose what you work on. I just think it's important that people considering academia understand what the job actually consists of.

ulber commented on DeepMind: A Generalist Agent   deepmind.com/publications... · Posted by u/extr
colemannugent · 4 years ago
What I really want to know is what kind of robot arm motion is produced when the network is given a cat image to classify. More specifically, what kind of insights has it learned from one control domain that it then applied to another?

I imagine that the simulated 3D environment and the actual control of the robot arm must have some degree of interconnection neurally.

ulber · 4 years ago
You could also train for this kind of interconnectedness by designing tasks that are explicitly multi-modal. For example, you could:

- Stack boxes collaboratively by controlling your own arm and communicating with another agent helping you.

- First produce a plan in text that another agent has to use to predict how you're going to control the arm. You'd get rewarded for both stacking correctly and being predictable based on the stated plan.

u/ulber

KarmaCake day609September 25, 2012View Original