If you believe the bitter lesson, all the handwavy "engineering" is better done with more data. Someone likely would have written the same thing as this 8 years ago about what it would take to get current LLM performance.

So I don't buy the engineering angle, I also don't think LLMs will scale up to AGI as imagined by Asimov or any of the usual sci-fi tropes. There is something more fundamental missing, as in missing science, not missing engineering.

I am working on a project that uses LLM to pull certain pieces of information from semi-structured documents and then categorize/file them under the correct account. it's about 95% accurate and we haven't even begun to fine tune it. i expect it will require human in the loop checks for the foreseeable future, but even with a human approval of each item, its going to save the clerical staff hundreds of hours per year. There are a lot of opportunities in automating/semi-automating processes like this, basically just information extraction and categorization tasks.

People here are giving you mathematical answers which is what you are asking for, but I want to challenge your intuition here.

In construction, grading a site for building is a whole process involving surveying. If you dropped a person on a random patch of earth that hasn't previously been levelled and gave them no tools, it would be a significant challenge for that person to level the ground correctly.

What I'm saying is, your intuition that "I can look around me and find the minimum of anything" is almost certainly wrong, unless you have a superpower that no other person has.

When you look at, for instance, a bowl, or even one of those egg carton mattress things, and you want to find the global minimum, you are looking at a surface which is 2 dimensions in and 1 dimension out. It's easy enough for your brain to process several thousand points and say ok the bottom of the bowl is right here.

When a computer has a surface which is 2 dimensions in and 1 dimension out, you can actually just do the same thing. Check like 100 values in the x/y directions and you only have to check like 10000 values. A computer can do that easy peasy.

When a computer does ML with a deep neural network, you don't have 2 dimensions in and 1 dimension out. You have thousands to millions of dimensions in and thousands to millions of dimensions out. If you have 100000 inputs, and you check 1000 values for each input, the total number of combinations is 1000^100000. Then remember that you also have 100000 outputs. You ain't doin' that much math. You ain't.

So we need fancy stuff like Jacobians and backtracking.

Evaluating a function using a densely spaced grid and plotting it does work. This is brute-force search. You will see the global minima immediately in the way you describe, provided your grid is dense enough to capture all local variation.

It's just that when the function is implemented on the computer, evaluating so many points takes a long time, and using a more sophisticated optimization algorithm that exploits information like the gradient is almost always faster. In physical reality all the points already exist, so if they can be observed cheaply the brute force approach works well.

Edit: Your question was good. Asking superficially-naive questions like that is often a fruitful starting point for coming up with new tricks to solve seemingly-intractable problems.

I've been saying this on HN since 2022:

For all the pro-WFH/fully remote developers on HN who live in North America, you're going to be in for a surprise when your company decides to replace you with someone living in another country. Why hire you when the company can hire someone who costs 1/5 of you and is willing to work harder without complaining? Both of you are remote anyway. So what if the new hire works at night and sleeps during the day?

For all pro-WFH/fully remote developers living in North America, you should be cheering for return to office mandates. It'll probably save your career long-term.