Hey Paul, I'm a Mentat author.

> I also notice that the instructions prompt that mentat uses seems to be inspired by the aider benchmark? Glad to see others adopting similar benchmarking approaches.

We were inspired by you to use Exercism as a benchmark, thank you! We will add attribution for that. We switched our original instruction prompts for that benchmark to be similar to Aiders to allow for fair comparison.

> After looking around a bit, there seems to be a bunch of aider code in your repo. Some attribution would be appreciated.

We have an unused implementation of your output response format (https://github.com/AbanteAI/mentat/blob/main/mentat/parsers/...), but I don't know what else you are seeing? We implemented that to compare with our response formats and didn't find much difference in performance.

I’ve been using the new model with Aider since it was released, and my anecdata agrees—the “edits applied successfully “ failure rate is much lower than classic gpt4.

Also THANK YOU for Aider! I talk it up to all my programmer friends; it really feels like a glimpse into the future of coding.

Isn't it a good thing that of the benchmarks they ran, the newer model has fewer of the answers memorized (aka, its parroting less)?

Wouldn't this actually be exactly proof that the model has improved over its predecessor by having to solve the problem itself rather than rely on memory?

What use is a model that memorizes the answers to all the benchmarks (see the 7b models on open llm leaderboard for more info on that).

I feel like I see this A LOT these days. If you do a Show HN (for example) and your project is directly inspired by somebody else's who came before you, the least you can do is give nominal attribution.

What is it about software development in particular that makes people so seemingly ethically unfettered by blatant plagiarism?

I am also noticing a massive improvement over the old model

Sorry about that. We updated the blog with attribution and put an attributing comment in our code base where we use your benchmarking prompts. We'll probably delete our implementation of your response format later today since we just had it for benchmarking.

Does aider work with c# at all?

  - C
  - C#
  - C++
  - Emacs Lisp
  - Elixir
  - Elm
  - Go
  - Java
  - Javascript
  - OCaml
  - PHP
  - Python
  - QL
  - Ruby
  - Rust
  - Typescript

Yes. I see people make this mistake time and again when evaluating LLMs. For a proper comparison, it's not enough to simply throw less than a hundred questions at it and point to a single digit difference. Not to mention that LLMs have some inherent randomness, so even if you passed the exact same tasks to the same model you would expect some variance.

I see a lot of room of improvement in how we apply statistics to understanding LLM performance.

I’m not surprised, most people can’t even tell the median from the mean.

> I think the evidence from these tests are not strong enough to draw conclusions from.

I've used gpt4 turbo for some coding problems yesterday. It was worse. That's enough to draw conclusions for me.

Probably because it's too expensive. Prompt can be processed quickly but output tokens are much slower (and that means more expensive).

From my local test on a 13B model, output tokens are 20-30x more expensive than input tokens. So OpenAI's pricing structure is based on expectation that there's much more input than output tokens in an average response. It didn't matter too much if a small percentage of requests used all 4k tokens for output, but with 128k it's a different story.

I believe openai wants to lower the time it takes for requests to finish to be able to accept more requests per server/gpu. ie money.

if i'm not mistaken, the model has to be trained for a specific context window

Do you have a link or gist of an example run you tried? I'd be curious to try something similar.

Memorization often gets a bad rap as the underachiever's shortcut. However, it's a fundamental component of any learning process! Our ability to reason, solve problems, and innovate is all built upon a foundation of memorized information. In fact, it's precisely the reason humans have thrived for so long; we were able to memorize and pass down knowledge culturally long before the written word, not because we were 100 times smarter than our nearest cousins. Without memorization, be it in our brains or AI algorithms, there's no foundation to build upon for higher reasoning.

It's hard to decide for me without seeing the data. Even if you don't know the exact exercise, seeing the title and the function name/parameters is often enough for me to guess what the challenge is. I checked the public questions on exercism and almost all of those (that I spot checked) that contained the function name were extremely obvious. Knowing it's a programming challenge would also improve my guessing chances.

For example the function stubs I can find are "value_of_card(<card>)" in exercise "Black Jack", or "generate_seat_letters(<number>)" in exercise "Plane Tickets". I think I could guess those without seeing the rest of the question.

You can call it whatever you want, all I know is I used to write programs in lines of code, then blocks of code at a time, spit out by LLMs

Using GPT-4 Turbo yesterday, I feel like I'm moving to pages of code at a time now.

Taking the ideas in my head and turning them into reality is so easy now

So how can it solve novel problems? Internet does not have all combinations for every possible task with any random programming language, library or constraints. It can even solve problems with non-existing programming languages and libraries if you describe them, if that's just memorization then I don't know what it isn't.

If that's your takeaway from this then you really missed the point. The implication here is that gpt-4 to gpt-4-turbo represents a leap away from memorization and toward better reasoning with a more complete world model.

"They memorized all the problems" is not what was found here and still a wrong overcorrection.

Ok but you understand there's a body of literature that shows that LLMs don't "just" memorize

+100 to that. My biggest scepticism is people actually creating a new problem while thinking they are solving problem. Don't get me wrong, translating natural language ideas into code is fun and all, the truth it is also code, yet in ambiguous language format given to the machine.

When did natural language became better for expressing development ideas than code? I know – when you don't know how to code in the first place. Then you will have to bet on all of the ambiguities of the language, cultural and meta-physical which words carry in order to hack your thing together instead of expressing yourself directly and explicitly.

Finally what is beautiful about strict code format we are so used to - it is truly the fastest and shortest path to get your thing done, in case you possess the knowledge needed.

From a black box point of view and one angle, gpt is a web filter where it will try to find you the exact thing you are looking for but from memory. Vs google you have to distill all the info into what you need

"memorize" implies they can only recite things verbatim and that's ignoring the massive leap in being able to synthesize disjoint "memories" in new ways.

even if it's not true AI or even an architecture with the potential to become AI, LLMs are already good enough to provide real world value. Obviously "super autocomplete" isn't as sexy as true AI, but still very useful

if the benchmark means replicating the experience of taking technical interviews by most people, then this is a spot on approach and serves the potential user right.

LLMs are lossy compression

People here spent a lot of time (and money) in school learning to do things that can now be automated. The whining is just beginning.

Or the programming quiz problems it tried to "solve" were in fact posted elsewhere also so it cheated on the ones it got right too.