I feel the advent of GPT-3 / Codex and the power that comes with it has been surprisingly underestimated - maybe because the folks who would benefit most from it (people who don't write code, like myself) haven't really caught up with it, while the ones using it (mostly people who do write code) maybe don't see the immediate benefits as much (and maybe are a little bit concerned about what this will mean to their economic position in all this...?)
I've played around a ton with simple GPT-3 code over the last few weeks and am in total awe of the possibilities this opens up to folks like me (who don't write code)
E.g. I had GPT-3 write me app script integrations of itself (!) and stablediffusino into a Google Sheet, allowing me to build a massive sandbox playground for various prototyping experiments.
That is the opposite of my experience. Being a skilled coder means I can verify the outputs quickly. Further using a beginner coding “style” actually encourages the model to output similarly “beginner” code.
It has been the largest boon to my productivity in years. I code exclusively in Python where copilot is quite good. In my experience the comments of supposed “experienced” coders mad about copilot are either GPL proponents or people who work with mostly statically typed languages. Almost none of them have actually used Copilot and certainly didn’t use it long enough to ascertain a “good faith” criticism of it (so not just “it outputs subtle bugs throw it in the trash!”)
There’s also just a natural reaction to hate things poised to make you redundant ( probably should have considered this when becoming a coder, though).
yes. reading some of your twitter posts I'd say I share your sentiment on how codex will upend the programming world. but I think better integrations and statement of intents need to be more fleshed out, given openAI & friends are tech companies and not user companies, the UX part will take time.
Thank you for posting this. Code generation is an interest of mine and a core foundation of the technology that I build.
It will take me some time to read through this blog, but I have a question:
Is there any research being done where people are using large language models to generate or transform syntax trees for programs as opposed to operating with programs as simply streams of tokens?
Is there any research being done where people are using large language models to generate or transform syntax trees for programs
I saw a mention of this on Twitter the other day. I have not read the paper, and it's anonymous, so I don't know how serious and/or credible this is. But it was apparently an actual ICLR submission, so it might be worth a look. Not sure it's exactly what you're interested in, but it seems like it might be in the same neighborhood.
Thanks for the link. This is interesting work, although not related to operations on program syntax trees.
From what I understand, they are using reinforcement learning techniques inspired by AutoML to learn architectures for language models that are effective at code generation.
They have reinforcement learning models optimizing the neural networks that generate code.
Pretty cool and reminds of work that followed on the heels of AutoML to build neural networks to automatically solve arbitrary ML problems (with the problem as the input parameter).
Can you clarify what this means? In this context seems like it could mean either machine learning language model or programming language grammar/semantics.
Not sure if there is a technical definition of the term. I am using it to refer to the transformer-based neural networks that are used to build models like Open AI's Codex (which GitHub Copilot uses IIRC).
I believe they are called large because they contain orders of magnitude more parameters than the models that came before them. They do not fit in memory on a single training device (GPU or TPU usually) and require serious infrastructure work to train effectively.
The models are also trained on large datasets. These datasets are usually text streams. In the case of GitHub Copilot, it was the textual source code of a large percentage of the git repositories on GitHub. (Maybe supplemented with other code?)
I am curious if anyone is parsing source code into syntax trees and then training large language models to operate on the resulting graph structures.
Edit: On a related note - GitHub Copilot is extremely good at generating syntactically correct code. Has anyone studied which parameters in the language model are related to learning programming language syntax? Are the parameters in the model related to javascript syntax "close to" the parameters in the model which learn Go syntax?
Edit 2: For GitHub Copilot, did they use the content of git diffs to learn the kinds of edits people make to existing code?
Edit 3: I realize that a random HN post about some blog is probably not the best place to be asking these questions. If anyone could point me to a better place, I would appreciate it.
it is very related, and I spent an ungodly amount of time to make sure it is visually compact and to the point, so I can deliver knowledge directly into your eye holes with minimal friction. It's super heartwarming to see someone appreciating this aspect.
Readit News
I feel the advent of GPT-3 / Codex and the power that comes with it has been surprisingly underestimated - maybe because the folks who would benefit most from it (people who don't write code, like myself) haven't really caught up with it, while the ones using it (mostly people who do write code) maybe don't see the immediate benefits as much (and maybe are a little bit concerned about what this will mean to their economic position in all this...?)
I've played around a ton with simple GPT-3 code over the last few weeks and am in total awe of the possibilities this opens up to folks like me (who don't write code)
E.g. I had GPT-3 write me app script integrations of itself (!) and stablediffusino into a Google Sheet, allowing me to build a massive sandbox playground for various prototyping experiments.
Here's a thread on that process if you're curious: https://twitter.com/fabianstelzer/status/1577416649514295297
It has been the largest boon to my productivity in years. I code exclusively in Python where copilot is quite good. In my experience the comments of supposed “experienced” coders mad about copilot are either GPL proponents or people who work with mostly statically typed languages. Almost none of them have actually used Copilot and certainly didn’t use it long enough to ascertain a “good faith” criticism of it (so not just “it outputs subtle bugs throw it in the trash!”)
There’s also just a natural reaction to hate things poised to make you redundant ( probably should have considered this when becoming a coder, though).
Feel free to ask me questions uhhh... How does hacker news work lmao do i get notifications even hmm.
I'll uhh... refresh this page once in awhile and see if I can help!
if that don't work prolly just ping me on twitter.
It will take me some time to read through this blog, but I have a question:
Is there any research being done where people are using large language models to generate or transform syntax trees for programs as opposed to operating with programs as simply streams of tokens?
I saw a mention of this on Twitter the other day. I have not read the paper, and it's anonymous, so I don't know how serious and/or credible this is. But it was apparently an actual ICLR submission, so it might be worth a look. Not sure it's exactly what you're interested in, but it seems like it might be in the same neighborhood.
https://openreview.net/forum?id=SKat5ZX5RET
From what I understand, they are using reinforcement learning techniques inspired by AutoML to learn architectures for language models that are effective at code generation.
They have reinforcement learning models optimizing the neural networks that generate code.
Pretty cool and reminds of work that followed on the heels of AutoML to build neural networks to automatically solve arbitrary ML problems (with the problem as the input parameter).
Can you clarify what this means? In this context seems like it could mean either machine learning language model or programming language grammar/semantics.
I believe they are called large because they contain orders of magnitude more parameters than the models that came before them. They do not fit in memory on a single training device (GPU or TPU usually) and require serious infrastructure work to train effectively.
The models are also trained on large datasets. These datasets are usually text streams. In the case of GitHub Copilot, it was the textual source code of a large percentage of the git repositories on GitHub. (Maybe supplemented with other code?)
I am curious if anyone is parsing source code into syntax trees and then training large language models to operate on the resulting graph structures.
Edit: On a related note - GitHub Copilot is extremely good at generating syntactically correct code. Has anyone studied which parameters in the language model are related to learning programming language syntax? Are the parameters in the model related to javascript syntax "close to" the parameters in the model which learn Go syntax?
Edit 2: For GitHub Copilot, did they use the content of git diffs to learn the kinds of edits people make to existing code?
Edit 3: I realize that a random HN post about some blog is probably not the best place to be asking these questions. If anyone could point me to a better place, I would appreciate it.
The operations that transform syntax trees are just programs as well which... Can be represented as a steams of tokens.
But this is likely helpful if it makes it easier to learn by the NN. I'm certain there are works that do this, but not off the top of my head rn