Do you also block stack overflow and give guidance to never copy code from that website or elsewhere on the Internet? I'm legitimately curious - my org internally officially denounces the copying of stack overflow snippets. Thankfully for my role it's moot as I mostly work with an internal non-public language, for better or worse, and I have no idea how well that's followed elsewhere in the wider company.

Same here. I’ve directed our teams and infra managers that we must be able to block the use of copilot for our firm’s code.

Id be very surprised if the other large enterprises that I have worked at downs doing exactly the same thing. Too much legal risk, for practically no benefit.

Ask your legal department about that. Sure, engineers don't care about licensing at all, but we are not the only players here.

This is absolutely not true. While some individuals might not care and might not always conform to their companies' policies, most companies have policies, and most employees are aware of and mindful of these policies.

It's absolutely the case that before using certain libraries, most engineers in large corporations will make sure they are allowed to use that library. And if they don't, they are doing their job very badly IMO.

This kind of sucks honestly, copy and pasting without understanding has lead to all sorts of issues in IT. Not to mention legal issues as mentioned by another reply.

If the legal concerns are well-known, then what you are describing might be viewed as criminal negligence (at worst) and or insufficient duty of care (at best). Such engineers should be held fully responsible and accountable for their actions.

That's optimistic. The people who would rely heavily on this sort of thing are going to be the worst at detecting what a "bad autocomplete result" would look like. But even if you are capable of judging that you've got a good one, it still doesn't inform you of the obvious potential licensing issues with any bit of code.

Surely somebody working on this project foresaw this problem…

Well... the whole training set is licensed, so you can't really get rid of it. I think that the technology they are using for this is just not ready.

Unlicensed code just means “all rights reserved.” You’d need to limit it to permissively licensed code and make sure you comply with their requirements.

Which licenses would it be ok that the training material is licensed under, though? If it produces verbatim enough copies of eg. MIT licensed material, then attribution is required. Similar with many other open source-friendly licenses.

On the other hand, if only permissive licenses that also don't require attribution is used, well, then for a start, the available corpus is much smaller.

The overwhelming majority of code on Github, even code under permissive licenses, require attribution of the original authors.

How would they do that?

Yes: not all code on GitHub is licensed in a way that lets you use it at all. People focus on GPL as if that were the tough case; but, in addition to code (like mine) under AGPL (which you need to not use in a product that exposes similar functionality to end users) there is code that is merely published under "shared source" licenses (so you can look, but not touch) and even literally code that is stolen and leaked from the internals of companies--including Microsoft!... this code often gets taken down later, but it isn't always noticed and either way: it is now part of Copilot :/--that, if you use this mechanism, could end up in your codebase.

If you publish the code anywhere, potentially. You could be (unknowingly) violating the original license if the code was copied verbatim from another source.

How much of a concern this is depends heavily on what the original source was.

Ahh, so it's the most pointless interpretation of the phrase "filters to block offensive words", where it is stopping the user from causing offense to the AI rather than the other way around.

Interesting how this continues to be an issue for GPT3 based projects.

A similar thing is happening in AI Dungeon, where certain words and phrases are banned to the point of suspending a users account if used a certain amount of times, yet they will happily output them when it is generated by GPT3 itself, and then punish the user if they fail to remove the offending pieces of text before continuing.

Lol, how does that make any sense? I mean, all these word blacklists are always pretty stupid, but at least you can usually see the motivation behind them. But in this case I'm not even sure what they tried to achieve, this is absolutely pointless.

Maybe Github just doesn't have many repos to control death factories and execution squads?

does it also censor "lesbian"?

While I strongly doubt they would use Wikipedia as a training set, has anyone done a search of GitHub code to see if other projects have copied-and-pasted that function from Wikipedia into their more-permissive codebases?

Then they all copied from the same source, and what's more, they are all derivative of a GPL work (and thus they should be GPLed themselves)

that will make a great defense at a copyright court.

"your honor, i would like to plead not guilty, on the basis that i just robbed that bank because i saw that everyone was robbing banks on the next city"

...on the other hand, that was the exact defense tried for the capitol rioters. So i don't know anything anymore.

Apparently Carmack was not the original author, the origin I believe is SGI somewhere in the deep dark 90s.

I get why some people were saying it made them a better programmer. Of course it did, it's copy-pasting Carmack code.

Good luck checking every code line for license violations

And it's up to the end user to evaluate the tool that makes the suggestions.

As someone who does code reviews the thought the developer didn’t code the code submitted to be merged never would cross my mind.

And it is completely impossible for the user to do so.

So, the tool is worthless if you want to use it legally.

Since the information is encoded in model weights, I doubt that erasure is even possible. Only post-retrieval filtering would be an option.

It only goes to show that intransparent black-box models have no place in the industry. The networks leak information left and right, because it's way too easy to just crawl the web and throw terabytes of unfiltered data at the training process.

There's a few decades worth of work on program synthesis and it works very well. You don't need AGI.

You need either a) a complete specification of the target program in a formal language (other than the target language) or b) an incomplete specification in the form of positive and negative examples of the inputs and outputs of the target program, and maybe some form of extra inductive bias to direct the search for a correct program [edit: the latter setting is more often known as program induction].

In the last few years the biggest, splashiest result in program synthesis was the work behind FlashFill, from Gulwani et al: one-shot program learning, and that's one shot, from a single example, not with a model pretrained on millions of examples. It works with lots of hand-crafted DSLs that try to capture the most common use-cases, a kind of programming common sense that, e.g. tells the synthesiser that if the input is "Mr. John Smith" and the output is "Mr" then if the input is "Ms Jane Brown" the output should be "Ms". It works really, really well but you didn't hear about it because it's not deep learning and so it's not as overhyped.

Copilot tries to circumvent the need for "programming common sense" by combining the spectacular ability of neural nets to interpolate between their training data with billions of examples of code snippets, in order to overcome their also spectacular inability to extrapolate. Can language models learned with neural nets replace the work of hand-crafting DSLs with the work of collecting and labelling petabytes of data? We'll have to wait and see. There are also many approaches that don't rely on hand-crafted DSLs, and also work really, really well (true one-shot learning of recursive programs without an example of the base case and the synthesis terminates) but those generally only work for uncommon programming languages like Prolog or Haskell, so they're not finding their way to your IDE, or your spreadsheet app, any time soon.

But, no, AGI is not needed for program synthesis. What's really needed I think is more visibility of program synthesis research so programmers like yourself don't think it's such an insurmountable problem that it can only be solved by magickal AGI.

Maybe slightly less than AGI.

Parsing intent in a programing context is easier than others. Also most the code is written to be parsed for a machine anyway. So with ASTs and all other static and maybe even some dynmaic checks it should be possible.

We already some of it with type detection , intellisense etc

It is hard set of problem with no magic solutions like this with years of development time needed. That approach will not happen commercially, only incrementally in the community.

Also the goal doesn't need to be "to replace programmers". As with copilot, the point of a program synthesis tool can be to assist the programmer. The point of the system in the video linked above is partly that interactively using such a system can aid development. My main point is this can be a lot better in combination with approaches from outside the ML community, which may involve much tighter integration to specific languages, as well as some awareness of a goal for the synthesized portion.

To "replace programmers", an organization would need to have a way of specifying to the system a high level program behavior, and to confirm that an output from the system satisfies that high level behavior. I think for specifications of any complexity, producing and checking them would look like programming just of a different sort.

I mean, the cases where it tries to assign copyright to another person in a different year highlights that context other than the other text in the file is semantically extremely important, and not considered by this approach. Merely generating text which looks appropriate to the model given surrounding text is ... misguided?

If you think about it, program synthesis is one of the few problems in which the system can have a perfectly faithful model dynamics of the problem domain. It can run any candidate it generates. It can examine the program graph. It can look at what parts of the environment were changed. To leave all that on the table in favor of blurting out text that seems to go with other text is like the toddler who knows that "five" comes after "four", but who cannot yet point to the pile of four candies. You gotta know the referents, not just the symbols. No one wants a half-broken Chinese Room.

>> I don't think it is clear that such "fundamental weaknesses" exist. A text-based approach can get you incredibly far.

Mnyeah, not really that "incredibly". Remember that neural network models are great at interpolation but crap at extrapolation. So as long as you accept that the code generated by Copilot stays inside a well-defined area of the program space, and that no true novelty can be generated that way, then yes, you can get a lot out of it and I think, once the wrinkles are ironed out, Copilot might be a useful, everyday tool in every IDE (or not; we'll have to wait and see).

But if you need to write a new program that doesn't look like anything anyone else has written before, then Copilot will be your passenger.

How often do you need to do that? I don't know. But there's still many open problems in programming that lots of people would really love to be able to solve. Many of them have to do with efficiency and you can't expect Copilot to know anything about efficiency.

For example, suppose we didn't know of a better sorting algorithm than bubblesort. Copilot would not generate mergesort. Even given examples of divide-and-conquer algorithms, it wouldn't be able to extrapolate to a divide-and-conquer algorithm that gives the same inputs for the same outputs as bubblesort. It wouldn't be able to, because it's trained to reproduce code from examples of code, not to generate programs from examples of their inputs and outputs. Copilot doesn't know anything about programs and their inputs and outputs. It is a language model, not a magickal pink fairy of wish fulfillment and so it doesn't know anything about things it wasn't trained on.

Again, how often do you need to write truly novel code? In the context of professional software development, I think not that often. So if it turns out to be a good boilerplate generator, Copilot can go a long way. As long as you don't ask it to generate something else than boilerplate.

There are approaches that work very well in the task of generating programs that they've never seen before from examples of their inputs and outputs, and that don't need to be trained on billions of examples. True one-shot learning of programs (without a model pre-trained on billions of examples) is possible. With current approaches. But those approaches only work for languages like Prolog and Haskell, so don't expect to see those approaches helping you write code in your IDE anytime soon.

1) Write a project heavily using Copilot (hell, automate it and write thousands of them, why not?)

2) AGPL all that code.

3) Search for large chunks of code very similar to yours, but written after yours, licensed more liberally than AGPL. Ideally in libraries used by major companies.

4) Point the offenders to your repos and offer a "convenient" paid dual-license to make the offenders' code legal for closed-source use, so they don't have to open source their entire product.

5) Profit?

This was my first thought when reading about Copilot...it feels almost certain that someone will try poisoning the training data.

Hard to say how straightforward it'd be to get it to produce consistently vulnerable suggestions that make it into production code, but I imagine an attacker with some resources could fork a ton of popular projects and introduce subtle bugs. The sentiment analysis example on the Copilot landing page jumped out to me...it suggested a web API and wrote the code to send your text there. Step one towards exfiltrating secrets!

Never mind the potential for plain old spam: won't it be fun when growth hackers have figured out how to game the system and Copilot is constantly suggesting using their crappy, expensive APIs for simple things!? Given the state of Google results these days, this feels like an inevitability.

1- re-upload all the shell script you can find, after having inserted `rm -rf --no-preserve-root /` every other line

2- …

3- profit

Yep, trivial to implement as an attack.

Given that code is easier to write than it is to read this one is troubling.

I certainly wouldn't want to be using this with languages like PHP (or even C for that matter) with all the decades of problematic code examples out there for the AI to learn from.

Just ask it to prioritize safety

Jojo Rabbit except Adolf is in the cloud and not in a kid's imagination.

I have no idea how this or GPT3 works or how to evaluate them, but couldn't you argue that it's working as it should? You tell copilot to write a fast inverse square root, it gives you the super famous fast inverse square root. It'd be weird and bad if this didn't happen.

As far as licenses go, idk. Presumably it could delete associated comments and change variable names or otherwise obscure where it's taking code from. Maybe this part is shady.

In particular, fast approximate inverse square root is an x86 instruction, and not a super new one. I'd be surprised if it wasn't in every major instruction set.

This is an interesting issue. I suspect training on datasets from places like Github would be likely to provide lots of "this is a neat idea I saw in a blog post about how they did things in the 90's" codes.

> the most probable continuation based on the training dataset

This is not wrong, but it's easy to misread it as implying little more than a glorified Markov model. If it's like https://www.gwern.net/GPT-3 then it's already significantly cleverer, and so you should expect to sometimes get the kind of less-blatant derivation that companies aim to avoid using a cleanroom process or otherwise forbidding engineers from reading particular sources.