I've been building a ChatGPT project this week and this explanation is so true it made me actually lol. Sometimes it's impressively flawless, sometimes it gets stuck in broken patterns... Even if the inputs are identical.
"Monkey around with the prompt and pray" has replaced unit testing.
From authors website: "I graduated high school in May of 2022..". Elsewhere, he writes he placed first at a university level drone programming challenge at IIT-Bombay Techfest. Very impressive.
The JEE exam in India is very advanced, I doubt most Europeans or US students could pass it, even after a year or two at university. So those who enter IIT are already quite good very early.
We also use GPT to perform actions in the software I build at work and we hit the same issue of inconsistency which lead me down a long rabbit hole to see if I could force an LLM to only emit grammatically correct output that follows a bespoke DSL (a DSL ideally safer and more precise than just eval'ing random AI-produced Python).
I just finished writing up a long post [1] that describes how this can work on local models. It's a bit tricky to do via API efficiently, but hopefully OpenAI will give us the primitives one day to appropriately steer these models to do the right things (tm).
Copilot has an option (maybe beta users only) to see alternative generations for a prompt. It's really handy because some generations are one-liners while others are entire functions.
Perhaps this is infeasible with the current cost per generation but a multiple choice display could be handy (perhaps split the screen into quadrants and pick the one that most fits what you prefer).
Google Bard has a similar feature, and you can ask the OpenAI API to generate multiple completions and pick the best one (best_of) or return some number of the generated completions (n). In OpenAI's case, best of means "the one with the highest log probability per token".
I know this tech is terrifying to actual 3D artist who don’t want be a “prompt engineer”, but as someone who has never used Blender, I think its cool that I can create something using tools like this and use them in my projects, ex background animation on a website hero section.
I don't think this is "terrifying" to 3D artists any more than github copilot is terrying to us haha, I'm not sure why we engineers have this tendency of imagining every other industry as being populated by superstitious peasants from the dark ages who fear new tools.
In my experience the vast majority of people are excited about GPT including artists.
I don't know, maybe I am one of those "superstitious peasants from the dark ages who fear new tools", but I'm increasingly terrified of GPT-4 and future iterations of it as applied to the industry I work with (i.e. software). It does seem to threaten to suddenly eliminate all the interesting parts of the job, a good chunk of openings, and to significantly reduce salaries for the openings that remain - all at the same time, and rather suddenly.
I think it's because if you're not in that particular industry you have a super simplified model of what the person does - something like "writing code" as the only activity a developer does.
You don't understand the difficulties and problems that people in the profession face, which I think is also why so many developers are convinced they can replace/"disrupt" other people's jobs with software.
I was speaking recently to a graphic artist who is terrified of Stable Diffusion and the like. I mentioned that these tools can augment their ability to do work instead of just replacing, but their point is that they are a graphic artist because they like doing the things that the AI will be replacing. Being a prompt engineer wasn't really the reason they studied and learned to become an artist. To me, that is a completely reasonable way to feel.
my main issue is the people making hiring and budget decisions aren't 3D artists, they're managers. my work might be objectively better than GPT's, but is that going to matter when my boss compares the cost difference and decides GPT is 'good enough'?
It just raises the bar so that 3D artists who are only capable of reproducing the same boring things they saw in YouTube tutorials are no longer considered proficient.
The more I think about this trend, the more I think it might be good.
Bootcamp devs are no longer good enough for junior roles since a GPT could replace them. Digital media people who learnt via YouTube and have no real talent are no longer skilled enough. Writers who can only churn out mediocre blog spam are now jobless.
This isn't how it will play out though. At first chess AIs got enough to beat unskilled players. They they were good enough to beat intermediate players. Then they challenged the grandmasters. Today no human, no matter how long they practise or how hard they train will ever get close to beating an AI at chess.
Your assumption that this raises the bar for human 3D artists is correct today, but it won't be long before human 3D artists are seen as much slower and less competent than AI artists, and there will be no going back.
I don't really care how or where someone learned something, so long as they learned it well and can apply it.
My experience in the software industry (20 years now) showed me that the best ones were the ones who got into it out of genuine interest. They tended to write software as a hobby.
There was no shortage of CS grads who couldn't be nearly as productive.
The self-taught ones, or the ones with genuine interest who also completed a degree program were the best.
I wouldn't discriminate against "boot camp coders" or people who learn things from YouTube.
There's a lot of people who live in a different world where an expensive college/university education is not an option.
>It just raises the bar so that 3D artists who are only capable of reproducing the same boring things they saw in YouTube tutorials are no longer considered proficient.
I use blender a lot. I know what the api can do. I'm not terrified. Getting anything significant done will still take real work/knowledge because the number of potential 'parameters' of what you can do/make requires that you have the language to describe what you want. Having the posession of that language and knowing how to use blender is a very positive correlation.
Imagine being able to sketch out the outline of what you are after, then say hey blender model it up. Thats what it will get to.
You have a beach, and you want a jetty, just grease pencil approximately where you want it and go Hey thats a jetty build it. and if its not quite right, generate me 50 different versions and ill pick the best.
> I know this tech is terrifying to actual 3D artist who don’t want be a “prompt engineer”
My predication: in the next 5~10 years, most artists won't be "prompt engineers". Instead they'll focus on fix small details on AI-generated art.
It's still kinda sad tho, because it's usually the most tedious and boring part of the process. Now AI is taking the fun part and leaving the unfun part to humans.
Yeah, but it's kinda like making business software. Most working people need to do boring work to sustain their lifestyles.
I hope that these task-specific implementations of AI can reduce the tedium in these fields, like the way PCs did. Certainly, this advancement is leading to the ability for practically anyone to program a computer, in the general sense. Things will shift, but there will be opportunities to exploit those abilities for personal gain.
This is exactly what this post from Reddit posted to HN [0] was complaining about.
I think many people hear and see these complaints and think that people are being luddites or being afraid of losing their job. People should be looking at it for what it is - someone complaining that their entire job is changing into something they no longer enjoy.
This is terrifying to us, but not in the way you think. Tools like this place selective pressure so that increasingly only the best of the best get paid to make stuff, and their stuff influences the data sets in the future. The rest of us get forced out simply due to cost unless we're doing it as a hobby. Some artists know all too well their limits, and those that do are the most vulnerable to things like this, because these learning machines use the ever higher ceiling of human ability to determine the floor of their ability. You can't go back to the funky weird pasta men of Stable Diffusion's 2021 iteration as the default for example, but someone who started painting in 2002 might still have the same level of skill now as they did in 2010.
Text was paralleled first. Then sound was paralleled next. And now image is being paralleled. It will be on level with highest percentile of human ability just as text and sound were before it. Game devs and comic artists are already replacing texture and background artists with AI generated images, just as they used AI to create hundreds of thousands of lines of fluff dialogue before then.
Most people aren't worried about being a prompt engineer-- they're worried about being a Doordash driver if it turns out that one prompt engineer can replace many professional, well-paid people in their industry. That's pretty rational.
If that job will even be available. The other day my sister sent me a photo of a little food delivery robot she spotted on the streets[0], and mind you, we're not living in Silicon Valley, but in Poland.
There is not much to prompt engineer; the obvious trivial thing that is already happening and will make the glorious job of prompt ‘engineer’ completely redundant is AI doing the ‘engineering’ for you. One person (or AI in the future; this is already possible now for humans and gpt though) that the AI has been trained to ‘know’ and work with will blurb things into a mic and the AI will instruct millions of brains to whatever the ‘boss’ wants, translating his mumbling into actionable user and system prompts. There won’t be prompt ‘engineers’ for long.
I asked my friend (renowned vfx artist) what he thought about AI in his field and he mentioned some niche cases where it was useful for blending models with film etc, but overall seemed uninterested.
He offered the explanation that so much of their time is consumed with nit picking through purely aesthetic decisions that AI would not be capable of the artistic reasoning required to produce work that could even get to the "pass or reject" stage.
As someone who had tried my hand at Blender and failed miserably/given up years ago, I am so excited to use this to make up for my Blender skill deficiency
I see a lot of people asking how does this work. The method he is using is one shot learning. He has a prompt and an example of what the interaction should look like.
It is really easy to build this kind of thing - I've got a very simple command line chatbot that should be very understandable and you can easily play with the prompt.
See how this file quickly got hacked together all in one file. It's refreshing to see and note-worthy as it appears when new exciting world-changing tech emerges that makes programmers let-go and become hackers again.
Did something similar as a natural language interface to DuckDB this weekend, and it's surprising how nice it works[0].
That said, through my testing I found it tended to trail off during longer exchanges if I only added the guidelines to the system prompt, so I ended up adding them to the first user message + adding a small reminder at the end of each further user message.
Ultimately there's a relationship between the preciseness in which you want to control something and the underlying information, as conveyed in language, to describe such precision.
Whether you use plain english, or code - ultimately to do things of sufficient precision you will have to be equally precise in your description. I'm sure someone with more time and more knowledge on such things have already formalized this in some information theory paper, but...
The point I'm making here is that this is great because a lot of people are doing "simple" things, and now they will be able to do those things without understanding the idiosyncrasies of Blender APIs, but I'm convinced that this will ultimately turn into something equally difficult as blender APIs to do novel things. and WHEN (not if) that happens, I hope users are prepared to learn the Blender APIs, because it will be inevitable.
edit:
one other thought. I think "language models" are not the right solution ultimately. I think kind of like AI didn't boom until the proper compute was available even though theoretical models and algorithms existed, language models are the crud solution.
once we have a loseless way to simply "think" what we want, then a "large thought model" will have less trouble, as there will be less ambiguity in what you want to what is said.
It could also be that something like this is just another tool in the toolbox. Sure, you could spend time trying to understand Blender's Python API, but lots of Blender users are not programmers. It could be really helpful for them to be able to say "Place 10 spot lights at random position within the boundaries of Mesh 3 with random intensity and color" and just have that appear, rather than having to go looking for a plugin that does it for them.
I dont know why people make this harder than it needs to be, make group of lights, change colors and duplicate, its good enough, no need to use python api.
Well, you get precision with imprecise communication via iteration. You walk together toward the correct mutual understanding by successive clarifications.
There’s also a parameter you can set called “temperature”. This is how you control the free-form aspect of the model. Set it close to 1 and it will bullshit a lot, or rather it will take bigger chances in filling in outside of context and maybe a bit more hyperactive. A value closer to 0 keeps it tight by taking less risk off the context, but can also miss out a bit and loses color.
Sure but GPT certainly cannot actually do this. It can iterate, but in GPT-world that means "give me another response" not "learn from what i'm saying, and give me another response"
The issue is, sometimes those two things seem to be the same thing!
Chatbots use random number generators to create a variety of output, so it would be a terrible idea to use natural language as "source code" for a code generation tool. Running a chatbot shouldn't happen within a build process, because the results aren't reproducible.
Once you have the source code though, you can use a variety of tools to manipulate it and save the result. Using chatbots to make modifications under supervision is fine. You discard bad modifications and save the good ones.
This is using natural language for one-offs and source code for reproducible results. It's looking like they will go well together.
With a shared vector database and semantic search plugin you can see bunch of prompts other people have created.
with a search plugin you can have it find the api docs and have it output interesting parameters and how to use them with examples.
with a python REPL plugin you can have it generate 10 variations and run the code for each.
with GPT4 and plugins you could describe the output you want to midjourney or something and give the prompt to it to generate it in blender(feed the outputs to some image similarity vector to compare) and have it search through parameter space(or vector space of the prompt) until it finds something pretty close to what you want.
If the command is not restricted to text only, but use text and geometrical context, you can remove a lot of ambiguities. This is often done in video games with contextual interactions.
After all, using the Blender GUI, you can do a lot using only a 2D mouse coordinate and two boutons. So 2D mouse coordinates and text could be better.
A nice evolution would be an AI model that can understand natural language instructions, while taking into account where your mouse pointer, how the model is zoomed and oriented, and that has geometric insight of the 3D scene built so far.
Or in other words, the "create an adversary capable of defeating Data" problem... just a bit less of an issue (for now) if you get the instructions wrong.
One interesting thing would be to describe a scene and get the rough print. But do it in sections, such that you can select and begin to refine sections and elements within the scene to whittle down to the preciseness that pleases you for each element...
What would be really interesting will be how geometry nodes can be managed using gpt.
Looking at the underlying prompt [1], it is a one-shot prompt, i.e. it contains one example of a natural language task along with the corresponding Python code (plus a "system" prompt for overall context and direction). Amazing how much you can do with a simple prompt.
Imagine Jupyter notebooks with this capability. Or Photoshop. Or Davinci Resolve. We live in amazing times.
Did you run into issues that made you tweak the prompt? You often tell the tool not to do extra work despite having it in the prompt, did you find that it often tries to do a lot of "boilerplate" work like setting up lights and cameras?
system_prompt = """You are an assistant made for the purposes of helping the user with Blender, the 3D software.
- Respond with your answers in markdown (```).
- Preferably import entire modules instead of bits.
- Do not perform destructive operations on the meshes.
- Do not use cap_ends. Do not do more than what is asked (setting up render settings, adding cameras, etc)
- Do not respond with anything that is not Python code.
Famous quote: "Wouldn't it be nice if our machines were smart enough to allow programming in natural language?". Well, natural languages are most suitable for their original purposes, viz. to be ambiguous in, to tell jokes in and to make love in, but most unsuitable for any form of even mildly sophisticated precision. And if you don't believe that, either try to read a modern legal document and you will immediately see how the need for precision has created a most unnatural language, called "legalese", or try to read one of Euclid's original verbal proofs (preferably in Greek). That should cure you, and should make you realize that formalisms have not been introduced to make things difficult, but to make things possible. And if, after that, you still believe that we express ourselves most easily in our native tongues, you will be sentenced to the reading of five student essays.
- Dijkstra From EWD952
On the other hand, we already specify programs in natural language, and in this case chatGPT is really taking a "specification language" as input.
When a client wants a button on a webpage, they don't send the web designer a legaleze document describing the dimensions of the button. They usually don't even tell the designer what font to use.
The web designer pattern matches the client's english request to the dozens of websites they've built, similar buttons they've seen and used, and then either asks for clarification, or specifies it clearly to the machine in a more specific way.
Is that different from the chatGPT flow?
Honestly, we also already mostly use english for programming too, not just design. Most of programming now is glueing together libraries, and libraries don't provide a formal logical specification of how each function works. No, they provide english documentation saying something like "http.get(url) returns an httpresponse object or an error". That's far from an actual mathematical specification of how it works, but the plain english definition is enough that most programmers won't ever look at the implementation, the actually correct specification, because the english docs are fine.
The designer knows the context of the question, the website, the previous meetings about the design styles, possibly information about the visitor demographics and goals, knows the implicit rules about approvals and company hierarchy, knows the toolset used, the project conventions, the previous issues, the test procedures, etc.
The equivalent of telling a designer where you want a new button would be equivalent to feeding a small book of the implicit context into ChatGPT and without access to visual feedback you could still end up with an off-screen button that passes all the tests and doesn't do anything. The "fun" part is that for simple tasks 90% of the time it will work every time - then it will do something completely stupid.
> they don't send the web designer a legaleze
That's the implicit context. (And yeah, bad assumptions about what both sides agree on causes problems for people too)
Also, chatgpt will try to make you happy. You want a green button here? You'll get a green button here. A designer instead will tell you it's a terrible idea and breaks accessibility.
Yes and no. You've provided an excellent introduction to the problem space, but I think natural language has a larger role in formalisms than you might expect.
The most familiar formal language grammars to most people here are programming languages. The difference between them and natural language has been categorized as the difference between "context-free grammar" and "context-dependent grammar".
The most popular context-free language is mathematics. The language of math provides an excellent grammar for expressing logical relationships. You can take an equation, write it in math, and transform it into a different equivalent representation. But why? Arithmetic. The Pythagorean Theorem would be wholly inconsequential if we didn't have an interest in calculating triangles. The application of math exists outside the grammar itself. This is why you, and everyone else here, grew up with story problems in math class.
Similarly, programming languages provide excellent utility for describing explicit computational behavior. What they are missing is the reason why that behavior should exist at all. Programs are surrounded by moats of incompatible context: it takes explicit design to coordinate them together.
If we can be explicit about the context in which a formalism exists, we could eliminate the need for ambiguity. With that work done, the incompatibility between software could be factored out. We could be precise about what we mean, and clear about what we infer. We could factor out all semantic arguments, and all logically fallacious positions. We could make empathy itself into software. That is the dream of Natural Language Processing.
I think that dream is achievable, but certainly not through implicit text models (LLMs). We need an explicit symbolic approach like parsing. If you're interested, I have been chewing on an idea that might work.
> Similarly, programming languages provide excellent utility for describing explicit computational behavior. What they are missing is the reason why that behavior should exist at all. Programs are surrounded by moats of incompatible context: it takes explicit design to coordinate them together.
Such a well written reply! This puts into words a lot of my thoughts around programming today and how NLP can help.
In (language model-backed) natural language interfaces the loss of precision can be made up via iteration. If it were about getting it right on the first try this would be a dead end but there's no need for that restriction.
And specifically, if the agent implementing the requested actions has a mental model of the domain that is similar to the mental model held by the imprecise specification writer, then the specification doesn't need to be precise to have a high probability of being implemented right the first time! The miracle here is that LLMs don't even need fine-tuning to get to that level - and that's unprecedented for non-human agents.
What makes us think that encoding all functionality using natural language is somehow more compact than using code? Sure, describing a website with a button is easy, but I don’t quite see how you would describe Kubernetes or a device driver without writing a book in “legalese”.
Or try formulating a math proof with natural language.
I’ve brought this up in another thread, but we already have evidence that in certain cases it is desirable to trade off specificity for just “getting something done”, that evidence is Python. Python succeeds despite not having a serious type system, because often times you can get something that achieves your goal despite the 25 bugs in your program that would have been caught by a real type system.
On the other side of the coin, there’s C++, which is usually doing the heavy lifting underneath the underspecified-but-sufficient Python code.
My guess is that as LLMs evolve, they will more naturally fill this niche and you will have high-level, underspecified “code” (prompts) that then glued together more formal libraries (like OpenAI’s plugins).
I think there exists room for an intermediate layer, a framework for AI to write code in specifically. With larger "target" code blocks it can stitch together. Code is designed around humans to allow them to write specific, abritrary and precise logic with a lot of minuate, but LLMs might be better off thinking in larger business logic blocks that are testable and "sanctioned" by the framework makers for the specific problem domain.
Not only that, but one of the big successes of chat AIs is their conversation. When a program does something wrong, it's hard to figure out what you said wrong.
When a char AI misunderstood you, it's often quite easy to explain where the misunderstanding happened and the AI will correct itself.
While I agree, I don't think this is what Dijkstra was speaking about. A lot of the "AI interfaces" are just underpinning the real technical representation. For example, if you say "Which customers recently bought product #721?" this might generate a SQL query like "select * from customers where ...".
The interface is not the ChatGPT text box; it's SQL. The ChatGPT text box is just an assistant to help you do the correct thing (or at least, that's the way it should be used).
Your example isn't very comparible to tasks involved in creating, texturing, lighting, animating and rendering 3D models using spline and mesh topologies.
I really feel this. Whenever I try to use ChatGPT, I feel both slow and strained. My usual workflow involves me thinking very few English sentences in my head - I move faster thanks to abstract thought, and I don't think I'm alone.
So, really, all we need then is a language sufficiently precise enough to specify what a program needs to do and we can then feed it into a program which can write software that implements that specification, possibly adopting safe transformations of it into equivalent forms.
Now, that safely describes a modern, optimizing C compiler.....
If you want something unambiguous in a single pass, sure. But interaction with some back and forth can dig into the ambiguities that matter quickly, ignoring those that don’t. Dialog is a whole separate model. So maybe natural language is a terrible specification language but a great interaction language.
Natural language doesn't NEED to be precise- that's what theory of mind is for, to allow efficient inference of meaning from objectively-ambiguous communication.
It's not perfect, but LLMs appear capable of making the same kind of inference, sufficiently that it'll inevitably be possible to program in natural language sooner or later with minimal or no manual double-checking
"The Sketchpad system makes it possible for a man and a computer to converse rapidly through the medium of line drawings. Heretofore, most interaction between men and computers has been slowed down by the need to reduce all communication to written statements that can be typed" - Sutherland
We're nearing the precipice of more natural human-computer interaction that will need to rethink the interfaces and conventions.
Alexa and Siri seem like Model T Fords when there's a jet aircraft flying overhead. I'm thinking these agents need to be replaced by more natural agents who can co-create a language with their human counterparts rather than relying on fixed, awkward, and sometimes unhelpful commands. It would behoove us to expose APIs and permissions delegation in a more consistent and self-describing (OpenAPI + OAuth / SAML possibly) manner for all possible services one would wish to grant to an agent. If a natural language agent is uncertain, it should ask for clarification. And on results, it is necessary to capture ever-more-precise feedback from users because positive and negative prompts aren't good enough.
> “The way it functioned was very interesting. When the Drink button was pressed it made an instant but highly detailed examination of the subject's taste buds, a spectroscopic analysis of the subject's metabolism and then sent tiny experimental signals down the neural pathways to the taste centers of the subject's brain to see what was likely to go down well. However, no one knew quite why it did this because it invariably delivered a cupful of liquid that was almost, but not quite, entirely unlike tea.”
I think Douglas Adams was closer to the truth on the subject of AIs and tea. I don't want to be overly cynical but I suspect we'll just get used to saying "OK, close enough" when dealing with LLMs.
Readit News
"Monkey around with the prompt and pray" has replaced unit testing.
I just finished writing up a long post [1] that describes how this can work on local models. It's a bit tricky to do via API efficiently, but hopefully OpenAI will give us the primitives one day to appropriately steer these models to do the right things (tm).
[1] https://github.com/newhouseb/clownfish
Perhaps this is infeasible with the current cost per generation but a multiple choice display could be handy (perhaps split the screen into quadrants and pick the one that most fits what you prefer).
There’s two approaches I use, I’m sure there are more.
* Do multiple completions and, filter the ones that successfully run, and take the most common result.
* Do a completion, if it fails, ask the Llm to find and correct the bug.
In my experience the vast majority of people are excited about GPT including artists.
You don't understand the difficulties and problems that people in the profession face, which I think is also why so many developers are convinced they can replace/"disrupt" other people's jobs with software.
It sounds more like you're just seriously underestimating the direction GitHub Copilot is headed.
https://www.reddit.com/r/blender/comments/121lhfq/i_lost_eve...
Edit; I see it was discussed indeed on HN yesterday. I have already seen tons more of these in my circles and not only 3d/artists.
The more I think about this trend, the more I think it might be good.
Bootcamp devs are no longer good enough for junior roles since a GPT could replace them. Digital media people who learnt via YouTube and have no real talent are no longer skilled enough. Writers who can only churn out mediocre blog spam are now jobless.
This seems like it might be a net benefit.
Your assumption that this raises the bar for human 3D artists is correct today, but it won't be long before human 3D artists are seen as much slower and less competent than AI artists, and there will be no going back.
My experience in the software industry (20 years now) showed me that the best ones were the ones who got into it out of genuine interest. They tended to write software as a hobby.
There was no shortage of CS grads who couldn't be nearly as productive.
The self-taught ones, or the ones with genuine interest who also completed a degree program were the best.
I wouldn't discriminate against "boot camp coders" or people who learn things from YouTube.
There's a lot of people who live in a different world where an expensive college/university education is not an option.
That is of course, until GPT-6 surpasses them.
You have a beach, and you want a jetty, just grease pencil approximately where you want it and go Hey thats a jetty build it. and if its not quite right, generate me 50 different versions and ill pick the best.
My predication: in the next 5~10 years, most artists won't be "prompt engineers". Instead they'll focus on fix small details on AI-generated art.
It's still kinda sad tho, because it's usually the most tedious and boring part of the process. Now AI is taking the fun part and leaving the unfun part to humans.
I hope that these task-specific implementations of AI can reduce the tedium in these fields, like the way PCs did. Certainly, this advancement is leading to the ability for practically anyone to program a computer, in the general sense. Things will shift, but there will be opportunities to exploit those abilities for personal gain.
I think many people hear and see these complaints and think that people are being luddites or being afraid of losing their job. People should be looking at it for what it is - someone complaining that their entire job is changing into something they no longer enjoy.
0: https://news.ycombinator.com/item?id=35308498
Text was paralleled first. Then sound was paralleled next. And now image is being paralleled. It will be on level with highest percentile of human ability just as text and sound were before it. Game devs and comic artists are already replacing texture and background artists with AI generated images, just as they used AI to create hundreds of thousands of lines of fluff dialogue before then.
If that job will even be available. The other day my sister sent me a photo of a little food delivery robot she spotted on the streets[0], and mind you, we're not living in Silicon Valley, but in Poland.
--
[0] - https://www.deliverycouple.com/ - based on the markings on that robot, its these people.
He offered the explanation that so much of their time is consumed with nit picking through purely aesthetic decisions that AI would not be capable of the artistic reasoning required to produce work that could even get to the "pass or reject" stage.
You can see the prompt here: https://github.com/gd3kr/BlenderGPT/blob/main/__init__.py
It is really easy to build this kind of thing - I've got a very simple command line chatbot that should be very understandable and you can easily play with the prompt.
https://github.com/atomic14/command_line_chatgpt
I would also recommend that people try out the openai playgrounds. They are great for experimenting with parameters.
See how this file quickly got hacked together all in one file. It's refreshing to see and note-worthy as it appears when new exciting world-changing tech emerges that makes programmers let-go and become hackers again.
That said, through my testing I found it tended to trail off during longer exchanges if I only added the guidelines to the system prompt, so I ended up adding them to the first user message + adding a small reminder at the end of each further user message.
[0]: https://github.com/cube2222/duckgpt
Why didn't you do the same for OctoSQL? Are you considering DuckDB for your compute engine for OctoSQL?
Ultimately there's a relationship between the preciseness in which you want to control something and the underlying information, as conveyed in language, to describe such precision.
Whether you use plain english, or code - ultimately to do things of sufficient precision you will have to be equally precise in your description. I'm sure someone with more time and more knowledge on such things have already formalized this in some information theory paper, but...
The point I'm making here is that this is great because a lot of people are doing "simple" things, and now they will be able to do those things without understanding the idiosyncrasies of Blender APIs, but I'm convinced that this will ultimately turn into something equally difficult as blender APIs to do novel things. and WHEN (not if) that happens, I hope users are prepared to learn the Blender APIs, because it will be inevitable.
edit:
one other thought. I think "language models" are not the right solution ultimately. I think kind of like AI didn't boom until the proper compute was available even though theoretical models and algorithms existed, language models are the crud solution.
once we have a loseless way to simply "think" what we want, then a "large thought model" will have less trouble, as there will be less ambiguity in what you want to what is said.
right now it's thought -> language -> model.
later it will be thought -> model.
The issue is, sometimes those two things seem to be the same thing!
Once you have the source code though, you can use a variety of tools to manipulate it and save the result. Using chatbots to make modifications under supervision is fine. You discard bad modifications and save the good ones.
This is using natural language for one-offs and source code for reproducible results. It's looking like they will go well together.
with a search plugin you can have it find the api docs and have it output interesting parameters and how to use them with examples.
with a python REPL plugin you can have it generate 10 variations and run the code for each.
with GPT4 and plugins you could describe the output you want to midjourney or something and give the prompt to it to generate it in blender(feed the outputs to some image similarity vector to compare) and have it search through parameter space(or vector space of the prompt) until it finds something pretty close to what you want.
given your budget of course.
After all, using the Blender GUI, you can do a lot using only a 2D mouse coordinate and two boutons. So 2D mouse coordinates and text could be better.
A nice evolution would be an AI model that can understand natural language instructions, while taking into account where your mouse pointer, how the model is zoomed and oriented, and that has geometric insight of the 3D scene built so far.
One interesting thing would be to describe a scene and get the rough print. But do it in sections, such that you can select and begin to refine sections and elements within the scene to whittle down to the preciseness that pleases you for each element...
What would be really interesting will be how geometry nodes can be managed using gpt.
Imagine Jupyter notebooks with this capability. Or Photoshop. Or Davinci Resolve. We live in amazing times.
[1]: https://github.com/gd3kr/BlenderGPT/blob/main/__init__.py
https://www.cs.utexas.edu/~EWD/transcriptions/EWD09xx/EWD952...
When a client wants a button on a webpage, they don't send the web designer a legaleze document describing the dimensions of the button. They usually don't even tell the designer what font to use.
The web designer pattern matches the client's english request to the dozens of websites they've built, similar buttons they've seen and used, and then either asks for clarification, or specifies it clearly to the machine in a more specific way.
Is that different from the chatGPT flow?
Honestly, we also already mostly use english for programming too, not just design. Most of programming now is glueing together libraries, and libraries don't provide a formal logical specification of how each function works. No, they provide english documentation saying something like "http.get(url) returns an httpresponse object or an error". That's far from an actual mathematical specification of how it works, but the plain english definition is enough that most programmers won't ever look at the implementation, the actually correct specification, because the english docs are fine.
The designer knows the context of the question, the website, the previous meetings about the design styles, possibly information about the visitor demographics and goals, knows the implicit rules about approvals and company hierarchy, knows the toolset used, the project conventions, the previous issues, the test procedures, etc.
The equivalent of telling a designer where you want a new button would be equivalent to feeding a small book of the implicit context into ChatGPT and without access to visual feedback you could still end up with an off-screen button that passes all the tests and doesn't do anything. The "fun" part is that for simple tasks 90% of the time it will work every time - then it will do something completely stupid.
> they don't send the web designer a legaleze
That's the implicit context. (And yeah, bad assumptions about what both sides agree on causes problems for people too)
Also, chatgpt will try to make you happy. You want a green button here? You'll get a green button here. A designer instead will tell you it's a terrible idea and breaks accessibility.
The most familiar formal language grammars to most people here are programming languages. The difference between them and natural language has been categorized as the difference between "context-free grammar" and "context-dependent grammar".
The most popular context-free language is mathematics. The language of math provides an excellent grammar for expressing logical relationships. You can take an equation, write it in math, and transform it into a different equivalent representation. But why? Arithmetic. The Pythagorean Theorem would be wholly inconsequential if we didn't have an interest in calculating triangles. The application of math exists outside the grammar itself. This is why you, and everyone else here, grew up with story problems in math class.
Similarly, programming languages provide excellent utility for describing explicit computational behavior. What they are missing is the reason why that behavior should exist at all. Programs are surrounded by moats of incompatible context: it takes explicit design to coordinate them together.
If we can be explicit about the context in which a formalism exists, we could eliminate the need for ambiguity. With that work done, the incompatibility between software could be factored out. We could be precise about what we mean, and clear about what we infer. We could factor out all semantic arguments, and all logically fallacious positions. We could make empathy itself into software. That is the dream of Natural Language Processing.
I think that dream is achievable, but certainly not through implicit text models (LLMs). We need an explicit symbolic approach like parsing. If you're interested, I have been chewing on an idea that might work.
Such a well written reply! This puts into words a lot of my thoughts around programming today and how NLP can help.
Or try formulating a math proof with natural language.
On the other side of the coin, there’s C++, which is usually doing the heavy lifting underneath the underspecified-but-sufficient Python code.
My guess is that as LLMs evolve, they will more naturally fill this niche and you will have high-level, underspecified “code” (prompts) that then glued together more formal libraries (like OpenAI’s plugins).
When a char AI misunderstood you, it's often quite easy to explain where the misunderstanding happened and the AI will correct itself.
The interface is not the ChatGPT text box; it's SQL. The ChatGPT text box is just an assistant to help you do the correct thing (or at least, that's the way it should be used).
Now, that safely describes a modern, optimizing C compiler.....
Considering they also predicted iPads, we might want to take it at face value.
It's not perfect, but LLMs appear capable of making the same kind of inference, sufficiently that it'll inevitably be possible to program in natural language sooner or later with minimal or no manual double-checking
"The Sketchpad system makes it possible for a man and a computer to converse rapidly through the medium of line drawings. Heretofore, most interaction between men and computers has been slowed down by the need to reduce all communication to written statements that can be typed" - Sutherland
Sometimes smart people say dumb things and it takes a while to figure out they are wrong.
An is-versus-ought style mistake.
We're nearing the precipice of more natural human-computer interaction that will need to rethink the interfaces and conventions.
Alexa and Siri seem like Model T Fords when there's a jet aircraft flying overhead. I'm thinking these agents need to be replaced by more natural agents who can co-create a language with their human counterparts rather than relying on fixed, awkward, and sometimes unhelpful commands. It would behoove us to expose APIs and permissions delegation in a more consistent and self-describing (OpenAPI + OAuth / SAML possibly) manner for all possible services one would wish to grant to an agent. If a natural language agent is uncertain, it should ask for clarification. And on results, it is necessary to capture ever-more-precise feedback from users because positive and negative prompts aren't good enough.
I think Douglas Adams was closer to the truth on the subject of AIs and tea. I don't want to be overly cynical but I suspect we'll just get used to saying "OK, close enough" when dealing with LLMs.