For the past 6 or 7 years I've had a side gig as an external examiner for CS students, and I've gotten GPT to pass most of the programming questions they ask the students in the programming exam. It's things like building a double linked list, sorting a binary tree and so on, and it's very good at it. On the flip side, I've yet to get it to do anything useful for me in my real job that isn't writing documentation. It's likely because it has a lot of data on how to pass CS courses, but not so much data on how to get entity Framework, Asp Versioning and Odata to work together. Probably because very few people have been crazy enough to attempt the latter.
I think it will be very interesting to see what the teachers, and institutions, do about it though. It's not like students couldn't find these things on the internet before GPT, but it's so much easier now. Over all I personally think that a nice shake-up of the way we educate people might turn out to be a good thing though. That is of course easy for me to say, as someone who doesn't actually have to do any of the shaking-up.
Students found way to cheat exams before too, what is great about Chat GPT is that is a great tool for a student that actually wants to understand and learn. With Google you can find an answer, maybe a good explanation but with a good Chat AI you can ask back for details, paste your broken solution and ask for the AI to spot the problem and explain it to you.
I see that some teachers will no longer allow projects printed on paper, it needs to be written by hand to force the student to read the content, I think this is bad for some subjects where the student does not care about, they will just forget instantly what they wrote.
> CS students, and I've gotten GPT to pass most of the programming questions they ask the students in the programming exam.
Highschool level questions aren't an issue for a bot trained specifically to answer such questions. A google search query could pass with flying colours.
I’ve gotten ChatGPT to write a lot of mostly correct code for me. Mostly around the AWS SDK in different languages. There is so much sample code out there it helps.
It can also write CloudFormation, convert existing CF to Terraform and idiomatically correct CDK code in Python, convert code from Python to all of the target languages I’ve given it and convert Python AWS SDK (boto3) code to bash/AWS CLI.
Maybe you are much better at prompting it than me, but I’ve gotten absolutely no value out of it in regards to my professional coding. Not for lack of trying either. I’ll give you a few examples of what I mean.
Friday we decided to split our EF migration into multiple context files, something we should’ve done from the beginning to be fair. Anyway, we deploy with Azure DevOps and we migrate for our more permanent testing, staging and production slots through a VM agent on our VNET. We don’t have a DevOps engineer, and we don’t completely know what we’re doing but it works. I figure I could copy paste the code four teams, giving us 5, one for each context, but I didn’t want to do that. So I asked ChatGPT and it gave me an answer involving Bash, which doesn’t work in our DevOps pipeline, as in, it doesn’t exist in the context. I told GPT this, and fed it more information and it proceeded to give me the exact same code four times before it gave me something different, but also unusable code. That’s one of the good examples of it failing. Because it made things up and they didn’t work. So no harm done, aside from the wasted time.
A few months back we tasked it with writing something for our Odata client library, and it did. It also worked, only, it didn’t reaaaaalllyyy work because it was flawed in a way you might not notice if you weren’t an experienced developer. Basically what it had build wouldn’t scale effectively, or at least not in a cost efficient manner. It would have been efficient enough to run for a long time without anyone noticing, however, but it would have been rather expensive and it would’ve broken down completely under “normal” loads, if we ever grow to twice the size we currently are. But it worked and it looked fine, and that’s sort of the scary scenarios. Or maybe they aren’t scary because we’ll likely be well paid to clean up those messes for years to come.
I do think it’s likely to become more and more useful as time passes. I mentioned documentation in my GP, and I am genuinely amazed at how good it is at writing code documentation. You obviously don’t feed it sensitive code, but it’s been able to accurately describe functions from their name, input and return values alone. Seriously, it’s taken a function name and written a piece of JSDoc that was miles better than what I had written, accurately guessing complex context (which it only saw maybe 10% of) from the function and variable names alone.
It's an interesting and legitimate question. But just as it takes business time to pick up on new technologies and adapt, it might take institutions as well. I've seen reddit posts of people getting writing flagged as AI written when it wasn't. The question is, how long until they start embracing it as a tool to use to write with?
Many of the institutions have billions of dollars put back. Harvard for example google tells me 53 billion. So I don't think they are just going to go away, they will adapt somehow.
No, thanks. The coding exams I've been in were either handwritten, or in a text field which loses focus every time you pess Tab. Both were a miserable experience and absolutely pointless.
How about this: test actual understanding instead of desperately making trivial tasks needlessly challenging.
That ChatGPT can pass exams, do your homework, and write your essays is a great thing.
Education has been fundamentally broken pretty much my entire life. The cracks have been showing through at an increasing rate since the internet went mainstream.
If I had to pick between the current gulag and a self-paced world where nearly every child has access to the library of Alexandria with a personal AI tutor, I’d chose the later in a heartbeat.
I have a very strong feeling that the current system isn’t going to survive the AI + Internet 1-2 punch. I believe that what replaces it will be significantly better and waste less of my descendants’ childhoods.
Sure, you get access to the library of Alexandria, but every book you need is wedged between copies of "The Big Book of distractions" and the "Encyclopedia of young girls with barely any clothes on". Some of those books play loud music, too, not to mention the books written by hacks and disguised as the book you actually want. You'd have to have the tenacity of a monk to get anything done.
It is possible for a person to be self-disciplined and learn by themselves. It is also possible, and even likely, to get lost in an ocean of distractions and/or depressed due to the lack of human interaction. As a physical person living in a physical space I wouldn't discount the positive influence of bringing your body somewhere with a purpose.
This is the big thing all the introverted self-learners miss about college. They reduce it to a number of CS classes they’d like to take, see the material on YouTube, then conclude that the college experience can be reduced to viewing some YouTube videos.
Lost in all of this is the notion that learning is a coop game, not single player; it’s more fun with friends! That’s the whole point!
Curious to hear why you feel that education has been broken throughout your life. I totally believe you, it's your own experience after all! Just feel that I was very lucky and had a good education at several great institutions with good mentors, so curious to hear what could have gone wrong.
I personally had a miserable experience too. I can focus for about 10-20 minutes in a typical class setting before my brain drifts off, which made most of education feel like a waste of time. That kind of daydreaming also pisses off certain teachers to a point they make it their goal to make one's life hell.
Part of the reason I finished school at all is that making a student fail causes more paperwork than most teachers care to do, which makes it quite easy to get barely passing grades. Of course this backfires in various ways like constantly declining standards for grades. Teachers in their last years before retirement are either so unmotivated they might as well stay at home (one of my teachers literally arrived 30 minutes late every time, another one simply didn't teach, ever, and was just present), others are just miserable people and decide none of their students ever deserves a better grade than a C. Very motivating.
Of course it's not a great system for teachers either and a lot of them work really hard.
I'm not the GP, but treating students as tabula rasa is a big part of the problem. Another is treating students as the average of their cohort.
> with good mentors
Most people don't get mentors until the latter part of undergraduate, at best. They don't even get people asking pertinent questions as to what they (the students) want to do, or are interested in, and what problems or issues they are having. Asking these questions is not an easy process, as the asker doesn't know about the student in order to ask pertinent questions, and the student often lacks the life knowledge to give the most pertinent answers. But even a bad processes would be preferable to nothing.
I know now what guidance I should have sought. But other students aren't me. It's a sad fact that it's pretty much only those who successfully navigated the system who are in a position to give informed advice and mentorship to students today. Our teachers are the successes. Those who failed are, at best, object lessons who we never hear from.
Some pedagogues try to research successes versus failures to generate lessons on how to instruct and credential better. And I wish them well.
The pessimist in me doesn’t expect those in power to change education for the better or react accordingly, but the optimist in me really hopes you’re right
It's always a generational problem for progress in anything isn't it? When people are entrenched in certain ways it's nigh impossible to change anything, especially when there's no gain for them personally. Real changes happen only when the current incumbents gradually die off. This is probably one of the main reasons mortality was evolutionarily selected for: easier adaptation to changing environments.
It's become glaringly apparent in the last century with progress vastly outpacing generational swaps. Old geezers in charge, trying to control a world they don't fundamentally understand anymore.
For thousands of generations we effectively learned through individual, tailored mentorship and trial-and-error. A hundred or two generations ago mass instruction was adopted for scions of certain of the powerful, or for everyone in organizations such as militaries. It's only in the last couple of hundred years that this mass instruction generalized to almost the entire populace of countries.
Mass instruction effectively treats students as the average of their cohort, without really getting to know each particular student. This is the fundamental break. It treats students more as blank states to be filled, than as collaborators in their own development.
As a specific anecdote, when I was in 4th/5th grade I didn't work ahead in math much because the assignments were to do every even problem on a page (one page, the potato page, as it had pictures of potatoes on it, had over 100 problems, I still remember it), which was tedious as heck. Because I didn't work ahead I was mainstreamed in math in 6th grade, and felt absolute horror at repeating yet again what I had already tediously repeated so many times.
Humans made unimaginable progress in the last 150ish years. Many of those who contribute were raised on farms or taught in third world countries. What magic system are you envisioning that could do better than what got us to the moon?
Solutions that involve tossing everyone into the deep end of the pool and letting them sink or swim on their own would probably work for some people assuming it exists as part of a system where there are no educational credentials (though presumably companies would then come up with their own credentialing systems).
But I expect a lot of people would drop out of the system in the absence of structure.
This reminds me of a gem I picked up in my pedagogical studies: the aim of effective learning is for the student to ask from the teacher to explain something they don't understand. That both guarantees relevance and is highly likely to leave a lasting impression.
However, in most classrooms it's the teacher doing the asking. "Why are you asking me if you already know the answer?" is a completely valid response. A teacher should try to avoid that and instead ask e.g. "what are you having trouble with?"
With regard to ChatGPT passing tests, IMO the root problem is the overemphasis on tests with these kinds of redundant questions (and constant grading) that have always had limited correlation with learning.
My favorite teacher, for math would give us a paper with soem optional homework math problems, we would xerox the paper and we will try at hoem to solve them, then next day we would ask each other how X or Y was solved, compare results, we would teach each other. And only the hard problems that we could not solve we would ask the teacher to show us the solution.
What is weird is that 4 years later my brother was in a class at the same teacher and the atmosphere was completely different, nobody was working on optional stuff, nobody was asking questions, nobody was offering to go to the blackboard and participate. It was same teacher, same school , same level of smart students. Not sure if maybe there is one ot two students that with their enthusiasm or the opposite can cause this drastic change.
The teacher of course brings the material to the student, typically by presenting it by making statements. If the emvironment is right and the teacher has good pedagogical skills, that should spur questions from the students.
I think the most interesting use of LLMs I’ve heard of in education is to use them as adversaries for real students, e.g. find factual errors in GPT output in a journalism class, or finding edge cases in GPT code.
As an educator, the way I’m responding to this is that the bar has been raised. Whatever ChatGPT can do is considered a C. If your responses are at ChatGPT level or above, you pass and can move on. If you can’t meet the bar set by ChatGPT, you will need to get to that level before moving on.
Where appropriate, yea. It’s the same as any other resource, like a calculator. If the assignment isn’t about calculating numbers, fine, use a calculator. But if the whole point is to make sure you can do long division, then no calculator allowed.
You can use one of GPTs to write essays, programs, solve math problems. But all you will learn is how to write a prompt. And this knowledge will be outdated in one year or less. Do you want it?
Professional use, on the other hand, saves time, at least in some cases.
Don't know if it has much of an effect on the answers but the prompts refer to the "hundredth digit" where they should read the "hundreds digit".
The "hundreds digit" being the third from the right, whereas the "hundredth digit" would be one hundred digits from (presumably) the left, as in "the hundredth digit of pi".
This is English as I would use it. YMMV.
Having written the above and with my hand hoverig over the "send" button, I wondered what GPT would think so I asked it:
'What is the difference between "the hundreds digit" and "the hundredth digit"?'
It answered:
'"The hundreds digit" refers to the digit in the hundreds place of a number, while "the hundredth digit" refers to the second digit after the decimal point in a number.'
An interesting interpretation that hadn't occurred to me!
It seems to me that if you are an educator testing your students with questions in English, you owe it to them to at least make sure the questions are accurately posed and free of typos. This means if you are not a native speaker, you should get a native speaker conversant in your topic area to check your writing.
“also omitting numbers whose hundredth digit is 2 or 3”
I know this kind of post is frowned on, so downvote away if that’s your thing, but I think the point could also be relevant to GPT’s performance here, although it generally seems to handle typos pretty well.
Readit News
I think it will be very interesting to see what the teachers, and institutions, do about it though. It's not like students couldn't find these things on the internet before GPT, but it's so much easier now. Over all I personally think that a nice shake-up of the way we educate people might turn out to be a good thing though. That is of course easy for me to say, as someone who doesn't actually have to do any of the shaking-up.
I see that some teachers will no longer allow projects printed on paper, it needs to be written by hand to force the student to read the content, I think this is bad for some subjects where the student does not care about, they will just forget instantly what they wrote.
Highschool level questions aren't an issue for a bot trained specifically to answer such questions. A google search query could pass with flying colours.
It can also write CloudFormation, convert existing CF to Terraform and idiomatically correct CDK code in Python, convert code from Python to all of the target languages I’ve given it and convert Python AWS SDK (boto3) code to bash/AWS CLI.
Friday we decided to split our EF migration into multiple context files, something we should’ve done from the beginning to be fair. Anyway, we deploy with Azure DevOps and we migrate for our more permanent testing, staging and production slots through a VM agent on our VNET. We don’t have a DevOps engineer, and we don’t completely know what we’re doing but it works. I figure I could copy paste the code four teams, giving us 5, one for each context, but I didn’t want to do that. So I asked ChatGPT and it gave me an answer involving Bash, which doesn’t work in our DevOps pipeline, as in, it doesn’t exist in the context. I told GPT this, and fed it more information and it proceeded to give me the exact same code four times before it gave me something different, but also unusable code. That’s one of the good examples of it failing. Because it made things up and they didn’t work. So no harm done, aside from the wasted time.
A few months back we tasked it with writing something for our Odata client library, and it did. It also worked, only, it didn’t reaaaaalllyyy work because it was flawed in a way you might not notice if you weren’t an experienced developer. Basically what it had build wouldn’t scale effectively, or at least not in a cost efficient manner. It would have been efficient enough to run for a long time without anyone noticing, however, but it would have been rather expensive and it would’ve broken down completely under “normal” loads, if we ever grow to twice the size we currently are. But it worked and it looked fine, and that’s sort of the scary scenarios. Or maybe they aren’t scary because we’ll likely be well paid to clean up those messes for years to come.
I do think it’s likely to become more and more useful as time passes. I mentioned documentation in my GP, and I am genuinely amazed at how good it is at writing code documentation. You obviously don’t feed it sensitive code, but it’s been able to accurately describe functions from their name, input and return values alone. Seriously, it’s taken a function name and written a piece of JSDoc that was miles better than what I had written, accurately guessing complex context (which it only saw maybe 10% of) from the function and variable names alone.
Many of the institutions have billions of dollars put back. Harvard for example google tells me 53 billion. So I don't think they are just going to go away, they will adapt somehow.
How about this: test actual understanding instead of desperately making trivial tasks needlessly challenging.
Education has been fundamentally broken pretty much my entire life. The cracks have been showing through at an increasing rate since the internet went mainstream.
If I had to pick between the current gulag and a self-paced world where nearly every child has access to the library of Alexandria with a personal AI tutor, I’d chose the later in a heartbeat.
I have a very strong feeling that the current system isn’t going to survive the AI + Internet 1-2 punch. I believe that what replaces it will be significantly better and waste less of my descendants’ childhoods.
Sure, you get access to the library of Alexandria, but every book you need is wedged between copies of "The Big Book of distractions" and the "Encyclopedia of young girls with barely any clothes on". Some of those books play loud music, too, not to mention the books written by hacks and disguised as the book you actually want. You'd have to have the tenacity of a monk to get anything done.
It is possible for a person to be self-disciplined and learn by themselves. It is also possible, and even likely, to get lost in an ocean of distractions and/or depressed due to the lack of human interaction. As a physical person living in a physical space I wouldn't discount the positive influence of bringing your body somewhere with a purpose.
Lost in all of this is the notion that learning is a coop game, not single player; it’s more fun with friends! That’s the whole point!
Of course it's not a great system for teachers either and a lot of them work really hard.
Deleted Comment
> with good mentors
Most people don't get mentors until the latter part of undergraduate, at best. They don't even get people asking pertinent questions as to what they (the students) want to do, or are interested in, and what problems or issues they are having. Asking these questions is not an easy process, as the asker doesn't know about the student in order to ask pertinent questions, and the student often lacks the life knowledge to give the most pertinent answers. But even a bad processes would be preferable to nothing.
I know now what guidance I should have sought. But other students aren't me. It's a sad fact that it's pretty much only those who successfully navigated the system who are in a position to give informed advice and mentorship to students today. Our teachers are the successes. Those who failed are, at best, object lessons who we never hear from.
Some pedagogues try to research successes versus failures to generate lessons on how to instruct and credential better. And I wish them well.
It's become glaringly apparent in the last century with progress vastly outpacing generational swaps. Old geezers in charge, trying to control a world they don't fundamentally understand anymore.
Go on Wikipedia, Khan Academy, MDN, etc and open ChatGPT right now.
Mass instruction effectively treats students as the average of their cohort, without really getting to know each particular student. This is the fundamental break. It treats students more as blank states to be filled, than as collaborators in their own development.
As a specific anecdote, when I was in 4th/5th grade I didn't work ahead in math much because the assignments were to do every even problem on a page (one page, the potato page, as it had pictures of potatoes on it, had over 100 problems, I still remember it), which was tedious as heck. Because I didn't work ahead I was mainstreamed in math in 6th grade, and felt absolute horror at repeating yet again what I had already tediously repeated so many times.
But I expect a lot of people would drop out of the system in the absence of structure.
However, in most classrooms it's the teacher doing the asking. "Why are you asking me if you already know the answer?" is a completely valid response. A teacher should try to avoid that and instead ask e.g. "what are you having trouble with?"
With regard to ChatGPT passing tests, IMO the root problem is the overemphasis on tests with these kinds of redundant questions (and constant grading) that have always had limited correlation with learning.
What is weird is that 4 years later my brother was in a class at the same teacher and the atmosphere was completely different, nobody was working on optional stuff, nobody was asking questions, nobody was offering to go to the blackboard and participate. It was same teacher, same school , same level of smart students. Not sure if maybe there is one ot two students that with their enthusiasm or the opposite can cause this drastic change.
Professional use, on the other hand, saves time, at least in some cases.
The "hundreds digit" being the third from the right, whereas the "hundredth digit" would be one hundred digits from (presumably) the left, as in "the hundredth digit of pi".
This is English as I would use it. YMMV.
Having written the above and with my hand hoverig over the "send" button, I wondered what GPT would think so I asked it:
'What is the difference between "the hundreds digit" and "the hundredth digit"?'
It answered:
'"The hundreds digit" refers to the digit in the hundreds place of a number, while "the hundredth digit" refers to the second digit after the decimal point in a number.'
An interesting interpretation that hadn't occurred to me!
“also omitting numbers whose hundredth digit is 2 or 3”
I know this kind of post is frowned on, so downvote away if that’s your thing, but I think the point could also be relevant to GPT’s performance here, although it generally seems to handle typos pretty well.