I teach math at a large university (30,000 students) and have also gone “back to the earth”, to pen-and-paper, proctored and exams.
Students don’t seem to mind this reversion. The administration, however, doesn’t like this trend. They want all evaluation to be remote-friendly, so that the same course with the same evaluations can be given to students learning in person or enrolled online. Online enrollment is a huge cash cow, and fattening it up is a very high priority. In-person, pen-and-paper assessment threatens their revenue growth model. Anyways, if we have seven sections of Calculus I, and one of these sections is offered online/remote, then none of the seven are allowed any in person assessment. For “fairness”. Seriously.
LLMs aren't destroying the University or the essay.
LLMs are destroying the cheap University or essay.
Cheap can mean a lot of things, like money or time or distance. But, if Universities want to maintain a standard, then they are going to have to work for it again.
No more 300+ person freshman lectures (where everyone cheated anyways). No more take-home zoom exams. No more professors checked out. No more grad students doing the real teaching.
I guess, I'm advocating for the Oxbridge/St. John's approach with under 10 class sizes where the proctor actually knows you and if you've done the work. And I know, that is not a cheap way to churn out degrees.
>I guess, I'm advocating for the Oxbridge/St. John's approach with under 10 class sizes where the proctor actually knows you and if you've done the work. And I know, that is not a cheap way to churn out degrees.
I could understand US tuition if that were the case. These days with overworked adjuncts make it McDonalds at Michelin star prices.
Believe it or not, 300-person freshman lectures can be done well. They just need a talented instructor who's willing to put in the prep, and good TAs leading sections. And if the university fosters the right culture, the students mostly won't cheat.
But yeah, if the professor is clearly checked out and only interested in his research, and the students are being told that the only purpose of their education is to get a piece of paper to show to potential employers, you'll get a cynical death-spiral.
(I've been on both sides of this, though back when copy-pasting from Wikipedia was the way to cheat.)
Over here in Finland, higher education is state funded, and the funding is allocated to universities mostly based on how many degrees they churn out yearly. Whether the grads actually find employment or know anything is irrelevant.
So, it's pretty hard for universities over here to maintain standards in this GenAI world, when the paying customer only cares about quantity, and not quality. I'm feeling bad for the students, not so much for foolish politicians.
After a short stint as a faculty member at a McU institution, I agree with much of this.
Provide machine problems and homework as exercises for students to learn, but assign a very low weight to these as part of an overall grade. Butt in seat assessments should be the majority of a course assessment for many courses.
This is depressing. I'm late GenX, I didn't cheat in college (engineering, RPI), nor did my peers. Of course, there was very little writing of essays so that's probably why, not to mention all of our exams were in person paper-and-pencil (and this was 1986-1990, so no phones). Literally impossible to cheat. We did have study groups where people explained the homework to each other, which I guess could be called "cheating", but since we all shared, we tended to oust anyone who didn't bring anything to the table. Is cheating through college a common millenial / gen z thing?
Cheap "universities" are fine for accreditation. Exams can be administered via in-person proctoring services, which test the bare minimum. The real test would be when students are hired, in the probationary period. While entry-level hires may be unreliable, and even in the best case not help the company much, this is already a problem (perhaps it can be solved by the government or some other outside organization paying the new hire instead of the company, although I haven't thought about it much).
Students can learn for free via online resources, forums, and LLM tutors (the less-trustworthy forums and LLMs should primarily be used to assist understanding the more-trustworthy online resources). EDIT: students can get hands-on-experience via an internship, possibly unpaid.
Real universities should continue to exist for their cutting-edge research and tutoring from very talented people, because that can't be commodified. At least until/if AI reaches expert competence (in not just knowledge but application), but then we don't need jobs either.
There are excellent 1000-student lecture courses and shitty 15-student lecture courses. There are excellent take-home exams and shitty in-class exams. There are excellent grad student teaching assistants and shitty tenured credentialed professors. You can't boil quality down to a checklist.
The masses get the cheap AI education. The elite get the expensive, small class, analog education. There won't be a middle class of education, as in the current system - too expensive for too little gain.
10 is a small number. There's a middle ground. When I studied, we had lectures for all students, and a similar amount of time in "work groups," as they were called. That resembled secondary education: one teacher, around 30 students, but those classes were mainly focused on applying the newly acquired knowledge, making exercises, asking questions, checking homework, etc. Later, I taught such classes for programming 101, and it was perfectly doable. Work group teachers were also responsible for reviewing their students' tests.
But that commercially oriented boards are ruining education, that's a given. That they would stoop to this level is a bit surprising.
I see that pressure as well. I find that a lot of the problems we have with AI are in fact AI exposing problems in other aspects of our society. In this case, one problem is that the people who do the teaching and know what needs to be learned are the faculty, but the decisions about how to teach are made by administrators. And another problem is that colleges are treating "make money" as a goal. These problems existed before AI, but AI is exacerbating them (and there are many, many more such cases).
I think things are going to have to get a lot worse before they get better. If we're lucky, things will get so bad that we finally fix some shaky foundations that our society has been trying to ignore for decades (or even centuries). If we're not lucky, things will still get that bad but we won't fix them.
Instructors and professors are required to be subject matter experts but many are not required to have a teaching certification or education-related degree.
So they know what students should be taught but I don't know that they necessarily know how any better than the administrators.
I've always found it weird that you need teaching certification to teach basic concepts to kindergartners but not to teach calculus to adults.
I totally agree. I think the neo-liberal university model is the real culprit. Where I live, Universities get money for each student who graduates. This is up to 100k euros for a new doctorate. This means that the University and its admin want as many students to graduate as possible. The (BA&MA) students also want to graduate in target time: if they do, they get a huge part of their student loans forgiven.
What has AI done? I teach a BA thesis seminar. Last year, when AI wasn't used as much, around 30% of the students failed to turn in their BA thesises. 30% drop-out rate was normal. This year: only 5% dropped out, while the amount of ChatGPT generated text has skyrocketed. I think there is a correlation: ChatGPT helps students write their thesises, so they're not as likely to drop out.
The University and the admins are probably very happy that so many students are graduating. But also, some colleagues are seeing an upside to this: if more graduate, the University gets more money, which means less cuts to teaching budgets, which means that the teachers can actually do their job and improve their courses, for those students who are actually there to learn. But personally, as a teacher, I'm at loss of what to do. Some thesises had hallucinated sources, some had AI slop blogs as sources, the texts are robotic and boring. But should I fail them, out of principle on what the ideal University should be? Nobody else seems to care. Or should I pass them, let them graduate, and reserve my energy to teach those who are motivated and are willing to engage?
In Australia Universities that have remote study have places where people can do proctored exams in large cities. The course is done remotely but the exam, which is often 50%+ of the final grade, is done in a place that has proctored exams as a service.
The Open University in the UK started in 1969. Their staff have a reputation for good interaction with students, and I have seen very high quality teaching materials produced there. I believe they have always operated on the basis of remote teaching but on-site evaluation. The Open University sounds like an all-round success story and I'm surprised it isn't mentioned more in discussions of remote education.
Variations in this system are in active use in the US as well.
Do you feel it is effective?
It seems to me that there is a massive asymmetry in the war here: proctoring services have tiny incentives to catch cheaters. Cheaters have massive incentives to cheat.
I expect the system will only catch a small fraction of the cheating that occurs.
Where I'm studying its proctored-online. They have a custom browser and take over your computer while you're doing the exam. Creepy AF but saves travelling 1,300 km to sit an exam.
Can you tell us: Is "remote study" a relatively recent phenom in AU -- COVID era, or much older? I am curious to learn more. And, what is the history behind it? Was it created/supported because AU is so vast and many people a state might not live near the campus?
Also: I think your suggestion is excellent. We may see this happen in the US if AI cheating gets out of control (which it well).
Proctoring services done well could be valuable, but it’s smaller rural and remote communities that would benefit most. Maybe these services could be offered by local schools, libraries, etc.
Those I ask are unanimously horrified that this is the choice they are given. They are devastated that the degree for which they are working hard is becoming worthless yet they all assert they don't want exams back. Many of them are neurodivergent who do miserably in exam conditions and in contrast excel in open tasks that allow them to explore, so my sample is biased but still.
They don't have a solution. As the main victims they are just frustrated by the situation, and at the "solutions" thrown at it by folks who aren't personally affected.
It is always interesting to me when people say they are "bad test takers". You mean you are bad at the part where we find out how much you know? Maybe you just don't know the material well enough.
caveat emptor - I am not ND so maybe this is a real concern for some, but in my experience the people who said this did not know the material. And the accommodations for tests are abused by rich kids more than they are utilized by those that need them.
I don't think I understand, as a terrible test taker myself.
The solution I use when teaching is to let evaluation primarily depend on some larger demonstration of knowledge. Most often it is CS classes (e.g. Machine Learning), so I don't really give much care for homeworks and tests and instead be project driven. I don't care if they use GPT or not. The learning happens by them doing things.
This is definitely harder in other courses. In my undergrad (physics) our professors frequently gave takehome exams. Open book, open notes, open anything but your friends and classmates. This did require trust, but it was usually pretty obvious when people worked together. They cared more about trying to evaluate and push us if we cared than if we cheated. They required multiple days worth of work and you can bet every student was coming to office hours (we had much more access during that time too). The trust and understanding that effort mattered actually resulted in very little cheating. We felt respected, there was a mutual understanding, and tbh, it created healthy competition among us.
Students cheat because they know they need the grade and that at the end of the day they won't won't actually be evaluated on what they learned, but rather on what arbitrary score they got. Fundamentally, this requires a restructuring, but that's been a long time coming. The cheating literally happens because we just treated Goodhart's Law as a feature instead of a bug. AI is forcing us to contend with metric hacking, it didn't create it.
IMO exams should be on the easier side and not require much computing (mainly knowledge, and not unnecessary memorization). They should be a baseline, not a challenge for students who understand the material.
Students are more accurately measured via long, take-home projects, which are complicated enough that they can’t be entirely done by AI.
Unless the class is something that requires quick thinking on the job, in which case there should be “exams” that are live simulations. Ultimately, a student’s GPA should reflect their competence in the career (or possible careers) they’re in college for.
We have an Accessible Testing Center that will administer and proctor exams under very flexible conditions (more time, breaks, quiet/privacy, …) to help students with various forms of neurodivergence. They’re very good and offer a valuable service without placing any significant additional burden on the instructor. Seems to work well, but I don’t have first hand knowledge about how these forms of accommodations are viewed by the neurodivergent student community. They certainly don’t address the problem of allowing « explorer » students to demonstrate their abilities.
> Many of them are neurodivergent who do miserably in exam conditions
I mean, for every neurodivergent person who does miserably in exam conditions you have one that does miserably in homework essays because of absence of clear time boundaries.
In my undergraduate experience, the location of which shall remain nameless, we had amble access to technology but the professors were fairly hostile to it and insisted on pencil and paper for all technical classes. There were some English or History classes here and there that allowed a laptop for writing essays during an "exam" that was a 3 hour experience with the professor walking around the whole time. Anyway, when I was younger I thought the pencil and paper thing to be silly. Why would we eschew brand new technology that can make us faster! And now that I'm an adult, I'm so thankful they did that. I have such a firm grasp of the underlying theory and the math precisely because I had to write it down, on my own, from memory. I see what these kids do today and they have been so woefully failed.
Teachers and professors: you can say "no". Your students will thank you in the future.
I have a Software Engineering degree from Harvard Extension and I had to take quite a few exams in physically proctored environments. I could very easily manage in Madrid and London. It is not too hard for either the institution or the student.
I am now doing an Online MSc in CompSci at Georgia Tech. The online evaluation and proctoring is fine. I’ve taken one rather math-heavy course (Simulation) and it worked. I see the program however is struggling with the online evaluation of certain subjects (like Graduate Algorithms).
I see your point that a professor might prefer to have physical evaluation processes. I personally wouldn’t begrudge the institution as long as they gave me options for proctoring (at my own expense even) or the course selection was large enough to pick alternatives.
Professional proctored testing centers exist in many locations around the world now. It's not that complicated to have a couple people at the front, a method for physically screening test-takers, providing lockers for personal possessions, providing computers for test administration, and protocols for checking multiple points of identity for each test taker.
This hybrid model is vastly preferable to "true" remote test taking in which they try to do remote proctoring to the student's home using a camera and other tools.
is it ok for students to submit images of hand-written solutions remotely?
seriously it reminds me of my high school days when a teacher told me i shouldn’t type up my essays because then they couldn’t be sure i actually wrote them.
maybe we will find our way back to live oral exams before long…
Centralization and IT-ification has made flouting difficult. There’s one common course site on the institution’s learning management system for all sections where assignments are distributed and collected via upload dropbox, where grades are tabulated and communicated.
So far, it’s still possible to opt out of this coordinated model, and I have been. But I suspect the ability to opt out will soon come under attack (the pretext will be ‘uniformity == fairness’). I never used to be an academic freedom maximalists who viewed the notion in the widest sense, but I’m beginning to see my error.
I attended Purdue. Since I graduated, it launched its "Purdue Global" online education. Rankings don't suggest it's happened yet, but I'm worried it will cheapen the brand and devalue my degree.
I remember sitting with the faculty in charge of offering online courses when I visited as an alum back in 2014. They seemed to look at it as a cash cow in their presentation. They were eager to be at the forefront of online CS degrees at the time.
Remote learning also opens up a lot of opportunities to people that would not otherwise be able to take advantage of them. So it's not _just_ the cash cow that benefits from it.
Some US universities do this remotely via proctoring software. They require pencil and paper to be used with a laptop that has a camera. Some do mirror scans, room scans, hand scans, etc. The Georgia Tech OMS CS program used to do this for the math proofs course and algorithms (leet code). It was effective and scalable. However, the proctoring seems overly Orwellian, but I can understand the need due to cheating as well as maintaining high standards for accreditation.
Maybe we should consider the possibility that this isn't a good idea? Just a bit? No? Just ignore how obviously comparable this is to the most famous dystopian fiction in literary history?
Just wow. If you're willing to do that, I don't know what to tell you.
Stanford requires pen & paper exams for their remote students; the students first need to nominate an exam monitor (a person) who in turn receives and prints the assignments, meets the student at an agreed upon place, the monitor gives them the printed exams and leaves, then collects the exam after allotted time, scans it and sends it back to Stanford.
When it was generally accepted by our society that the goal of all work is victory, not success. Capitalism frames everything as a competition, even when collaboration is obviously superior. Copyright makes this an explicit rule.
Hand written essays are inherently ableist. I would be at a massive disadvantage. I grew up during the 60's, but handwriting was alway slow and error prone for me. As soon as I could use a word processor I blossomed.
It's probably not as bad for mathematical derivations. I still do those by hand since they are more like drawing than expression.
So is testing; people who don't have the skills don't do well. Hell, the entire concept of education is ableist towards learning impaired kids. Let's do away with it entirely.
Would you hire someone as a writer who is completely illiterate? Of course that's an extreme edge case, but at some point equality stops and the ability to do the work is actually important.
I was a slow handwriter, too. I always did badly on in-class essay exams because I didn't have time to write all that I knew needed to be said. What saved my grade in those classes was good term papers.
Having had much occasion to consider this issue, I would suggest moving away from the essay format. Most of the typical essay is fluff that serves to provide narrative cohesion. If knowledge of facts and manipulation of principles are what is being evaluated, presentation by bullet points should be sufficient.
how would you propose to filter out able cheaters instead? there's also in person one on one verbal exam, but economics and logistics of that are insanely unfavorable (see also - job interviews.)
I teach computer science / programming, and I don't know what a good AI policy is.
On the one hand, I use AI extensively for my own learning, and it's helping me a lot.
On the other hand, it gets work done quickly and poorly.
Students mistake mandatory assignments for something they have to overcome as effortlessly as possible. Once they're past this hurdle, they can mind their own business again. To them, AI is not a tutor, but a homework solver.
I can't ask them to not use computers.
I can't ask them to write in a language I made the compiler for that doesn't exist anywhere, since I teach at a (pre-university) level where that kind of skill transfer doesn't reliably occur.
So far we do project work and oral exams: Project work because it relies on cooperation and the assignment and evaluation is open-ended: There's no singular task description that can be plotted into an LLM. Oral exams because it becomes obvious how skilled they are, how deep their knowledge is.
But every year a small handful of dum-dums made it all the way to exam without having connected two dots, and I have to fail them and tell them that the three semesters they have wasted so far without any teachers calling their bullshit is a waste of life and won't lead them to a meaningful existence as a professional programmer.
Teaching Linux basics doesn't suffer the same because the exam-preparing exercise is typing things into a terminal, and LLMs still don't generally have API access to terminals.
Maybe providing the IDE online and observing copy-paste is a way forward. I just don't like the tendency that students can't run software on their own computers.
I'm not that old, and yet my university CS courses evaluated people with group projects, and in-person paper exams. We weren't allowed to bring computers or calculators into the exam room (or at least, not any calculators with programming or memory). It was fine.
I don't see why this is so hard, other than the usual intergenerational whining / a heaping pile of student entitlement.
If anything, the classes that required extensive paper-writing for evaluation are the ones that seem to be in trouble to me. I guess we're back to oral exams and blue books for those, but again...worked fine for prior generations.
Yup. ~25 years ago competitions / NOI / leet_coding as they call it now were in a proctored room, computers with no internet access, just plain old borland c, a few problems and 3h of typing. All the uni exams were pen & paper. C++ OOP on paper was fun, but iirc the scoring was pretty lax (i.e. minor typos were usually ignored).
> I don't see why this is so hard, other than the usual intergenerational whining / a heaping pile of student entitlement.
You know that grading paper exams is a lot more hassle _for the teachers_?
Your overall point might or might not still stand. I'm just responding to your 'I don't see why this is so hard'. Show some imagination for why other people hold their positions.
(I'm sure there's lots of other factors that come into play that I am not thinking of here.)
I'm not too old either and in my university, CS was my major, we did group projects and in person paper exams as well.
We wrote c++ on paper for some questions and were graded on it. Ofcourse the tutors were lenient on the syntax they cared about the algorithm and the data structures not so much for the code. They did test syntax knowledge as well but more in code reasoning segments, i.e questions like what's the value of a after these two statements or after this loop is run.
We also had exams in the lab with computers disconnected from the internet. I don't remember the details of the grading but essentially the teaching team was in the room and pretty much scored us then and there.
> Students mistake mandatory assignments for something they have to overcome as effortlessly as possible.
It has been interesting to see this idea propagate throughout online spaces like Hacker News, too. Even before LLMs, the topic of cheating always drew a strangely large number of pro-cheating comments from people arguing that college is useless, a degree is just a piece of paper, knowledge learned in classes is worthless, and therefore cheating is a rational decision.
Meanwhile, whenever I’ve done hiring or internships screens for college students it’s trivial to see which students are actually learning the material and which ones treat every stage of their academic and career as a game they need to talk their way through while avoiding the hard questions.
I teach computer science / programming, and I know what a good AI policy is: No AI.
(Dramatic. AI is fine for upper-division courses, maybe. Absolutely no use for it in introductory courses.)
Our school converted a computer lab to a programming lab. Computers in the lab have editors/compilers/interpreters, and whitelist documentation, plus an internal server for grading and submission. No internet access otherwise. We've used it for one course so far with good results, and and extending it to more courses in the fall.
An upside: our exams are now auto-graded (professors are happy) and students get to compile/run/test code on exams (students are happy).
>Students mistake mandatory assignments for something they have to overcome as effortlessly as possible.
This is the real demon to vanquish. We're approaching course design differently now (a work in progress) to tie coding exams in the lab to the homework, so that solving the homework (worth a pittance of the grade) is direct preparation for the exam (the lion's share of the grade).
> Our school converted a computer lab to a programming lab. Computers in the lab have editors/compilers/interpreters, and whitelist documentation, plus an internal server for grading and submission. No internet access otherwise. We've used it for one course so far with good results, and and extending it to more courses in the fall.
Excellent approach. It requires a big buy-in from the school.
Thanks for suggesting it.
I'm doing something for one kind of assignment inspired by the game "bashcrawl" where you have to learn Linux commands through an adventure-style game. I'm bundling it in a container and letting you submit your progress via curl commands, so that you pass after having run a certain set of commands. Trying to make the levels unskippable by using tarballs. Essentially, if you can break the game instead of beating it honestly, you get a passing grade, too.
>Our school converted a computer lab to a programming lab. Computers in the lab have editors/compilers/interpreters, and whitelist documentation, plus an internal server for grading and submission. No internet access otherwise. We've used it for one course so far with good results, and extending it to more courses in the fall.
As an higher education (university) IT admin who is responsible for the CS program's computer labs and is also enrolled in this CS program, I would love to hear more about this setup, please & thank you. As recently as last semester, CS professors have been doing pen'n paper exams and group projects. This setup sounds great!
Isn't auto-grading cheating by the instructors? Isn't part of their job providing their expert feedback by actually reading the code the students have generating and providing feedback and suggestions for improvement even at for exams? A good educational program treats exams as learning opportunities, not just evaluations.
So if the professors can cheat and they're happy about having to do less teaching work, thereby giving the students a lower-quality educational experience, why shouldn't the students just get an LLM to write code that passes the auto-grader's checks? Then everyone's happy - the administration is getting the tuition, the professors don't have to grade or give feedback individually, and the students can finish their assignments in half an hour instead of having to stay up all night. Win win win!
> Oral exams because it becomes obvious how skilled they are, how deep their knowledge is.
Assuming you have access to a computer lab, have you considered requiring in-class programming exercises, regularly? Those could be a good way of checking actual skills.
> Maybe providing the IDE online and observing copy-paste is a way forward. I just don't like the tendency that students can't run software on their own computers.
And you'll frustrate the handful of students who know what they're doing and want to use a programmer's editor. I know that I wouldn't have wanted to type a large pile of code into a web anything.
You can provide vscode, vim and emacs all in some web interface, and those are plenty good enough for those use cases. Choosing the plugin list for each would also be a good bikeshedding exercise for the department.
> But every year a small handful of dum-dums made it all the way to exam without having connected two dots, and I have to fail them and tell them that the three semesters they have wasted so far without any teachers calling their bullshit is a waste of life
Yeah, I've had teachers like that, who tell you that you're a "waste of life" and "what are you doing here?" and "you're dumb", so motivational.
I guess this "tough love" attitude helps for some people? But I think mostly it's just that people think it works for _other_ people, but rarely people think that this works when applied to themselves.
Like, imagine the school administration walking up to this teacher and saying "hey dum dum, you're failing too many students and the time you've spent teaching them is a waste of life."
Many teachers seem to think that students go to school/university because they're genuinely interested in motivated. But more often then not, they're there because of societal pressure, because they know they need a degree to have any kind of decent living standard, and because their parents told them to. Yeah you can call them names, call them lazy or whatever, but that's kinda like pointing at poor people and saying they should invest more.
> I use AI extensively for my own learning, and it's helping me a lot. On the other hand, it gets work done quickly and poorly.
> small handful of dum-dums made it all the way to exam without having connected two dots, and I have to fail them ... won't lead them to a meaningful existence
I don't see a problem, the system is working.
The same group of people that are going to loose their job to an LLM arent getting smarter because of how they are using LLM's.
Ideally the system would encourage those dum-dums to realize they need to change their ways before they're screwed. Unless the system working is that people get screwed and cause problems for the rest of society.
> The same group of people that are going to loose their job to an LLM arent getting smarter because of how they are using LLM's.
Students who use LLMs and professional programmers who use LLMs: I wouldn't say it's necessarily the same group of people.
Sure, their incentives are the same, and they're equally unlikely to maintain jobs in the future.
But students can be told that their approach to become AI secretaries isn't going to pan out. They're not actively sacrificing a career because they're out of options. They can still learn valuable skills, because what they were taught has not been made redundant yet, unlike mediocre programmers who can only just compete with LLM gunk.
I ran one semester embracing AI, and... I don't know, I don't have enough to compare with, but clearly it leaves a lot of holes in people's understanding. They generate stuff that they don't understand. Maybe it's fine. But they're certainly worse programmers than I was after having spent the same time without LLMs.
When I was studying games programming we used an in house framework developed by the lecturers for OGRE.
At the time it was optional, but I get the feeling that if they still use that framework, it just became mandatory, because it has no internet facing documentation.
That said, I imagine they might have chucked it in for Unity before AI hit, in which case they are largely out of luck.
>But every year a small handful of dum-dums made it all the way to exam without having connected two dots, and I have to fail them and tell them that the three semesters they have wasted so far without any teachers calling their bullshit is a waste of life and won't lead them to a meaningful existence as a professional programmer.
This happened to me with my 3d maths class, and I was able to power through a second run. But I am not sure I learned anything super meaningful, other than I should have been cramming better.
If there is another course where students design their own programming language, maybe you could use the best of the previous year's. That way LLMs are unlikely to be able to (easily) produce correct syntax. Just a thought from someone who teaches in a totally different neck of the mathematical/computational woods.
Modern LLMs can one-shot code in a totally new language, if you provide the language manual. And you have to provide the language manual, because otherwise how can the students learn the language.
I had numerous in person paper exams in CS (2009 - 2013) where we had to not only pseudo code an algorithm from a description, but also do the reverse of saying/describing what a chunk of pseudo code would do.
There you go. Actually that would be a great service, wouldn't it? Having them explain to an LLM what they are doing, out loud, while doing it, online. On a site that you trust to host it.
> Teaching Linux basics doesn't suffer the same because the exam-preparing exercise is typing things into a terminal, and LLMs still don't generally have API access to terminals.
Huh, fighting my way through a Linux CLI is exactly the kind of thing I use Chatgpt for professionally.
I did study it in compsci, but those commands are inherently not memorable.
Yes, LLMs have had API access to terminals for quite a while now. I've been using Windsurf and Claude Code to type terminal commands for me for a long while (and `gh copilot suggest` before that) and couldn't be happier. I still manually review most of them before approving, but I've seen that the chances of the AI getting an advanced incantation right on the first try are much higher than mine, and I haven't yet once had it make a disastrous one, while that's happened to me quite a few times with commands I typed on my own.
> their bullshit is a waste of life and won't lead them to a meaningful existence as a professional programmer
That's where you're wrong. Being a professional programmer is 10% programming, 40% office politics, and 50% project management. If your student managed to get halfway through college without any actual programming skills, they're perfect candidate, because they clearly own the 90% of skills needed to be a professional programmer.
In my experience, it's 70% programming, 20% office politics, and 10% project management. People who realize late they're no good at programming, or don't enjoy it, will pivot towards other kinds of work, like project management. But people who think they'll have luck managing people without having any grasp of the skill set of the people they manage, they either need really good people skills, or they're obnoxiously incompetent in both humans and computers.
This is only temporary. It will be able to code like anyone in time. The only way around this will be coding in-person, but only in elementary courses. Everyone in business will be using AI to code, so that will be the way in most university courses as well.
IMO no amount of AI should be used during an undergrad education, but I can see how people would react more strongly to its use in these intro to programming courses. I don't think there's as much of an issue with using it to churn out some C for an operating systems course or whatever. The main issue with it in programming education is when learning rudiments of programming IS the point of the course. Same with using to it crank out essays for freshman English courses. These courses are designed to introduce fundamental raw skills that everything else builds on. Someone's ability to write good code isn't as big a deal for classes in OS, algs, compilers, ML, etc., as the main concepts of those courses are.
I’m enrolled in an undergraduate CS program as an experienced (10 year) dev. I find AI incredibly useful as a tutor.
I usually ask it to grade my homework for me before I turn it in. I usually find I didn’t really understand some topic and the AI highlights this and helps set my understanding straight. Without it I would have just continued on with an incorrect understanding of the topic for 2-3 weeks while I wait for the assignment to be graded. As an adult with a job and a family this is incredibly helpful as I do homework at 10pm and all the office hours slots are in the middle of my workday.
I do admit though it is tough figuring out the right amount to struggle on my own before I hit the AI help button. Thankfully I have enough experience and maturity to understand that the struggle is the most important part and I try my best to embrace it. Myself at 18 would definitely not have been using AI responsibly.
When I was in college if AI was available I would have abused it way too much and been much worse off for it.
This is my biggest concert about GenAI in our field. As an experienced dev I've been around the block enough times to have a good feel of how things should be done and can catch when and LLM goes off on a tangent that is a complete rabbit hole, but if this had been available 20 years ago I would have never learned and become an experienced dev because I absolutely would have over relied on an LLM. I worry that 10 years from now getting mid career dev will be like trying to get a COBOL dev now, except COBOL is a lot easier to learn.
I’m wondering how the undergrad CS course is as an experienced dev and why you decided to do that? I have been a software developer for 5 years with an EE degree, and as I do more software engineering and less EE I feel like I am missing some CS concepts that my colleagues have. Is this your situation too or did you have another reason? And why not a masters?
A mix of feeling I’m “missing” some CS concepts and just general intellectual curiosity.
I am planning on doing a masters but I need some undergrad CS credits to be a qualified candidate. I don’t think I’m going to do the whole undergrad.
Overall my experience has been positive. I’ve really enjoyed Discrete Math and coming to understand how I’ve been using set theory without really understanding it for years. I’m really looking forward to my classes on assembly/computer architecture, operating systems, and networks. They did make me take CS 101-102 as prereqs which was a total waste of time and money, but I think those are the only two mandatory classes with no value to me.
Not GP, but in my experience most MSC programs will require that you have substantial undergrad CS coursework in order to be accepted. There are a few programs designed for those without that background.
I have a friend who is self-medicating untreated adhd with street amphetamines and he talks about it similarly. I can't say with any certainty that either of you is doing anything wrong or even dangerous. But I do think you both are overconfident in your assessment of the risks.
A bit off-topic, but I think AI has the potential to supercharge learning for the students of the future.
Similar to Montessori, LLMs can help students who wander off in various directions.
I remember often being “stuck” on some concept (usually in biology and chemistry), where the teacher would hand-wave something as truth, this dismissing my request for further depth.
Of course, LLMs in the current educational landscape (homework-heavy) only benefit the students who are truly curious…
My hope is that, with new teaching methods/styles, we can unlock (or just maintain!) the curiosity inherent in every pupil.
(If anyone knows of a tool like this, where an LLM stays on a high-level trajectory of e.g. teaching trigonometry, but allows off-shoots/adventures into other topical nodes, I’d love to know about it!)
>>> Of course, LLMs in the current educational landscape (homework-heavy) only benefit the students who are truly curious
I think you hit on a major issue: Homework-heavy. What I think would benefit the truly curious is spare time. These things are at odds with one another. Present-day busy work could easily be replaced by occupying kids' attention with continual lessons that require a large quantity of low-quality engagement with the LLM. Or an addictive dopamine reward system that also rewards shallow engagement -- like social media.
I'm 62, and what allowed me to follow my curiosity as a kid was that the school lessons were finite, and easy enough that I could finish them early, leaving me time to do things like play music, read, and learn electronics.
And there's something else I think might be missing, which is effort. For me, music and electronics were not easy. There was no exam, but I could measure my own progress -- either the circuit worked or it didn't. Without some kind of "external reference" I'm not sure that in-depth research through LLMs will result in any true understanding. I'm a physicist, and I've known a lot of people who believe that they understand physics because they read a bunch of popular books about it. "I finally understand quantum mechanics."
> I'm 62, and what allowed me to follow my curiosity as a kid was that the school lessons were finite, and easy enough that I could finish them early, leaving me time to do things like play music, read, and learn electronics.
I see both sides of this. When I was a teenager, I went to a pretty bad middle school where there were fights everyday, and I wasn’t learning anything from the easy homework. On the upside, I had tons of free time to teach myself how to make websites and get into all kinds of trouble botting my favorite online games.
My learning always hit a wall though because I wasn’t able to learn programming on my own. I eventually asked my parents to send me to a school that had a lot more structure (and a lot more homework), and then I properly learned math and logic and programming from first principles. The upside: I could code. The downside: there was no free time to apply this knowledge to anything fun
>I'm 62, and what allowed me to follow my curiosity as a kid was that the school lessons were finite, and easy enough that I could finish them early, leaving me time to do things like play music, read, and learn electronics.
Yeah I feel like teachers are going to try and use LLMs as an excuse to push more of the burden of schooling to their pupils homelife somehow. Like, increasing homework burdens to compensate.
Spare time, haha, most people nowadays have a hard time having some dead time. The habitual checking of socials or feeds has killed the mind wandering time. People feel uncomfortable or consiser life boring with the device induced dopamine fix. Corporations got us by the balls.
The last thing I need when researching a hard problem is an interlocutor who might lie to me, make up convincing citations to nowhere, and tell me more or less what I want to hear.
Still better than the typical classroom experience. And you can always ask again, there's no need to avoid offending the person who has a lot of power over you.
The longer I go without seeing cases of ai supercharging learning, the more suspicious I get that it just won’t. And no, self reports that it makes internet denizens feel super educated, don’t count.
The problem is that many students come to university unequipped with the discipline it takes to actually study. Teaching students how to effectively learn is a side-effect of university education.
> I remember often being “stuck” on some concept (usually in biology and chemistry), where the teacher would hand-wave something as truth, this dismissing my request for further depth.
This resonates with me a lot. I used to dismiss AI as useless hogwash, but have recently done a near total 180 as I realised it's quite useful for exploratory learning.
Not sure about others but a lot of my learning comes from comparison of a concept with other related concepts. Reading definitions off a page usually doesn't do it for me. I really need to dig to the heart of my understanding and challenge my assumptions, which is easiest done talking to someone. (You can't usually google "why does X do Y and not Z when ABC" and then spin off from that onto the next train of reasoning).
Hence ChatGPT is surprisingly useful. Even if it's wrong some of the time. With a combination of my baseline knowledge, logic, cross referencing, and experimentation, it becomes useful enough to advance my understanding. I'm not asking ChatGPT to solve my problem, more like I'm getting it to bounce off my thoughts until I discover a direction where I can solve my problem.
I am building such an AI tutoring experience, focusing on a Socratic style with product support for forking conversations onto tangents. Happy to add you to the waitlist, will probably publish an MVP in a few weeks.
I haven’t personally tried it, but the high-level demos of “khanmigo” created by khan academy seem really promising. I’ll always have a special place in my heart (and brain) for the work of Sal Khan and the folks at khan academy.
yeah this is a good point, just adjust coursework from multiple choice tests and fill in the blank homework to larger scale projects.
Putting together a project using the AI help will be a very close mimicry of what real work will be like and if the teacher is good they will learn way more than being able to spout information from memory.
I've always though that the education system was broken and next to worthless. I've never felt that teachers ever tried to _teach_ me anything, certainly not how to think. In fact I saw most attempts at thought squashed because they didn't fit neatly into the syllabus (and so couldn't be graded).
The fact that AI can do your homework should tell you how much your homework is worth. Teaching and learning are collaborative exercises.
> The fact that AI can do your homework should tell you how much your homework is worth.
Homework is there to help you practise these things and have help you progress, find the areas where you're in need of help and more practise. It is collaborative, it's you, your fellow students and your teachers/professors.
I'm sorry that you had bad teachers, or had needs that wasn't being meet by the education system. That is something that should be addressed. I just don't think it's reasonable to completely dismiss a system that works for the majority. Being mad at the education system isn't really a good reason for say "AI/computers can do all these things, so why bother practising them?"
Schools should learn kids to think, but if the kids can't read or reasonably do basic math, then expecting them to have independent critical thinking seems a way of. I don't know about you, but one of the clear lessons in "problem math" in schools was to learn to reason about numbers and result, e.g. is it reasonable that a bridge span 43,000km? If not, you probably did something wrong in your calculations.
These conversations are always eye-opening for the number of people who don’t understand homework. You’re exactly right that it’s practice. The test is the test (obviously) and the homework is practice with a feedback loop (the grade).
Giving people credit for homework helps because it gives students a chance to earn points outside of high pressure test times and it also encourages people to do the homework. A lot of people need the latter.
My friends who teach university classes have experimented with grading structures where homework is optional and only exam scores count. Inevitably, a lot of the class fails the exams because they didn’t do any practice on their own. They come begging for opportunities to make it up. So then they circle back to making the homework required and graded as a way to get the students to practice.
ChatGPT short circuits this once again. Students ChatGPT their homework then fail the first exam. This time there is little to do, other than let those students learn the consequences of their actions.
> The fact that AI can do your homework should tell you how much your homework is worth.
A lot of people who say this kind of thing have, frankly, a very shallow view of what homework is. A lot of homework can be easily done by AI, or by a calculator, or by Wikipedia, or by looking up the textbook. That doesn't invalidate it as homework at all. We're trying to scaffold skills in your brain. It also didn't invalidate it as assessment in the past, because (eg) small kids don't have calculators, and (eg) kids who learn to look up the textbook are learning multiple skills in addition to the knowledge they're looking up. But things have changed now.
Completely agree - I always thought the framing of "exercises" is the right one, the point is that your brain grows by doing. It's been possible for a long time to e.g. google a similar algebra problem and find a very relevant math stackexchange post, doesn't mean the exercises were useless.
"The fact that forklift truck can lift over 500kg should tell you how worthwhile it is for me to go to a gym and lift 100kg." - complete non-sequitur.
Then maybe the homework assignment has been poorly chosen. I like how the article's author has decided to focus on the process and not the product and I think that's probably a good move.
I remember one of my kids' math teachers talked about wanting to switch to in inverted classroom. The kids would be asked to read a some part of their textbook as homework and then they would work through exercise sheets in class. To me, that seemed like a better way to teach math.
> But things have changed now.
Yep. Students are using AIs to do their homework and teachers are using AIs to grade.
Yep, making time to sit down to do homework, forming an understanding of planning the doing part, forming good habits of doing them, knowing how to look up stuff, in a book index or on Wikipedia or by searching or asking AI.
The expectation is still that some kind of text output needs to be found and then read, digested.
> The fact that AI can do your homework should tell you how much
you still have to learn. The goal of learning is not to do a job. It's to enrich you, broaden your mind, and it takes work on your part.
In similar reasoning, you could argue that you can take a car to go anywhere, or let everything be delivered on your doorstep, so why should I my child learn to walk?
The fact that AI can replace the work that you are measured on should tell you something about the measurement itself.
The goal of learning should be to enrich the learner. Instead, the goal of learning is to pass measure. Success has been quietly replaced with victory. Now LLMs are here to call that bluff.
As a student, you can make "getting the diploma" the only goal, and so it rests entirely on the educators and the institution to ensure that the only way you can do that is by learning the material and becoming competent in its applications.
However, you can instead recognize the difficulty and time that this would require on the part of the educator, and therefore expense to the student, and you can recognize that you have the goal of not just obtaining a piece of paper but actually learning a skill. With this mindset, it makes sense to take the initiative to treat the homework as an opportunity to learn and practice. It's is one of those things that's worth as much as you put into it. Of course, one can use their judgement to decide which homework is worth spending time on to learn the material, and which can be safely sailed through with minimum effort.
Having a skilled teacher that you can really collaborate with and who can spend the time to evaluate your skills in a personal way is of course going to lead to better learning outcomes than the traditional education system. It will also be far more expensive. Although, AI is offering something somewhat akin to this experience at a much lower price, to those who are able to moderate their usage so that they are learning from the AI instead of just offloading tasks to it.
Homework isn't about doing the homework, it's teaching you to learn and evidence that you have and can learn. Yeah you can have an AI do it just as much as you can have someone else do it, but that doesn't teach you anything and if you earn the paper at the end of it, it's effectively worthless.
Unis should adjust their testing practices so that their paper (and their name) doesn't become worthless. If AI becomes a skill, it should be tested, graded, and certified accordingly. That is, separate the computer science degree from the AI Assisted computer science degree.
Current AI can ace math and programming psets at elite institutions, and yet prior to GPT not only did I learn loads from the homework, I often thoroughly enjoyed it too. I don’t see how you can make that logical leap.
Its a problem of incentives. For many courses the psets make up a large chunk of your grade. Grades determine your suitability for graduate school, internships, jobs, etc. So if your final goal is one of those then you are highly incentivized to get high grades, not necessarily to learn the material.
> The fact that AI can do your homework should tell you how much your homework is worth.
I mean... if you removed the substring "home" from that sentence, is it still true in your opinion?
That is, do you believe that because AI can perform some task, that task must not have any value? If there's a difference, help me understand it better please.
No that's not exactly what I meant. It's not that "If AI can do X then X is worthless", but rather in _this_ case it's inappropriate.
Homework should be a part of the collaborative process of learning (as others above have already elaborated on). If teachers are having a problem with AI generated homework being submitted, it shows that the system is broken because they couldn't have been collaborating with students on their learning then.
> Teaching and learning are collaborative exercises.
That's precisely where we went wrong. Capitalism has redefined our entire education system as a competition; just like it does with everything else. The goal is not success, it's victory.
If the trend continues, it seems like most college degrees will be completely worthless.
If students using AI to cheat on homework are graduating with a degree, then it has lost all value as a certificate that the holder has completed some minimum level of education and learning. Institutions that award such degrees will be no different than degree mills of the past.
I’m just grateful my college degree has the year 2011 on it, for what it’s worth.
All of the best professors I had either did not grade homework or weighted it very small and often on a did-you-do-it-at-all basis and did not grade attendance at all. They provided lectures and assignments as a means to learn the material and then graded you based on your performance in proctored exams taken either in class or at the university testing center.
For most subjects at the university level graded homework (and graded attendance) has always struck me as somewhat condescending and coddling. Either it serves to pad out grades for students who aren't truly learning the material or it serves to force adult students to follow specific learning strategies that the professor thinks are best rather than giving them the flexibility they deserve as grown adults.
Give students the flexibility to learn however they think is best and then find ways to measure what they've actually learned in environments where cheating is impossible. Cracking down on cheating at homework assignments is just patching over a teaching strategy that has outgrown its usefulness.
> rather than giving them the flexibility they deserve as grown adults
I have had so many very frustrating conversations with full grown adults in charge of teaching CS. I have no faith at all that students would be able to choose an appropriate method of study.
My issue with the instruction is the very narrow belief in the importance of certain measurable skills. VERY narrow. I won’t go into details, for my own sanity.
> All of the best professors I had either did not grade homework or weighted it very small and often on a did-you-do-it-at-all basis and did not grade attendance at all. They provided lectures and assignments as a means to learn the material and then graded you based on your performance in proctored exams taken either in class or at the university testing center.
I have the opposite experience - the best professors focused on homework and projects and exams were minimal to non-existent. People learn different ways, though, so you might function better having the threat/challenge of an exam, whereas I hated having to put everything together for an hour of stress and anxiety. Exams are artificial and unlike the real world - the point is to solve problems, not to solve problems in weirdly constrained situations.
Maybe schools and universities need to stop considering homework to be evidence of subject matter mastery. Grading homework never made sense to me. What are you measuring, really, and how confident are you of that measurement?
You can't put the toothpaste back into the tube. Universities need to accept that AI exists, and adjust their operations accordingly.
Provide an incentive for students to do the thing they should be doing anyway.
Give an opportunity to provide feedback on the assignment.
It is totally useless as an evaluation mechanic, because of course the students that want to can just cheat. It’s usually pretty small, right? IIRC when I did tutoring we only gave like 10-20% for the aggregate homework grade.
> If the trend continues, it seems like most college degrees will be completely worthless.
I suspect the opposite: Known-good college degrees will become more valuable. The best colleges will institute practices that confirm the material was learned, such as emphasizing in-person testing over at-home assignments.
Cheating has actually been rampant at the university level for a long time, well before LLMs. One of the key differentiators of the better institutions is that they are harder to cheat to completion.
At my local state university (where I have friends on staff) it’s apparently well known among the students that if they pick the right professors and classes they can mostly skate to graduation with enough cheating opportunity to make it an easy ride. The professors who are sticklers about cheating are often avoided or even become the targets of ratings-bombing campaigns
I've tried re-enrolling in a STEM major last year, after a higher education "pause" of 16-ish years. 85% of the class used GPTs to solve homework, and it was quite obvious most of them haven't even read the assignment.
The immediate effect was the distrust of the professors towards most everyone and lots classes felt like some kind of babysitting scheme, which I did not appreciate.
> I’m just grateful my college degree has the year 2011 on it, for what it’s worth.
College students still cram and purge. Nobody forced to sit through OChem remembers their Diels-Alder reaction except the organic chemists.
College degrees probably don't have as much value as we've historically ascribed to them. There's a lot of nostalgia and tradition pent up in them.
The students who do the best typically fill their schedule with extra-curricular projects and learning that isn't dictated by professors and grading curves.
I've been hiring people for the better part for 15 years and I never considered them to be valuable outside of the fact that it appears you're able to do one project for a sustained period of time. My impressions was unless your degree confers something such that you are in a job that human risk can be involved, most degrees are worth very little and most serious people know that.
To be clear, I think that most college degrees were generally low value (even my own), but still had some value. The current trend will be towards zero value unless something changes.
This is not related to "AI", but I have an amusing story about online cheating.
* I have a nephew who was switched into online college classes at the beginning of the pandemic.
* As soon as they switched to online, the class average on the exams shot up, but my nephew initially refused to cheat.
* Eventually he relented (because everyone else was doing it) and he pasted a multitude of sticky notes on the wall at the periphery of his computer monitor.
* His father walks into his room, looks at all the sticky notes and declares, "You can't do this!!! It'll ruin the wallpaper!"
If you’re hiring humans just to use AI, why even hire humans? Either AI will replace them or employers will realize that they prefer employees who can think. In either case, being a human who specializes in regurgitating AI output seems like a dead end.
TBF this problem doesn’t seem that new to me. I was forced to do my lab work in Vim and C via SSH because the faculty felt that Java IDEs with autocomplete were doing a disservice to learning.
It was (to some degree), and could still be. The status quo was more effective, relatively speaking, before the AI boom. The status quo appears to be trending towards ineffective, post-AI boom.
So in order to remain useful, the status quo of higher education will probably have to change in order to adapt to the ubiquity of AI, and LLMs currently.
Just because you can cheat at something doesn't mean doing it legitimately isn't useful.
Thinking of that. We have build these expensive machines with massive investments to be able to output what we expect college students to output... Wouldn't that tell us that well maybe that output has some value, intent or use? Or we would not have spend those resources...
Just because machine can do things, doesn't mean humans should be able to do it too. Say reading a text aloud.
I mean this seems a solved problem: hand-and-paper written onsite exams + blackboard-and-chalk oral onsite exams. If this is too costly (is it? many countries manage), make students take them less often.
I teach on a small university. These are some of the measures we take:
- Hand written midterms and exams.
- The students should explain how they designed and how they coded their solutions to programming exercises (we have 15-20 students per class, with more students it become more difficult).
- Presentations of complex topics (after that the rest of the students should comment something, ask some question, anything related to the topic)
- Presentation of a handwritten one page hand written notes, diagram, mindmap, etc., about the content discussed.
- Last minute changes to more elaborated programming labs that should be resolved in-class (for example, "the client" changed its mind about some requirement or asked a new feature).
The real problem is that it is a (lot) more work for the teachers and not everyone is willing to "think outside of the box".
Back when I was doing my BSc in Software Engineering, we had a teacher who did her Data Structure and Algorithms exams with pen and paper. On one of them, she basically wrote 4 coding problems (which would be solved in 4 short ~30 LOC).
We had to write the answer with pen and paper, writing the whole program in C. And the teacher would score it by transcribing the verbatim text in her computer, and if it had one single error (missed semicolon) or didn't compile for some reason, the whole thing was considered wrong (each question was 25% of the exam score)
I remember I got 1 wrong (missed semicolon :( ) and got a 75% (1-100 pointing system). It's crazy how we were able to do that sort of thing in the old days.
We definitely exercised our attention to detail and concentration muscles with that teacher.
Yes, pen and paper. The approach is to pseudocode the solution, minor syntax errors aren’t punished (and indeed are generally expected anyway). The point is to simply show that you understand and can work through the concepts involved, it’s not being literally compiled.
Writing a small algorithm with pen & paper on programming exams in universities of all sizes was still common when I was in uni in the 2010s and there’s no reason to drop that practice now.
One of the most offensive words in the anthropomophization of LLMs is: hallucinate.
It's not only an anthropomorphism, it's also a euphemism.
A correct interpretation of the word would imply that the LLM has some fantastical vision that it mistakes for reality. What utter bullsh1t.
Let's just use the correct word for this type of output: wrong.
When the LLM generates a sequence of words, that may or may not be grammatically correct, but infers a state or conclusion that is not factually correct; lets state what actually happened: the LLM generated text was WRONG.
It didn't take a trip down Alice's rabbit hole, it just put words together into a stream that inferred a piece of information that was incorrect, it was just WRONG.
The euphemistic aspect of using this word is a greater offense than the anthropomorphism, because it's painting some cutesy picture of what happened, instead of accurately acknowledging that the s/w generated an incorrect result. It's covering up for the inherent short comings of the tech.
When a person hallucinates a dragon coming for them, they are wrong, but we still use a different word to more precisely indicate the class of error.
Not all llm errors are hallucinations - if an llm tells me that 3 + 5 is 7, It's just wrong. If it tells me that the source for 3 + 5 being 7 is a seminal paper entitled "On the relative accuracy of summing numbers to a region +-1 from the fourth prime", we would call that a hallucination. In modern parlance " hallucination" has become a term of art to represent a particular class of error that llms are prone to. (Others have argued that "confabulation" would be more accurate, but it hasn't really caught on.)
It's perfectly normal to repurpose terms and anthropomorphizations to represent aspects of the world or systems that we create. You're welcome to try to introduce other terms that don't include any anthropomorphization, but saying it's "just wrong" conveys less information and isn't as useful.
I think your defense of reusing terms for new phenomenon is fair.
But in this specific case, I would say the reuse of this particular word, to apply to this particular error, is still incorrect.
A person hallucinating is based on a many leveled experience of consciousness.
The LLM has nothing of the sort.
It doesn't have a hierarchy of knowledge which it is sorting to determine what is correct and what is not. It doesn't have a "world view" based on a lifetime of that knowledge sorting.
In fact, it doesn't have any representation of knowledge at all. Much less a concept of whether that knowledge is correct or not.
What it has is a model of what words came in what order, in the training set on which it was "trained" (another, and somewhat more accurate, anthropomorphism).
So without anything resembling conscious thought, it's not possible for an LLM to do anything even slightly resembling human hallucination.
As such, when the text generated by an LLM is not factually correct, it's not an hallucination, it's just wrong.
I teach an "advanced" shell scripting course with an exam.
I mark "hallucinations" as "LLM Slop" in my grading sheets, when someone gives me a 100-character sed filter that just doesn't work that there is no way we discussed in class/in examples/in materials, or a made up API endpoint, or non-nonsensical file paths that reference non-existent commands.
Slop is an overused term these days, but it sums it up for me. Slop, from a trough, thrown out by an uncaring overseer, to be greedily eaten up by the piggies, who don't care if its full of shit.
Readit News
Students don’t seem to mind this reversion. The administration, however, doesn’t like this trend. They want all evaluation to be remote-friendly, so that the same course with the same evaluations can be given to students learning in person or enrolled online. Online enrollment is a huge cash cow, and fattening it up is a very high priority. In-person, pen-and-paper assessment threatens their revenue growth model. Anyways, if we have seven sections of Calculus I, and one of these sections is offered online/remote, then none of the seven are allowed any in person assessment. For “fairness”. Seriously.
LLMs aren't destroying the University or the essay.
LLMs are destroying the cheap University or essay.
Cheap can mean a lot of things, like money or time or distance. But, if Universities want to maintain a standard, then they are going to have to work for it again.
No more 300+ person freshman lectures (where everyone cheated anyways). No more take-home zoom exams. No more professors checked out. No more grad students doing the real teaching.
I guess, I'm advocating for the Oxbridge/St. John's approach with under 10 class sizes where the proctor actually knows you and if you've done the work. And I know, that is not a cheap way to churn out degrees.
I could understand US tuition if that were the case. These days with overworked adjuncts make it McDonalds at Michelin star prices.
But yeah, if the professor is clearly checked out and only interested in his research, and the students are being told that the only purpose of their education is to get a piece of paper to show to potential employers, you'll get a cynical death-spiral.
(I've been on both sides of this, though back when copy-pasting from Wikipedia was the way to cheat.)
So, it's pretty hard for universities over here to maintain standards in this GenAI world, when the paying customer only cares about quantity, and not quality. I'm feeling bad for the students, not so much for foolish politicians.
Provide machine problems and homework as exercises for students to learn, but assign a very low weight to these as part of an overall grade. Butt in seat assessments should be the majority of a course assessment for many courses.
This is depressing. I'm late GenX, I didn't cheat in college (engineering, RPI), nor did my peers. Of course, there was very little writing of essays so that's probably why, not to mention all of our exams were in person paper-and-pencil (and this was 1986-1990, so no phones). Literally impossible to cheat. We did have study groups where people explained the homework to each other, which I guess could be called "cheating", but since we all shared, we tended to oust anyone who didn't bring anything to the table. Is cheating through college a common millenial / gen z thing?
Students can learn for free via online resources, forums, and LLM tutors (the less-trustworthy forums and LLMs should primarily be used to assist understanding the more-trustworthy online resources). EDIT: students can get hands-on-experience via an internship, possibly unpaid.
Real universities should continue to exist for their cutting-edge research and tutoring from very talented people, because that can't be commodified. At least until/if AI reaches expert competence (in not just knowledge but application), but then we don't need jobs either.
The masses get the cheap AI education. The elite get the expensive, small class, analog education. There won't be a middle class of education, as in the current system - too expensive for too little gain.
But that commercially oriented boards are ruining education, that's a given. That they would stoop to this level is a bit surprising.
certainly not anywhere close to ten!
I think things are going to have to get a lot worse before they get better. If we're lucky, things will get so bad that we finally fix some shaky foundations that our society has been trying to ignore for decades (or even centuries). If we're not lucky, things will still get that bad but we won't fix them.
So they know what students should be taught but I don't know that they necessarily know how any better than the administrators.
I've always found it weird that you need teaching certification to teach basic concepts to kindergartners but not to teach calculus to adults.
What has AI done? I teach a BA thesis seminar. Last year, when AI wasn't used as much, around 30% of the students failed to turn in their BA thesises. 30% drop-out rate was normal. This year: only 5% dropped out, while the amount of ChatGPT generated text has skyrocketed. I think there is a correlation: ChatGPT helps students write their thesises, so they're not as likely to drop out.
The University and the admins are probably very happy that so many students are graduating. But also, some colleagues are seeing an upside to this: if more graduate, the University gets more money, which means less cuts to teaching budgets, which means that the teachers can actually do their job and improve their courses, for those students who are actually there to learn. But personally, as a teacher, I'm at loss of what to do. Some thesises had hallucinated sources, some had AI slop blogs as sources, the texts are robotic and boring. But should I fail them, out of principle on what the ideal University should be? Nobody else seems to care. Or should I pass them, let them graduate, and reserve my energy to teach those who are motivated and are willing to engage?
Can't this be done in the US as well ?
Do you feel it is effective?
It seems to me that there is a massive asymmetry in the war here: proctoring services have tiny incentives to catch cheaters. Cheaters have massive incentives to cheat.
I expect the system will only catch a small fraction of the cheating that occurs.
Also: I think your suggestion is excellent. We may see this happen in the US if AI cheating gets out of control (which it well).
Those I ask are unanimously horrified that this is the choice they are given. They are devastated that the degree for which they are working hard is becoming worthless yet they all assert they don't want exams back. Many of them are neurodivergent who do miserably in exam conditions and in contrast excel in open tasks that allow them to explore, so my sample is biased but still.
They don't have a solution. As the main victims they are just frustrated by the situation, and at the "solutions" thrown at it by folks who aren't personally affected.
caveat emptor - I am not ND so maybe this is a real concern for some, but in my experience the people who said this did not know the material. And the accommodations for tests are abused by rich kids more than they are utilized by those that need them.
The solution I use when teaching is to let evaluation primarily depend on some larger demonstration of knowledge. Most often it is CS classes (e.g. Machine Learning), so I don't really give much care for homeworks and tests and instead be project driven. I don't care if they use GPT or not. The learning happens by them doing things.
This is definitely harder in other courses. In my undergrad (physics) our professors frequently gave takehome exams. Open book, open notes, open anything but your friends and classmates. This did require trust, but it was usually pretty obvious when people worked together. They cared more about trying to evaluate and push us if we cared than if we cheated. They required multiple days worth of work and you can bet every student was coming to office hours (we had much more access during that time too). The trust and understanding that effort mattered actually resulted in very little cheating. We felt respected, there was a mutual understanding, and tbh, it created healthy competition among us.
Students cheat because they know they need the grade and that at the end of the day they won't won't actually be evaluated on what they learned, but rather on what arbitrary score they got. Fundamentally, this requires a restructuring, but that's been a long time coming. The cheating literally happens because we just treated Goodhart's Law as a feature instead of a bug. AI is forcing us to contend with metric hacking, it didn't create it.
Isn't this part of life? Learning to excel anyway?
Students are more accurately measured via long, take-home projects, which are complicated enough that they can’t be entirely done by AI.
Unless the class is something that requires quick thinking on the job, in which case there should be “exams” that are live simulations. Ultimately, a student’s GPA should reflect their competence in the career (or possible careers) they’re in college for.
I mean, for every neurodivergent person who does miserably in exam conditions you have one that does miserably in homework essays because of absence of clear time boundaries.
if "many" are "divergent" then... are they really divergent? or are they the new typical?
Less stress at the end of the term, and the student can't leave everything to the last minute, they need to do a little work every week.
Teachers and professors: you can say "no". Your students will thank you in the future.
I am now doing an Online MSc in CompSci at Georgia Tech. The online evaluation and proctoring is fine. I’ve taken one rather math-heavy course (Simulation) and it worked. I see the program however is struggling with the online evaluation of certain subjects (like Graduate Algorithms).
I see your point that a professor might prefer to have physical evaluation processes. I personally wouldn’t begrudge the institution as long as they gave me options for proctoring (at my own expense even) or the course selection was large enough to pick alternatives.
This hybrid model is vastly preferable to "true" remote test taking in which they try to do remote proctoring to the student's home using a camera and other tools.
seriously it reminds me of my high school days when a teacher told me i shouldn’t type up my essays because then they couldn’t be sure i actually wrote them.
maybe we will find our way back to live oral exams before long…
So far, it’s still possible to opt out of this coordinated model, and I have been. But I suspect the ability to opt out will soon come under attack (the pretext will be ‘uniformity == fairness’). I never used to be an academic freedom maximalists who viewed the notion in the widest sense, but I’m beginning to see my error.
Wow.
Maybe we should consider the possibility that this isn't a good idea? Just a bit? No? Just ignore how obviously comparable this is to the most famous dystopian fiction in literary history?
Just wow. If you're willing to do that, I don't know what to tell you.
Since endowments got huge.
It's probably not as bad for mathematical derivations. I still do those by hand since they are more like drawing than expression.
So is testing; people who don't have the skills don't do well. Hell, the entire concept of education is ableist towards learning impaired kids. Let's do away with it entirely.
Having had much occasion to consider this issue, I would suggest moving away from the essay format. Most of the typical essay is fluff that serves to provide narrative cohesion. If knowledge of facts and manipulation of principles are what is being evaluated, presentation by bullet points should be sufficient.
Doing anything is inherently based on your ability to do it. Running is inherently ableist. Swimming is ableist. Typing is inherently ableist.
Pointing this out is just a thought terminating cliche. Ok, it's ableist. So?
> As soon as I could use a word processor I blossomed.
You understand this is inherently ableist to people that can't type?
> I still do those by hand since they are more like drawing than expression.
Way to do ableist math.
yes.
> I would be at a massive disadvantage.
yes.
...but.
how would you propose to filter out able cheaters instead? there's also in person one on one verbal exam, but economics and logistics of that are insanely unfavorable (see also - job interviews.)
On the one hand, I use AI extensively for my own learning, and it's helping me a lot.
On the other hand, it gets work done quickly and poorly.
Students mistake mandatory assignments for something they have to overcome as effortlessly as possible. Once they're past this hurdle, they can mind their own business again. To them, AI is not a tutor, but a homework solver.
I can't ask them to not use computers.
I can't ask them to write in a language I made the compiler for that doesn't exist anywhere, since I teach at a (pre-university) level where that kind of skill transfer doesn't reliably occur.
So far we do project work and oral exams: Project work because it relies on cooperation and the assignment and evaluation is open-ended: There's no singular task description that can be plotted into an LLM. Oral exams because it becomes obvious how skilled they are, how deep their knowledge is.
But every year a small handful of dum-dums made it all the way to exam without having connected two dots, and I have to fail them and tell them that the three semesters they have wasted so far without any teachers calling their bullshit is a waste of life and won't lead them to a meaningful existence as a professional programmer.
Teaching Linux basics doesn't suffer the same because the exam-preparing exercise is typing things into a terminal, and LLMs still don't generally have API access to terminals.
Maybe providing the IDE online and observing copy-paste is a way forward. I just don't like the tendency that students can't run software on their own computers.
I don't see why this is so hard, other than the usual intergenerational whining / a heaping pile of student entitlement.
If anything, the classes that required extensive paper-writing for evaluation are the ones that seem to be in trouble to me. I guess we're back to oral exams and blue books for those, but again...worked fine for prior generations.
Yup. ~25 years ago competitions / NOI / leet_coding as they call it now were in a proctored room, computers with no internet access, just plain old borland c, a few problems and 3h of typing. All the uni exams were pen & paper. C++ OOP on paper was fun, but iirc the scoring was pretty lax (i.e. minor typos were usually ignored).
You know that grading paper exams is a lot more hassle _for the teachers_?
Your overall point might or might not still stand. I'm just responding to your 'I don't see why this is so hard'. Show some imagination for why other people hold their positions.
(I'm sure there's lots of other factors that come into play that I am not thinking of here.)
There’s such a shortfall of teachers globally, and the role is a public good, so it’s constantly underpaid.
And if you are good - why would you teach ? You’d get paid to just take advantage of your skills.
And now we have a tool that makes it impossible to know if you have taught anyone because they can pass your exams.
We wrote c++ on paper for some questions and were graded on it. Ofcourse the tutors were lenient on the syntax they cared about the algorithm and the data structures not so much for the code. They did test syntax knowledge as well but more in code reasoning segments, i.e questions like what's the value of a after these two statements or after this loop is run.
We also had exams in the lab with computers disconnected from the internet. I don't remember the details of the grading but essentially the teaching team was in the room and pretty much scored us then and there.
Deleted Comment
It has been interesting to see this idea propagate throughout online spaces like Hacker News, too. Even before LLMs, the topic of cheating always drew a strangely large number of pro-cheating comments from people arguing that college is useless, a degree is just a piece of paper, knowledge learned in classes is worthless, and therefore cheating is a rational decision.
Meanwhile, whenever I’ve done hiring or internships screens for college students it’s trivial to see which students are actually learning the material and which ones treat every stage of their academic and career as a game they need to talk their way through while avoiding the hard questions.
(Dramatic. AI is fine for upper-division courses, maybe. Absolutely no use for it in introductory courses.)
Our school converted a computer lab to a programming lab. Computers in the lab have editors/compilers/interpreters, and whitelist documentation, plus an internal server for grading and submission. No internet access otherwise. We've used it for one course so far with good results, and and extending it to more courses in the fall.
An upside: our exams are now auto-graded (professors are happy) and students get to compile/run/test code on exams (students are happy).
>Students mistake mandatory assignments for something they have to overcome as effortlessly as possible.
This is the real demon to vanquish. We're approaching course design differently now (a work in progress) to tie coding exams in the lab to the homework, so that solving the homework (worth a pittance of the grade) is direct preparation for the exam (the lion's share of the grade).
Excellent approach. It requires a big buy-in from the school.
Thanks for suggesting it.
I'm doing something for one kind of assignment inspired by the game "bashcrawl" where you have to learn Linux commands through an adventure-style game. I'm bundling it in a container and letting you submit your progress via curl commands, so that you pass after having run a certain set of commands. Trying to make the levels unskippable by using tarballs. Essentially, if you can break the game instead of beating it honestly, you get a passing grade, too.
As an higher education (university) IT admin who is responsible for the CS program's computer labs and is also enrolled in this CS program, I would love to hear more about this setup, please & thank you. As recently as last semester, CS professors have been doing pen'n paper exams and group projects. This setup sounds great!
So if the professors can cheat and they're happy about having to do less teaching work, thereby giving the students a lower-quality educational experience, why shouldn't the students just get an LLM to write code that passes the auto-grader's checks? Then everyone's happy - the administration is getting the tuition, the professors don't have to grade or give feedback individually, and the students can finish their assignments in half an hour instead of having to stay up all night. Win win win!
Assuming you have access to a computer lab, have you considered requiring in-class programming exercises, regularly? Those could be a good way of checking actual skills.
> Maybe providing the IDE online and observing copy-paste is a way forward. I just don't like the tendency that students can't run software on their own computers.
And you'll frustrate the handful of students who know what they're doing and want to use a programmer's editor. I know that I wouldn't have wanted to type a large pile of code into a web anything.
I might not have liked that, but I sure would have liked to see my useless classmates being forced to learn without cheating.
Even IntelliJ has gateway
Wow.
I guess this "tough love" attitude helps for some people? But I think mostly it's just that people think it works for _other_ people, but rarely people think that this works when applied to themselves.
Like, imagine the school administration walking up to this teacher and saying "hey dum dum, you're failing too many students and the time you've spent teaching them is a waste of life."
Many teachers seem to think that students go to school/university because they're genuinely interested in motivated. But more often then not, they're there because of societal pressure, because they know they need a degree to have any kind of decent living standard, and because their parents told them to. Yeah you can call them names, call them lazy or whatever, but that's kinda like pointing at poor people and saying they should invest more.
Deleted Comment
> small handful of dum-dums made it all the way to exam without having connected two dots, and I have to fail them ... won't lead them to a meaningful existence
I don't see a problem, the system is working.
The same group of people that are going to loose their job to an LLM arent getting smarter because of how they are using LLM's.
Students who use LLMs and professional programmers who use LLMs: I wouldn't say it's necessarily the same group of people.
Sure, their incentives are the same, and they're equally unlikely to maintain jobs in the future.
But students can be told that their approach to become AI secretaries isn't going to pan out. They're not actively sacrificing a career because they're out of options. They can still learn valuable skills, because what they were taught has not been made redundant yet, unlike mediocre programmers who can only just compete with LLM gunk.
What’s changed is that “some working code” is no longer proof that a student understands the material.
You’re going to need a new way to identify students that understand the material.
At the time it was optional, but I get the feeling that if they still use that framework, it just became mandatory, because it has no internet facing documentation.
That said, I imagine they might have chucked it in for Unity before AI hit, in which case they are largely out of luck.
>But every year a small handful of dum-dums made it all the way to exam without having connected two dots, and I have to fail them and tell them that the three semesters they have wasted so far without any teachers calling their bullshit is a waste of life and won't lead them to a meaningful existence as a professional programmer.
This happened to me with my 3d maths class, and I was able to power through a second run. But I am not sure I learned anything super meaningful, other than I should have been cramming better.
Huh, fighting my way through a Linux CLI is exactly the kind of thing I use Chatgpt for professionally.
I did study it in compsci, but those commands are inherently not memorable.
That's where you're wrong. Being a professional programmer is 10% programming, 40% office politics, and 50% project management. If your student managed to get halfway through college without any actual programming skills, they're perfect candidate, because they clearly own the 90% of skills needed to be a professional programmer.
I'd say that really depends on your job.
At smaller companies, your job will likely be 60% programming at a minimum.
Only at ~100 employees do companies fall into lots of meetings and politics.
Some of my students are naturally talented.
Others achieve great results through hard work.
Some half-assedly make it.
And some don't even try.
Those are the dum-dums.
They just play games and think everything is going to work out without effort.
This is only temporary. It will be able to code like anyone in time. The only way around this will be coding in-person, but only in elementary courses. Everyone in business will be using AI to code, so that will be the way in most university courses as well.
I usually ask it to grade my homework for me before I turn it in. I usually find I didn’t really understand some topic and the AI highlights this and helps set my understanding straight. Without it I would have just continued on with an incorrect understanding of the topic for 2-3 weeks while I wait for the assignment to be graded. As an adult with a job and a family this is incredibly helpful as I do homework at 10pm and all the office hours slots are in the middle of my workday.
I do admit though it is tough figuring out the right amount to struggle on my own before I hit the AI help button. Thankfully I have enough experience and maturity to understand that the struggle is the most important part and I try my best to embrace it. Myself at 18 would definitely not have been using AI responsibly.
This is my biggest concert about GenAI in our field. As an experienced dev I've been around the block enough times to have a good feel of how things should be done and can catch when and LLM goes off on a tangent that is a complete rabbit hole, but if this had been available 20 years ago I would have never learned and become an experienced dev because I absolutely would have over relied on an LLM. I worry that 10 years from now getting mid career dev will be like trying to get a COBOL dev now, except COBOL is a lot easier to learn.
I am planning on doing a masters but I need some undergrad CS credits to be a qualified candidate. I don’t think I’m going to do the whole undergrad.
Overall my experience has been positive. I’ve really enjoyed Discrete Math and coming to understand how I’ve been using set theory without really understanding it for years. I’m really looking forward to my classes on assembly/computer architecture, operating systems, and networks. They did make me take CS 101-102 as prereqs which was a total waste of time and money, but I think those are the only two mandatory classes with no value to me.
Not GP, but in my experience most MSC programs will require that you have substantial undergrad CS coursework in order to be accepted. There are a few programs designed for those without that background.
Dead Comment
Deleted Comment
Similar to Montessori, LLMs can help students who wander off in various directions.
I remember often being “stuck” on some concept (usually in biology and chemistry), where the teacher would hand-wave something as truth, this dismissing my request for further depth.
Of course, LLMs in the current educational landscape (homework-heavy) only benefit the students who are truly curious…
My hope is that, with new teaching methods/styles, we can unlock (or just maintain!) the curiosity inherent in every pupil.
(If anyone knows of a tool like this, where an LLM stays on a high-level trajectory of e.g. teaching trigonometry, but allows off-shoots/adventures into other topical nodes, I’d love to know about it!)
I think you hit on a major issue: Homework-heavy. What I think would benefit the truly curious is spare time. These things are at odds with one another. Present-day busy work could easily be replaced by occupying kids' attention with continual lessons that require a large quantity of low-quality engagement with the LLM. Or an addictive dopamine reward system that also rewards shallow engagement -- like social media.
I'm 62, and what allowed me to follow my curiosity as a kid was that the school lessons were finite, and easy enough that I could finish them early, leaving me time to do things like play music, read, and learn electronics.
And there's something else I think might be missing, which is effort. For me, music and electronics were not easy. There was no exam, but I could measure my own progress -- either the circuit worked or it didn't. Without some kind of "external reference" I'm not sure that in-depth research through LLMs will result in any true understanding. I'm a physicist, and I've known a lot of people who believe that they understand physics because they read a bunch of popular books about it. "I finally understand quantum mechanics."
I see both sides of this. When I was a teenager, I went to a pretty bad middle school where there were fights everyday, and I wasn’t learning anything from the easy homework. On the upside, I had tons of free time to teach myself how to make websites and get into all kinds of trouble botting my favorite online games.
My learning always hit a wall though because I wasn’t able to learn programming on my own. I eventually asked my parents to send me to a school that had a lot more structure (and a lot more homework), and then I properly learned math and logic and programming from first principles. The upside: I could code. The downside: there was no free time to apply this knowledge to anything fun
Yeah I feel like teachers are going to try and use LLMs as an excuse to push more of the burden of schooling to their pupils homelife somehow. Like, increasing homework burdens to compensate.
The specific examples I recall most vividly were from 4th grade and 7th grade.
This resonates with me a lot. I used to dismiss AI as useless hogwash, but have recently done a near total 180 as I realised it's quite useful for exploratory learning.
Not sure about others but a lot of my learning comes from comparison of a concept with other related concepts. Reading definitions off a page usually doesn't do it for me. I really need to dig to the heart of my understanding and challenge my assumptions, which is easiest done talking to someone. (You can't usually google "why does X do Y and not Z when ABC" and then spin off from that onto the next train of reasoning).
Hence ChatGPT is surprisingly useful. Even if it's wrong some of the time. With a combination of my baseline knowledge, logic, cross referencing, and experimentation, it becomes useful enough to advance my understanding. I'm not asking ChatGPT to solve my problem, more like I'm getting it to bounce off my thoughts until I discover a direction where I can solve my problem.
Eg. it's easy to ask copilot: can you give me a list of free, open source mqtt brokers and give me some statistics in the form of a table
And copilot (or any other ai) does this quite nicely. This is not something that you can ask a traditional search engine.
Offcourse you do need to know enough of the underlying material and double check what output you get for when the AI is hallucinating.
Putting together a project using the AI help will be a very close mimicry of what real work will be like and if the teacher is good they will learn way more than being able to spout information from memory.
The fact that AI can do your homework should tell you how much your homework is worth. Teaching and learning are collaborative exercises.
Homework is there to help you practise these things and have help you progress, find the areas where you're in need of help and more practise. It is collaborative, it's you, your fellow students and your teachers/professors.
I'm sorry that you had bad teachers, or had needs that wasn't being meet by the education system. That is something that should be addressed. I just don't think it's reasonable to completely dismiss a system that works for the majority. Being mad at the education system isn't really a good reason for say "AI/computers can do all these things, so why bother practising them?"
Schools should learn kids to think, but if the kids can't read or reasonably do basic math, then expecting them to have independent critical thinking seems a way of. I don't know about you, but one of the clear lessons in "problem math" in schools was to learn to reason about numbers and result, e.g. is it reasonable that a bridge span 43,000km? If not, you probably did something wrong in your calculations.
Giving people credit for homework helps because it gives students a chance to earn points outside of high pressure test times and it also encourages people to do the homework. A lot of people need the latter.
My friends who teach university classes have experimented with grading structures where homework is optional and only exam scores count. Inevitably, a lot of the class fails the exams because they didn’t do any practice on their own. They come begging for opportunities to make it up. So then they circle back to making the homework required and graded as a way to get the students to practice.
ChatGPT short circuits this once again. Students ChatGPT their homework then fail the first exam. This time there is little to do, other than let those students learn the consequences of their actions.
A lot of people who say this kind of thing have, frankly, a very shallow view of what homework is. A lot of homework can be easily done by AI, or by a calculator, or by Wikipedia, or by looking up the textbook. That doesn't invalidate it as homework at all. We're trying to scaffold skills in your brain. It also didn't invalidate it as assessment in the past, because (eg) small kids don't have calculators, and (eg) kids who learn to look up the textbook are learning multiple skills in addition to the knowledge they're looking up. But things have changed now.
"The fact that forklift truck can lift over 500kg should tell you how worthwhile it is for me to go to a gym and lift 100kg." - complete non-sequitur.
Then maybe the homework assignment has been poorly chosen. I like how the article's author has decided to focus on the process and not the product and I think that's probably a good move.
I remember one of my kids' math teachers talked about wanting to switch to in inverted classroom. The kids would be asked to read a some part of their textbook as homework and then they would work through exercise sheets in class. To me, that seemed like a better way to teach math.
> But things have changed now.
Yep. Students are using AIs to do their homework and teachers are using AIs to grade.
you still have to learn. The goal of learning is not to do a job. It's to enrich you, broaden your mind, and it takes work on your part.
In similar reasoning, you could argue that you can take a car to go anywhere, or let everything be delivered on your doorstep, so why should I my child learn to walk?
The fact that AI can replace the work that you are measured on should tell you something about the measurement itself.
The goal of learning should be to enrich the learner. Instead, the goal of learning is to pass measure. Success has been quietly replaced with victory. Now LLMs are here to call that bluff.
However, you can instead recognize the difficulty and time that this would require on the part of the educator, and therefore expense to the student, and you can recognize that you have the goal of not just obtaining a piece of paper but actually learning a skill. With this mindset, it makes sense to take the initiative to treat the homework as an opportunity to learn and practice. It's is one of those things that's worth as much as you put into it. Of course, one can use their judgement to decide which homework is worth spending time on to learn the material, and which can be safely sailed through with minimum effort.
Having a skilled teacher that you can really collaborate with and who can spend the time to evaluate your skills in a personal way is of course going to lead to better learning outcomes than the traditional education system. It will also be far more expensive. Although, AI is offering something somewhat akin to this experience at a much lower price, to those who are able to moderate their usage so that they are learning from the AI instead of just offloading tasks to it.
Unis should adjust their testing practices so that their paper (and their name) doesn't become worthless. If AI becomes a skill, it should be tested, graded, and certified accordingly. That is, separate the computer science degree from the AI Assisted computer science degree.
I mean... if you removed the substring "home" from that sentence, is it still true in your opinion?
That is, do you believe that because AI can perform some task, that task must not have any value? If there's a difference, help me understand it better please.
Homework should be a part of the collaborative process of learning (as others above have already elaborated on). If teachers are having a problem with AI generated homework being submitted, it shows that the system is broken because they couldn't have been collaborating with students on their learning then.
That's precisely where we went wrong. Capitalism has redefined our entire education system as a competition; just like it does with everything else. The goal is not success, it's victory.
If students using AI to cheat on homework are graduating with a degree, then it has lost all value as a certificate that the holder has completed some minimum level of education and learning. Institutions that award such degrees will be no different than degree mills of the past.
I’m just grateful my college degree has the year 2011 on it, for what it’s worth.
For most subjects at the university level graded homework (and graded attendance) has always struck me as somewhat condescending and coddling. Either it serves to pad out grades for students who aren't truly learning the material or it serves to force adult students to follow specific learning strategies that the professor thinks are best rather than giving them the flexibility they deserve as grown adults.
Give students the flexibility to learn however they think is best and then find ways to measure what they've actually learned in environments where cheating is impossible. Cracking down on cheating at homework assignments is just patching over a teaching strategy that has outgrown its usefulness.
I have had so many very frustrating conversations with full grown adults in charge of teaching CS. I have no faith at all that students would be able to choose an appropriate method of study.
My issue with the instruction is the very narrow belief in the importance of certain measurable skills. VERY narrow. I won’t go into details, for my own sanity.
I have the opposite experience - the best professors focused on homework and projects and exams were minimal to non-existent. People learn different ways, though, so you might function better having the threat/challenge of an exam, whereas I hated having to put everything together for an hour of stress and anxiety. Exams are artificial and unlike the real world - the point is to solve problems, not to solve problems in weirdly constrained situations.
I agree that something will have to change to avert the current trend.
You can't put the toothpaste back into the tube. Universities need to accept that AI exists, and adjust their operations accordingly.
Provide an incentive for students to do the thing they should be doing anyway.
Give an opportunity to provide feedback on the assignment.
It is totally useless as an evaluation mechanic, because of course the students that want to can just cheat. It’s usually pretty small, right? IIRC when I did tutoring we only gave like 10-20% for the aggregate homework grade.
I suspect the opposite: Known-good college degrees will become more valuable. The best colleges will institute practices that confirm the material was learned, such as emphasizing in-person testing over at-home assignments.
Cheating has actually been rampant at the university level for a long time, well before LLMs. One of the key differentiators of the better institutions is that they are harder to cheat to completion.
At my local state university (where I have friends on staff) it’s apparently well known among the students that if they pick the right professors and classes they can mostly skate to graduation with enough cheating opportunity to make it an easy ride. The professors who are sticklers about cheating are often avoided or even become the targets of ratings-bombing campaigns
The immediate effect was the distrust of the professors towards most everyone and lots classes felt like some kind of babysitting scheme, which I did not appreciate.
College students still cram and purge. Nobody forced to sit through OChem remembers their Diels-Alder reaction except the organic chemists.
College degrees probably don't have as much value as we've historically ascribed to them. There's a lot of nostalgia and tradition pent up in them.
The students who do the best typically fill their schedule with extra-curricular projects and learning that isn't dictated by professors and grading curves.
This is not related to "AI", but I have an amusing story about online cheating.
* I have a nephew who was switched into online college classes at the beginning of the pandemic.
* As soon as they switched to online, the class average on the exams shot up, but my nephew initially refused to cheat.
* Eventually he relented (because everyone else was doing it) and he pasted a multitude of sticky notes on the wall at the periphery of his computer monitor.
* His father walks into his room, looks at all the sticky notes and declares, "You can't do this!!! It'll ruin the wallpaper!"
Sounds laughably naive now, doesn’t it?
It’s the same old story with a new set of technology.
So in order to remain useful, the status quo of higher education will probably have to change in order to adapt to the ubiquity of AI, and LLMs currently.
Just because you can cheat at something doesn't mean doing it legitimately isn't useful.
Just because machine can do things, doesn't mean humans should be able to do it too. Say reading a text aloud.
^ Why many go to Harvard. Very nice club.
- Hand written midterms and exams.
- The students should explain how they designed and how they coded their solutions to programming exercises (we have 15-20 students per class, with more students it become more difficult).
- Presentations of complex topics (after that the rest of the students should comment something, ask some question, anything related to the topic)
- Presentation of a handwritten one page hand written notes, diagram, mindmap, etc., about the content discussed.
- Last minute changes to more elaborated programming labs that should be resolved in-class (for example, "the client" changed its mind about some requirement or asked a new feature).
The real problem is that it is a (lot) more work for the teachers and not everyone is willing to "think outside of the box".
(edit: format)
We had to write the answer with pen and paper, writing the whole program in C. And the teacher would score it by transcribing the verbatim text in her computer, and if it had one single error (missed semicolon) or didn't compile for some reason, the whole thing was considered wrong (each question was 25% of the exam score)
I remember I got 1 wrong (missed semicolon :( ) and got a 75% (1-100 pointing system). It's crazy how we were able to do that sort of thing in the old days.
We definitely exercised our attention to detail and concentration muscles with that teacher.
Writing a small algorithm with pen & paper on programming exams in universities of all sizes was still common when I was in uni in the 2010s and there’s no reason to drop that practice now.
It's not only an anthropomorphism, it's also a euphemism.
A correct interpretation of the word would imply that the LLM has some fantastical vision that it mistakes for reality. What utter bullsh1t.
Let's just use the correct word for this type of output: wrong.
When the LLM generates a sequence of words, that may or may not be grammatically correct, but infers a state or conclusion that is not factually correct; lets state what actually happened: the LLM generated text was WRONG.
It didn't take a trip down Alice's rabbit hole, it just put words together into a stream that inferred a piece of information that was incorrect, it was just WRONG.
The euphemistic aspect of using this word is a greater offense than the anthropomorphism, because it's painting some cutesy picture of what happened, instead of accurately acknowledging that the s/w generated an incorrect result. It's covering up for the inherent short comings of the tech.
Not all llm errors are hallucinations - if an llm tells me that 3 + 5 is 7, It's just wrong. If it tells me that the source for 3 + 5 being 7 is a seminal paper entitled "On the relative accuracy of summing numbers to a region +-1 from the fourth prime", we would call that a hallucination. In modern parlance " hallucination" has become a term of art to represent a particular class of error that llms are prone to. (Others have argued that "confabulation" would be more accurate, but it hasn't really caught on.)
It's perfectly normal to repurpose terms and anthropomorphizations to represent aspects of the world or systems that we create. You're welcome to try to introduce other terms that don't include any anthropomorphization, but saying it's "just wrong" conveys less information and isn't as useful.
But in this specific case, I would say the reuse of this particular word, to apply to this particular error, is still incorrect.
A person hallucinating is based on a many leveled experience of consciousness.
The LLM has nothing of the sort.
It doesn't have a hierarchy of knowledge which it is sorting to determine what is correct and what is not. It doesn't have a "world view" based on a lifetime of that knowledge sorting.
In fact, it doesn't have any representation of knowledge at all. Much less a concept of whether that knowledge is correct or not.
What it has is a model of what words came in what order, in the training set on which it was "trained" (another, and somewhat more accurate, anthropomorphism).
So without anything resembling conscious thought, it's not possible for an LLM to do anything even slightly resembling human hallucination.
As such, when the text generated by an LLM is not factually correct, it's not an hallucination, it's just wrong.
Deleted Comment
I mark "hallucinations" as "LLM Slop" in my grading sheets, when someone gives me a 100-character sed filter that just doesn't work that there is no way we discussed in class/in examples/in materials, or a made up API endpoint, or non-nonsensical file paths that reference non-existent commands.
Slop is an overused term these days, but it sums it up for me. Slop, from a trough, thrown out by an uncaring overseer, to be greedily eaten up by the piggies, who don't care if its full of shit.