I’ve been teaching in higher education for 30 years and am soon retiring. I teach math. In every math course there is massive amounts of cheating on everything that is graded that is not proctored in a classroom setting. Locking down browsers and whatnot does not prevent cheating.
The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test. But any teacher doing this will end up with no students signing up for their class. The only solution I see is the Higher Learning Commission mandating this for all classes.
But even requiring in person proctored exams is not the full solution. Students are not used to doing the necessary work to learn. They are used to doing the necessary work to pass. And that work is increasingly cheating. It’s a clusterfuck. I have calculus students who don’t know how to work with fractions. If we did truly devise a system that prevents cheating we’ll see that a very high percentage of current college students are not ready to be truly college educated.
My personal take, we’ve made the cost of failure to high and cheating too easy.
As a student, the only thing the next institution will see is GPA, school, major. Roughly in that order. If the cost of not getting an A is exclusion from future opportunities- then students will reject exclusion by taking easier classes or cheating.
As someone who studied physics and came out with a 2.7 GPA due to studying what I wanted (the hard classes) and not cheating (as I did what I wanted) - I can say that there are consequences to this approach.
In my opinion, the solution is to reduce the reliance on assessments which are prone to cheating or which in the real world would be done by computer.
This really can't be emphasized enough. Universities and the initial hiring process really optimize for a score and not for learning. Those could be, and sometimes are, correlated, but it isn't necessarily the case.
Really focusing on stretching yourself necessarily means lower grades. Why is that penalized? TBH, in software engineering a lot of people with lower grades tutor the ones with 4.0 averages. The skillsets required to code and the skillsets required to get a good grade on a test are different.
I just want to second this (also did an undergrad in physics funny enough). I specifically sought out the harder professors in my undergrad and for the most part I'm happy I did it, but it's also a good thing that I'm not very motivated by money or prestige because I saw many of my colleagues who had gotten into better schools or jobs (even just the return calls on applications) who chose the easier routes or cheated. They are without a doubt wealthier. What mattered the most was the line items on their resumes and networking, but there is feedback in this so one begets the other. Fwiw, I had a 3.3.
So it then becomes hard for me to make suggestions to juniors. It isn't difficult to sniff out those like you or me who are motivated by the rabbit holes themselves, nor difficult to tell those who are entirely driven by social pressures (money, prestige, family, etc), but what about those on the edge? I think it's the morally best option to encourage learning for learning but it's naive to also not recognize that their peers who will cheat will be rewarded for that effort. It's clear that we do not optimize for the right things and we've fallen victim to Goodhart's Law, but I just hope we can recognize it because those systems are self reinforcing and the longer we work in them the harder they are to escape. Especially because there are many bright students who's major flaw is simply a lack of opportunity. For me? I'm just happy if I can be left to do my research, read papers and books, and have sufficient resources -- which is much more modest than many of my peers (ML). But it'd be naive to not recognize the costs and I'm a big believer in recognizing incentive structures and systematic issues. Unfortunately these are hard to resolve because they're caused by small choices by all of us collectively, but fortunately that too means they can be resolved by small choices each of us make.
Employers need to wake up to this in hiring, too. You can get a 4.0 with a degree in computer science from a top school, and still not be able to program at all.
Some organizations still hire software engineers just based on resume and a nontechnical interview. This can easily be a disaster! You need to do a real assessment during the interview of how well software engineers can code.
I think grading is obsolete. Grade inflation increased a lot the past 30 years. Ironically, it has increased the least at the least prestigious colleges. Pass/fail is the way to go. Don’t know if this would mess up things like applying for graduate school or jobs but let’s end the farce that grading has become.
> My personal take, we’ve made the cost of failure to high and cheating too easy.
I agree with the first part, but I think the second follows from it.
Take a class like organic chemistry. When I was in school, the grade was based on 5 exams, each worth 20% of your grade. Worse still, anything less than an A was seen as a failure for most students dreaming of medical/vet school.
Of course you are going to have people that are going to cheat. You've made the stakes so high that the consequences of getting caught cheating are meaningless.
On top of that, once enough students are cheating, you need to cheat just to keep up.
> As a student, the only thing the next institution will see is GPA, school, major. Roughly in that order. If the cost of not getting an A is exclusion from future opportunities- then students will reject exclusion by taking easier classes or cheating.
That's not the cost of not getting an A, it's the cost of appearing to underperform compared to too many of your peers. Which is directly tied to how many of them cheat. If not enough cheaters got an A then the cost would no longer be tied to not getting an A, it would be tied to whatever metric they appeared to outperform you on.
> As someone who studied physics and came out with a 2.7 GPA due to studying what I wanted (the hard classes) and not cheating (as I did what I wanted) - I can say that there are consequences to this approach.
I can, too. I wanted to learn, but I also wanted to achieve a high GPA. I had a privileged background, so I got to retake classes after earning Cs or Bs until I got an A, without cheating.
The consequences: My degree took a long time to get, cost more money than my peers in the same program, and I now have a deep-seated feeling of inadequacy.
> My personal take, we’ve made the cost of failure to high and cheating too easy.
This is so true. I was recently pondering about the impact of AI cheating in Africa and came up with the conclusion that it won't be as significant as in EU/US precisely because most evaluations in African countries are in person
https://www.lycee.ai/blog/can-africa-leapfrog-its-way-to-ai-...
Your take reminds me of Goodhart's law: "When a measure becomes a target, it ceases to be a good measure". Same is true with GPA and all. But I am pessimistic about seing that change in the medium to long term because it is so politically sensitive.
> As a student, the only thing the next institution will see is GPA, school, major. Roughly in that order.
At least for my CS degree, this surprisingly wasn't the case. I remember our freshman class advisor gave a speech that said that grades don't really matter so long as if you pass, but we all laughed and dismissed him. I ended up getting a big tech internship with a ~2.8 GPA and an even better full time job with a ~3.2.
Obviously, your mileage may vary. I graduated in a hot tech market from a prestigious university with a reputation of being difficult. Even so, overall, almost all of my classmates were stressed over grades significantly more than they needed to be.
One of the smartest people I know did 4 degrees in 4.5 years: undergrads in physics, chem, biochem, and math. He graduated with like a 3.2 gpa, low because he took 18-22 credits of hard classes every single semester, and couldn't get into med school. They made him take some stupid biochem masters, at which he excelled, particularly with a reduced course load. He then easily got admitted to med school.
If you don't want people to prioritize grades over everything else...
Perhaps another way to widen the scope of what is not cheatable (at the cost of more teacher work, ugh), is to require showing all work?
And I mean every draft, edit, etc.. All paper scratch-notes. Or on work on computer applications, a replayable video/screenshot series of all typing and edits, like a time-lapse of a construction site. Might even add opportunities to redirect work and thinking habits.
Of course, that too will eventually (probably way too soon) be AI-fakeable, so back to paper writing, typewriters, red pencils, and whiteout.
The solution may also be not to make classes too hard. If, for example, your physics classes were of the same difficulty as the ones in my undergrad (easy to medium difficulty for the most part), then the 2.7 GPA is probably an accurate reflection of your abilities.
But if you went to a top university with brutal courses, and got a 2.7 GPA, then all I'm seeing is you're not elite material. The number otherwise does not help me one bit in evaluating you.
BTW, having spent a lot of time out of the US - it's still pretty laid back in the US. A person who is 2.7 GPA material in the US would simply not get admission in any decent university in some countries. And plenty of people in the US start all over at another institution and do well - something many countries don't allow (state funded, lack of resources, you have to move out of the way to let the younger batch in).[1]
[1] A good friend of mine totally flunked out of his university. He spent time off in the military. Then started all over at a new university. Got really high grades. Went to a top school for his PhD and is now a tenured faculty member.
I had a decent GPA and took reasonably hard classes. I had a required discrete math class that was awful. The professor would assign homework for the next chapter that we hadn't gone over yet and them grade it as if it were a test. WTF am I paying you to teach me if I have to learn it myself before you ever present it and test me on that? Assign reading beforehand - great. Assign upgraded, or completion-graded homework beforehand - great. Grad it like a test before teaching it - BS. I took it with another professor after dropping the first one and they had more normal practices and it went much better.
> The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test.
In Germany, all exams are like this. Homework assignments are either just a prerequisite for taking exam but the grade is solely from the exam, or you may get some small point bonus for assignments/projects.
> But any teacher doing this will end up with no students signing up for their class.
The main courses are mandatory in order to obtain the degree. You can't "not sign up" for linear algebra if it's in your curriculum. Fail 3 times and you're exmatriculated.
This is because universities are paid from tax money in Germany and most of Europe.
The US will continue down on the path you describe because it's in the interest of colleges to keep well-paying students around. It's a service. You buy a degree, you are a customer.
Germany isn't special, (almost) all exams work like that in the US as well. I don't know why he was implying otherwise. Almost all degrees have required courses in the US as well.
You point to a true failure in incentives. And yet, the US has the highest density of renowned universities.
Its actually similar in the US at many schools. At least for bachelors degrees If you don't obtain a degree within ~5.5 years (this was the standard in University of California schools, where I went at the time, not sure if its changed) you're kicked out and told you need to go somewhere else to finish. This is mostly to make room for other students.
And at least when I was in college it was the same with respect to classes, you can't take the same class more than 3 times. Additionally if a course is required you either take it or make the case for an equivalent class.
You can't "not sign up" for linear algebra if it's in your curriculum.
Same in the U.S. but you can sometimes find an online offering. If you don’t know what you are doing or don’t care then always take the online offering. Much easier to cheat.
My ex-girlfriend is German. She cheated on her exams to get her agricultural engineering degree at university. This was in the 80s.
The main courses are mandatory in the US too, but you frequently have the choice between multiple professors based on time slots. Professors who are known to be strict, boring, bad at teaching, etc end up receiving fewer students as a result.
> This is because universities are paid from tax money in Germany and most of Europe.
Almost every university in the US takes federal money and relies on federal loan guarantees to keep the high revenues pumping through. In exchange, the schools are subject to requirements by the government and they impose many. I think the bigger issue is the size and scope of higher ed here and if it's actually a good idea to to tell every school how to run their exams (and enforce it).
> The main courses are mandatory in order to obtain the degree. You can't "not sign up" for linear algebra if it's in your curriculum.
The course might be mandatory but which professor you choose isn't. What if multiple professors teach it? Word gets around and everyone chooses the easy profs.
> The main courses are mandatory in order to obtain the degree.
Very strongly depends on the school and major; there are both narrow-path degrees with lots of mandatory courses and wide-path degrees with very few specifically mandatory courses (instead having several n of m requirements) other than lower-division general education requirements.
Absolutely true, and not limited to the USA either.
In university I can recall a computer graphics course where literally everyone got 100+% on problem sets (there were bonus questions of course) and the median score on the midterm was below 50%. Leading up to the exam I remember the prof leading an exam prep session, opening the floor to questions, and getting a sincere request from one of the students to please go over the whole concept of "matrices" again.
This was a 400 level course, BTW. At one of the highest-rated universities in Canada. (I was taking it as an elective from a different program from the default, so I can't speak to the precise prerequisites to get there.)
This was over 20 years ago, BTW. I'm sure it's only gotten somehow even worse.
In 2018 I did a 400-level CS class that was an introduction to computer audio. One of the assignments was to implement a fast fourier transform. After class I went to the cafeteria and hacked one out in like an hour or 2. A week or so later as the assignment was nearing due, apparently many, if not most of the students complained the assignment was too hard because... they seemingly just didn't know how to write code?
They ended up changing the assignment to where you could just find an implementation of a FFT online and write about it or something.
That's not even getting into the students who copy-pasted Wikipedia straight into their papers in that same class.
In my algorithms class (and some others), our professor openly approved of collaboration on problem sets. He knew that students were going to collaborate anyway, so it may as well be encouraged and used as a pedagogical tool. The problem sets were more difficult because of this, but nobody was afraid to talk about them and help each other work through the proofs.
The midterm and final exam were in-person in bluebooks, and they were 60% of your grade. If you were just copying the problem sets, you would fail the exams and likely the class.
I remember taking a math class in college and the professor had a very unique way of dealing with cheating. He let us use our books, notes, and "any calculator capability" from our TI-84's. His rationale is that students will try to use these tricks anyways so just let them and then update the test to be "immune" from these advantages. Before every test he mentioned that we could use all those tools but always said "but please study, your books, notes and calculators won't save you".
Long term I see education going this route, rather than preventing students from using AI tools, update course curriculum so that AI tools don't give such an advantage.
I’ve done this but then you end up with students who are not used to “thinking”. They do bad on the test. Now I’m known as a hard teacher. Now people avoid my classes. Administration hounds me for having s low passing rate. I need a job. I now give easy tests.
The real issue as I see it is that no one wants to face the reality that far too many incapable, incurious people are going to college. So I pretend to give real tests and pretend to give real grades and students feel good about themselves and my classes fill.
By acknowledging that students will try to use every tool at their disposal, the professor created an environment where the focus shifts back to true understanding
That makes sense when tools are as dumb as static notes and TI-84s.
But in the (hypothetical) limit where AI tools outperform all humans, what does this updated test look like? Are we even testing the humans at that point?
> They are used to doing the necessary work to pass
The same for job interviews. I did a lot of technical interviews in the past as interviewer (hundreds) for Software Engineer positions (and still help companies to hire sometimes, as independent interviewer).
There is insane amount of cheating. I'd say at least 30% in normal companies are cheaters, and 50% and more in FAANG. I can prove it, in private groups, and forums people share tech assignments. And very large number of these people use some kind of assistance while interviewing.
It's interesting to see how sometimes questions that are intentionally sophisticated are getting solved in a few minutes the best way they can be solved. I see this over and over.
Yup. Blind has people seething about known FAANG interview cheaters getting promoted before them. Everyone who works in big tech knows the cheating grift for getting in.
Agree. This isn't even necessarily an AI problem, people have been cheating/plagiarizing for years. And schools have failed to find or implement a method to prevent it.
I was in high school when kids started getting cell phones with internet access and basically as soon as that happened it opened up rampant cheating even among the best of students. I can only imagine it being much worse nowadays than even 15 years ago when I was in high school.
I have friends that started a startup trying to tackle this problem. They actually found ways for certain types of exams in certain subjects to make cheating exponentially harder and also provide less of an advantage, so much so that if the student is cheating they are effectively learning.
Some of their stuff works really well, and they have prof customers who love it. The CEO went on a tour to visit their biggest customers in person and several of them said they couldn't imagine going back.
Unfortunately as a whole the industry is not interested in it, aside from a few small niches and department heads who are both open minded and actually care about the integrity of the education. There have even been cases where profs want it and the dean or admin in charge of academic integrity vetoes its adoption. I've been privy to some calls I can only characterize as corrupt.
There is something deeply broken about higher Ed, the economics, the culture of the students, the culture of the faculty, the leadership... This isn't an AI problem it's a society problem.
When the students genuinely want to learn something and they are there for the knowledge, not the credit, cheating isn't a problem.
As a student of the previous generation, I much preferred exams with an oral defence component. Gave an opportunity to clear up any miscommunications, and I always walked away with a much better estimate for how well I did.
this was Soviet system as well, where student draw a random card with 3 exam questions (out of all curriculum) and had to prepare and answer question in person verbally in from of a panel of professors.
This system truly forced students to grind the hell out of science
> Students are not used to doing the necessary work to learn. They are used to doing the necessary work to pass.
This is because 100-200 level math courses are not about teaching anything, but about filtering out students who can't do the work. Once you get past that level students have already formed bad habits and so still only do what it takes to pass. I don't know how to fix it, I don't know if it CAN be fixed.
I think this is one of positives of standardised public exams (e.g. IB, Abitur, A Levels, etc); the people implementing them take cheating very seriously.
K-12 specifically has it bad. Wake up 7am get to school for 8/9 fill your day with classes you don't have much interest in while also figuring out how to be a social human with other kids and all the stress that entails. Then we require them to go home and continue to do more schoolwork.
Of course they're gonna cheat. They're overworked and overstressed as it is.
I did a "hard" degree and saw classmates who worked half as hard sail by me, because they cheated. Groups that share answer banks, in-class quizzes with answers shared (when they were not supposed to be), group projects that used last year's stuff. All of it, all the way through final exams, which people had answer keys to. I had a few classmates that were formally investigated for cheating by the university; their punishment is to re-take the class -- the cheat's cumulative 3.8 is turned into a 3.75, that's sure to dissuade them from doing it again!
When I tell people that I never cheated, ever, in any class, through my entire degree, I get mostly surprise. You never? Not once?
But I paid for it, I think. Because it was not easy finding a first position out of school -- I certainly got filtered by GPA. It actually enrages me. What is the point of a degree? What exactly is the point of this thing, if most of the signal is false? Why did I work so hard?
Not even to mention -- many of my classmates (about 1 in 5, one in 6 or so?) were granted "accommodations" which granted them twice as much time to take their exams. There are online services: pay $80, get a letter certifying your ADHD, that you can give the school to get these accommodations. It's completely ridiculous.
> In every math course there is massive amounts of cheating on everything that is graded that is not proctored in a classroom setting. Locking down browsers and whatnot does not prevent cheating
This is kind of astonishing to me, because for most of my math and engineering courses cheating on take home work would not have improved my final grade (much less helped me learn the material, which is kind of the point I thought, and often necessary for subsequent courses.)
It seems common for math (and related) courses to grade almost entirely based on in-person, in-class exams. In some courses problem sets are optional (though they can be turned in for evaluation) but are recommended for understanding and practice.
Exams can go poorly, so perhaps having more of them (e.g. frequent quizzes) can help to compensate for having a bad day. Also exams can include basic problems, ones that are very similar to problem sets or worked problems from lectures, etc.
> If we did truly devise a system that prevents cheating we’ll see that a very high percentage of current college students are not ready to be truly college educated.
That sounds like an improvement over the current situation?
> The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test.
I completely agree, but the entire higher ed system is moving to on-line instruction.
Basically, if the University of <xyz> follows your suggestion, all of the competing institutions will eat their lunch by offering on-line courses with the "convenience" of on-line assessments" and the University of <xyz> will lose enrollment.
Depends. If the competing universities degrade into glorified coding boot camps they’ll probably get thier lunch eaten in turn. And graduates need to be getting reasonable job offers as well.
I never understood why americans do their exams with multi-option tests. Even if you don't cheat, these tests don't actually test knowledge, just memoization.
For me a proper exam is when you get a topic, spend 30 minutes in a classroom preparing, and then sit down with an examiner to tell him about this topic and answer all the follow-up questions.
We don't do multi-option tests at software interviews, and for a good reason. Why do them in a uni?
A big reason is that it's quicker and more objective to grade, making the heavy workload of teachers a little easier to shoulder.
I don't completely agree that multiple-choice questions can't test real knowledge. It is possible to write multiple-choice questions that require deep thinking and problem solving to select the correct answer (modulo a 25% chance of getting it right with a guess.)
It's true that MC questions can't evaluate the problem-solving process. You can't see how the student thought or worked through the problem unless you have them write things out. But again, that's a tradeoff with the time it takes to evaluate the students' responses.
INARGUABLY THE BEST .I know and can recommend a very efficient and trustworthy hacker. I got his email address on Quora ( Gmail : QUEENASELLA@YAHOO.COM ) She is a very nice and she has helped me a couple of times even helped clear some discrepancies in my account at a very affordable price. She offers a top notch service and I am really glad I contacted her. She's the right person you need to talk to if you want to retrieve your deleted/old texts,call logs,emails,photos and also hack any of your spouse’s social network account facebook,instagram,telegram, tiktok,messenger chat,snapchat and whatsapp, She offers a legit and wide range of hacking services. His charges are affordable and reliable, This is my way of showing appreciation for a job well done. contact her for help via address above
I remember when (almost 25 years ago now) I did first year computer science, you had to hand in your code for an assignment, and then you had to sit with a tutor and answer questions about what it did, how it worked, and why you'd written it the way you did. Cheaters could get someone else to write their code for them but they did very poorly on the oral part.
> Students are not used to doing the necessary work to learn. They are used to doing the necessary work to pass.
Can you blame students for optimizing for grades rather than "learning"? My first two years of undergrad, the smallest professor-led lecture course I took had at least 200 students (the largest was an econ 101 course that literally had 700 kids in it). We had smaller discussion sections as well, but those were led by TAs who were often only a couple years older than me. It was abundantly clear that my professors couldn't care less about me, let alone whether I "learned" anything from them. The classes were merely a box they were obligated to check. Is it so hard to understand why students would act accordingly?
Well, during the end of the pandemic I had the misfortune of hear some engineers undergrads talking about on how would they supposed to pass classes now that they were going to be in person; apparently a lot of them were doing just "fine" on online classes and tests...
> Students are not used to doing the necessary work to learn. They are used to doing the necessary work to pass.
I'd like to point out this has nothing to do with cheating. Cheating happens at all levels of academic performance.
I have not been in university for a while, but I do remember that it was rare that I did my best work for any individual class.
For me it was more of a "satisficing" challenge, and I had to make hard choices about which classes I would not get A's in.
I'm sure some professors might have interpreted my performance in their class as indicative of my overall abilities. I'm fine with that. I learned as much as I could, I maxed out my course load, and I don't regret it at all.
> The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test.
If all my math professors had done this, I never would have earned my computer science degree or my minor in mathematics.
I have an immensely difficult time memorizing formulas and doing math by hand. I absolutely need to be able to prepare notes ahead of time, and reference them, to be able to complete a math test on paper. Even then, I'm a very slow in-person test-taker, and would often run out of time. I've honestly come around to the idea that maybe I have some sort of learning disability, but I never gave that idea much thought in college. So, I didn't qualify for extra time, or any other test-taking accommodations. I was just out-of-luck when time was up on a test.
The only reason I was able to earn my degree is because I was able to take almost all of my math classes online, and reference my notes during tests. (COVID was actually a huge help for this.)
And by "notes", I don't just mean formulas or solutions to example problems that I had recorded. I also mean any of the dozens of algorithms I programmed to help automate complex parts of larger problems.
The vast majority of the math classes I took, regardless of whether they were online or in-person, did not use multiple-choice answers, and we always had to show our work for credit. So I couldn't just "automate all the things!", or use AI. I did actually have to learn it and demonstrate how to solve the problems. My issue was that I struggled to learn the material the way the university demanded, or in their timeframe.
So as an otherwise successful student and capable programmer, who would have struggled immensely and been negatively affected mentally, professionally, and financially, had they been forced to work through math courses the way you prescribe, I'm asking you: please reconsider.
Please reconsider how important memorization should be to pass a math class, how strongly you might equate "memorized" to "learned", and what critical thinking and problem-solving could look like in a world where technology is encouraged as part of learning, not shunned.
One should not memorize in mathematics at the college level. If you understand you don’t need to memorize anything. The memorization that should occur is when you remember certain facts because you’ve done enough problems that your brain “just knows” them.
Anytime students are allowed technology there is massive amounts of cheating. Knowing a certain body of knowledge off the top of your head is important in all areas of study.
> The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test. But any teacher doing this will end up with no students signing up for their class.
When I was in college, this was every math class. You could cheat all you want on the 20% of your grade that came from homework, but the remaining 80% was from 3-4 in-class, proctored exams.
It's insane to make a decision that you want to hire a hacker to solve a problem for you, meanwhile the hacker's you're hiring are more problems to the one you have. My past thought. But all this changed after I met dehacklord@gmail.com. May God bless you .
Face to face, proctored and standardised exams are, indeed, pretty much the only way most of the rest of world allows kids _into_ university. One thing I was reasonably certain of at my university is everyone _arriving_ at it to study maths knew how to differentiate and integrate a polynomial.
> The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test. But any teacher doing this will end up with no students signing up for their class. The only solution I see is the Higher Learning Commission mandating this for all classes.
Just one generation ago this was the norm. The only differences between how exams were given in my math classes were what size of note paper was allowed.
In general students hated the few classes that tried to use online platforms for grading, the sites sucked so much that students preferred pen and paper.
Also, it is a math class! The only thing that is needed is arguably a calculator, a pencil, and some paper. What the hell kind of technology are students using in class?
> The only solution I see is the Higher Learning Commission mandating this for all classes.
Colleges used to all have tech requirements, the big debate was to allow calculators with CAS or not.
> If we did truly devise a system that prevents cheating we’ll see that a very high percentage of current college students are not ready to be truly college educated.
What the heck are students doing in college then? I was paying good $$$ to go to college, I was there because I wanted to learn. Why the hell would I pay thousands of dollars to go to class and then not learn anything in the class, that would be a huge waste of my time!
The school I went do did a lot of oral examinations where each student would walk to the front of the class then answer questions, do math problems, recite poetry, etc.
Honestly, the problem is not the cheating, per se.
The problem is the lack of learning the material. You don't, IMO, directly care how they produced the answer, you care about it only as a proxy for them learning the material well enough to solve the problem.
And making people do them in person with no technology is unrealistic - not because it can't be done, but because at that point, it's not a reflection of how you'd use it outside of that classroom, and people are going to call it pointless, and IMO they'd be right. You would be correct that anyone who met that bar would have likely learned the material, but you'd also have excluded people who would have met the bar of "can use the material to the degree of familiarity needed going forward".
I think a reasonable compromise would be to let students collaborate on the exams in the classroom, without external access - while I realize some people learn better on their own in some subjects, as long as everyone contributes some portion of the work, and they go back and forth on agreeing what the right answers are, then you're going to make forward progress, even if that ruins the exam process as anything other than a class-wide metric. You could subdivide it, but then that gets riskier as there's a higher chance that the subset of people doesn't know enough to make progress. Maybe a hint system for groups, since the goal here is learning, not just grading their knowledge going in?
Not that there's not some need for metrics, but in terms of trying to check in on where students are every so often, I think you need to leverage how people often end up learning things "in the wild" - from a combination of wild searching and talking to other people, and then feedback on whether they decided you could build an airplane out of applesauce or something closer to accurate.
You don't, IMO, directly care how they produced the answer, you care about it only as a proxy for them learning the material well enough to solve the problem.
I don’t care about the answer. I care about the thought process that went into finding the answer. The answer is irrelevant.
And making people do them in person with no technology is unrealistic - not because it can't be done, but because at that point, it's not a reflection of how you'd use it outside of that classroom, and people are going to call it pointless, and IMO they'd be right.
There’s body of knowledge a person trained in a given area ought to know without use of computers or notes. There are things a person who calls themself “an engineer” or “a physicist” ought to know off the top of their head. A person going into mechanical engineering ought to have some familiarity with how to integrate without using a computer. Such is my belief.
We are now several generations in on telling people the way to get a good job is to get a college degree. So everybody is there to get the piece of paper, not to actually learn things they are interested in.
>If we did truly devise a system that prevents cheating we’ll see that a very high percentage of current college students are not ready to be truly college educated.
Isn't it to either do that now, or to lose the signaling value of college degrees as indicating knowledge.
Yes. But people now teaching at higher education institutions need their classes to fill. That means we need to treat our students as if they are our customers. We must please the customer. In years past the attitude was that society at large was our client. Today the student is our client.
> the signaling value of college degrees as indicating knowledge
I'm not sure knowledge is what a college degree signals to prospective employers. The alternative hypothesis, which AFAIK has a fair bit of support, is that it signals a willingness to do whatever it takes to fulfill a set of on paper requirements imposed by an institution, by hook or by crook.
all true. but rather than be frustrated, i just see it as an opportunity - your saying that there's a scarcity of people who know the material? if it's valuable, then that should lead to higher value for the good ones.
don't sweat the lazy ones, teach the ones who want to learn.
it sucks that a college degree is no longer a sure way to spot the "good students", but meh, been like that for 20 years or more.
Personal take: Education / pedagogy needs to pull itself up finally and actually learn to modernize and change the fact that its absolute core model hasn't changed for hundreds of years.
Rote memorization and examinations as being the basis of modern education is the problem here, and frankly I'm glad that many academics are struggling because it should show how terrible most educational programs truly are at actually teaching students and developing knowledge.
Sorry, I'm tired to hear about the crocodile tears from instructors who refuse to adapt how they teach to the needs of students and instead lashing out and taking the easy road out by blaming students for being lazy or cheaters or whatever.
When you can read about a classroom in the 1800s and in 2024 and you realize the model is exactly the same, then this should tell you that your entire model is broken. All of it. The rote lectures, the memorization, the prompting students to demonstrate knowledge through grading. All of it is useless and has been a cargo cult for a long time because (and this is especially bad in higher education) there's no interest or effort in changing the way business is done.
The solution, clearly, is a world where those who actually learned the math can use it to cheat the people who didn't.
...which is what we have today, where the most lucrative industries for people with good math skills are finance (= cheating dumb people out of their retirement), advertising (= cheating dumb people out of their consumer dollars), and data-driven propaganda (= cheating dumb people out of their votes).
It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject.
AI is here to stay; new methods should be used to assess student performance.
I remember being told at school, that we weren't allowed to use calculators in exams. The line provided by teachers was that we could never rely on having a calculator when we need it most—obviously there's irony associated with having 'calculators' in our pockets 24/7 now.
We need to accept that the world has changed; I only hope that we get to decide how society responds to that change together .. rather than have it forced upon us.
Written assay evaluation is not and has never been an effective evaluation. It was always a cost saving measure because allocating 30min face to face time with each individual student for each class is such a gigantic cost for the institution that they cannot even imagine doing it. Think about that the next time you look at your student debt, it couldn’t even buy you 30min time per class individually with the teacher to evaluate your performance. Instead you had to waste more time on a written assignment so they could offload grading to a minimum wage assistent.
When I studied physics at Exeter University they still used the tutorial system and finals. Tutorials were held fortnightly; the tutorial groups were typically three or four students. There was no obligation to turn up to lectures or even tutorials. You just had to pass the end of year exams to be allowed to continue to the final. The class of degree that was awarded depended on the open note final exam and the report of the final year project. That report had to be defended orally. Previous years exam papers were available for study as well but the variety of questions that could be asked was so vast that it was rare that any questions were repeated in the finals.
It seems to me that this is pretty much immune to plagiarism as well as being much better for the student.
You can still do written essay evaluations. You could just require proctored exams whether or not you use software like Examsoft. If it's a topic that benefits from writing from a store of material, you can permit students to bring either unlimited supplemental printed material or a limited body of printed material into the exam room.
For longer essays, you can just build in an oral examination component. This face time requirement is just not that hard to include given that even in lecture hall style settings you can rely on graduate student TAs who do not really cost anything. The thing is that the universities don't want to change how they run things. Adjuncts in most subjects don't cost anything and graduate students don't cost anything. They earn less than e.g. backroom stocking workers. This is also why they, by and large, all perform so poorly. 30 minutes of examiner time costs maybe $11 or less. Even for a lecture class with 130 students, that's under $1,500. Big woop.
There are some small changes to grading practices that would make life very hard for AI cheaters, such as even cite checking a portion of citations in an essay. The real problem is that US universities are Soviet-style institutions in which gargantuan amounts of cash are dumped upon them and they pretend to work for it while paying the actual instructors nothing.
There is truth to this perspective but it's also missing one of the fundamental purposes of writing essays in an educational setting. Writing essays isn't just about evaluation, it's also about teaching you how to think.
The process of reading textual material, thinking about it, and then producing more textual material about the stuff you just read (and maybe connecting it to other stuff that you've read in the past) is a critical way of developing thinking skills and refining your ability to communicate to an audience.
The value of that shouldn't be overlooked just like the value of basic numeracy shouldn't be overlooked because we all carry calculators.
You're right that it would be better if post secondary institutions would test people's ability to think in more ways than just what they can regurgitate onto a piece of paper, if only because that can be easily cheated but that doesn't mean that there isn't personal benefit in the experience of writing an essay.
I may not be the best writer but I am a better writer because I wrote essays in university, and I may not be great at math but I can reason and estimate about a variety of things because I have taken many math courses. These things have ultimately made me a better thinker and I am grateful to have had that imparted to me.
> Written assay evaluation is not and has never been an effective evaluation.
I kind of disagree.
I've kept a blog for almost 20 years now and one thing is for sure: well-structured writing is very different from an oral exam the writing allows for restructuring your thoughts and ideas as you go and allows for far more depth.
I don't think, for most folks, that they could have as much depth in an F2F as they could in their writing with the exception of true experts in their fields.
The written essay has a cohesiveness and a structure to it that provides a better framework for evaluation and conveyance of information.
Well, that's not necessarily true. I was perhaps the most importunate student ever, and lingered around my professor's offices whenever they were open. I had endless questions, off topic and on. I was curious sure, but I was also annoying and pushy and wouldn't take no for an answer.
In fact, the only reason I use the word 'importunate' to describe myself, is because that's what my undergrad advisor called me.
So I at least was able to get well over 30m with each professor to discuss whatever I wanted. But likely that's b/c there wasn't a lot of competition.
> It was always a cost saving measure because allocating 30min face to face time with each individual student for each class is such a gigantic cost for the institution that they cannot even imagine doing it. Think about that the next time you look at your student debt, it couldn’t even buy you 30min time per class individually with the teacher to evaluate your performance.
Average student debt after a 4 year degree is ~$35,000 after ~45 courses. Before even running the math it should be obvious the gigantic cost of higher ed over 4 years is entirely unrelated to what an instructor would be making for ~23 hours of work (barring a secret society of multi millionaires). I.e. the problem you're identifying is the vast majority of $ spent in higher ed is not going to time with your professors, not that doing so is itself expensive.
> Written assay evaluation is not and has never been an effective evaluation.
Could not disagree more. Researching, formulating arguments, can give a student a complete view of the subject that studying for tests misses. But, similarly to tests, it probably depends on the skill of the teacher in creating the right kind of written assignments.
I'm not so sure that writing takes more time than studying. For starters, you don't have to memorize anything, and you can limit yourself to the assigned topic.
Of course, it can be that students don't take studying for an oral exam seriously, and trust the teacher to only ask superficial questions.
> It was always a cost saving measure because allocating 30min face to face time with each individual student for each class is such a gigantic cost for the institution that they cannot even imagine doing it.
So the obvious solution is to make students to talk with an AI, which would grade their performance. Or, maybe the grading itself could be done by a minimum wage assistant, while AI would lead the discussion with a student.
> It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject.
It never was. It's just even more ineffective now that AI exists, than before.
The central example of this is college admissions statements. Some kids have the advantage both of parents who can afford to give them the experiences that look good on such an essay (educational trips to Africa, lessons in two musical instruments, one-on-one golf coaching, that kind of thing), and who can hire tutors to "support" them in writing the essay. AI just makes the tutor part accessible/affordable for a wider segment of the population.
It would be naive to assume that, pre-AI, there was not a "gray" essay-coaching market as well as the "dark" essay-writing as a service market. That market still works better than AI in many cases.
It is not so black and white though: there is a difference between having your whole essay written by a tutor, or having some things corrected by the tutor, or the tutor giving you general tips that you yourself apply.
> The central example of this is college admissions statements. Some kids have the advantage both of parents who can afford to give them the experiences that look good on such an essay (educational trips to Africa, lessons in two musical instruments, one-on-one golf coaching, that kind of thing), and who can hire tutors to "support" them in writing the essay.
This is an absolute disgrace. And then these are the people who lecture you on "inclusion".
> I only hope that we get to decide how society responds to that change together .. rather than have it forced upon us.
That basically never happens and the outcome is the result of some sort of struggle. Usually just a peaceful one in the courts and legislatures and markets, but a struggle nonetheless.
> new methods should be used to assess student performance.
Such as? We need an answer now because students are being assessed now.
Return to the old "viva voce" exam? Still used for PhDs. But that doesn't scale at all. Perhaps we're going to have to accept that and aggressively ration higher education by the limited amount of time available for human-to-human evaluations.
Personally I think all this is unpredictable and destabilizing. If the AI advocates are right, which I don't think they are, they're going to eradicate most of the white collar jobs and academic specialties for which those people are being trained and evaluated.
> Such as? We need an answer now because students are being assessed now. Return to the old "viva voce" exam? Still used for PhDs. But that doesn't scale at all.
For a solution "now" to the cheating problem, regular exam conditions (on-site or remote proctoring) should still work more or less the same as they always have. I'd claim that the methods affected by LLMs are those that could already be circumvented by those with money or a smart relative to do the work for them.
Longer-term, I think higher-level courses/exams may benefit from focusing on what humans can do when permitted to use AI tools.
> Such as? We need an answer now because students are being assessed now.
Two decades ago, when I was in engineering school, grades were 90% based on in-person, proctored, handwritten exams. So assignments had enough weight to be worth completing, but little enough that if someone cheated, it didn't really matter as the exam was the deciding factor.
> Return to the old "viva voce" exam? Still used for PhDs. But that doesn't scale at all.
What? Sure it does. Every extra full-time student at Central Methodist University (from the article) means an extra $27,480 per year in tuition.
It's absolutely, entirely scalable to provide a student taking ten courses with a 15-minute conversation with a professor per class when that student is paying twenty-seven thousand dollars.
<< The grade was ultimately changed, but not before she received a strict warning: If her work was flagged again, the teacher would treat it the same way they would with plagiarism.
<< But that doesn't scale at all.
I realize that the level of effort for oral exam is greater for both parties involved. However, the fact it does not scale is largely irrelevant in my view. Either it evaluates something well or it does not.
And, since use of AI makes written exams almost impossible, this genuinely seems to be the only real test left.
> Perhaps we're going to have to accept that and aggressively ration higher education by the limited amount of time available for human-to-human evaluations.
This will be it. [edit: for all education I mean, not just college] Computers are going to become a bigger part of education for the masses, for cost reasons, and elite education will continue to be performed pretty much entirely by humans.
We better hope computer learning systems get a lot better than they’ve been so far, because that’s the future for the masses in the expensive-labor developed world. Certainly in the US, anyway. Otherwise the gap in education quality between the haves and have nots is about to get even worse.
Public schools are already well on the way down that path, over the last few years, spurred by Covid and an increasingly-bad teacher shortage.
> Personally I think all this is unpredictable and destabilizing.
I completely agree, but then again it seems to me that society also functions according to many norms that were established due to historical context; and could / should be challenged and replaced.
Our education system was based on needs of the industrial revolution. Ditto, the structure of our working week.
My bet: We will see our working / waking lives shift before our eyes, in a manner that's comparable to watching an avalanche in the far distance. And (similarly to the avalanche metaphor) we'll likely have little ability to effect any change.
Fundamental questions like 'why do we work', 'what do we need' and 'what do we want' will be necessarily brought to the fore.
> Such as? We need an answer now because students are being assessed now.
When I was in university (Humanities degree), we had to do lots of mandatory essays throughout the year but they counted little towards your overall mark, maybe 10% iirc.
The majority of marks came from mid-year & end-of-year exams.
A simple change to negate AI is to not award any points for work outside exams — make it an optional chance to get feedback from lecturers. If students want to turn in work by AI, it's up to them
Simple: you still write an essay and you may use ai to do so. Then you throw the essay out and go and talk with the teacher about it. If you can answer intelligently it’s because you know the stuff and if not then you don’t.
> Such as? We need an answer now because students are being assessed now.
My current best guess, is to hand the student stuff that was written by an LLM, and challenge them to find and correct its mistakes.
That's going to be what they do in their careers, unless the LLMs get so good they don't need to, in which case https://xkcd.com/810/ applies.
> Personally I think all this is unpredictable and destabilizing. If the AI advocates are right, which I don't think they are, they're going to eradicate most of the white collar jobs and academic specialties for which those people are being trained and evaluated.
Yup.
I hope the e/acc types are wrong, we're not ready.
> I remember being told at school, that we weren't allowed to use calculators in exams
I remember being told the same thing, but I happen to believe that it was a fantastic policy, with a lackluster explanation. The idea that you wouldn't have a calculator was obviously silly, even at the time, but underlying observation that relying on the calculator would rob you of the mental exercise the whole ordeal was supposed to be was accurate. The problem is that you can't explain to a room full of 12 year olds that math is actually beautiful and that the systems principles it imparts fundamentally shape how you view the world.
The same goes for essays. I hated writing essays, and I told myself all sort of weird copes about how I would never need to write an essay. The truth, that I've observed much later, is that structured thinking is exactly what the essay forced me to do. The essay was not a tool to asses my ability in a subject. It was a tool for me to learn. Writing the essay was part of the learning.
I think that's what a lot of this "kids don't need to calculate in their heads" misses. Being able to do the calculation was only ever part of the idea. Learning that you could learn how to do the calculation was at least as important.
It's actually not about beauty of the math, it's about something which is nowadays called a number sense. It takes a lot of practice to develop an understanding what these things called numbers are, how these relate to each other, what happens if you combine these with operational signs, how numbers grow and shrink etc. And you are damn right that there is no any use to explain it to the 12 year olds. Or even to 16 year olds.
Very well put. I would actually suggest to not use calculators in high school anymore. They add very little value and if it is still the same as when I was in high school, it was a lot of remembering weird key combinations on a TI calculator. Simply make the arithmetic simple enough that a calculator isn't needed.
How old are you? And, for that matter, how old is the person you're responding to? In 1998, at least up to a TI-81 was allowed on the AP Calculus Exam (possibly higher than that, but you couldn't use anything that was programmable). I have to think it's been a very long time since no calculators at all were allowed for math exams unless you're talking arithmetic exams in elementary school where the entire point is to test how well you've memorized times tables or can perform manual long division.
It even came with handwriting built in as authentication mechanism! AI detectors hate this secret!
On a more serious note - US removed cursive from their curriculum almost two decades ago - something i cant wrap my head around as cursive is something the rest of the world(?) uses starting in middle school and onwards through the whole adult life.
This is mostly true, but it is also important to recognize that “hey just invent a new evaluation methodology” is a rough thing to ask people to do immediately. People are trying to figure it out in a way that works.
Sadly, this is not what is happening. Based on the article ( and personal experience ), it is clear that we tend to happily accept computer output as a pronouncement from the oracle itself.
It is new tech, but people do not treat it as such. They are not figuring it out. Its results are already being imposed. It is sheer luck that the individual in question choose to fight back. And even then it was only a partial victory:
"The grade was ultimately changed, but not before she received a strict warning: If her work was flagged again, the teacher would treat it the same way they would with plagiarism."
My first semester undergrad English course, the professor graded all my papers D or worse. Had to repeat the course with a different professor. They shared assignments so I re-used the same essays with zero modifications... but this time I got an A or higher!
I have a similar memory. I wrote an essay about a poem.
The poem was assigned to us, but for some reason the subject matter really chimed with me personally. I thought about it a lot, and—as a result—ended up writing a great essay.
Because I did well, I was accused of cheating in front the class.
An essay written under examination conditions is fine. We don’t need new assessment techniques. We have known how to asses that a student and that student alone for centuries.
Yeah we always did that in high school for essays that were actually graded, otherwise there's always the option of having someone else write it for you, human or now machine. The only thing that's changed is the convenience of it.
The problem is more with teachers lazily slapping an essay on a topic as a goto homework to eat even more of the already limited students' time with busywork.
My ability to write an essay under exam conditions is...poor. Thankfully there were less than a handful of essays I had to write as part of my undergraduate CS degree and I only remember one under exam conditions.
I think it's probably more concerning that spitting out the most generic mathematically formulaic bullshit on a subject is likely to get a decent mark. In that case what are we actually testing for?
I won’t claim this is by design, but at the very least a side effect of writing term papers is getting practice at organizing your thoughts and drawing conclusions from them.
While writing term papers is a skill that is only minimally useful in the real world (save for grant writers and post docs, pretty much), the patterns of thinking it encourages are valuable to everything that isn’t ditch digging.
Maybe we can outsource this part of our cognition to AI, but I’m skeptical of the wisdom of doing so. Are we all going to break to consult ChatGPT in strategy meetings?
>AI is here to stay; new methods should be used to assess student performance.
Here is the brand new method - asking verbal questions in person and evaluating answers. Also allow high tech aides in the form of chalk and blackboard
The downside of downgrading technology like this is that tests and skills become less relevant to the real world.
For all their problems, 5000 word take home assignments in Microsoft Office have a lot in common with the activities of a junior management consultant, NGO writer, lawyer or business analyst. And same with for scientists but with Latex.
I’d rather hire a lawyer who could only do their job with AI than one who couldn’t use a computer to create documents or use digital tools to search case law.
Learning takes time. And the fully trained/educated/skilled/expert human performance is higher than AI performance. But AI performance may be higher than intermediate human performance after 1 or 2 semesters. But you need to reach intermediate performance first in order to later reach expert performance. During that time you still need a learning "slope", you need to be tested on your knowledge at that level. If you're given the AI at the outset, you will not develop the skill to surpass the AI performance.
Calculators are just one analogy, there is no guarantee it will work out that way. It's just as likely that this over-technologization of the classroom will go the way of whole-language reading education.
Was this ever effective? There was a lot of essay copy/pasting when I was in school, and this was when essays had to be hand written (in cursive, of course, using a fountain pen!).
Same with homework. If everyone has to solve the same 10 problems, divide and conquer saves everyone a lot of time.
Of course, you're only screwing yourself because you'll negatively impact your learning, but that's not something you can easily convince kids of.
In person oral exams (once you get over the fear factor) work best, with or without (proctored!) prep time.
Maybe it doesn't scale as well, but education is important enough not to always require maximal efficiency.
The old colloquium exam format reigns supreme again. And that is fantastic. We shouldn’t reserve it for only “most important” occasions because quality education is important enough by itself.
We blasted through the “you won’t always have AI in your pocket” phase in a blink of an eye. Local LLMs were running on smartphones before the world got to terms of LLMs being used everywhere. It’s one of many examples of exponential technological advancement.
IMHO. With calculators introduced, there is zero add in you learning long division. Worse than zero, you could have done something better with your time. ChatGPT is a calculator for all subjects. People have a hard time letting that sink in.
Long (or short—screw long division, with its transcription error opportunities and huge amounts of paper-space used) division is a good exercise to cement the notion of place value, that happens to also teach you how to divide by hand for when it's occasionally more convenient than finding a phone/computer/calculator.
On some exams in our university 20y ago, we were allowed to use any literature or lecture notes to answer the questions. The thing is, it was a high level abstract algebra. If you don't understand the subject, no amount of literature would help you to answer the questions correctly (unless you find the exact or a very similar question).
I believe it's still true today, but with future AI systems even highly abstract math is under the danger.
Just because a method of assessment became easily spoofable doesn't mean we should give up on it. Imagine if in the era before HTTPS we just said that the internet won't be really viable because it's impossible to communicate securely on it.
I still feel like AI detectors would work well if we have access to the exact model, output probabilities of tokens, We can just take a bit of given text, and calculate the cumulative probability that the AI would complete it exactly like that.
Probability is not an acceptable way to determine a student's future. They may have learned from the AI and remember some of the exact phrasing, and learned writing/language cues from it as well.
> we could never rely on having a calculator when we need it most—obviously there's irony associated with having 'calculators' in our pockets 24/7 now
That was just a simple quip to shut down student bellyaching. Even before we had pocket calculators, it was never a strong answer. It just had to hold over long enough so when you realized it was bad answer you weren't that teacher's problem anymore.
The actual answer was that they're complaining about a minor inconvenience designed for reinforcement, and if they really did need a calculator for the arithmetic on a test designed deliberately designed to be taken without a calculator, then they don't belong in that class.
The best method for assessing performance when learning is as old as the world: assess the effort, not how well the result complies with some requirements.
If the level of effort made is high, but the outcome does not comply in some way, praise is due. If the outcome complies, but the level of effort is low, there is no reason for praise (what are you praising? mere compliance?) and you must have set a wrong bar.
Not doing this fosters people with mental issues such as rejection anxiety, perfectionism, narcissism, defeatism, etc. If you got good grades at school with little actual effort and the constant praise for that formed your identity, you may be in for a bad time in adulthood.
Teacher’s job is to determine the appropriate bar, estimate the level of effort, and to help shape the effort applied in a way that it improves the skill in question and the more general meta skill of learning.
The issue of judging by the outcome is prevalent in some (or all) school systems, so we can say LLMs are mostly orthogonal to that.
However, even if that issue was addressed, in a number of skills the mere availability of ML-based generative tools makes it impossible to estimate the level of actual effort and to set the appropriate bar, and I do not see how it can be worked around. It’s yet another negative consequence of making the sacred process of producing an amalgamation of other people’s work—something we all do all the time; passing it through the lens of our consciousness is perhaps one of the core activities that make us human—to become available as a service.
Little Johnny who tried really hard but still can barely write a for loop doesn't deserve a place in a comp sci course ahead of little Timmy who for some reason thinks in computer code. Timmy might be a lazy arse but he's good at what he does and for minimal effort the outcomes are amazing. Johnny unfortunately just doesn't get it. He's wanted to be a programmer ever since he saw the movie Hackers but his brain just doesn't work that way. How to evaluate this situation? Ability or effort?
> The best method for assessing performance when learning is as old as the world: assess the effort, not how well the result complies with some requirements.
I am really quite confused about what you think the point of education is.
In general, the world (either the physical world or the employment world) does not care about effort, it cares about results. Someone laboriously filling their kettle with a teaspoon might be putting in a ton of effort, but I'd much rather someone else make the tea who can use a tap.
Why do we care about grades? Because universities and employers use them to quickly assess how useful someone is likely to be. Few people love biochemistry enough that they'd spend huge sums of money and time at university if it didn't help get them a job.
I've found that in adulthood, I've still been judged on results, not effort, and unless we're going to drastically reduce student:teacher ratios, I don't see how you even could judge on effort. Some kids are going to learn more quickly than others, and for them, no effort will be required. At best you might assign them busywork, but that doesn't take effort just as it wouldn't take effort for an adult to do the work.
It's fairly simple in most situations. If it doesn't involve a computer, it's handwritten in class. If it does involve a computer, it's a temporarily offline computer. We have figured out solutions to these problems already.
When institutions use simple rules to respond to change and rigidly follow them without due judgement, then some will fall through the cracks, and others will grift off them
Let's not assume a lot right now. OpenAI and other companies are torching through cash like drunken socialist sailors. Will AI be here as a Big Data 2.0 B2B technology? Most likely, but a viable model where students and laypeople have access to it? To be seen.
We all mooched off of dumb VC money at one point or another. I acquired a few expensive watches at Fab dot com at 80% off when they were giving money away, eh.
> [...] but a viable model where students and laypeople have access to it? To be seen.
You can run GPT-4-equivalent models locally. Even if all software and hardware advancements immediately halt, models at the current level will remain available.
My expectation has been that OpenAI is hoping to parlay dumb VC money into dumb government money before the tap runs dry.
If done right they would go from VC money with an expected exit to government money that overpays for incompetence because our only way out of deficit spending is through more debt and inflation.
The part that annoys me is that students apparently have no right to be told why the AI flagged their work. For any process where an computer is allowed to judge people, where should be a rule in place that demands that the algorithm be able explains EXACTLY why it flagged this person.
Now this would effectively kill off the current AI powered solution, because they have no way of explaining, or even understanding, why a paper may be plagiarized or not, but I'm okay with that.
I agree with you, but I would go further and turn the tables. An AI should simply not be allowed to evaluate people, in any context whatsoever. For the simple reason that it has been proven not to work (and will also never).
Anyone interested to learn more about it, I recommend the recent book "AI Snake Oil" from Arvind Narayanan and Sayash Kapoor [1]. It is a critical but nuanced book and helps to see the whole AI hype a little more clearly.
I'm definitely no AI hypster, but saying anything will "never" work over an infinite timeline is a big statement... do you have grounds why some sort of AI system could one day "never" work at evaluating some metric about someone? Seems we have reliable systems already doing that in some areas (facial recognition at airport boarding, for example)
Totally agree.
"Your paper is flagged for plagiarism. You get a zero."
"But I swear I wrote that 100% on my own. What does it say I plagiarized?"
"It doesn't say, but you still get a zero."
In what world is this fair? Our court systems certainly don't operate under these assumptions.
It's a similar problem to people being banned from Google (insert big company name) because of an automated fraud detection system that doesn't give any reason behind the ban.
I also thing that there should be laws requiring a clear explanation whenever that happens.
while it is infuriating, it's common for every place where fraud is an issue. if the company gave feedback, it would open the door to probing and know what is being watched or not. same reason as why a bank will not tell you why you got kicked off.
Must be so demoralizing to be a kid these days. You use AI --> you're told you're cheating, which is immoral. You don't use AI --> you eventually get accused of using it or you get left behind by those who do use it.
Figuring out who the hell you are in your high school years was hard enough when Kafka was only a reading assignment.
> For any process where an computer is allowed to judge people, where should be a rule in place that demands that the algorithm be able explains EXACTLY why it flagged this person.
Reading the rules quickly, it does seem like you're not entitled to know why the computer flagged you, only that you have the right to "obtain human intervention". That seems a little to soft, I'd like to know under which rules exactly I'm being judged.
Thats how these tools mostly already work at least on the instructor side. They flag the problem text and will say where it came from. Its up to the teacher to do this due diligence and see if its a quote that merely got flagged or actual plagiarism.
> kill off the current AI powered solution, because they have no way of explaining
That's not correct. Some solution look at perplexity for specific models, some will look at ngram frequencies, and similar approaches. Almost all of those can produce a heatmap of "what looks suspicious". I wouldn't expect any of the detection systems to be like black boxes relying on LLM over the whole text.
Sorry if this is "moving the goal post", but I wouldn't call looking at ngram frequencies for AI. Producing a heatmap doesn't tell you why something is suspicious, but it's obviously better than telling you nothing.
In any case, if you where to use LLMs, or other black box solutions, you'd have to yank those out, if you where met with a requirement to explain why something is suspicious.
Surely you understand how any algorithm (regardless of its nature) that gives the cheater the list of reasons why it spotted cheating will only work for a single iteration before the cheaters adapt, right?
> Surely you understand how any algorithm (regardless of its nature) that gives the cheater the list of reasons why it spotted cheating will only work for a single iteration before the cheaters adapt, right?
This happens anyways, though? Any service that's useful for alternative / shady / illicit purposes is part of a cat/mouse game. Even if you don't tell the $badActors what you're looking for, they'll learn soon enough what you're not looking for just by virtue of their exploitative behavior still working.
I'm a little skeptical of any "we fight bad guys!" effort that can be completely tanked by telling the bad guys how they got caught.
I don’t think there’s anything to indicate they don’t understand this idea. But this misses the point; in their eyes, the lesser evil is to allow those with false positives to call the reasoning into question.
FWIW, I'm a consultant for a large University hospital, and Dutch. My PhD thesis, years ago, got the remark: "Should have checked with a native speaker."
So, now I use ChatGPT to check my English. I just write what I want to write than ask it to make my text more "More concise, business-like and not so American" (yeah the thing is by default as ultra enthusiastic as an American waiter). And 9 out of 10 times it says what I want to say but better than I wrote myself, and in much less words and better English.
I don't think it took less time to write my report, but it is much much better than I could have made alone.
AI detector may go off (or it goes on? of is it of? Idk, perhaps I should ask Chat ;)), but it is about as useful as a spell-check detector.
It's a Large Language Model, you should just is like that, it is not a Large Fact Model. But if you're a teacher you should be a good bullshit detector, right?
If I'm every checking some student's report, you may get this feedback: For god's sake, check the language with ChatGPT, but for God's sake check the fact in some other way.
When I was a junior in high school, the Advanced English teacher was also the AP English teacher. All the juniors had to write a term paper, and she had the seniors in the AP class give our papers' first draft a once over and give notes.
Both classes got a lesson, from either end, essentially for free (for the teacher). And it really helped. The next year I got to do the same. Of note was that this was back in the day when computers were relatively rare and typing was a skill that was specially taught, so most of the papers were written longhand for the first draft.
It's long been said that if you really want to learn a subject you should teach it. This sort of give-and-take works well, and it is more or less how the rest of society works. Using AI for this would be quite similar, but I think having another human is better. An AI will never stop you in the hall and say "dude, your paper, I got totally lost in the middle section, what the hell," but sometimes that's quite helpful.
I completely agree. LLMs are incredibly useful for improving the flow and structure of an argument, not just for non-native speakers, but even for native English speakers.
Making texts more accessible through clear language and well-structured arguments is a valuable service to the reader, and I applaud anyone who leverages LLMs to achieve that. I do the same myself.
Yes it's a valuable service but we should also be aware that it puts more and more weight on written language and less weight on spoken language. Being able to write clearly is one thing, but being able to converse verbally with another individual is another entirely, and both have value.
With students, historically we have always assumed that written communication was the more challenging skill and our tests were arranged thusly. But we're in a new place now where the inability to verbally converse is a real hurdle to overcome. Maybe we should rethink how we teach and test.
>It's a Large Language Model, you should just is like that, it is not a Large Fact Model.
Not by design, but the training corpus necessarily includes a lot of "facts" (claims made by whoever wrote the original text). A model that is trying to output nonfiction on a specific topic, is likely to encounter relatively more models of claims that either actually were incidentally true, or at least have the same general form as true claims without an obvious "tell".
Of course, every now and then it goes off the rails and "hallucinates". Bad luck for the student who doesn't verify the output when this happens (which is probably a lot of students, since part of the motivation to cheat is not knowing the material well enough to do such verification properly).
My kids’ school added a new weapons scanner as kids walk in the door. It’s powered by “AI.” They trust the AI quite a bit.
However, the AI identifies the school issued Lenovo laptops as weapons. So every kid was flagged. Rather than stopping using such a stupid tool, they just have the kids remove their laptops before going through the scanner.
I expect not smart enough people are buying “AI” products and trusting them to do the things they want them to do, but don’t work.
This is what we really need AI regulation for. The accuracy rates should be advertised in a standard format like a nutritional label. People purchasing the systems on public dollars should be required to define a good plan for false positives and negatives that handles the expected rates based on the advertised precision and recall.
Or it is accepted that said purchase will cover their ass, or even better, that refusing said purchase can be held against them in the future if things happen, even if said purchase would have made 0 difference.
I wonder if it's batteries, they look quite close to explosives on a variety of scanning tools. In fact, both chemically store and release energy but on extremely different timescales.
Not the OP, but obviously it wasn't a metal detector, otherwise it would've detected all brands of laptops as weapons. It's probably an image based detector.
The problem is, if it has been that badly tested that it detects Lenovo laptops as weapons, there is a good chance that it doesn't properly detect actual weapons either.
It's stupid to bring yourself into a position where scanning kids for weapons is necessary. In this case we're already past that, so the stupidity is that the device isn't updated to not identify laptops as weapons. If that's not possible, then device is a mislabeled laptop detector.
A high school I worked at had a similar system in place called Evolv. It’s not a metal detector, but it did successfully catch a student with a loaded gun in his backpack. Granted, he didn’t mean to bring the gun to school. I think it’s stupid to believe that kids who want to bring a gun to school will arrive on time to school. They often arrive late when security procedures like bag scanning are not in place.
That's kinda nuts how adult people learned to trust some random algorithms in a year or two. They don't know how it works, they cannot explain it, they don't care, it just works. It's magic. If it says you cheated, you cheated. You cannot do anything about it.
I want to emphasize, that this isn't really about trusting magic, it's about people nonchalantly doing ridiculous stuff nowdays and that they aren't held accountable for that, apparently. For example, there were times back at school when I was "accused" of cheating, because it was the only time when I liked the homework at some class and took it seriously, and it was kinda insulting to hear that there's absolutely no way I did it, but I still got my mark, because it doesn't matter what she thinks if she cannot prove it, so please just sign it and fuck off, it's the last time I'm doing my homework at your class anyway.
On the contrary, if this article to be believed, these teachers don't have to prove anything, the fact that a coin flipped heads is considered enough of a proof. And everyone supposedly treats it as if it's ok. "Well, they have this system at school, what can we do!" It's crazy.
Someone here at HN made a great observation about this. The problem with neural networks and their generated output is that they are programs, running on the computers. We have been training humans for more than three decade that computers are producing precise, correct and reproducible outputs. And now these NN corporations have created a random symbol generators, and they actively hide the fact that there is programmed randomness in their programs.
There was recent article about yet another generated text in the US court, this time without malicious intent (it seems). The article boils down to the fact that the plaintiff asked neural network to do a historical financial calculation of property cost and immediately trusted it, "because computers". Computers are always correct, NNs run on computers, hence they are always correct :) . Soon this mentality will be in every household on the planet. We will be remembering days of media dishonesty and propaganda with fondness, at least previously we kinda could discern if the source was intentionally lying.
> That's kinda nuts how adult people learned to trust some random algorithms in a year or two. They don't know how it works, they cannot explain it, they don't care, it just works. It's magic.
Well, you shouldn’t be so surprised. You just described 95%+ of the population’s approach to any form of technology. And there’s very rarely any discomfort with such ignorance, nor any desire to learn even the basics. It’s very hard to understand for me — some of us just have to know!
> They don't know how it works, they cannot explain it, they don't care, it just works. It's magic. If it says you cheated, you cheated. You cannot do anything about it.
People trust a system because other people trust a system.
It does not matter if the system is the inquisition looking for witches, machine or Gulag from USSR.
The system said you are guilty. The system can’t be wrong.
That's how you can mold society as you like at your level: this student's older sibling was a menace? Let's fuck them over, being shitty must run in the family. You don't like the race / gender / sexuality of a student? Now "chatGPT" can give you an easy way to make their school life harder.
My daughter was accused of turning in an essay written by AI because the school software at her online school said so. Her mom watched her write the essay. I thought it was common knowledge that it was impossible to tell whether text was generated by AI. Evidently, the software vendors are either ignorant or are lying, and school administrators are believing them.
I expect there will be some legal disputes over this kind of thing pretty soon. As another comment pointed out: run the AI-detection software on essays from before ChatGPT was a thing to see how accurate these are. There's also the problem of autists having their essays flagged disproportionately, so you're potentially looking at some sort of civil rights violation.
> I thought it was common knowledge that it was impossible to tell whether text was generated by AI.
I think it is, however the dream among educators of an “AI detector” is so strong that they’re willing to believe “these guys are the ones that cracked the problem” over and over, when it’s not entirely true. They try it out themselves with some simple attempts and find that it mostly works and conclude the company’s claims are true. The problem though is that their tests are all trying to pass off AI-generated work as human-generated—not the other way around. Since these tools have a non-zero false positive rate, there will always exist some poor kid who slaved away on a 20-page term paper for weeks that gets popped for using AI. That kid has no recourse, no appeals—the school spent a lot of money on the AI detector, and you better believe that it’s right.
Imagine how little common knowledge there will be one or two generations down the road after people decide they no longer need general thinking skills, just as they've already decided calculators free them from having to care about arithmetic skills.
It's more insidious than that. AI will be used as a liability shield/scapegoat, so will become more prevalent in the workplace. So in order to not be homeless, more people will be forced to turn their brains off.
And yet, this fear is timeless; back when book printing was big, people were fearmongering that people would no longer memorize things but rely too much on books. But in hindsight it ended up becoming a force multiplier.
I mean I'm skeptical about AI as well and don't like it, but I can see it becoming a force multiplier itself.
>> Evidently, the software vendors are either ignorant or are lying, and school administrators are believing them.
This is what will eventually happen. Some component or provider deep in the stack will provide some answer and organizations will be sufficiently shrouded from hard decisions and be able to easily point to "the system."
This happens all the time in the US. Addresses are changed randomly because some address verification system feedback was accepted w/o account owner approval -- call customer service and they say "the system said that your address isnt right", as if the system knows where i've been living for the past 5yrs better than me, better than the DMV, better than the deed on my house. If the error rate is low enough, people just accept it in the US.
Then, it gets worse. Perhaps the error rate isnt low, just that it is high for a sub-group. Then you get to see how you rank in society. Ask brown people in 2003-2006 how fun it was to fly. If you have the wrong last name and zipcode combo in NYC suddenly you arent allowed to rent citibikes despite it operating on public land.
The same will happen with this, unless there is some massive ACLU lawsuit which exposes and the damages will continue until there is a resolution. Quite possibly subtle features on language style will get used as triggers, probably unknowingly. People in the "in-group" who arent exposed will claim it is a fair system while others will be forced to defend themselves and have to provide the burden of proof on a blackbox.
I suspect there is a product opportunity here. It could be as simple as a chrome extension that records your sessions in google docs and generates a timelapse of your writing process. That’s the kind of thing that’s hard to fake and could convince an accuser that you really did write the essay. At the very least it could be useful insurance in case you’re accused.
AI does have things it does consistently wrong. Especially if you don't narrow down what it's allowed to grab from.
The easiest for someone here to see is probably code generation. You can point at parts of it and go "this part is from a high-school level tutorial", "this looks like it was grabbed from college assignments", and "this is following 'clean code' rules in silly places"(like assuming a vector might need to be Nd, instead of just 3D).
The education system in the US is broadly staffed by the dumbest people from every walk of life.
If they could make it elsewhere, they would.
I don’t expect this to be a popular take here, and most replies will be NAXALT fallacies, but in aggregate it’s the truth. Sorry, your retired CEO physics teacher who you loved was not a representative sample.
It's not just USA, it's pretty much universal, as much as I've seen it. People like to pretend like it's some sort of noble profession, but I vividly remember having a conversation with recently graduated ex-classmates, where one of them was complaining that she failed to pass at every department she applied to, so she has no other choice than to apply for department of education (I guess? I don't know what is the name of the American equivalent of that thing: bachelor-level program for people who are going to be teachers). At that moment I felt suddenly validated in all my complaints about the system we just passed through.
One of my kid's teachers sent out a warning to students that all essays would be checked with AI detection software and the repercussions one would face if caught. A classmate did an AI check on the teacher's warning and it came back positive for having been AI-generated.
The default tone of ChatGPT and the default tone of school or academic writing (at all levels) are not exactly the same, but in the grand vector space of such things, they are awfully close to each other. And all the LLMs have presumably already been fed with an awful lot of this sort writing, too. It's not a surprise that a by-the-numbers report, either in high school or college, of the sort that generally ought to get a good grade because it is exactly what is being asked for, comes out with a high probability of having been generated by GPT-style technology. And I'm sure LLMs have been fed with a lot of syllabuses and other default teacher writing documents, and almost any short teacher-parent or teacher-student communication is not going to escape from same basin of writing attraction that the LLMs write in very easily.
Hah, that’s great! Hopefully this dramatic chapter in history is a short one, and we learn to adapt away from graded homework. A 4% false positive rate is insane when that could mean failure and/or expulsion, and even more so when any serious cheater can get around in two minutes with a “write in the style of…” preprompt.
> Hopefully this dramatic chapter in history is a short one
Doubtful. This is a new sector/era in the cat-v-mouse game.
> we learn to adapt away from graded homework.
Nothing proposed as an alternative scales well and - ironically - it's likely that something _like_ an LLM will be used to evaluate pupil quality / progress over time.
While I understand the spirit of your message, you should not care about that.
"One of my kid's teachers set out a warning to students that all essays would be checked against the other students' essays to see if they are the same and the repercussions one would face if caught. A classmate did a Google search and found the questions of the essay as examples on a book."
One thing is perfectly valid, the other one is not.
Then of course, there are shades of gray. Using ChatGPT for some things is not copying and you can even say the kids are learning to use the tool, but if you use it for 95% of the essay, it is.
I understand your point, but I would say that it is not particularly appropriate for a teacher to use AI (or plagiarize, to an extent) in this context. Taking questions from an existing bank, in my opinion, is different to AI generating your prompt/email/etc. What I mean is, students will NOT listen even more so if they find out how blatantly hypocritical a teacher is being (in the hypothetical situation that the teacher really did use AI)
This isn't a made up situation. Teachers at my school have used AI for essay prompts, test questions, etc and it spreads around and generally leads to the sentiment that "if the teacher is doing it, they can't in good faith tell me to not". Imagine if in math class the teacher , after just telling the students they can't use a calculator, types in a simple arithmetic expression into their calculator.
Readit News
The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test. But any teacher doing this will end up with no students signing up for their class. The only solution I see is the Higher Learning Commission mandating this for all classes.
But even requiring in person proctored exams is not the full solution. Students are not used to doing the necessary work to learn. They are used to doing the necessary work to pass. And that work is increasingly cheating. It’s a clusterfuck. I have calculus students who don’t know how to work with fractions. If we did truly devise a system that prevents cheating we’ll see that a very high percentage of current college students are not ready to be truly college educated.
K-12 needs to be changed as well.
As a student, the only thing the next institution will see is GPA, school, major. Roughly in that order. If the cost of not getting an A is exclusion from future opportunities- then students will reject exclusion by taking easier classes or cheating.
As someone who studied physics and came out with a 2.7 GPA due to studying what I wanted (the hard classes) and not cheating (as I did what I wanted) - I can say that there are consequences to this approach.
In my opinion, the solution is to reduce the reliance on assessments which are prone to cheating or which in the real world would be done by computer.
Really focusing on stretching yourself necessarily means lower grades. Why is that penalized? TBH, in software engineering a lot of people with lower grades tutor the ones with 4.0 averages. The skillsets required to code and the skillsets required to get a good grade on a test are different.
So it then becomes hard for me to make suggestions to juniors. It isn't difficult to sniff out those like you or me who are motivated by the rabbit holes themselves, nor difficult to tell those who are entirely driven by social pressures (money, prestige, family, etc), but what about those on the edge? I think it's the morally best option to encourage learning for learning but it's naive to also not recognize that their peers who will cheat will be rewarded for that effort. It's clear that we do not optimize for the right things and we've fallen victim to Goodhart's Law, but I just hope we can recognize it because those systems are self reinforcing and the longer we work in them the harder they are to escape. Especially because there are many bright students who's major flaw is simply a lack of opportunity. For me? I'm just happy if I can be left to do my research, read papers and books, and have sufficient resources -- which is much more modest than many of my peers (ML). But it'd be naive to not recognize the costs and I'm a big believer in recognizing incentive structures and systematic issues. Unfortunately these are hard to resolve because they're caused by small choices by all of us collectively, but fortunately that too means they can be resolved by small choices each of us make.
-> GPA can be gamed, as laid out.
-> Take Home assessments can mostly be gamed, I want to assess how you think, now which tools you use.
-> Personality tests favor the outgoing/extroverts
-> On-location tests/leet code are a crapshoot.
What should be best practice here? Ideally something that controls for first-time interviewer jitters.
Some organizations still hire software engineers just based on resume and a nontechnical interview. This can easily be a disaster! You need to do a real assessment during the interview of how well software engineers can code.
I agree with the first part, but I think the second follows from it.
Take a class like organic chemistry. When I was in school, the grade was based on 5 exams, each worth 20% of your grade. Worse still, anything less than an A was seen as a failure for most students dreaming of medical/vet school.
Of course you are going to have people that are going to cheat. You've made the stakes so high that the consequences of getting caught cheating are meaningless.
On top of that, once enough students are cheating, you need to cheat just to keep up.
That's not the cost of not getting an A, it's the cost of appearing to underperform compared to too many of your peers. Which is directly tied to how many of them cheat. If not enough cheaters got an A then the cost would no longer be tied to not getting an A, it would be tied to whatever metric they appeared to outperform you on.
I can, too. I wanted to learn, but I also wanted to achieve a high GPA. I had a privileged background, so I got to retake classes after earning Cs or Bs until I got an A, without cheating.
The consequences: My degree took a long time to get, cost more money than my peers in the same program, and I now have a deep-seated feeling of inadequacy.
At least for my CS degree, this surprisingly wasn't the case. I remember our freshman class advisor gave a speech that said that grades don't really matter so long as if you pass, but we all laughed and dismissed him. I ended up getting a big tech internship with a ~2.8 GPA and an even better full time job with a ~3.2.
Obviously, your mileage may vary. I graduated in a hot tech market from a prestigious university with a reputation of being difficult. Even so, overall, almost all of my classmates were stressed over grades significantly more than they needed to be.
And every non-educational institution after that will see school, degree as a checkbox.
If you don't want people to prioritize grades over everything else...
Perhaps another way to widen the scope of what is not cheatable (at the cost of more teacher work, ugh), is to require showing all work?
And I mean every draft, edit, etc.. All paper scratch-notes. Or on work on computer applications, a replayable video/screenshot series of all typing and edits, like a time-lapse of a construction site. Might even add opportunities to redirect work and thinking habits.
Of course, that too will eventually (probably way too soon) be AI-fakeable, so back to paper writing, typewriters, red pencils, and whiteout.
Just an idea; useful?
But if you went to a top university with brutal courses, and got a 2.7 GPA, then all I'm seeing is you're not elite material. The number otherwise does not help me one bit in evaluating you.
BTW, having spent a lot of time out of the US - it's still pretty laid back in the US. A person who is 2.7 GPA material in the US would simply not get admission in any decent university in some countries. And plenty of people in the US start all over at another institution and do well - something many countries don't allow (state funded, lack of resources, you have to move out of the way to let the younger batch in).[1]
[1] A good friend of mine totally flunked out of his university. He spent time off in the military. Then started all over at a new university. Got really high grades. Went to a top school for his PhD and is now a tenured faculty member.
If it's a job, the order will be school, school, major, everything else on the résumé, grades maybe.
I had a decent GPA and took reasonably hard classes. I had a required discrete math class that was awful. The professor would assign homework for the next chapter that we hadn't gone over yet and them grade it as if it were a test. WTF am I paying you to teach me if I have to learn it myself before you ever present it and test me on that? Assign reading beforehand - great. Assign upgraded, or completion-graded homework beforehand - great. Grad it like a test before teaching it - BS. I took it with another professor after dropping the first one and they had more normal practices and it went much better.
In Germany, all exams are like this. Homework assignments are either just a prerequisite for taking exam but the grade is solely from the exam, or you may get some small point bonus for assignments/projects.
> But any teacher doing this will end up with no students signing up for their class.
The main courses are mandatory in order to obtain the degree. You can't "not sign up" for linear algebra if it's in your curriculum. Fail 3 times and you're exmatriculated.
This is because universities are paid from tax money in Germany and most of Europe.
The US will continue down on the path you describe because it's in the interest of colleges to keep well-paying students around. It's a service. You buy a degree, you are a customer.
You point to a true failure in incentives. And yet, the US has the highest density of renowned universities.
And at least when I was in college it was the same with respect to classes, you can't take the same class more than 3 times. Additionally if a course is required you either take it or make the case for an equivalent class.
Same in the U.S. but you can sometimes find an online offering. If you don’t know what you are doing or don’t care then always take the online offering. Much easier to cheat.
My ex-girlfriend is German. She cheated on her exams to get her agricultural engineering degree at university. This was in the 80s.
Almost every university in the US takes federal money and relies on federal loan guarantees to keep the high revenues pumping through. In exchange, the schools are subject to requirements by the government and they impose many. I think the bigger issue is the size and scope of higher ed here and if it's actually a good idea to to tell every school how to run their exams (and enforce it).
The course might be mandatory but which professor you choose isn't. What if multiple professors teach it? Word gets around and everyone chooses the easy profs.
Very strongly depends on the school and major; there are both narrow-path degrees with lots of mandatory courses and wide-path degrees with very few specifically mandatory courses (instead having several n of m requirements) other than lower-division general education requirements.
In university I can recall a computer graphics course where literally everyone got 100+% on problem sets (there were bonus questions of course) and the median score on the midterm was below 50%. Leading up to the exam I remember the prof leading an exam prep session, opening the floor to questions, and getting a sincere request from one of the students to please go over the whole concept of "matrices" again.
This was a 400 level course, BTW. At one of the highest-rated universities in Canada. (I was taking it as an elective from a different program from the default, so I can't speak to the precise prerequisites to get there.)
This was over 20 years ago, BTW. I'm sure it's only gotten somehow even worse.
They ended up changing the assignment to where you could just find an implementation of a FFT online and write about it or something.
That's not even getting into the students who copy-pasted Wikipedia straight into their papers in that same class.
The midterm and final exam were in-person in bluebooks, and they were 60% of your grade. If you were just copying the problem sets, you would fail the exams and likely the class.
Long term I see education going this route, rather than preventing students from using AI tools, update course curriculum so that AI tools don't give such an advantage.
The real issue as I see it is that no one wants to face the reality that far too many incapable, incurious people are going to college. So I pretend to give real tests and pretend to give real grades and students feel good about themselves and my classes fill.
But in the (hypothetical) limit where AI tools outperform all humans, what does this updated test look like? Are we even testing the humans at that point?
The same for job interviews. I did a lot of technical interviews in the past as interviewer (hundreds) for Software Engineer positions (and still help companies to hire sometimes, as independent interviewer).
There is insane amount of cheating. I'd say at least 30% in normal companies are cheaters, and 50% and more in FAANG. I can prove it, in private groups, and forums people share tech assignments. And very large number of these people use some kind of assistance while interviewing.
It's interesting to see how sometimes questions that are intentionally sophisticated are getting solved in a few minutes the best way they can be solved. I see this over and over.
I interview a lot of people and I rarely see anything I'd describe as cheating. Maybe my company is not famous enough to be worth cheating at.
Dead Comment
I was in high school when kids started getting cell phones with internet access and basically as soon as that happened it opened up rampant cheating even among the best of students. I can only imagine it being much worse nowadays than even 15 years ago when I was in high school.
Some of their stuff works really well, and they have prof customers who love it. The CEO went on a tour to visit their biggest customers in person and several of them said they couldn't imagine going back.
Unfortunately as a whole the industry is not interested in it, aside from a few small niches and department heads who are both open minded and actually care about the integrity of the education. There have even been cases where profs want it and the dean or admin in charge of academic integrity vetoes its adoption. I've been privy to some calls I can only characterize as corrupt.
There is something deeply broken about higher Ed, the economics, the culture of the students, the culture of the faculty, the leadership... This isn't an AI problem it's a society problem.
When the students genuinely want to learn something and they are there for the knowledge, not the credit, cheating isn't a problem.
This system truly forced students to grind the hell out of science
An argument perhaps that there should not be hundreds in a room.
This is because 100-200 level math courses are not about teaching anything, but about filtering out students who can't do the work. Once you get past that level students have already formed bad habits and so still only do what it takes to pass. I don't know how to fix it, I don't know if it CAN be fixed.
This is 100% incorrect.
K-12 specifically has it bad. Wake up 7am get to school for 8/9 fill your day with classes you don't have much interest in while also figuring out how to be a social human with other kids and all the stress that entails. Then we require them to go home and continue to do more schoolwork.
Of course they're gonna cheat. They're overworked and overstressed as it is.
When I tell people that I never cheated, ever, in any class, through my entire degree, I get mostly surprise. You never? Not once?
But I paid for it, I think. Because it was not easy finding a first position out of school -- I certainly got filtered by GPA. It actually enrages me. What is the point of a degree? What exactly is the point of this thing, if most of the signal is false? Why did I work so hard?
Not even to mention -- many of my classmates (about 1 in 5, one in 6 or so?) were granted "accommodations" which granted them twice as much time to take their exams. There are online services: pay $80, get a letter certifying your ADHD, that you can give the school to get these accommodations. It's completely ridiculous.
Deleted Comment
This is kind of astonishing to me, because for most of my math and engineering courses cheating on take home work would not have improved my final grade (much less helped me learn the material, which is kind of the point I thought, and often necessary for subsequent courses.)
It seems common for math (and related) courses to grade almost entirely based on in-person, in-class exams. In some courses problem sets are optional (though they can be turned in for evaluation) but are recommended for understanding and practice.
Exams can go poorly, so perhaps having more of them (e.g. frequent quizzes) can help to compensate for having a bad day. Also exams can include basic problems, ones that are very similar to problem sets or worked problems from lectures, etc.
> If we did truly devise a system that prevents cheating we’ll see that a very high percentage of current college students are not ready to be truly college educated.
That sounds like an improvement over the current situation?
I completely agree, but the entire higher ed system is moving to on-line instruction.
Basically, if the University of <xyz> follows your suggestion, all of the competing institutions will eat their lunch by offering on-line courses with the "convenience" of on-line assessments" and the University of <xyz> will lose enrollment.
:-(
For me a proper exam is when you get a topic, spend 30 minutes in a classroom preparing, and then sit down with an examiner to tell him about this topic and answer all the follow-up questions.
We don't do multi-option tests at software interviews, and for a good reason. Why do them in a uni?
I don't completely agree that multiple-choice questions can't test real knowledge. It is possible to write multiple-choice questions that require deep thinking and problem solving to select the correct answer (modulo a 25% chance of getting it right with a guess.)
It's true that MC questions can't evaluate the problem-solving process. You can't see how the student thought or worked through the problem unless you have them write things out. But again, that's a tradeoff with the time it takes to evaluate the students' responses.
Can you blame students for optimizing for grades rather than "learning"? My first two years of undergrad, the smallest professor-led lecture course I took had at least 200 students (the largest was an econ 101 course that literally had 700 kids in it). We had smaller discussion sections as well, but those were led by TAs who were often only a couple years older than me. It was abundantly clear that my professors couldn't care less about me, let alone whether I "learned" anything from them. The classes were merely a box they were obligated to check. Is it so hard to understand why students would act accordingly?
I'd like to point out this has nothing to do with cheating. Cheating happens at all levels of academic performance.
I have not been in university for a while, but I do remember that it was rare that I did my best work for any individual class.
For me it was more of a "satisficing" challenge, and I had to make hard choices about which classes I would not get A's in.
I'm sure some professors might have interpreted my performance in their class as indicative of my overall abilities. I'm fine with that. I learned as much as I could, I maxed out my course load, and I don't regret it at all.
If all my math professors had done this, I never would have earned my computer science degree or my minor in mathematics.
I have an immensely difficult time memorizing formulas and doing math by hand. I absolutely need to be able to prepare notes ahead of time, and reference them, to be able to complete a math test on paper. Even then, I'm a very slow in-person test-taker, and would often run out of time. I've honestly come around to the idea that maybe I have some sort of learning disability, but I never gave that idea much thought in college. So, I didn't qualify for extra time, or any other test-taking accommodations. I was just out-of-luck when time was up on a test.
The only reason I was able to earn my degree is because I was able to take almost all of my math classes online, and reference my notes during tests. (COVID was actually a huge help for this.)
And by "notes", I don't just mean formulas or solutions to example problems that I had recorded. I also mean any of the dozens of algorithms I programmed to help automate complex parts of larger problems.
The vast majority of the math classes I took, regardless of whether they were online or in-person, did not use multiple-choice answers, and we always had to show our work for credit. So I couldn't just "automate all the things!", or use AI. I did actually have to learn it and demonstrate how to solve the problems. My issue was that I struggled to learn the material the way the university demanded, or in their timeframe.
So as an otherwise successful student and capable programmer, who would have struggled immensely and been negatively affected mentally, professionally, and financially, had they been forced to work through math courses the way you prescribe, I'm asking you: please reconsider.
Please reconsider how important memorization should be to pass a math class, how strongly you might equate "memorized" to "learned", and what critical thinking and problem-solving could look like in a world where technology is encouraged as part of learning, not shunned.
Anytime students are allowed technology there is massive amounts of cheating. Knowing a certain body of knowledge off the top of your head is important in all areas of study.
When I was in college, this was every math class. You could cheat all you want on the 20% of your grade that came from homework, but the remaining 80% was from 3-4 in-class, proctored exams.
Can you blame them? If they do the necessary work to learn, but do poorly on an exam for some reason, will you still give them a passing grade?
Just one generation ago this was the norm. The only differences between how exams were given in my math classes were what size of note paper was allowed.
In general students hated the few classes that tried to use online platforms for grading, the sites sucked so much that students preferred pen and paper.
Also, it is a math class! The only thing that is needed is arguably a calculator, a pencil, and some paper. What the hell kind of technology are students using in class?
> The only solution I see is the Higher Learning Commission mandating this for all classes.
Colleges used to all have tech requirements, the big debate was to allow calculators with CAS or not.
> If we did truly devise a system that prevents cheating we’ll see that a very high percentage of current college students are not ready to be truly college educated.
What the heck are students doing in college then? I was paying good $$$ to go to college, I was there because I wanted to learn. Why the hell would I pay thousands of dollars to go to class and then not learn anything in the class, that would be a huge waste of my time!
-Social media Hacking & Recovery.
-Local Location detection.
-Spy remotely on android & IOS devices
-Cellphone monitoring 24/7
-Cloning (WhatsApp, social media account, website, cellular phones, documents) remotely to your device
-Hacking Spouse’s devices.
-Cryptocurrency Wallet Recovery
-GRADE HACK ( Transcript hack. Examsoft , Canvas e.t.c) EM AIL- ( DEHACKLORD@GMAIL.COM )
TELEGRAM : DEHACKLORD01
Hope this information helps someone!
The problem is the lack of learning the material. You don't, IMO, directly care how they produced the answer, you care about it only as a proxy for them learning the material well enough to solve the problem.
And making people do them in person with no technology is unrealistic - not because it can't be done, but because at that point, it's not a reflection of how you'd use it outside of that classroom, and people are going to call it pointless, and IMO they'd be right. You would be correct that anyone who met that bar would have likely learned the material, but you'd also have excluded people who would have met the bar of "can use the material to the degree of familiarity needed going forward".
I think a reasonable compromise would be to let students collaborate on the exams in the classroom, without external access - while I realize some people learn better on their own in some subjects, as long as everyone contributes some portion of the work, and they go back and forth on agreeing what the right answers are, then you're going to make forward progress, even if that ruins the exam process as anything other than a class-wide metric. You could subdivide it, but then that gets riskier as there's a higher chance that the subset of people doesn't know enough to make progress. Maybe a hint system for groups, since the goal here is learning, not just grading their knowledge going in?
Not that there's not some need for metrics, but in terms of trying to check in on where students are every so often, I think you need to leverage how people often end up learning things "in the wild" - from a combination of wild searching and talking to other people, and then feedback on whether they decided you could build an airplane out of applesauce or something closer to accurate.
I don’t care about the answer. I care about the thought process that went into finding the answer. The answer is irrelevant.
And making people do them in person with no technology is unrealistic - not because it can't be done, but because at that point, it's not a reflection of how you'd use it outside of that classroom, and people are going to call it pointless, and IMO they'd be right.
There’s body of knowledge a person trained in a given area ought to know without use of computers or notes. There are things a person who calls themself “an engineer” or “a physicist” ought to know off the top of their head. A person going into mechanical engineering ought to have some familiarity with how to integrate without using a computer. Such is my belief.
Deleted Comment
"It’s a clusterfuck."
Isn't it to either do that now, or to lose the signaling value of college degrees as indicating knowledge.
I'm not sure knowledge is what a college degree signals to prospective employers. The alternative hypothesis, which AFAIK has a fair bit of support, is that it signals a willingness to do whatever it takes to fulfill a set of on paper requirements imposed by an institution, by hook or by crook.
don't sweat the lazy ones, teach the ones who want to learn.
it sucks that a college degree is no longer a sure way to spot the "good students", but meh, been like that for 20 years or more.
Rote memorization and examinations as being the basis of modern education is the problem here, and frankly I'm glad that many academics are struggling because it should show how terrible most educational programs truly are at actually teaching students and developing knowledge.
Sorry, I'm tired to hear about the crocodile tears from instructors who refuse to adapt how they teach to the needs of students and instead lashing out and taking the easy road out by blaming students for being lazy or cheaters or whatever.
When you can read about a classroom in the 1800s and in 2024 and you realize the model is exactly the same, then this should tell you that your entire model is broken. All of it. The rote lectures, the memorization, the prompting students to demonstrate knowledge through grading. All of it is useless and has been a cargo cult for a long time because (and this is especially bad in higher education) there's no interest or effort in changing the way business is done.
Yeah sorry, no sympathy from me here.
...which is what we have today, where the most lucrative industries for people with good math skills are finance (= cheating dumb people out of their retirement), advertising (= cheating dumb people out of their consumer dollars), and data-driven propaganda (= cheating dumb people out of their votes).
/dystopia
Advertising absolutely works on you regardless of how smart or educated you are.
How it has to work to do that can change, but the idea that advertising only impacts dumb people is pernicious as shit.
I like the phrasing you used.
AI is here to stay; new methods should be used to assess student performance.
I remember being told at school, that we weren't allowed to use calculators in exams. The line provided by teachers was that we could never rely on having a calculator when we need it most—obviously there's irony associated with having 'calculators' in our pockets 24/7 now.
We need to accept that the world has changed; I only hope that we get to decide how society responds to that change together .. rather than have it forced upon us.
It seems to me that this is pretty much immune to plagiarism as well as being much better for the student.
For longer essays, you can just build in an oral examination component. This face time requirement is just not that hard to include given that even in lecture hall style settings you can rely on graduate student TAs who do not really cost anything. The thing is that the universities don't want to change how they run things. Adjuncts in most subjects don't cost anything and graduate students don't cost anything. They earn less than e.g. backroom stocking workers. This is also why they, by and large, all perform so poorly. 30 minutes of examiner time costs maybe $11 or less. Even for a lecture class with 130 students, that's under $1,500. Big woop.
There are some small changes to grading practices that would make life very hard for AI cheaters, such as even cite checking a portion of citations in an essay. The real problem is that US universities are Soviet-style institutions in which gargantuan amounts of cash are dumped upon them and they pretend to work for it while paying the actual instructors nothing.
The process of reading textual material, thinking about it, and then producing more textual material about the stuff you just read (and maybe connecting it to other stuff that you've read in the past) is a critical way of developing thinking skills and refining your ability to communicate to an audience.
The value of that shouldn't be overlooked just like the value of basic numeracy shouldn't be overlooked because we all carry calculators.
You're right that it would be better if post secondary institutions would test people's ability to think in more ways than just what they can regurgitate onto a piece of paper, if only because that can be easily cheated but that doesn't mean that there isn't personal benefit in the experience of writing an essay.
I may not be the best writer but I am a better writer because I wrote essays in university, and I may not be great at math but I can reason and estimate about a variety of things because I have taken many math courses. These things have ultimately made me a better thinker and I am grateful to have had that imparted to me.
I've kept a blog for almost 20 years now and one thing is for sure: well-structured writing is very different from an oral exam the writing allows for restructuring your thoughts and ideas as you go and allows for far more depth.
I don't think, for most folks, that they could have as much depth in an F2F as they could in their writing with the exception of true experts in their fields.
The written essay has a cohesiveness and a structure to it that provides a better framework for evaluation and conveyance of information.
In fact, the only reason I use the word 'importunate' to describe myself, is because that's what my undergrad advisor called me.
So I at least was able to get well over 30m with each professor to discuss whatever I wanted. But likely that's b/c there wasn't a lot of competition.
Average student debt after a 4 year degree is ~$35,000 after ~45 courses. Before even running the math it should be obvious the gigantic cost of higher ed over 4 years is entirely unrelated to what an instructor would be making for ~23 hours of work (barring a secret society of multi millionaires). I.e. the problem you're identifying is the vast majority of $ spent in higher ed is not going to time with your professors, not that doing so is itself expensive.
Could not disagree more. Researching, formulating arguments, can give a student a complete view of the subject that studying for tests misses. But, similarly to tests, it probably depends on the skill of the teacher in creating the right kind of written assignments.
I'm not so sure that writing takes more time than studying. For starters, you don't have to memorize anything, and you can limit yourself to the assigned topic.
Of course, it can be that students don't take studying for an oral exam seriously, and trust the teacher to only ask superficial questions.
That strikes me as a workable bottom line.
So the obvious solution is to make students to talk with an AI, which would grade their performance. Or, maybe the grading itself could be done by a minimum wage assistant, while AI would lead the discussion with a student.
It never was. It's just even more ineffective now that AI exists, than before.
The central example of this is college admissions statements. Some kids have the advantage both of parents who can afford to give them the experiences that look good on such an essay (educational trips to Africa, lessons in two musical instruments, one-on-one golf coaching, that kind of thing), and who can hire tutors to "support" them in writing the essay. AI just makes the tutor part accessible/affordable for a wider segment of the population.
It would be naive to assume that, pre-AI, there was not a "gray" essay-coaching market as well as the "dark" essay-writing as a service market. That market still works better than AI in many cases.
This is an absolute disgrace. And then these are the people who lecture you on "inclusion".
That basically never happens and the outcome is the result of some sort of struggle. Usually just a peaceful one in the courts and legislatures and markets, but a struggle nonetheless.
> new methods should be used to assess student performance.
Such as? We need an answer now because students are being assessed now.
Return to the old "viva voce" exam? Still used for PhDs. But that doesn't scale at all. Perhaps we're going to have to accept that and aggressively ration higher education by the limited amount of time available for human-to-human evaluations.
Personally I think all this is unpredictable and destabilizing. If the AI advocates are right, which I don't think they are, they're going to eradicate most of the white collar jobs and academic specialties for which those people are being trained and evaluated.
For a solution "now" to the cheating problem, regular exam conditions (on-site or remote proctoring) should still work more or less the same as they always have. I'd claim that the methods affected by LLMs are those that could already be circumvented by those with money or a smart relative to do the work for them.
Longer-term, I think higher-level courses/exams may benefit from focusing on what humans can do when permitted to use AI tools.
Two decades ago, when I was in engineering school, grades were 90% based on in-person, proctored, handwritten exams. So assignments had enough weight to be worth completing, but little enough that if someone cheated, it didn't really matter as the exam was the deciding factor.
> Return to the old "viva voce" exam? Still used for PhDs. But that doesn't scale at all.
What? Sure it does. Every extra full-time student at Central Methodist University (from the article) means an extra $27,480 per year in tuition.
It's absolutely, entirely scalable to provide a student taking ten courses with a 15-minute conversation with a professor per class when that student is paying twenty-seven thousand dollars.
<< But that doesn't scale at all.
I realize that the level of effort for oral exam is greater for both parties involved. However, the fact it does not scale is largely irrelevant in my view. Either it evaluates something well or it does not.
And, since use of AI makes written exams almost impossible, this genuinely seems to be the only real test left.
This will be it. [edit: for all education I mean, not just college] Computers are going to become a bigger part of education for the masses, for cost reasons, and elite education will continue to be performed pretty much entirely by humans.
We better hope computer learning systems get a lot better than they’ve been so far, because that’s the future for the masses in the expensive-labor developed world. Certainly in the US, anyway. Otherwise the gap in education quality between the haves and have nots is about to get even worse.
Public schools are already well on the way down that path, over the last few years, spurred by Covid and an increasingly-bad teacher shortage.
I completely agree, but then again it seems to me that society also functions according to many norms that were established due to historical context; and could / should be challenged and replaced.
Our education system was based on needs of the industrial revolution. Ditto, the structure of our working week.
My bet: We will see our working / waking lives shift before our eyes, in a manner that's comparable to watching an avalanche in the far distance. And (similarly to the avalanche metaphor) we'll likely have little ability to effect any change.
Fundamental questions like 'why do we work', 'what do we need' and 'what do we want' will be necessarily brought to the fore.
When I was in university (Humanities degree), we had to do lots of mandatory essays throughout the year but they counted little towards your overall mark, maybe 10% iirc.
The majority of marks came from mid-year & end-of-year exams.
A simple change to negate AI is to not award any points for work outside exams — make it an optional chance to get feedback from lecturers. If students want to turn in work by AI, it's up to them
It doesn't scale if performed by a human. But what if... we employ AI to conduct the voice exams?
My current best guess, is to hand the student stuff that was written by an LLM, and challenge them to find and correct its mistakes.
That's going to be what they do in their careers, unless the LLMs get so good they don't need to, in which case https://xkcd.com/810/ applies.
> Personally I think all this is unpredictable and destabilizing. If the AI advocates are right, which I don't think they are, they're going to eradicate most of the white collar jobs and academic specialties for which those people are being trained and evaluated.
Yup.
I hope the e/acc types are wrong, we're not ready.
On the contrary; with AI it scales better than ever before.
I remember being told the same thing, but I happen to believe that it was a fantastic policy, with a lackluster explanation. The idea that you wouldn't have a calculator was obviously silly, even at the time, but underlying observation that relying on the calculator would rob you of the mental exercise the whole ordeal was supposed to be was accurate. The problem is that you can't explain to a room full of 12 year olds that math is actually beautiful and that the systems principles it imparts fundamentally shape how you view the world.
The same goes for essays. I hated writing essays, and I told myself all sort of weird copes about how I would never need to write an essay. The truth, that I've observed much later, is that structured thinking is exactly what the essay forced me to do. The essay was not a tool to asses my ability in a subject. It was a tool for me to learn. Writing the essay was part of the learning.
I think that's what a lot of this "kids don't need to calculate in their heads" misses. Being able to do the calculation was only ever part of the idea. Learning that you could learn how to do the calculation was at least as important.
I especially agree that essay writing is hugely useful. I'd even go as far as saying, the ability to think clearly is fundamental to a happy life.
On a more serious note - US removed cursive from their curriculum almost two decades ago - something i cant wrap my head around as cursive is something the rest of the world(?) uses starting in middle school and onwards through the whole adult life.
It is new tech, but people do not treat it as such. They are not figuring it out. Its results are already being imposed. It is sheer luck that the individual in question choose to fight back. And even then it was only a partial victory:
"The grade was ultimately changed, but not before she received a strict warning: If her work was flagged again, the teacher would treat it the same way they would with plagiarism."
The poem was assigned to us, but for some reason the subject matter really chimed with me personally. I thought about it a lot, and—as a result—ended up writing a great essay.
Because I did well, I was accused of cheating in front the class.
Teachers are definitely fallible.
Can't perform any research, compare conflicting sources, or self-reflection.
The problem is more with teachers lazily slapping an essay on a topic as a goto homework to eat even more of the already limited students' time with busywork.
I think it's probably more concerning that spitting out the most generic mathematically formulaic bullshit on a subject is likely to get a decent mark. In that case what are we actually testing for?
While writing term papers is a skill that is only minimally useful in the real world (save for grant writers and post docs, pretty much), the patterns of thinking it encourages are valuable to everything that isn’t ditch digging.
Maybe we can outsource this part of our cognition to AI, but I’m skeptical of the wisdom of doing so. Are we all going to break to consult ChatGPT in strategy meetings?
Here is the brand new method - asking verbal questions in person and evaluating answers. Also allow high tech aides in the form of chalk and blackboard
For all their problems, 5000 word take home assignments in Microsoft Office have a lot in common with the activities of a junior management consultant, NGO writer, lawyer or business analyst. And same with for scientists but with Latex.
I’d rather hire a lawyer who could only do their job with AI than one who couldn’t use a computer to create documents or use digital tools to search case law.
Calculators are just one analogy, there is no guarantee it will work out that way. It's just as likely that this over-technologization of the classroom will go the way of whole-language reading education.
Same with homework. If everyone has to solve the same 10 problems, divide and conquer saves everyone a lot of time.
Of course, you're only screwing yourself because you'll negatively impact your learning, but that's not something you can easily convince kids of.
In person oral exams (once you get over the fear factor) work best, with or without (proctored!) prep time.
Maybe it doesn't scale as well, but education is important enough not to always require maximal efficiency.
This assumes that homework helps kids learn, or that the knowledge required to succeed in school will help kids once they graduate.
This is overdue - we should be using interactive technology and not boring kids to death with a whiteboards.
Bureaucracy works to protect itself and protect ease of administration. Even organising hand on practical lessons is harder
I believe it's still true today, but with future AI systems even highly abstract math is under the danger.
I still feel like AI detectors would work well if we have access to the exact model, output probabilities of tokens, We can just take a bit of given text, and calculate the cumulative probability that the AI would complete it exactly like that.
That was just a simple quip to shut down student bellyaching. Even before we had pocket calculators, it was never a strong answer. It just had to hold over long enough so when you realized it was bad answer you weren't that teacher's problem anymore.
The actual answer was that they're complaining about a minor inconvenience designed for reinforcement, and if they really did need a calculator for the arithmetic on a test designed deliberately designed to be taken without a calculator, then they don't belong in that class.
Deleted Comment
You are in a room with a sheet of paper and a pen. Go.
You’re acting as if 2010 was a hundred years ago.
If the level of effort made is high, but the outcome does not comply in some way, praise is due. If the outcome complies, but the level of effort is low, there is no reason for praise (what are you praising? mere compliance?) and you must have set a wrong bar.
Not doing this fosters people with mental issues such as rejection anxiety, perfectionism, narcissism, defeatism, etc. If you got good grades at school with little actual effort and the constant praise for that formed your identity, you may be in for a bad time in adulthood.
Teacher’s job is to determine the appropriate bar, estimate the level of effort, and to help shape the effort applied in a way that it improves the skill in question and the more general meta skill of learning.
The issue of judging by the outcome is prevalent in some (or all) school systems, so we can say LLMs are mostly orthogonal to that.
However, even if that issue was addressed, in a number of skills the mere availability of ML-based generative tools makes it impossible to estimate the level of actual effort and to set the appropriate bar, and I do not see how it can be worked around. It’s yet another negative consequence of making the sacred process of producing an amalgamation of other people’s work—something we all do all the time; passing it through the lens of our consciousness is perhaps one of the core activities that make us human—to become available as a service.
I am really quite confused about what you think the point of education is.
In general, the world (either the physical world or the employment world) does not care about effort, it cares about results. Someone laboriously filling their kettle with a teaspoon might be putting in a ton of effort, but I'd much rather someone else make the tea who can use a tap.
Why do we care about grades? Because universities and employers use them to quickly assess how useful someone is likely to be. Few people love biochemistry enough that they'd spend huge sums of money and time at university if it didn't help get them a job.
Let's not assume a lot right now. OpenAI and other companies are torching through cash like drunken socialist sailors. Will AI be here as a Big Data 2.0 B2B technology? Most likely, but a viable model where students and laypeople have access to it? To be seen.
We all mooched off of dumb VC money at one point or another. I acquired a few expensive watches at Fab dot com at 80% off when they were giving money away, eh.
You can run GPT-4-equivalent models locally. Even if all software and hardware advancements immediately halt, models at the current level will remain available.
If done right they would go from VC money with an expected exit to government money that overpays for incompetence because our only way out of deficit spending is through more debt and inflation.
Sorry, English is not my first langage what is this expression ? Why does the sailor as to be socialist ? Google didn't help me with this one.
Dead Comment
Now this would effectively kill off the current AI powered solution, because they have no way of explaining, or even understanding, why a paper may be plagiarized or not, but I'm okay with that.
Anyone interested to learn more about it, I recommend the recent book "AI Snake Oil" from Arvind Narayanan and Sayash Kapoor [1]. It is a critical but nuanced book and helps to see the whole AI hype a little more clearly.
[1] https://press.princeton.edu/books/hardcover/9780691249131/ai....
Examples: Spam detection, copyrighted material detection, etc.
In what world is this fair? Our court systems certainly don't operate under these assumptions.
I also thing that there should be laws requiring a clear explanation whenever that happens.
Figuring out who the hell you are in your high school years was hard enough when Kafka was only a reading assignment.
This is a big part of GDPR.
Reading the rules quickly, it does seem like you're not entitled to know why the computer flagged you, only that you have the right to "obtain human intervention". That seems a little to soft, I'd like to know under which rules exactly I'm being judged.
> The data subject shall have the right not to be subject to a decision based solely on automated processing [...]
[1]: https://gdpr.eu/article-22-automated-individual-decision-mak...
That's not correct. Some solution look at perplexity for specific models, some will look at ngram frequencies, and similar approaches. Almost all of those can produce a heatmap of "what looks suspicious". I wouldn't expect any of the detection systems to be like black boxes relying on LLM over the whole text.
Deleted Comment
In any case, if you where to use LLMs, or other black box solutions, you'd have to yank those out, if you where met with a requirement to explain why something is suspicious.
GDPR to the rescue!
https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-re...
You must identify whether any of your processing falls under Article 22 [automated decision making, including AI] and, if so, make sure that you:
* give individuals information about the processing;
* introduce simple ways for them to request human intervention or challenge a decision;
* carry out regular checks to make sure that your systems are working as intended.
Why in gods name has the USA not adopted similar common sense legislation?
This happens anyways, though? Any service that's useful for alternative / shady / illicit purposes is part of a cat/mouse game. Even if you don't tell the $badActors what you're looking for, they'll learn soon enough what you're not looking for just by virtue of their exploitative behavior still working.
I'm a little skeptical of any "we fight bad guys!" effort that can be completely tanked by telling the bad guys how they got caught.
So, now I use ChatGPT to check my English. I just write what I want to write than ask it to make my text more "More concise, business-like and not so American" (yeah the thing is by default as ultra enthusiastic as an American waiter). And 9 out of 10 times it says what I want to say but better than I wrote myself, and in much less words and better English.
I don't think it took less time to write my report, but it is much much better than I could have made alone.
AI detector may go off (or it goes on? of is it of? Idk, perhaps I should ask Chat ;)), but it is about as useful as a spell-check detector.
It's a Large Language Model, you should just is like that, it is not a Large Fact Model. But if you're a teacher you should be a good bullshit detector, right?
If I'm every checking some student's report, you may get this feedback: For god's sake, check the language with ChatGPT, but for God's sake check the fact in some other way.
Both classes got a lesson, from either end, essentially for free (for the teacher). And it really helped. The next year I got to do the same. Of note was that this was back in the day when computers were relatively rare and typing was a skill that was specially taught, so most of the papers were written longhand for the first draft.
It's long been said that if you really want to learn a subject you should teach it. This sort of give-and-take works well, and it is more or less how the rest of society works. Using AI for this would be quite similar, but I think having another human is better. An AI will never stop you in the hall and say "dude, your paper, I got totally lost in the middle section, what the hell," but sometimes that's quite helpful.
Making texts more accessible through clear language and well-structured arguments is a valuable service to the reader, and I applaud anyone who leverages LLMs to achieve that. I do the same myself.
With students, historically we have always assumed that written communication was the more challenging skill and our tests were arranged thusly. But we're in a new place now where the inability to verbally converse is a real hurdle to overcome. Maybe we should rethink how we teach and test.
Not by design, but the training corpus necessarily includes a lot of "facts" (claims made by whoever wrote the original text). A model that is trying to output nonfiction on a specific topic, is likely to encounter relatively more models of claims that either actually were incidentally true, or at least have the same general form as true claims without an obvious "tell".
Of course, every now and then it goes off the rails and "hallucinates". Bad luck for the student who doesn't verify the output when this happens (which is probably a lot of students, since part of the motivation to cheat is not knowing the material well enough to do such verification properly).
However, the AI identifies the school issued Lenovo laptops as weapons. So every kid was flagged. Rather than stopping using such a stupid tool, they just have the kids remove their laptops before going through the scanner.
I expect not smart enough people are buying “AI” products and trusting them to do the things they want them to do, but don’t work.
Dead Comment
Deleted Comment
People are willing to believe almost anything as long as it makes their lives a little more convenient.
The problem is, if it has been that badly tested that it detects Lenovo laptops as weapons, there is a good chance that it doesn't properly detect actual weapons either.
I want to emphasize, that this isn't really about trusting magic, it's about people nonchalantly doing ridiculous stuff nowdays and that they aren't held accountable for that, apparently. For example, there were times back at school when I was "accused" of cheating, because it was the only time when I liked the homework at some class and took it seriously, and it was kinda insulting to hear that there's absolutely no way I did it, but I still got my mark, because it doesn't matter what she thinks if she cannot prove it, so please just sign it and fuck off, it's the last time I'm doing my homework at your class anyway.
On the contrary, if this article to be believed, these teachers don't have to prove anything, the fact that a coin flipped heads is considered enough of a proof. And everyone supposedly treats it as if it's ok. "Well, they have this system at school, what can we do!" It's crazy.
There was recent article about yet another generated text in the US court, this time without malicious intent (it seems). The article boils down to the fact that the plaintiff asked neural network to do a historical financial calculation of property cost and immediately trusted it, "because computers". Computers are always correct, NNs run on computers, hence they are always correct :) . Soon this mentality will be in every household on the planet. We will be remembering days of media dishonesty and propaganda with fondness, at least previously we kinda could discern if the source was intentionally lying.
Well, you shouldn’t be so surprised. You just described 95%+ of the population’s approach to any form of technology. And there’s very rarely any discomfort with such ignorance, nor any desire to learn even the basics. It’s very hard to understand for me — some of us just have to know!
People trust a system because other people trust a system.
It does not matter if the system is the inquisition looking for witches, machine or Gulag from USSR.
The system said you are guilty. The system can’t be wrong.
Kafka is rolling in his grave.
That's how you can mold society as you like at your level: this student's older sibling was a menace? Let's fuck them over, being shitty must run in the family. You don't like the race / gender / sexuality of a student? Now "chatGPT" can give you an easy way to make their school life harder.
Just introduce an incomprehensible process, Like applying for a Visa or planning permission, and then use it to your advantage.
From the victims perspective, there is no difference between bureaucracy and AI
I’ll give you a hint: they’re not ignorant.
I think it is, however the dream among educators of an “AI detector” is so strong that they’re willing to believe “these guys are the ones that cracked the problem” over and over, when it’s not entirely true. They try it out themselves with some simple attempts and find that it mostly works and conclude the company’s claims are true. The problem though is that their tests are all trying to pass off AI-generated work as human-generated—not the other way around. Since these tools have a non-zero false positive rate, there will always exist some poor kid who slaved away on a 20-page term paper for weeks that gets popped for using AI. That kid has no recourse, no appeals—the school spent a lot of money on the AI detector, and you better believe that it’s right.
At least not having to care about arithmetic leaves more time to care about mathematics.
We don't do calculations: computers do it for us.
We don't accumulate knowledge: we trust Google to give us the information when needed.
Everything in a small package everyone can wear all day long. We're at the second step of transhumanism.
I mean I'm skeptical about AI as well and don't like it, but I can see it becoming a force multiplier itself.
This is what will eventually happen. Some component or provider deep in the stack will provide some answer and organizations will be sufficiently shrouded from hard decisions and be able to easily point to "the system."
This happens all the time in the US. Addresses are changed randomly because some address verification system feedback was accepted w/o account owner approval -- call customer service and they say "the system said that your address isnt right", as if the system knows where i've been living for the past 5yrs better than me, better than the DMV, better than the deed on my house. If the error rate is low enough, people just accept it in the US.
Then, it gets worse. Perhaps the error rate isnt low, just that it is high for a sub-group. Then you get to see how you rank in society. Ask brown people in 2003-2006 how fun it was to fly. If you have the wrong last name and zipcode combo in NYC suddenly you arent allowed to rent citibikes despite it operating on public land.
The same will happen with this, unless there is some massive ACLU lawsuit which exposes and the damages will continue until there is a resolution. Quite possibly subtle features on language style will get used as triggers, probably unknowingly. People in the "in-group" who arent exposed will claim it is a fair system while others will be forced to defend themselves and have to provide the burden of proof on a blackbox.
I suspect there is a product opportunity here. It could be as simple as a chrome extension that records your sessions in google docs and generates a timelapse of your writing process. That’s the kind of thing that’s hard to fake and could convince an accuser that you really did write the essay. At the very least it could be useful insurance in case you’re accused.
The easiest for someone here to see is probably code generation. You can point at parts of it and go "this part is from a high-school level tutorial", "this looks like it was grabbed from college assignments", and "this is following 'clean code' rules in silly places"(like assuming a vector might need to be Nd, instead of just 3D).
If they could make it elsewhere, they would.
I don’t expect this to be a popular take here, and most replies will be NAXALT fallacies, but in aggregate it’s the truth. Sorry, your retired CEO physics teacher who you loved was not a representative sample.
Hey, he was Microsoft’s patent attorney who retired to teach calculus!
Because if you're putting forth the assertion "If they could make it elsewhere, they would." you've certainly had spent sometime teaching, yes?
I think it would be good to understand how much experience teaching it took for you to come to that conclusion.
Deleted Comment
Anyone who's been around AI generated content for more than five minutes can tell you what's legitimate and what isn't.
For example this: https://www.maersk.com/logistics-explained/transportation-an... is obviously an AI article.
to some degree of accuracy.
what.
I'm asking, because all this "AI" text-generation stuff isn't a technology problem. It's 101% a human problem.
Doubtful. This is a new sector/era in the cat-v-mouse game.
> we learn to adapt away from graded homework.
Nothing proposed as an alternative scales well and - ironically - it's likely that something _like_ an LLM will be used to evaluate pupil quality / progress over time.
"One of my kid's teachers set out a warning to students that all essays would be checked against the other students' essays to see if they are the same and the repercussions one would face if caught. A classmate did a Google search and found the questions of the essay as examples on a book."
One thing is perfectly valid, the other one is not.
Then of course, there are shades of gray. Using ChatGPT for some things is not copying and you can even say the kids are learning to use the tool, but if you use it for 95% of the essay, it is.
This isn't a made up situation. Teachers at my school have used AI for essay prompts, test questions, etc and it spreads around and generally leads to the sentiment that "if the teacher is doing it, they can't in good faith tell me to not". Imagine if in math class the teacher , after just telling the students they can't use a calculator, types in a simple arithmetic expression into their calculator.