Interestingly, through engaging with you I discovered that this is a cognitive bias called the "horn effect" and is the reverse of the more common "Halo effect": https://en.wikipedia.org/wiki/Horn_effect#:~:text=The%20horn....
They share a list of academic publications that have resulted from the project, and their Team page lists the full names a sizable large number of people.
Their FAQ indicates that the cost of the DNA Kit and other things are covered by the project funding. [1]
What made you think that it's engaging in fraud? I'm genuinely curious.
I'm not involved in the project but just from looking at the site for several minutes, it seems to be a fairly reasonable research project.
Or did you say "fraud" less to mean "these are people who are stealing money and e.g., hoarding it away" and more to mean "these are people engaging in a research project I disapprove of"?
[1] https://dogagingproject.zendesk.com/hc/en-us/articles/441699...
I'm not familiar with what you're referring to here. Happen to have a link?
At the bottom, they noted the following:
> SWE-Bench has recently modified their submission requirements, now asking for the full working process of our AI model in addition to the final results -their condition to have us appear on the offical leaderboard. This change poses a significant challenge for us, as our proprietary methodology is evident in these internal processes. Publicly sharing this information would essentially open-source our approach, undermining the competitive advantage we’ve worked hard to develop. For now, we’ve decided to keep our model’s internal workings confidential. However we’ve made the model’s final outputs publicly available on GitHub for independent verification. These outputs clearly demonstrate our model’s 30% success rate on the SWE-Bench tasks.
Their model outputs are here: https://github.com/CosineAI/experiments/tree/cos/swe-bench-s...
What are some frameworks for how to think about navigating this tension in emerging scientific or engineering fields?
Some ones I'm mulling over:
1. Rate of innovation: In rapidly evolving fields, imposing strict regulations too early can hinder innovation and progress. In such cases, it might be better to minimize restrictions early on to allow practitioners to explore new ideas. Then, as the field matures, regulations and standards can be gradually introduced.
2. Adaptive regulation: Implement a flexible regulatory framework that can be updated as new information becomes available.
3. Self-regulation: In some cases, maybe we should expect and encourage the industry to use self-regulation via developing guidelines and codes of conduct. This may be one way to try and strike a balance between responsible innovation while minimizing bureaucratic obstacles.
What do others think?
It was very entertaining and charming to hear him discuss his personal and professional life, and lessons he's learned throughout them often occasionally have very little to do with computer science.
I don't remember all of his "Thoughts for the Weekend", but I do remember one story he told about wishing he had apologized sooner to resolve some conflict he was in. That was a bit of wisdom that stuck with me from the class, beyond any of the computer science topics we covered.
[1]: https://www.theguardian.com/society/2022/jul/23/alzheimers-s...
"The hypothesis took another hit last July when a bombshell article in Science revealed that data in the influential 2006 Nature paper linking amyloid plaques to cognitive symptoms of Alzheimer’s disease may have been fabricated. The connection claimed by the paper had convinced many researchers to keep pursuing amyloid theories at the time. For many of them, the new exposé created a “big dent” in the amyloid theory, Patira said."
We must assume they don't care about learning, so if we want to cram some knowledge into them anyway, hit them the only place they apparently care about: their transcript. Either they'll shape up or be rightfully denied the (ever more scant) honor of a degree. The heroics of the author here cannot be scaled, and we shouldn't be asking anyone to go to that extent. Just fail the bastards. You're only recording what's already true.
Specifically, different systems of incentives and permissiveness will produce different behavior. I taught high school computer science for 4 years, and I can attest that cheating occurred in the classes I taught. I've also been enrolled part-time in Stanford's MS in CS, and have taken a number of the core undergraduate curriculum for CS majors.
I also went to a hypercompetitive US public high school with a number of brilliant classmates, many of whom also cheated.
My experiences have showed me that there is a wide spectrum of "cheating", ranging from students sharing things like, "I was at office hours and heard from the TA heard from the professor that topic X is going to be really emphasized on the exam, so you better study for it!" to outright blatant copying of other's code or answers.
What I've noticed as qualities of a learning environment that seems to increase the likelihood of cheating are:
1. The technological ease of which it is to cheat: it's easier to cheat on an asynchronous online exam than when you're taking it synchronously in a large classroom.
2. How "high stakes" the course is for students: for students at institutions like Stanford, where they be used to a certain level of academic success, failing a course isn't just a blow to their transcript -- it's a psychological blow to their identity as a "smart student." They may find it easier to cheat and maintain their self-image (and projected image to their family/friends) as a great student than to take the honest hit to their GPA, and have to give up their identity.
3. How "legitimate" the course feels: classes where the instructor is widely perceived as "unfair" or "incompetent" seem to have more cheating. Students feel disrespected ("How could she put X on the exam? We barely covered it!") or unvalued ("He doesn't even bother giving clear instructions on the homework assignments. Why should we respect his test?") may try to 'retaliate' by cheating.
4. How permissive the academic culture is around cheating: if there is widely perceived to be little-to-no consequences to cheating, or if cheating is seen as, "well everyone does it", then you will have a lot more cheating.
I'm sure the above is not an exhaustive list. My broader point is that in order to address the issues around cheating, we need to be more encompassing than simply punishing the cheaters. If the stakes are high enough, and the incentives strong enough, cheaters will still exist even if they are aware of the severity of the punishment.
If a student admitted to cheating, while they would face academic disciplinary action (i.e. receiving a failing or low grade), they would not be brought up to the administrative office that deals with issues of academic integrity, and therefore would not face consequences like expulsion or being on official academic probation.
However if a cheating student decided to risk it and not admit their guilt, they were at risk of a potentially even greater degree punishment. The course staff would run all students code through a piece of software to detect similarities between each other, as well as online solutions. Students who were flagged by this software would then have their code hand-checked by at least one course staff, who would make a judgement call as to whether it seemed like cheating.
I found this policy quite interesting. As a former high school teacher, I've certain encountered teaching in my own classes, and have historically oscillated between taking a very harsh stance, or perhaps an overly permissive one.
The one taken by the lecturers of this course offered a "second chance" to cheaters in a way I hadn't seen before.