I tried to do some Turking just for a little side-cash.
I stopped after I realized that buying a higher tier of tax software just to handle multiple income sources ate at least the first $15 of income (which is a significant time investment in Mechanical Turk).
In other words the income was so low that even paying the taxes on it ruined the whole venture. Even Bing's rewards have a higher rate of return. The $2/hour from this article entirely mirrors my experience (although my average might be a little lower).
I have an anecdotal story on the other side. I had a small startup and we had a list of URLs. We loaded them into mTurk and asked turkers to visit site and find an email address (probably could have automated this ourselves, but we didn't). Most of the responses were just info@<<websitename>>.com and we could see they completed the task in <10 seconds. So although info@ is a legit answer, after checking a few we found out the turker just guessed it would be a suitable answer and get paid for it. We basically couldn't trust any of the answers and just walked away from that service.
You set up the HIT the wrong way. The correct way to do this is to assign the same URL to multiple Turkers and look for cases where people disagree in order to verify the response quality. Granted it's still possible that both Turkers assigned to the HIT could commit fraud in the same way, but surely just as a matter of chance this bounds your error rate pretty substantially.
I tried MTurk as well out of curiosity. I only made a few dollars before giving up, but my observation was that like anything else in life, you get what you pay for. High paying jobs on MTurk require a impeccable record of a high number of accepted jobs and a near perfect acceptance ratio. You slowly (I can't emphasize how slow this is) build up your reputation as you complete jobs for pennies until you get to the category where you can accept jobs that pay $100 dollars or more.
Turkers who work hard to get the reputation to do high paying jobs won't risk losing the reputation by cheating. So if you want good results you need to pay up to get those Turkers on your job.
What's even worse is the people who do that simple guessing vs. actually doing the task ruins it for others -- eventually the Amazon mTurk AI thinks that your correct answers are 'wrong' due to a mass of others garbage responses. Thus your accuracy rate goes down and you can't qualify for many or most of the new mTurk tasks.
Amusing. Couldn't someone on the other end automate this, then cash in on it? I wonder if there's some potential in this thought: paying for automation as a service, based on the volume of output...
Seems to me the answer to this is a sliding scale of compensation that rises for consistently valid responses and decays to zero for these sorts. Of course, this can be gamed as well, but it seems like everything's a Red Queen's Race these days so the only way to avoid^H^H^H^H^Hminimize one's losses is to race.
Have seen similar things, as it sounds like a great way to use Mturk on the surface. Perhaps the answer is more direction and qualifiers, such as "must be a name, cannot be info@ or editor@"
Side note, if anyone has a way to automate the above in a way better than Hunter.io, there's big money in that.
Wouldn't tax software make for a perfect open source project? Has this been attempted? Any money I give to tax preparation companies upsets me, and my taxes are as simple as it gets.
Tax software would make for a terrible open source project because of the intersection of domain expertise: to do it right you need CPAs and tax lawyers. To do that right, you have to pay them (because there's no "open source" equivalent of a trained and certified CPA or lawyer). If you are paying domain experts, why shouldn't you also be paying software developers? Building a business plan for something like tax software is already hard enough, especially with such entrenched players already, but building a business plan for something like tax software that is also open source is extreme hard mode.
So many social science papers rely on MTurk which is concerning given the poor compensation. With such low payment structures, MTurk is incentivizing people to rush through responses and give responses that they anticipate the researchers want to see.
(Note: I am an academic social scientist who has done an MTurk pilot but never published anything based on MTurk responses)
Yes, this is true, but this is true outside MTurk as well. If you rely on the YouGov panel (people complete surveys to get points for t-shirts), or really anything else, there's a strong incentive to cheat.
This is why good survey research is going to involve attempting to bound fraud through a variety of measures: attention checks ("What's your favorite color? Ignore this question and answer yellow."), looking for straight-lining (respondents always picking the leftmost answer), looking for unusual contradictions (i.e. asking the same question in two, opposite ways and looking for people who don't have the expected relation in their responses), looking at the distribution of completion speed and scrutinizing the lower quantiles, attempting to log participants who take the survey multiple times through dummy accounts, etc.
Oddly, my experience was that Turkers were fairly conscientious. I know one theory for this is that many Turkers are in fact Turking on the job, and so their reserve wages are not about what they are being paid for the MTurk hit, they're about what they're being paid to sit in their desk and not work.
> "What's your favorite color? Ignore this question and answer yellow."
I’ve noticed something similar to that in some of the YouGov surveys I’ve answered over the last decade. I wonder what it says about me that I sometimes write, in the errors/comments section at the end of the surveys, that I have seen surprising questions such as “what do you think about Channel 4?” when I have previously answered that I haven’t watched Channel 4 recently.
I used to work in assessments (educational testing, employee satisfaction, etc.), and the MMPI and other psychological testing would sprinkle in items like "I have never had hair on any part of my body (agree/disagree)" and use them to calculate an honesty/reliability scale, to help assess whether the results were usable or the respondent was giving random or misleading answers.
Just last week, I saw a presentation about some research results that used MTurk subjects. Some of the results were counter-intuitive, albeit fairly consistent across experiments. However, the subject matter was such (basically trust in the results of ML models with differing degrees of complexity/transparency) that I couldn't help wondering if the results might be greatly affected if the outcomes were genuinely important to people.
As you say though, it's the same problem with just about any study of this type whether MTurks, some other online panel, or students earning a little bit of beer money.
Many of the problems in comments are from new requesters/workers using/having poor qualifications or no qualifications, having no quality control, and not researching the platform before using it. If you think you can just dip your toe into the Mturk pool and profit, you're wrong. There is a huge learning curve for both workers and requesters. If you do not invest the time in the platform, you will get garbage pay and garbage results.
MTurk sucks. I need to have over a thousand video transcripts manually fixed, and so far, only about 15% come back even close. Most MTurkers submit the original, unmodified transcript, or something completely irrelevant. I have to use diffing software just to quickly scan each submission for scammers trying to make a quick buck. Granted, I feel like a scammer myself for paying so little (but the decision for payout is coming from above me).
Does mTurk have some sort of quality-related metrics respectively higher rewards linked to it?
For example when I submit some work can I specify "I want only at least 4-stars-people (out of 5, where "5" is for people that rarely make mistakes) to work on this and yes I will pay 4$ extra fees for 4-stars and 8$ extra fees for 5-stars rated workers"?
Readit News
I stopped after I realized that buying a higher tier of tax software just to handle multiple income sources ate at least the first $15 of income (which is a significant time investment in Mechanical Turk).
In other words the income was so low that even paying the taxes on it ruined the whole venture. Even Bing's rewards have a higher rate of return. The $2/hour from this article entirely mirrors my experience (although my average might be a little lower).
Turkers who work hard to get the reputation to do high paying jobs won't risk losing the reputation by cheating. So if you want good results you need to pay up to get those Turkers on your job.
Side note, if anyone has a way to automate the above in a way better than Hunter.io, there's big money in that.
Yes, this is true, but this is true outside MTurk as well. If you rely on the YouGov panel (people complete surveys to get points for t-shirts), or really anything else, there's a strong incentive to cheat.
This is why good survey research is going to involve attempting to bound fraud through a variety of measures: attention checks ("What's your favorite color? Ignore this question and answer yellow."), looking for straight-lining (respondents always picking the leftmost answer), looking for unusual contradictions (i.e. asking the same question in two, opposite ways and looking for people who don't have the expected relation in their responses), looking at the distribution of completion speed and scrutinizing the lower quantiles, attempting to log participants who take the survey multiple times through dummy accounts, etc.
Oddly, my experience was that Turkers were fairly conscientious. I know one theory for this is that many Turkers are in fact Turking on the job, and so their reserve wages are not about what they are being paid for the MTurk hit, they're about what they're being paid to sit in their desk and not work.
I’ve noticed something similar to that in some of the YouGov surveys I’ve answered over the last decade. I wonder what it says about me that I sometimes write, in the errors/comments section at the end of the surveys, that I have seen surprising questions such as “what do you think about Channel 4?” when I have previously answered that I haven’t watched Channel 4 recently.
As you say though, it's the same problem with just about any study of this type whether MTurks, some other online panel, or students earning a little bit of beer money.
Does mTurk have some sort of quality-related metrics respectively higher rewards linked to it?
For example when I submit some work can I specify "I want only at least 4-stars-people (out of 5, where "5" is for people that rarely make mistakes) to work on this and yes I will pay 4$ extra fees for 4-stars and 8$ extra fees for 5-stars rated workers"?
Thx