Crowdsourcing: Convenience Or Confusion?

Crowdsourcing: Convenience or Confusion?

R Hi, this is Renato Beninatto.

M And this is Michael Stevens.

R Welcome to Globally Speaking. Today we’re going to talk about an old topic that is a favourite in our industry. How did we call this program?

M I refer to it as Crowdsourced Confusion.

R Yeah, everybody loves to talk about crowdsourcing. The other day I received a call from a recruiter asking me if I knew anybody, an executive, or somebody that could work for one of her clients but one of the requirements was that they wanted somebody that had experience with crowdsourcing because crowdsourcing is the way of the future in the language business.

I literally laughed because it’s one of those things that it’s like teenage sex, everybody talks about it, nobody is doing it.

M Absolutely. And you mean you didn’t laugh with joy because you felt like they had finally cracked the code as to where we’re going?

R Well, this happens so often it’s not even funny anymore. The whole story about crowdsourcing in the localization industry started in 2007 at LocWorld in Seattle. They were called Localization World then – they hadn’t changed their name yet – but there was this keynote speaker, Jeff Howe, he had written an article in Wired magazine in 2006 who set this whole movement of the idea of crowdsourcing. That was actually based on an earlier book called the “Wisdom of the Crowds” which is a fantastic book, I strongly recommend that you read it, if you haven’t read it.

M He starts with, isn’t it, the jelly jar principle?

R Absolutely, yes.

M Where they have a county fair and the idea is that everybody guesses…

R It wasn’t actually a jelly jar, it was a bull and they had to guess the weight of the bull.

M Same concept but you’re guessing the weight of the bull and the person who gets closest wins a prize – what do they call them, Colorado Oysters – you win some kind of prize.

R Rocky mountain!

M That’s it. So, they guess the weight of the bull and while the guesses that are the furthest off are way, way off, when you take a certain statistical percentage… R The average.

M The average, but you have to have a statistically relevant group who guess, it actually comes out pretty close to what the bull’s actual weight is.

R Yes. So, the whole idea of the “Wisdom of the Crowds” is that all of us know more than any one of us individually. So, what this guy did in the 1800s was – I keep thinking – he had thousands of tickets with guesses for the bull’s weight and he averaged them out and he found out that the average number of all the guesses was within very few pounds of the actual weight of the bull. So, this whole idea of the crowd knows best and that if you bring enough people together to find oil fields, to find cancer in certain images – it didn’t work with the Malaysian Air airplane, they tried to use crowdsourcing to find the location of the airplane, it didn’t work in that case.

But, anyway, this was brought into the language industry after this presentation at Localization World, and another element of confusion and that has had a lot of interesting repercussions in the language business is that crowdsourced work is done by non-experts, so when you want to find oil you don’t need to have seismic engineers and petroleum experts to find new fields of oil, you can have a mixture of plumbers and GI specialists, people who understand fluids, to go to that. But, when it gets applied to the localization space and the translation space, all you need is somebody that has some knowledge of the target language that you want to work with. They don’t need to be expert and professional translators because it’s supposed to be done in spare time with small – you mentioned the hits, Human Initiated Tasks, these very simple tasks. So you are going to translate three words, five words and so on. So, there was a big controversy, I think, around 2009, 2010 with LinkedIn.

M I was thinking of the same thing. That lesson had to be learned the hard way, you don’t necessarily… you’re not asking professionals to do stuff for free.

R Yes. And LinkedIn posted in their platform that they were looking for people to translate LinkedIn strings for free in exchange for… there actually was a survey “how would you like; would you like to translate for us and get a badge on your profile and get a free subscription to our services” or something like that. So, “you give us some of your language skills and we give you some of our freemium services”. And there was this backlash; it ended up with a letter to Time Magazine and people really getting very angry, translators getting very angry because “how could LinkedIn, a company that has so much money, try to get translations for free?”

M Yes, and I still get a little confused by the outrage that was there because I think the line for LinkedIn benefiting for profit form free activity professionals is much clearer. Yet, Facebook does a significant amount of volunteer crowdsourcing as well and they are professionals and non-professionals working in that area.

R I translated for Facebook at one point.

M I translated Pirate as well.

R I translated into Brazilian Portuguese right in the beginning, I wanted to see how the platform worked and it’s fascinating. But, the thing about crowdsourcing is that it’s not a reliable and something that you can predict and manage. So, if you have a deadline and you have a task that needs to be completed by a certain deadline, a certain volume of work that needs to be done by a certain date or any milestone that you have, there is no guarantee that the crowd is going to show up.

One of the things that I found out with Hassan Haddad who was the original creator of this platform for Facebook was that people translate strings with three to five words really, really fast. The moment you start putting sentences and more structured content it becomes a little complicated, there are variables involved; people just skip them and they go to the next. So, if you want to translate legal content, for example, that every website has disclaimers and terms and conditions, and things like that, people will avoid doing that because it requires more professional skill than just translating strings and simple sentences that are done on a day-to-day basis.

M So, timing is a big piece and reliability when you’re getting things done. In your opinion, Renato, why did LinkedIn cause such an outroar and yet Facebook is rarely criticised for their volunteer crowdsourcing.

R I think it had to do with timing and also with the way… they were early, so was Facebook but the way that people engage with it. So, LinkedIn is more of a professional environment and they targeted professionals, they targeted translators who offered their service for free.

M Facebook, if you login in another language they’re asking you whether you want to do it, whether you are a professional linguist, whether you are bilingual, whether you have no training, nothing; and it fits much more what you said, it’s the wisdom of the crowd, you’re not trying to get something for free.

R And here’s the beauty of automation and crowdsourcing. Facebook doesn’t pay their volunteers to translate but they have a platform where people can go, anybody can go, you can go there today and select your language and volunteer to start translating for them and they’re going to give you a leader-board and you can have your picture show up in the leader-board, it’s really cool. But, what they do is they don’t spend any time reviewing or doing quality control. What happens is you translate that content, you save and then they ask other people in the community to review that content.

So, let’s say if five people review the content and approve it; they put a like button on it; if there are 10 or – I don’t’ know what the algorithm is, how many people are required to approve it – if the majority says “yes it’s good” and one person says “no, it’s bad,” they assume it’s good and they take it forward. They are only going to have a control process when half of the people say it’s good and half say it’s bad, so they are going to send it to a professional linguist and say “please referee this dispute and find what is the best solution.”

So, you manage, which is the beauty of management today in the language business, what you want to do. It’s not manage the whole process, it’s manage by exception. You look at the things that don’t meet the norm.

M Yes, and the costs differ when it comes to crowdsourcing. You may be getting a volunteer group to do the work and you’re lucky if your company can do that or you may be paying them less for each task than you would if you were paying a professional company but you have to build in some automations; otherwise, you’re going to spending all your time weeding out the cheaters, trying to find people who are just putting garbage in just to get paid. You really waste time. So, you are going to spend some money and make some investment in automation to clean it up.

R Well, that’s the other myth is that crowdsourcing is free. It’s not, Michael, I’m so sorry to disappoint you. Crowdsourcing requires management on your side; it requires technical infrastructure. It requires technology, it requires a subscription to Amazon Turkers [sic], it requires... Lingotek is a company that has a technology platform for crowdsourcing. Several vendors, Lionbridge, Welocalize have crowdsourcing solutions. But, this is not a service that is profitable for the language service providers. It’s just a service that is provided to the client as part of the service stack that you provide. So, you will provide very professional transcreation services, you will provide traditional translation language services. And if the client wants crowdsourcing, you provide crowdsourcing. If they want you to wash their feet every Thursday, you will … it’s a service company, you will do that too.

M One of the people who I’ve read some of his articles, he teaches at the University of Pennsylvania, and it’s Chris Callison-Burch. Chris lays out some of how the algorithms work if you have two people do it, you have A and B, you are out to try and find one that matches A and until you get to that point you can’t eliminate whether the crowd has said it is good or not and verified it. He goes into detail as to how those mechanics play out in crowdsourcing. He has a lot of good articles, he’s worth checking out and we’ll give some links.

R Yes, we should provide the links and the website.

M But, if you had to crowdsource one thing in your life, what would you do?

R I don’t know if I would want to crowdsource. I’ve had professional situations where I had tasks that were repetitive and annoying enough that I didn’t want to do and I seriously thought of using crowdsourcing. Something like verifying email addresses or verifying web links and see if they’re working or not working, and things like that. But, I either gave up the idea because it wasn’t worth it… or it wasn’t that much work, it was like hundreds of things instead of thousands and we ended up doing it internally, manually.

But I think to wrap this, crowdsourcing is just one more tool in your toolbox. It’s something that might be useful. I think the value of crowdsourcing is more if you have a community of users and you want to engage that community with your product so that they feel they are involved and they belong in the creation of the product, and I think Facebook has that element, that it’s a way for them to engage with their users and make their product better. After all, the biggest expert in Facebook is whoever uses Facebook, it’s not somebody else.

I know that we had a funny situation at our company where we had a project manager working for a messaging app and he was the project manager for that app but he didn’t have the app on his phone so he couldn’t do a good job, right? So, we replaced him for somebody that used the app actively. So, one of the key takeaways, I think, from the concept of crowdsourcing is that as a standalone solution for the language business it’s outdated, it doesn’t create any additional value, it is good for certain localization-related tasks but not necessarily to some language-related tasks, so search relevance, testing, identification of certain features in global maps, namings in different languages. So, these are localization-related tasks but they’re not necessarily language-related tasks. So crowdsourcing, yes; crowdsourcing in translation, no.

M I liked what you said about it’s like teenage sex because it’s rapid and it’s messy.

R Nobody knows how to do it well.

M Yeah, no-one knows how to do it well and so you need to invest time in making sure your automation backbone is going to clean up some of that stuff for your because if you’re doing manual work in that area it’s not going to be worth the time and effort.

There are, also, some big questions out there and I mentioned Chris before; he’s doing research on his “crowdsourcing nothing more than a virtual sweatshop”. So, is this just something people can use to supplement their income and do while they’re watching TV? Great. But, are there people out there trying to earn a living and are they getting a fair wage while they’re doing it? I think that’s a real question that crowdsourcing has globally. You can have a hypothesis, test it quickly on a bunch of people and then figure out if you’re going to move forward with doing it.

R Let’s crowdsource that question! Michael, it was great talking to you again. Let’s see what we’re going to talk about next time.

M Awesome, thank you.

End of discussion