Meet BugFinders, the British company crowdsourcing the art of app testing

If you’re charged with finding critical bugs in the apps and websites of major clients such as the Met Office, Tesco and Bang & Olufsen, enlisting the help of testers sourced via the internet might not seem like a smart idea. According to BugFinders CTO and founder Martin Mudge, however, it’s the only sensible way of doing it.

BugFinders describes its job as “true crowdsourced testing”, employing freelancer testers from all over the globe to spot and report the flaws in its clients’ software. A community of 55,000 testers sit waiting for their next job, some earning as much as £800 per week, many doing it purely for the challenge.

Why does online outsourcing work so successfully for software testing? And how does BugFinders prevent sensitive flaws in big brands’ software becoming public knowledge? How might you, if you have the skills, join the team? We met with Martin Mudge to find out.

Numbers game

Mudge started testing apps in the days before crowdsourcing was even heard of, let alone desirable. He was head of the testing team at mobile network Orange at around the turn of the decade, when the number of devices you needed to test against was tiny compared to today. “At that time, you only really needed to cover 15 devices to get good coverage, and we’d use a team of two to do that,” he said.

Before long, however, app developers began complaining that bugs were being missed and Mudge couldn’t work out why. “It transpired that, in the same way spot-the-difference puzzles have only two pictures, if you give the tester more than three devices or three browsers to check the same application, they basically get bored by the fourth one and don’t see bugs unless they’re really big ones,” he said.

There was another problem: with the app boom triggered by the emergence of the iPhone and Android, Mudge’s team simply couldn’t keep up. “I and my co-founder [who happens to be Mudge’s wife, Donna] came up with the idea that what we needed to do was use a bigger team of people who are remote to do the work.” Donna already owned her own recruitment firm, giving them a pre-existing network of freelancers to tap, and consequently BugFinders was born in 2011.

Experienced testers

BugFinders isn’t like other crowdsourced labour projects, such as Amazon’s Mechanical Turk, where essentially anyone can earn a few pounds by completing a relatively menial task. BugFinders specifically recruits experienced testers, ideally those who already have at least three years’ experience working as a tester. They might top up their existing skills and salary, earning freelance income in their spare time, or they might be one of the elite testers who can earn a full-time salary by testing from home.

Finding the number of testers required to make crowdsourcing effective proved to be more difficult – and expensive – than Mudge thought, even with Donna’s experience in the recruitment industry.

Although they quickly enlisted a decent-sized community of part-time testers in the UK, Mudge needed testers available in multiple time zones, so that when clients dropped an app that required testing within a short time frame, BugFinders had people who were able to work around the clock. “We sent people all over the world at one stage to recruit people,” he said. “That worked really well.”

Indeed, it pushed tester numbers past the 25,000 mark – what Mudge described as the “critical point whereby we could then effectively take a project at any time, day or night, 365 days a year and deliver it”. That number has since swelled to 55,000, largely helped by existing testers referring colleagues, although anyone can sign up via the company’s website (head to bugfinders.com/register).  

A team of testers that would come close to filling Arsenal’s Emirates Stadium sounds like one heck of a workforce, but when you consider that you now need to be testing apps/websites on 152 devices to cover only 80% of traffic, you begin to see the necessity of having so many people to call upon.

“Most of our clients, when we start talking to them, are covering five or so devices internally,” said Mudge. “They’re missing out on massive opportunities to improve their mobile experience. Effectively, with 300 testers we’re covering sometimes 400 or more mobile devices – so way over the 80%.”