Cochrane’s 2016 Citation Screening Challenge: Turning a lonely task into the most fun you can have in evidence based healthcare

 This article is part of the network’s archive of useful research information. This article is closed to new comments due to inactivity. We welcome new content which can be done by submitting an article for review or take part in discussions in an open topic or submit a blog post to take your discussions online.

Being able to identify randomized trials (RCTs) is a critical part of most Cochrane reviews that evaluate the effects of health treatments. In May last year Cochrane launched Cochrane Crowd. This platform enables anyone with an interest in health to dive into tasks that help identify and describe health research.

By late December 2016, over 4,000 people had signed up to help and we were in sight of reaching 1 million individual classifications. For those of you new to Cochrane Crowd, each task is based on people answering a single question about a piece of research. Each piece of research is looked at multiple times by different people and a final decision is made based on a certain number of decisions that are in agreement. We call each decision made, a classification. We decided to try to reach this impressive milestone, of 1 million classifications, in style by running a 48-hour citation screening challenge. We also decided to make it a fundraiser, raising money for Medicine Sans Frontiers (MSF) and UNICIF.

Pre-challenge jitters
The challenge was set to start on Monday 19 Dec at 10am UK time, and finish at 10 am Wednesday 21 December. As the date drew closer I began to worry: what if no one joins in because it’s so close to Christmas? What if we’ve totally miscalculated and don’t reach 1 million during the 48 hours? What if the whole site goes down, and never works again…

And we’re off
Then suddenly it was too late to do anything but make a cup of tea, get settled comfortably and wonder if my first record would be an RCT. It wasn’t. In fact the first 100 or so records I screened during the challenge were all ‘rejects’ but it didn’t matter. The challenge was underway and it was thrilling to see so many people online and to watch the classifications counter steadily going up.

For the next few hours a good rhythm was established. Contributors in Australia, New Zealand and neighbouring countries of the Asia-Pacific then went offline to get some kip, whilst others in the US and Canada woke up to join the fun.

24 hours later
At the halfway point we’d done 26,169 classifications bringing our total to 985,362. Over 100 people had so far joined the collective effort. I began to allow myself to think we might actually do this. It seemed others were thinking the same as suddenly the pace picked up considerably, driven on in equal measure by the approaching milestone and by Gordon’s brilliant and encouraging banner messages (Gordon is our IT maestro extraordinaire!). By 4.20pm on day 2 we stood at 999,194 classifications. With my heart racing, and my fingers inaccurately trying to type frantic tweets whilst also screen, we crossed the 1 million mark just 7 minutes later at 4.27pm.

I threw my headset off (why was I even wearing it?) and did a victory lap round my tiny study. As I sat back down, remembering my responsibilities to tweet and email, I felt a little emotional. To reach this milestone for Cochrane Crowd through the efforts of so many, not just during this challenge but throughout the last few years, felt significant.

No time to sit back and relax - we still had work to do!
We crossed the 1 million mark around 17 hours before the challenge was due to end. The rest of the time would therefore be focused on raising as much money as we could for MSF and UNICEF. More tea was made, the headset inexplicably put back on, and the pace picked up once more. Together we went on to raise a further £2,000 bringing our total to over £7,000.

And now, relax
At 9.59am on 21 December 2016 I wondered if my last record would be an RCT. It wasn’t, but it didn’t matter, we’d already found hundreds of them. As the challenge ended I couldn’t help but feel we’d all become more than participants. We had been on an adventure together.

In numbers

In words
We received some really helpful feedback about what people felt worked well and what we could do to improve the experience next time. Here’s just a small sample of feedback that made me smile:

Why did you join the challenge?
“Erm, I actually simply joined Cochrane Crowd - and found myself in the middle of it :)”

Describe the experience:
“Great working with people around the world. Great sense of unity”
“I liked seeing that others were "out there" doing the same thing”
“The awareness that colleagues around the world were working on the same job at the same time”
“Very easy to do - even whilst eating a sandwich”
“It was serious fun”

Thank you

It most certainly was serious fun. If you took part in this event: thank YOU. There are lots of people that make Cochrane Crowd work, in fact, that’s exactly the point: lots of people, working together, can achieve brilliant things. We ended 2016 on a high. Let’s make 2017 even more exciting for Cochrane Crowd.

If you missed this event, don’t worry, there will be more challenges, and in the meantime why not pop over to Cochrane Crowd and get some practice in. And don’t forget to drop in and say hi on Twitter @cochrane_crowd.

Happy 2017 and happy citation screening.

From Anna, Emily, Gordon and the rest of the Cochrane Crowd team