peer review

I am in Helsinki, Finland, for a few days this week, sitting on a review panel for the ‘Joint Programme – Neurodegenerative Disease Research.’  It is the first time that this programme has involved patients in its review process for funding research so it feels a bit of a milestone moment.  This, after twelve months, of patients and the public coming together (lead ably by Mogens Horder in Denmark) to help develop an approach to public involvement that is practical, constructive and manageable for all concerned.  Another small step for public involvement you might say.

Helsinki Sparring PartnersThe meeting is taking place in the Helsinki Congress Paasitorni which, roughly translated, is the City’s Workers’ Hall. It is a rather magnificent building and last night I posted some of the art that adorns its walls, to my Instagram account.  Outside, in the square, is a sculpture of two sparring fighters as see here.  They certainly don’t seem that hostile towards one another do they?

Somehow it feels like a fitting metaphor for how patients and researchers should be working together in a review panel: sparring together to get better, challenging each other’s thinking, endeavouring to develop each other’s technique .  This is no bout between hostile combatants.  Or it shouldn’t be.

Yet, as well all know, what happens in the ring is dependent on many factors. From the quality of the referee (Chair) to how the ring is set up.  I am happy to report that this has been a good experience thus far.

Before I go….A good friend got in touch with me last week. ‘Have your ears been burning?’ they asked.  Before going on to tell me how my name had cropped up in conversation with a mutual acquaintance.

The saying – ”Have your ears been burning?’ – goes back to Roman times I understand.  It was believed that if your left ear burns, someone is speaking ill of you and; if your right ear burns someone is praising you.

But I am thinking we ought to update this bit of ancient lore to suit modern times.  Something along the lines of: if your right ear burns then someone is accessing your personal data for societal benefit; if your left ear burns someone is using it unwisely.  And if your nose itches then someone else is looking at your genome sequence.

Why? Because of late I have found myself reviewing a lot research proposals in which, almost without exception, researchers are proposing the collection of this or that sample, to generate data that might be linked or not linked with other data, to give new insights or not into this or that disease.  None of this is wrong or unethical of course.  Appropriate consent has been sought from the patient in every case.  Much of the proposed research could be highly significant in the search for disease causes and viable treatments.

In fact it is perhaps one of the less talked about practical benefits of research using personal data that: if people were being poked, prodded and questioned in person as much as their data can be, we might be facing research fatigue across the general population.  But it would still be good to know when, how and why in my opinion. Rather than relying on a feeling.

Still, you can take sayings too far. After all, the Romans also believed anybody could be a doctor.  And I don’t want to start a fight.

A little while ago someone suggested that I write something about my
approach to lay reviewing in health research.

Not so much a ‘how to’ guide as there are lots of these and very good
they are too. More, an insight into the questions I ask myself when
forming an opinion on an application for funding. Whether it be funding for a specific research project or a new research organisation.

For lay reviewing is an art, a matter of applying one’s judgement to
help a funder decide whether what is being proposed is not just going to work from a scientific point of view but from a patient and public one too.

This week I spent two days sitting on a panel for the Welsh Government
which was selecting their new Biomedical Research Centres and Units.
The lay reviewers were excellent. As was the way in which their review comments were listened to and respected by the panel Chair and other members.

So here’s my ten pointers or should I say pointers for ten!

Q1. Who will benefit from this research?
Has the team clearly set out how their proposal will addresses patient need and expressed what this need is? How do they know and how have they come to this view (see Q2 below). Normally I like to see some data here about numbers of people affected, limitations of current treatments and perhaps a sense of what this means to patients, their families and the health and social care system. Without this I think you are within your rights to ask
whether what is proposed is a patient priority? There may be other
valid reasons why the work should be undertaken but is that made
clear and the reason for not including patients and the public a sound one.

Q2. Have there been any pre-nuptials?
These days I would expect some evidence being put forward in the application that the team has already involved patients and the public to a) identify what they are doing is a priority, b) help design the project and it’s methodology and c) develop a public involvement plan that fits with the project. Time and again the evidence shows that public involvement falters during the course of a project if not enough time is given over to it at an early stage and, ideally, as the project is being put together. And there’s
lots of ways in which researchers can do this. Some funders like the
Wellcome Trust offer researchers small pots of money to develop plans around what they term ‘engagement’ and NIHR’s Research Design Service (RDS) does similar.

Q3. How much of the public involvement plan is in the ‘future perfect’ tense?
Let’s face it. This is a proposal so there’s going to be a lot
of ‘will-ing’ expressed. To an extent you can’t help but take some of
this on trust. But you should be look for some assurances that this trust is not misplaced. Clear statements about ‘how’ they will approach the task is one. Also, whether clear goals for public involvement are set out and a timeline for achieving these. At the very least, these are then things that can be monitored and tracked as the work progresses. You would also want to then see some involvement of patients and the public in this reporting and monitoring.

Q4. Will the public involvement operate under a glass ceiling?
Here we are really talking about issues around governance. Ask yourself howthis team or organisation will be making decisions? Then ask yourself
whether you can hear the patient voice in this? And will patients and
the public be able to raise concerns with the Board or leadership should they need to? Often the difficulty in public involvement is that when things go wrong it’s difficult to elevate the issue to a place where the issue
can be considered and decisions made. At the very least I would expect
patients and the public to be on the Board or equivalent forum. You might also want to ask yourself whether it is clear who is going to lead on public involvement within the team: a junior researchers or the Principal Investigator. That can be pretty telling on how important public involvement will be considered.

Q5. Is public involvement off-label?
I nearly called this ‘the dreaded advisory board.’ For it is the easiest mechanism for research teams or institutions to use as a place to put all those troublesome patients and the public and perhaps others as well. People that they are told they should work more closely with but don’t really want too. Don’t get me wrong, some of these boards work really well? The secret of their success – and the questions you should ask – are: where do they fit in with the governance? Who will chair them and sit on them? How are they supported? And what they will do? Even at the start of a project I would expect applicants to know the sorts of questions they need this group of ‘experts’ to address.

Q6. Is it a one-person operation?
A great friend and colleague of mine once said to me that when asked by people how they can find a way of not working with the usual suspects will say: ‘have you seen the unusual suspects?’ It is inevitable for the same names to crop up in people’s public involvement plans. Particularly if the applicants fall into the same condition area or part of the world. All I would say is that these days, and with the networks now available to people, there seems no excuse for researchers or research teams not to make an effort to source a wider view. Also, it concerns me when quite so much emphasis is put on one person not least because of the stress that can mean (see also Q8).

Q7. Is it charity or people they are after?
So we know how this one goes. Under the public involvement plan there is an eloquent description of the wonderful relationship people have formed with ‘x’ charity – ‘the leading voice of ‘x’ in the country.’ Of course, charities are very important in making research happen. They can bring patient voice but also important funds. But they can also be highly corporate in their view. The problems arise when this singular relationship with a charity is unaccommodating of other views. So, how will they reach the patient voice that might not be represented by the named charity?

Q8: Are they making a rod for their own back?
Let’s be realistic, no team or organisation or company for that matter has money to do everything? So, we should beware of the team that says it’s going to do 40 focus groups to write a patient leaflet as much as we should the team that says they will put it on the website for people to find and edit over a few days? Assuming we are content with the answers to Q1-7, what I really want to see now is a clear expression of the issues or questions that patients and the public will be asked to address and the methodology for doing this. But it seems important in this day and age that the plan feels appropriate, measured and a good use of money. To be honest I think you can tell quite quickly from an application – especially in its tone and style – when researchers and public colleagues have reached a state of mutuality in how they intend to work together.

Q9: ‘Is it a house built on sand?’
Nice research, shame about the budget. Actually, that should be: nice public involvement plan, shame about the budget. What you really want to see here is a clear presentation of the budget that will be provided to support public involvement. In particular; expenses and reimbursement of public involvement colleagues; the running of meetings and events and; learning and development. If it is not there, we have every right to ask why and where it is.

Q10. ‘How will patients and the public learn about this work when the results are out?’
These days everyone is asking about the impact of research. Quite rightly so. And most funders in their forms will ask teams to set out how they intend to disseminate the results of their research when it is completed. Cue two lines about publication in academic journals and presenting at international conferences. Or a line or two about building a website! But from a public involvement point of view neither really cut it. Unless things have changed vastly since I last looked, the public are not known for their readership of academic journals or attendance at scientific conferences. Websites also have their limitations. But there are a vast array of old and new tools in communications that are not expensive and remain better ways of reaching people – from articles in patient group newsletters to using social media. ‘Know thy audience’ is the motto here. Because if you know your audience you will likely know the places they are more likely to look for information.

So, there you go. It’s not intended to be exhaustive and i am sure many other questions could just as easily be included. In fact you have suggestions, please do add them as a comment to this piece.

One last point, And this is about behaviours.

It is simply that we all have our own way of doing things. In my book, politeness, decency and respect goes a long way. And I tend to think that the vast majority of people whose work lands in front of me are trying to do a difficult job in often trying circumstances. My role is therefore not to trip them up but to help where I can to make what they are doing even better.


So there I was this morning talking about peer review, when what should come through the Ovarian Cancer Action letterbox but our certificate of best practice from the Association of Medical Research Charities (AMRC) for our peer reivew practices.

It’s great to have this ‘quality-mark’ for all that we do as a research funder. If you want to read more about AMRC’s peer review audit process have a look here.

By the way I forgot to mention this article from The Guardian that covers the RAND Europe reports. AND you can vote on whether the peer review system should be changed or not.

I know some of you like to finish the week on a clinical trial so here’s a good local newspaper piece from East Lancashire about the trust there beating its targets for clinical trials. It’s also a good story about the role of the NIHR clinical research networks in helping to make it happen.

A few years back I recommended an NIHR report for pool-side summer reading. Ever since I have been inundated with requests to make similar recommendations each year. A little like Greta Garbo I have refused.

But not this year.

RAND Europe – who are a little like the McKinsey of research proffering consultancy and advice to everyone from the Department of Health and Wellcome Trust to smaller funders as well – brought out a rather interesting and curious report this past week entitled: ‘Alternatives to Peer Review in Research Project Funding.’

Interesting, because finding effective and efficient alternatives to peer review is often on the funders’ mind. Curious, because the report focuses on different ‘mechanisms’ and it is not clear to me that some of them fulfil what I have always understood the intentions of peer review to be about – namely the quality of the science.

I wondered also when reading it whether it is perhaps not alternative mechanisms we should always be looking for as opposed to additional participants such as patients and the public. That is why I was perhaps drawn to the ‘sand pit’ idea in there. I’ll let you read it and find out for yourself what that involves.

I shall add to this later today so you have something to read over the beach barbecue.

Tomorrow is Budget day.  Having just finished AMRC’s own budget and business plan for 2011-2012 all I can say is: George you know where I am if you want to crunch some last minute numbers?

The Financial Times is reporting today that the Chancellor of the Exchequer will announce a £100million cash injection for capital projects to ameliorate the effects on science of cut-backs announced by Pfizer and Novartis since the beginning of the year. 

Since the political message in the last few weeks has been about ‘reform’ I wonder whether we might also see some movement on the research regulation front.  Not a great deal since the word is it will take years rather than months to disassemble the current system before putting in back along the lines suggested by the Academy of Medical Sciences in its report earlier this year.  But something that suggests the Government is committed to speeding up the system and encouraging growth in the sector.  And if they don’t, then it will be a salutory lesson that given a choice between an octopus and myself to make predictions, always go for the octupus. 

Anyway, we shall be posting all the news from the budget here and on Becky’s fantastic policy blog – but visit here first please just so my stats are better than hers.

In other news, you may wish to look at AMRC’s submission to the House of Commons Science and Technology Select Committee inquiry into peer review published today.  As I have said many times before, the UK has a good story to tell on the commitment of research charities to using peer review to allocate their research funding.  Other written evidence submitted to the inquiry can be viewed on the committee’s website.

Plus, I should make mention of the final AMRC/UK National Stem Cell Network public engagement meeting – this time in York on 29th March 2011.  The event will feature the screening of the award winning film ‘Indestructible’ which documents the declining health of a young American motor neurone disease sufferer, including his trip to China for a stem cell therapy.  The film will be followed by a question and answer session with a panel of local experts.

Finally, a soft launch of an exciting new project that AMRC, together with the British Library, UKOLN at the University of Bath, Sage Bionetworks and the Digital Curation Centre and funded by JISC, will be undertaking over the next few months to investigate the potential of crowd- sourced “lay summaries”, derived from UK PubMed Central content, to enable the citizen- patient to better understand research.  The project now has its very own blog which will be populated with a great deal more information very, very soon.

As CEO of a membership association I am used to crowd-surfing.  So this is a new one on me and all the more exciting because of it.

A colleague in the office sent me this rather good blog on New Philanthropy Capital.  In it, Angela Kail talks about, or rather asks the question, why charities generally don’t do more to promote the real evidence of their effectiveness rather than rely – or allow others to rely on – anecdotes.

My belief is that research charities still do far too little to connect the two – hard evidence and the rich anecdote.   And by doing so we can often find ourselves falling short in the ‘selling game’ if I put it crudely like that.

AMRC and its member charities have a good story to tell about how we use peer review in terms of quality assurance, or to put it another way, ensuring that the sector funds science of the highest quality.  All AMRC’s members must abide by our principles of peer review and we audit how well they are doing every five years.  We are in the midst of this audit right now but you might wish to look at the results of our last audit on our website (see item for 2nd February 2007). 

…by the way some of you may have seen the announcement today that the House of Commons Science and Technology Select Committee is starting a new inquiry into ‘peer review’ and has called for written evidence to be submitted by 10th March….

But we have not been so good at telling the story of how the high quality science the sector funds is making a difference.  Indeed, this very issue, has cropped-up in every conversation I have had with members this week.  The basic line goes something like this:  We can demonstrate how we ensure we fund the best of the best; but following this through and being able to show impact is proving very difficult. 

My sense is that too often charities tend to separate the hard facts intended for a scientific audience, and the anecdotes that might give them greater meaning to a wider audience.  They think of reporting to their scientific colleagues and the public as two different activities.  They don’t think how the one can support if not enhance the other.  And they mistakenly make the assumption that what will interest one audience won’t interest another.  After all, most of us like a good story don’t we, particularly if it is true?

In simple terms it means that rather than say ‘We fund £x of research including x number of scientists at x laboratory..’ we should be saying ‘I am Simon and I am one of 12 scientists funded by x charity.  Their money enabled me to stay in science and devote mytime to trying to understand the causes of ‘x’ disease.’   Sorry, I know that is a poor example.  But I hope that you can see how the faceless, the purely numerical can become a person, an activity and a story that we can begin to visualise and very likely remember.

Last night I attended the BioIndustry Association’s annual where I heard a great example of this.  

Ahead of the evening’s silent auction, Neil Dickson, who founded the Samantha Dickson Brain Tumour Trust in honour of his daughter, told his story about the research the Trust has funded since 1996 so that it is now the leading funder of research into brain tumours in the UK.  That journey had seen Neil and his wife, Angela, travel far and wide.  Their latest trip had been to the leading international conference in the field which was held in Vienna.  Here, they found out that the UK was second-only to the US in the number of papers being presented and, of these, 60% was work that had been supported by the Trust.  It was a well-told story with an indisputable fact to support it.

Every fact can help tell a story.  A story without fact is merely that – a story.