Critical evaluation: Validity and bias

Last updated: Sunday, September 06, 2015

To evaluate validity we need to consider whether the research was done properly. All possible measures should be used to reduce the risk of bias in a clinical trial. The methods section of a paper should explain exactly what steps have been taken and the results should be fully and unambiguously reported. For simplicity, the following points describe a trial of placebo (control) versus drug treatment, but they also apply to trials that compare drug treatments (e.g. drug A versus drug B).

  • Selection of volunteers from a patient population should be random. This stops the researcher from choosing their preferred patient population, so affecting the outcome of their trial favourably.

  • Allocation of volunteers to placebo and treatment groups should be concealed from the researcher. This prevents the researcher choosing patients who they think will do best on their study treatment. For example, the early studies of diphtheria vaccine showed that more patients in the vaccine group died compared to placebo. This was because the sickest patients were chosen to receive the vaccine and the healthier patients were given a placebo.

  • Ideally as many people as possible involved in the trial should be 'blind' (or masked) to whether volunteers are receiving placebo or treatment. The opposite is ‘open-label’, when everyone knows what the volunteer is receiving. ‘Double-blind’ usually means the investigators and the volunteers do not know which arm of the study each volunteer is in, and ‘triple-blind’means the committee monitoring the data also do not know. However blinding is not always possible – for example with drugs that cause distinct side effects (e.g. peppermint oil capsules cause rectal burning) or if the treatment has a complicated dosage regime (e.g. warfarin dosed according to INR results). One way around this is to use something called a ‘PROBE’ design: prospective, randomised, open-label, blinded endpoint evaluation where the people doing the evaluation of the endpoints do not know which group the volunteers have been assigned to.

Courtesy of Simon Wills

  • Baseline characteristics of the groups under study should be as similar as possible. This helps to ensure that any effect seen in the treatment group is due to the treatment and not to pre-existing differences between the groups. The demographics of the groups should be described in the paper. If the baseline characteristics of the groups are very similar this can also be used as an indicator that allocation to groups was truly random.

  • Apart from the treatment or placebo, patients should be treated identically during the trial; they should receive the same number of blood tests, X-rays, and clinic appointments. 

  • Participant flow should be clearly reported, showing whether and why volunteers did not receive the treatment allocated, or were lost to follow-up or excluded after the trial had started. If this leads to imbalances between the groups it is known as attrition bias. It is important to know which and how many trial participants were included in the final analysis. If only those available for follow-up are included it is known as ‘on-treatment’ or ‘per protocol’ analysis. ‘Intention-to-treat’ analysis includes all participants who underwent randomisation in their originally allocated groups, no matter what happened during the trial. This is generally favoured because it reduces bias and is more like real life, where people change their minds, or change or stop treatments.