Previously I have discussed what might be concluded from the ganzfeld experiments. In this post I will address a notoriously contentious issue: The quality of the evidence.
Practitioners and proponents of parapsychology often voice concern that their experiments are held to a different standard than experiments in ordinary science. To alleviate these concerns I will be scrupulous in applying the standards of medicine.
Medical experiments are in many ways similar to parapsychological ones. They are conducted on humans with all the messiness and limitations that comes with that. There are also some important differences, of course. I will discuss that as I go along.
Medicine faces more than any other science, the task of drawing conclusions from multiple studies, at least if the goal is to give patients truly evidence-based advice. Guidelines have been developed for that. Of course, these guidelines are themselves based on research on what methods are best suited to drawing correct conclusions and are still evolving.
I will mainly be relying on the Cochrane Handbook (v 5.1.0). Much of the handbook is not relevant to our purpose since we are not preparing a Cochrane review. The relevant material for this post is to be found in Part 2 of the handbook.
What is Quality of evidence?
It should be obvious what is meant by quality of evidence. It means how reliable it is. If it is of high quality then we probably won’t have to revise our judgement later on. If it is of low quality then we can’t be certain of that. The conclusion we draw from it might still be true but we would probably not be willing to bet the farm on it.
Let’s look at a practical example of how a medical trial might go wrong:
Experimenter: We have just given you a very expensive treatment. Do you feel any better?
Patient: I… don’t know?
Experimenter: It was really very expensive. I’m sure it must have worked. You do feel better, right?
Patient: Err…. Sure.
Clearly, if that were to happen in a medical trial, the results would be completely worthless for anyone outside of the marketing department. You probably already know the remedy to the problems here: Blinding and a control group.
The control group will typically receive a placebo. That is something that looks almost exactly like the real treatment but with the essential part removed, for example a pill without the drug. Often the control group will receive the old treatment instead because one cannot ethically let people go without treatment. Besides, one is usually more interested in whether a new treatment is better and should replace the current treatment rather than if it is better than nothing at all.
Blinding means concealing who belongs to the treatment and who to the control group. A double-blind trial is one where neither subjects nor experimenters know who is in what group. Which in turn means that they cannot be influenced by their personal preferences.
But hold on a moment. Just because someone is not blinded, does not mean that he or she will go to such lengths to influence the patient. Maybe the patient will not be influenced at all. If we know that a trial was unblinded we know that there was a risk of getting a wrong result. We don’t know if that actually happened, though.
Risk of bias
The Cochrane Collaboration talks about “risk of bias”. A bias in science is something that causes a result to be false. It has nothing to do with the personal motivations of the people involved.A completely impartial experimenter may still conduct a biased study while a biased experimenter may conduct an unbiased experiment.
The reason that the collaboration talks about risk of bias rather than quality is because in some cases it may be impossible to conduct a study with all the necessary safe-guards. In such cases it would be unfair to talk about a low quality study, but unfair or not, such a study would be still be at risk to yield a false results, ie to be biased. Another reason is that study quality is often regarded to be about more than just the reliability of the results. Other factors, such as properly following ethical guidelines also come into play.
According to the Collaboration’s suggested scheme, studies are assigned either a ‘Low risk’ of bias, ‘High risk’ of bias, or ‘Unclear risk’ of bias. Note that there is no category for no risk. This will probably come as a surprise to many, particularly fans of parapsychology. When mainstream scientists regard a parapsychological study as potentially flawed, that is not because of some prejudice toward parapsychology, it is because they apply the same standard as is applied to all studies.
The basic reason for that is that we can never rule out human error. We can never presume to know everything. Particularly, the risk of bias is assessed only by reading the report from the experimenters. Such reports are always idealized. If you ever read a scientific paper reporting an experimenter, you will almost never find mention of equipment malfunction or such mishaps. Of course, they do happen but every student learns not to report them. They are not supposed to whine about how hard the experiment was to get right or what personal drama they experienced. No one wants to hear that. Only events that might compromise the results are to be reported.
That leaves a lot to the judgement of the experimenters who have every reason not to mention things which might reflect badly on them or imply that they wasted their time.
By the way, the FDA does conduct on-site inspections to ensure that proper protocol is followed but that is only for some trials connected to drug licensing and the like. Regulatory bodies around the world will not accept anything but high quality evidence. Be aware, though, that alternative medicine is not held to these standards.
Bias vs. Conflict of interest
Personal motivations, the biases of the people involved, are not assessed.
Scientific journals, particularly in medicine, require researchers to declare financial conflicts of interest. That means mainly financial conflicts, such as when the researcher owns a considerable amount of stock in a company that will benefit from a certain result, or is employed by one, or funded by one. Financial benefits can be reaped even if the result is unmasked as false within a short time.
Other conflicts of interest such as personal friendships or family associations generally receive less attention. religious or ideological commitments are generally not considered at all.
Studies show indeed that, for example, trials financed by industry, are a little more likely to find a beneficial effect for the drug. Insofar, extra care is advised when conflicts of interest exist. If the majority of studies being looked at are industry-funded then one may even take this as evidence that publication bias is likely.
I am going to spare a few words on conflict of interest related to the ganzfeld and let that be enough.
Declaring conflicts of interest is not done in parapsychology. I don’t know why but I think it is not widely practiced in psychology either (do not mix up psychology with psychiatry or psychotherapy, though).
With regards to the ganzfeld experiment, there may be financial conflicts of interest related to funding. Parapsychological research is financed mostly by private grants and donations. I suspect that receiving such moneys is closely related to being able to obtain positive results. However, if or how much such factors correlate to positive results is, of course, unknown.
More serious are perhaps the religious or spiritual convictions of parapsychologists. I have previously analyzed what one can conclude from the ganzfeld experiments and found that it does not allow conclusions about the correctness of any world-view. Nevertheless parapsychologists argue this and insofar may be strongly committed to obtaining certain results.
Fans of parapsychology and parapsychologists often take it for granted that people will go to any length, even falsifying their research to uphold their world-view. Of course, that is an accusation only made at people who are not able to reproduce parapsychological results.
I, personally, do not know that anyone actually does that and rather suspect it to be rare outside of parapsychology. However, those accusations make it clear that some parapsychologists regard this as a completely obvious and natural thing to do. Which in turn makes me suspect that they themselves may be prone to such behavior.
Such considerations are not evidence-based, though, and as such I will not take them into account when assessing the evidence. One could point to past failures of parapsychology as a sort of indirect evidence for such problems and, of course, many people do that in practice when they consider parapsychology to be ridiculous. Eventually that is not different in principle from displaying knee-jerk suspicion of industry-funded studies.
To be continued…