This post will examine the relatively extensive experiments conducted by Tricia Robertson and Archie Roy. In some ways, these are the best I know of. But I’m getting ahead of myself, for they were of to a rather shaky start.
Archie Roy is Professor Emeritus of Astronomy from Glasgow University (Emeritus means that he is retired). He has had an interest in the paranormal for several decades and was president of the Society for Psychical Research (SPR) from 1993-1995. Tricia Robertson is a past president of the Scottish SPR (SSPR) but I don’t know that she has mainstream qualifications.
The First Article
In 2001 they published the first paper that will interest us here. Interestingly, the results are presented not as evidence of mediums but as a test of “the sceptical hypothesis that the statements made by mediums to recipients are so general that they could as readily be accepted by non-recipients.“
To do so they conducted a “two-year study involving 10 mediums, 44 recipients and 407 non-recipients “. That’s really awesome in many ways. They really spent time and effort on this and they don’t even claim proof of mediumship. Compare that to the junk that Gary Schwartz has produced!
Unfortunately, despite all that the study is scientifically worthless. A complete waste of time and effort.
The first problem is obviously that no half-way knowledgable skeptic actually claims that. The second problem is that the experiment was carried out so badly that no conclusions are possible. It is indeed a maximally shaky start.
What they did
They assembled an audience and then had a medium perform a reading for one of the people in that audience. The reading was then broken down into individual items. All subjects (the entire audience), including the intended recipient of the reading, then rated these items as applicable to themselves or not.
This then allows them to determine if the recipient endorsed more items as applicable then the non-recipients. Which would mean that they found the reading as a whole more applicable. It also allows them to deduce how specific an item was. If many people say that an items applies to them it is quite general, if few, or just one, say so then it is specific.
They develop a fairly complicated method to statistically evaluate the readings based on both the number and the specificity of the accepted statements but we needn’t concern ourselves with the details now.
The First Problem
Now we face the first problem. What they test is if the entire reading is so general that it could be accepted by anyone. Yet, if we actually look at the literature on how to fake such readings we learn that this is only part of the art. An accomplished “faker” (actually, many mentalists, like magicians, are quite upfront about their use of trickery) will use any means at his or her disposal to tailor his or her statements specifically to the client.
But all right. Just because no one actually believes a certain hypothesis to be true, does not mean that it shouldn’t be tested. It just means that falsifying it is as exciting as finding that water is not dry.
This gets us to problem number two.
The Second Problem
Now it’s time to ask ourselves what results we would expect from the experiment, depending on whether the hypothesis is true or not.
If the medium makes statements that are true only for the recipient and for few or no others then we would find that the intended recipients accept more statements than non-recipients. Obvious.
But what if the medium only makes statements that are true for most or every one of the subjects?
We would expect exactly the same result. This is counter-intuitive but supported by ample psychological research. Such phenomena are called Barnum effect, illusory correlation, confirmation bias and a number of other names. It’s not necessary to go into detail now. Suffice it to say that Robertson and Roy will be able to confirm this expectation in a later experiment.
This simply means that the experiment did not test the hypothesis it was supposed to test. In fact, I don’t think any conclusion can be drawn from the data so collected. 2 years of research, 451 participants and all for nothing.
The Second Article
Others would have dug in, insisted on their worthless research and ranted about closed-minded skeptics. Robertson and Roy set out to make amends. A few months later they published a protocol for a new experiment that was supposed to demonstrate mediumship under conditions eliminating all ordinary, skeptical explanations.
Publishing the protocol in advance enabled them to take criticism into account before carrying out the experiments. Indeed, the experimental design found in their 3rd and last paper was improved over this one.
Basically, it is quite similar to the one in the first article except that the audience doesn’t learn for who the reading is and the medium performs in a separate room so that he or she does not get any usable clues.
I won’t bore you, dear readers, with the minutiae but as far as I can tell, in this final, rigorous protocol, skeptics would not expect the intended recipient to accept more statements in a reading than anyone else.
The Third Article
The third and final article in the series was published in 2004. It presents data collected in 13 sessions that took place over 2 and a half years. They involved some 300 participants and 10 mediums giving 73 readings.
However, few of the experiments conducted actually followed the rigorous protocol they took so much care designing. Ostensibly the reason for this is to assess other factors that may influence subject’s acceptance or rejection of statements but there is not much in the way of analysis of such factors. There’s not even a complete table with results to allow readers to perform their own analyses. But I’m getting ahead of myself.
One thing that they did find was that misleading the subjects on who the intended recipient is, really does affect the results. As I’ve said previously, psychology leads us to expect that someone who believes that a reading is intended for him or her will rate it as more applicable than someone who believes it is for someone else. This expectation was found to be true, confirming that the experiment presented in the first article was indeed a waste of time.
Despite the ample evidence that psychology has amassed that lead us to expect this, it is good to have it confirmed. The situation is not exactly the same as in the standard experiments demonstrating the relevant cognitive biases. This leaves a slight possibility that these biases are not operating under the conditions in which Robertson and Roy studied mediumship.
Now what about the results to the rigorous protocol. That’s, after all, the supposed main point of the exercise. I’m not sure how to put this but… they aren’t there. Yes. Seriously.
We never learn how mediums did under conditions precluding normal explanations.
All right, this needs some more explanation.
There is a lot of analysis presented in the article but it’s almost all completely pointless filler. The results come from experiments that differed in crucial design aspects. In principle, this allows one to determine if a variable factor correlates with greater mediumistic success. However, such analyses are almost completely absent with the notable exception mentioned.
Instead the results from these different experimental conditions are pooled without any reason being given or even inferable. Such pooled results are also compared with other pooled results, even though there is nothing to be learned from that. It’s just weird and bizarre. I can only conclude they don’t really have anything to report that they want to report and so stuff the paper with filler.
The reviewers seriously failed in not demanding that the analyses be cleaned up and all the results reported.
An Abstract Untruth
So where does the supposed main result come from that they report in the abstract?
Due to the design of the experiments the results cannot be due to normal factors such as body language and verbal response. The probability that the results are due to chance is one in a million.
In context one would be lead to believe that these results are actually the results of the rigorous protocol that was so much talked about. One would be wrong.
Indeed these results come from pooling data from several different experimental conditions. That includes the rigorous protocol but also data from experiments where normal factors operate. For one, the bias of the subjects introduced by whether they belief the reading to be for them or not. There are also other factors which might conceivably have influenced the result but one is enough.
To make this clear, if I have results that indicate that something is going on and I combine them with other data then the combination should also indicate something going on because that’s the case, even if there’s nothing in the added data.
In a very strict sense the quoted statement is true. The mentioned factors do not operate, different factors do. It is merely completely misleading in context.
Clearly Robertson and Roy confirmed one skeptical explanation despite their protestations. Moreover, all results that they report have ordinary, conventional explanations. Speculating about paranormal effects is not justified.
I personally believe that the results of the rigorous protocol must have been negative or else they would surely have been reported.
As such, I consider Robertson and Roy’s work the best evidence against mediumship obtained in modern times.
I also think that the misleading nature of their last article was not intentional. I think they just couldn’t face up to the results. Although, the fact that Tricia Robertson has threatened to sue people who claim that they fixed or rigged the result does not speak of a clean conscience. Not that I see why anyone would claim that, given that the available results are so ordinary.
Robertson, T. J. and Roy, A. E. (2001) A preliminary study of the acceptance by non-recipients of mediums’ statements to recipients. JSPR Vol 65.2
Roy, A. E. and Robertson, T. J. (2001) A double-blind procedure for assessing the relevance of a medium’s statements to a recipient. JSPR 65.3
Robertson, T. J. and Roy, A. E. (2004) Results of the application of the Robertson-Roy Protocol to a series of experiments with mediums and participants. JSPR 68.1
JSPR, Journal of the Society for Psychical Research, can be accessed via Lexscien. Skeptical minds, interested in double-checking my work, can do so for free by signing up for a free trial.
Tricia Robertson on critics: “I am aware that critics will say the tests were somehow rigged. But, rest assured, we could not have been more scientific in the way this was carried out. If anyone claims it is fixed or rigged, we would sue.”