[sound of sighing]

April 30, 2012 at 11:40 am (Believers, Daryl Bem, Reason 101, Richard Wiseman, Skeptics)
Tags: , , , , , , , ,

The May issue of The Psychologist carries an article by Stuart Ritchie, Richard Wiseman and Chris French titled Replication, Replication, Replication plus some reactions to it. The Psychologist is the official monthly publication of The British Psychological Society. And the article is, of course, about the problems the 3 skeptics had in getting their failed replications published.

Yes, replication is important

That the importance of replications receives attention is good, of course. Depositories for failed experiments are important and have the potential to aid the scientific enterprise.

What is sad, however, is that the importance of proper methodology is largely overlooked. Even the 3 skeptics who should know all about the dangers of data-dredging cavalierly dismiss the issue with these words:

While many of these methodological problems are worrying, we don’t think any of them completely undermine what appears to be an impressive dataset.

But replication is still not the answer

I have written about how replication cannot be the whole answer before. In a nutshell, by cunning abuse of statistical methods it is possible to give any mundane and boring result the impression of showing some amazing, unheard of effect. That takes hardly any extra work but experimentally debunking the supposed effect is a huge effort. It takes more searching to be sure that something is not there than to simply find it. For statistical reasons, an experiment needs more subjects to “prove” the absence of an effect with the same confidence as finding it.
But there’s also that there might be some difference between the original experiment and the replication that explain the lack of effect. In this case it was claimed that maybe the 3 failed because they did not believe in the effect. It takes just seconds to make such a claim. Disproving it requires finding a “believer” who will again run an experiment with more subjects that the original.

Quoth the 3 skeptics:

Most obviously, we have only attempted to replicate one of Bem’s nine experiments; much work is yet to be done.

It should be blindingly obvious that science just can’t work like that.

There are a few voices that take a more sensible approach. Daniel Bor writes a little of how neuroimaging which has, or had, extreme problems with useless statistics might improve by foster greater expertise among the practitioners. Neuroimaging seems to have made methodological improvements. What social psychology needs is a drink of the same cup.

The difficulty of publishing and the crying of rivers

On the whole, I find the article by the 3 skeptics to be little more than a whine about how difficult it is to get published, hardly an unusual experience. The first journal refused because they don’t publish replications.
Top journals are supposed to make sure that the results they publish are worthwhile. Showing that people can see into the future is amazing, not being able to show that is not. Back in the day it was simply so that there was only a limited number of pages that could be stuffed into an issue, these days, with online publishing, there’s still the limited attention of readers.
The second journal refused to publish because one of the peer-reviewers, who happened to be Daryl Bem, requested further experiments to be done. That’s a perfectly normal thing and it’s also normal that researchers should be annoyed by what they see as a frivolous request.
In this case, one more experiment should have made sure that the failure to replicate wasn’t due to the beliefs of the experimenters. The original results published by Bem were almost certainly not due to chance. Looking for a reason for the different results is good science.

I’ve given a simple explanation for the obvious reason here. If the 3 skeptics are unwilling or unable to actually give such an explanation they are hardly in a position to complain.

Beware the literature

As a general rule, failed experiments have a harder time to get published than successful ones. That’s something of a problem because it means that information about what doesn’t work is lost to the larger community. When there is an interesting result in the older literature that seems not to have been followed up on then it probably is the case that it didn’t work after all. The original report was a fluke and the “debunking” was never much published. Of course, one can’t be sure if it was not maybe overlooked, which is a problem.
One must be aware that the scientific literature is not a complete record of all available scientific information. Failures will mostly live on in the memory of professors and will still be available to their ‘apprentices’ but it would be much more desirable if the information could be made available to all. With the internet, this possibility now exists and that discussion about such means is probably the most valuable result of the Bem affair so far.

1 Comment

Modern Mediumship Research: Negative

June 2, 2011 at 9:12 am (Believers, Mediumship, Richard Wiseman, Skeptics)

In this post I am going to deal with results that speak against the reality of mediumship. We begin with mediumship tests where a negative outcome was admitted.
For starters there’s a study by Jensen and Cardena. It involved only one medium, however. So there’s not really much we can conclude about mediumship in general.

A larger study was published by O’Keefe and Wiseman. It involved five mediums and 5 sitters (a jargon term for client). It discusses methodological issues in great detail and can be downloaded from Wiseman’s website. It is recommended reading for anyone thinking about putting together their own medium test. In fact, similar methods could be used to test someone who claims telepathy or just merely being a good judge of character.

One argument against such negative studies is that they simply didn’t have a genuine medium on their hands. This, however, is not a very common argument in my experience. Perhaps because it is uncomfortably close to accusing the tested mediums of fraud, or perhaps because it leaves the implication that there is

The other more common argument is that the conditions didn’t permit mediumship. Usually, some factor is said to have blocked the ability, such as the presence of skeptics. This has by now turned into a muchly ridiculed cliché.
Others say it was simply that the unfamiliar surroundings of the lab and strict limitations of the protocol made the mediums uncomfortable. This has been compared (by males, of course) to the inability of people to have sex under such conditions. I find it amazing that in this day and age there are still men who are unaware of the existence of porn.
As a remedy, Julie Beischel of the Windbridge Institute argued that O’Keefe and Wiseman should have tested their mediums before testing them to ensure that they can pass the test. It sounds better if you call the pretest a “screening”.
Some more cerebral believers argue that the conditions for paranormal just so happen to be the same as for error or fraud (though they will, of course, not put it so bluntly). Such an argument is implicit in Kelly and Arcangel’s paper when they talk about “priming the pump”.

Challenges

Perhaps one might include skeptic challenges as another negative piece of evidence. These challenges are meant to call out people who claim abilities they don’t have. The most famous of these, and the blue-print for most others, is James Randi’s Million Dollar Challenge.
The rules are both simple and scrupulously fair. Whoever wishes to take up the challenge needs not only claim some ability but design himself a task, impossible to ordinary people. So rather than just claim mediumship, one would need to say, for example, ‘I can tell if someone’s parents are alive or dead.’ Of course, the conditions would have to be so as to preclude ordinary means of inference, however unlikely. And it would have to be stipulated how many tries there would be and how many would have to be successful.
Basically, one has to convince James Randi and his advisors that there is no way to do the promised feat with ordinary means. Once they accept this, the claimant has a legal contract that stipulates payout on success.

So far, the money has been quite safe.

The arguments against the challenge are numerous but ill-informed. Some think that whether or not someone wins is a judgement call by James Randi. In fact, the rules explicitly forbid any judgement calls. Success or failure must be obvious to anyone.
Others accuse Randi of dishonesty. They say he only allows weak claimants or even that he manipulates the result. This is some rather outrageous slander but even so, there are other challenges. In any case, one can also contact parapsychologists. Yet by all appearances, they don’t have any potential winners either
Naively, one would expect something like Randi’s challenge to be treated as the X-Prize of parapsychology. But what if there are no humans with super-powers? Then those who claim them would be frauds or lunatics. The reaction to the offer of a million dollars certainly does nothing to dissuade one from considering that possibility.

Still, it is true that just because no one has risen to the challenge, this does not disprove mediumship. One might claim, for example, that a true medumistic gift always comes with extreme shyness.

Science?

Finally, there is also the fact that what science has revealed over the last centuries about the world, about us and about our place in the world is rather inconsistent with mediumship. Not so long ago science believed that there was a life-force that distinguished living from non-living matter. Today this is considered refuted.
Something similar is true for the human mind, which today is considered to be the workings of the brain.

Not only has science not uncovered support for mediumship, it has refuted some of its core tenets.

In regards to this some will point out that science has been wrong before. This is quite true and, indeed, science is nothing if not a quest to be ever less wrong. But here we have a case where this quest simply leads away from the proffered answer.
There are even many examples where scientists held onto ideas that should have been recognized as discredited by the evidence. Who is the dogmatic hold-out in this story?

Still, the fact remains that while science as a whole offers strong arguments against the possibility of mediumship, it cannot be ruled out. Science could be wrong and that is a simple, unalterable fact.
If the evidence indicated that it was real then one would have to find a way to reconcile it with other seemingly contradictory evidence.
However, I have never so far come across a serious attempt at this. There are plenty of demands that mainstream science should take parapsychology seriously. Demands that parapsychology take mainstream science seriously are usually met with the charge of close-mindedness.

The Universal Negative

Eventually, what all this boils down to is quite simply that you can’t prove a universal negative. You can’t test everyone and even when you test someone it only shows that they failed in this specific test on this one occasion.

1 Comment

The Saga of Rupert Sheldrake and the Psychic Dog

April 22, 2011 at 1:57 pm (Believers, Richard Wiseman, Rupert Sheldrake)
Tags: ,

The saga starts  in 1994 with a book with the not-quite-modes promise in its title Seven experiments that could change the world.

Sheldrake relates how some pet owners think that their pets can tell when they are coming home, even if that should be impossible. He believes that there is a telepathic link between pet and owner. One of these seven world-changing experiments is to show this behavior.

Three Surveys

The saga continues with three surveys of pet owners in England and California, published in 1997/98. About 50% of dog owners said their pet anticipated the return of a family member and 30% of cat owners. Almost 20% of dog owners said that this behavior started more than 10 minutes before the person’s arrival.

Jaytee

Meanwhile a specific dog by the name of Jaytee was the center of an exhaustive investigation. Jaytee’s owner, Pam Smith, and her parents had noticed as early as 1991 that Jaytee was anticipating Pam’s return. They put this down to routine as she returned home from work at always the same time. However, the behavior seemed to persist even after Pam was laid off in 1993 and no longer followed a set routine.
Pam Smith learned of Rupert Sheldrake’s interest in psychic pets in April 1994 from a newspaper article. She volunteered for an experiment. In the following month her parents began taking notes.
The first observations seemed promising so the notes got more detailed and eventually lead to several specific tests. A few had Pam return by an unusual mode of transport so that the dog would not hear the familiar car. In two tests the return time was determined by coin toss.
There was also a test by austrian state television (ORF) for a documentary.

What Jaytee could do

Based on these observations and tests Sheldrake argued that Jaytee reacted whenever Pam Smith decided to journey home.
Sometimes the dog reacted before Pam Smith started journeying home but Sheldrake said that, in fact, this was because Jaytee had reacted when Pam prepared to travel home, rather than when the journey actually started. When the dog reacted late this might have been because the parents had not been paying attention and simply missed the proper time, so that the dog only seemed to have been late.
For some failures other reasons were found such as distractions outside (like a bitch on heat) or the dog being ill.

Such arguments to explain failures away may not be too convincing but Sheldrake could also point to some successes. Yet those successes relied greatly on the reliability of Pam Smith’ parents as unbiased observers, or in one case, of a film crew.

Videotaped experiments

The next step was to videotape the whole thing. The camera was trained on a certain spot, in front of a window. Going there and looking out was, according to the Smiths, how Jaytee anticipated his owner.
This would take place in several locations. On 30 occasions, Jaytee was left with Pam’s parents, as before. Five times, he was left with Pam’s sister, and 50 times he was left alone.
When Jaytee was with Pam’s sister at her place he spent altogether less time at the window but Sheldrake describes his behavior as being similar to when he was at the parents’ place.
There were also 50 such observations where Jaytee was alone. There he usually did not go to the window at all. Only in 15 cases he showed his usual response. However, no graphs, or other information, are given to support this statement.

Jaytee’s behavior when he was with Pam’s parents is shown here:

Graph from Sheldrake 1998

The 30 trials were separated according to how long Pam Smith was absent.
Each step on the x-axis represents a 10 minute (600 second) period. The y-axis tells us how many seconds of these
10 minutes, Jaytee spent at the window. The filled circle/square indicates the first 10 minutes of Pam’s return journey.  The lower line, marked with squares excludes 7 observations where Jaytee spent especially much time at the window before Pam returned, I won’t be using it.

But what does that mean?
For one thing it clearly contradicts Sheldrake’s earlier conclusion. Jaytee does not suddenly go to the window and wait there as soon as Pam starts returning. He simply spent more and more time there.

It does seem as if he had a rough idea of when Pam would return and behaved accordingly but maybe he was merely reacting to the parents anticipation. Even though she should not have told them they may have been able to guess from clues like what she took along. Or indeed, Jaytee may guessed himself.
I must admit, though, that I am not entirely certain if that may not be simply a statistical illusion.

Weird!

Now things get seriously weird. A normal person, or at least a normal scientist, faced with that data would now seriously reevaluate his assumptions. Maybe when the parents took the notes, they were picking up some different, more subtle clues from the dog. Maybe just looking at when the dog goes to a certain spot is not good enough.
Or maybe the telepathic link was between Pam and her parents in the first place.

That’s however not what Sheldrake does. Sheldrake argues that the data confirms his idea. Jaytee spends the most time at the window right before Pam returns therefor he’s psychic. That’s the argument. No kidding.
It gets worse.
Yes.
Really.

He is aware that if the dog goes to the window more and more this will also have him at the window most when Pam returns. And this is why he produced that graph. I took it right from his paper. And, you see, it shows how Jaytee did not go to the window more and more. You don’t see? Good for you.
His argument is simply wrong but for the morbidly curious here it is: He compares the short, medium and long absences. For example, when Pam returned after 80 minutes (short) the dog spent an average of about 300 of the last 600 seconds (10 minutes) at the window. But after 80 minutes in the medium and long absences, he only spent about 100 or 50 seconds there respectively.

That’s true, and as I said, might indicate that the dog knew something. It just tells us nothing about whether the dog really did go to the window more and more. It is obvious from the graph anyway but if you wanted to test that mathematically you would use a so-called linear regression. Based on an off-hand remark in a different section this seems to have been done (by Dean Radin) with expected results but not included.

This may seem like the end but the saga is not finished yet.

Randomization

There is one final experiment to be done. By determining a return time for Pam at random and only communicating it to her once she is on her way, we can make sure that Jaytee has no clue when she is going to return. Sheldrake performed 12 such experiments that naturally showed Jaytee being at the window most right before Pam’s return. He still thinks that this indicates telepathy, in complete defiance of the facts and any rational argument.

Richard Wiseman

When you hear this saga related elsewhere you will always hear of Richard Wiseman as well. Wiseman is a British psychologist with a well-known skeptical interest in the paranormal. The seemingly stunning performance of Jaytee that was filmed by the austrian television crew lead him to contact Sheldrake. Pam Smith graciously agreed to take part in his experiment and Sheldrake allowed him to use the same video-camera.
Wiseman, with the assistance of two colleagues, Matthew Smith and Julie Milton, performed four experiments.

Since Sheldrake had already done all the preliminary groundwork, Wiseman could jump right in. The dog was supposed to do a certain thing, that is, go to the spot at the window right when Pam Smith was about to return. He would simply test if that was the case. Wiseman would stay with the dog, filming him. Smith would go with Pam and tell her to return at the appointed time.
If the dog went to the window in the same 10-minute time frame as the return, the test would be a success.
As we would expect, the dog was much too early.
However, the dog stayed there only a brief moment, maybe because of some distraction outside. It was decided to try again but this time the dog would have to stay at the window for a full two minutes.
Same thing again, of course.
So it was decided to wait until winter with the next try, when there would be fewer distraction outside.
Yet again too early.
In the fourth experiment the dog didn’t ‘signal’ at all.

Of course, Jaytee’s pattern of going to the window more and more is present in this data as well. By Sheldrake’s twisted logic this means that Wiseman found evidence of telepathy. This is where the saga takes an unsavory turn.
Wiseman has bluntly stated that he failed to find evidence of Jaytee being psychic, moreover he finds Sheldrake’s own data unconvincing. To Sheldrake this is an outrage.
When Wiseman confirmed that he agreed that his data showed the same pattern as Sheldrake’s this was to Sheldrake an admission that Wiseman had found telepathy. To Sheldrake, Wiseman is simply being dogmatic and irrational in not saying so.
It may seem hard to believe that anyone could read through Sheldrake’s work and not see the foolishness in his logic but it isn’t just fans of Rupert Sheldrake who uncritically accept his twisted reality. It is also authors, such as Chris Carter and Robert McLuhan as well, who pride themselves on having investigated such issues. This has, by now, turned into a character assassination campaign against Wiseman.
I must add that Wiseman himself has largely ignored this and never criticized Sheldrake for his irrationality. He has only expressed disagreement and laid out his arguments.

Kane
There was also a small number of tests with another dog called Kane. His pattern seems to have been slightly different but there were only very few tests. That makes it virtually impossible to say anything with confidence.

Conclusion?

You may now think that all this psychic dog business is completely debunked. Well, in a way.
We have seen how Sheldrake’s original hypothesis seemingly collapsed with more stringent tests but one could claim that this was due to error on the part of the scientists.
Wiseman was with the dog, filming him, did that throw the dog off? Sheldrake switched to a different, nonsensical statistical analysis which may cover up evidence.

And even if Sheldrake’s hypothesis about how the dog expresses telepathy is completely wrong, the dog may still be telepathic but just expressing it in a different way.
There are any number of reasons why the tests would have failed to find telepathy.

Is there anything we could interpret as possibly evidence of telepathy?
When we look at the twelve highest quality experiments, those with videotape and random return tim, we find that in four of these Jaytee only went to the window when Pam was on her way home. Not any sooner Maybe that means something?
On the other hand, that only happened when the return time was very early. When the return time was late, he was always too soon (except in one case when he did not do anything at all). That makes it seem much less interesting. It makes it look like the “hits” depend more on the random time being just right than anything else.
The case for telepathy can be strengthened again by subtracting those times where there was some identifiable distraction that may have caused Jaytee to go to the window. Two tests then turn from failures to successes.
But how reliable is such a retrospective judgement? A worrying detail is that the graphs that were published in the parapsychological literature and those contained in Sheldrake’s books show slight differences.
Also there are Wiseman’s results which were all clear failures by this standard.

There’s another issue and it’s the most important one. These few cases that might be telepathy are the result of me going over the results in detail, searching for anything that, at least, doesn’t contradict telepathy. I had to completely ignore Sheldrake’s argument which is simply wrong.
I also had to ignore that the failed tests suggest that this was just chance.

That makes the whole evidence not very convincing. We’d need additional tests to determine if this idea stands up.

The question is, how much effort do we put in before giving up?

Most people, surely most scientists, would look at the track record of telepathy claims. Perhaps they would also look at Sheldrake’s track record who had a well-deserved reputation for irrationality, well before this episode. Based on that they would dismiss the whole thing from the start.
Wiseman gave it more of a chance than most would. His fate may hold something of an answer to those who wonder why people aren’t more open-minded.

How much effort would you personally expend?

Sheldrake thinks he has good evidence of telepathy in dogs. And yet he, too, has given up on research. One would think that by finding a telepathic dog the science would only begin. One would think that your average scientist would continue by uncovering the physiological basis for it.

If one wanted to pick up the work that Sheldrake dropped one would first have to find a psychic dog. Going by Sheldrake’s surveys, this should be easy, if people don’t fool themselves about their pets being telepathic.
There was one person who tried this. A former high-tech entrepreneur turned podcaster by the name of Alex Tsakiris. He put in quite some effort and money.
His plan was to turn the project over to professional scientists once he had found some suitable dogs but it never happened. He found candidates that seemed promising to him but nothing worked out. Eventually he quietly abandoned the project.

So here’s my personal conclusion: I am going to live my life as if there is no such thing as a psychic pet or telepathic dog or whatever. I am also going to be highly doubtful about anything coming from Rupert Sheldrake.
You draw your own conclusion.

Sources:
Papers by Rupert Sheldrake
Papers by Richard Wiseman
Dogs that know by Alex Tsakiris

5 Comments

That Wiseman Quote

April 13, 2011 at 6:26 pm (Richard Wiseman, Skeptics)

Richard Wiseman is a british psychologist known for his pop-sci books as well as his skeptical interest in paranormal claims. The Daily Mail quotes him thusly:

“I agree that by the standards of any other area of science that remote viewing is proven, but begs the question: do we need higher standards of evidence when we study the paranormal? I think we do.

“If I said that there is a red car outside my house, you would probably believe me.

“But if I said that a UFO had just landed, you’d probably want a lot more evidence.

“Because remote viewing is such an outlandish claim that will revolutionise the world, we need overwhelming evidence before we draw any conclusions. Right now we don’t have that evidence.”

This is frequently quoted by believers in the paranormal as support for their position. The spin is that Wiseman, the skeptic, admits that something paranormal is proven but then resorts to a double standard to deny this.

Is there any truth to this?

Those who know Wiseman will know he is usually a rather rational person. Those who know the Daily Mail will know that it is not the most sober or reliable newspaper, that is to say a british tabloid. One suspects that the quote was simply mangled beyond recognition.

Surely remote viewing is disproven by any normal standard! Besides what’s that talk about different standards in different areas of science? I think we can infer that he is not talking about having different standards in different sciences but rather for different claims. Also, he’s misusing the phrase begs the question.

And finally, I assume that when he talks about “revolutionising the world” he talks about the scientific world. IE he means that this would uncover glaring and massive holes and/or errors in our understanding rather than that it simply would unlock new technologies.

New technologies, of course, don’t require extraordinary evidence, they are extraordinary evidence. Everyone can test if they work. Those who employ new, effective techniques profit, the rest gets left behind.

A misquote?

The misquotation hypothesis receives a partial confirmation on another blog where Wiseman clarifies thusly:

“It is a slight misquote, because I was using the term in the more general sense of ESP — that is, I was not talking about remote viewing per se, but rather Ganzfeld, etc as well.  I think that they do meet the usual standards for a normal claim, but are not convincing enough for an extraordinary claim.”

So he is not talking about remote viewing but instead about something else. Unfortunately it is quite unclear what.

It doesn’t look like there is any real clarification from Wiseman forthcoming. I don’t know what he really meant to say and who was responsible for mangling it but I will critique it anyways.

“Extraordinary claims require extraordinary evidence” is a truism that I have previously justified. An extraordinary claim is one that is unlikely to be true. And this likelihood is something we judge from our previous knowledge. Taking this into account we could render the Wiseman quote like so:

“If ESP had still more evidence going for it, it would be proven.”

Or as:

“ESP would be proven if there was not also a lot of counter-evidence.”

Both statements are true but not very sensible.

Wiseman, according to a JREF thread, specifically mentions psychology as a realm where normal claims have as much evidence as whatever he was talking about. The exchange between the poster and Wiseman went so:

Kuko 4000:
The existing RV database does not convince you, ok. But at the same time you seem to say that RV has been proven by scientific standards, now I’m confused. I would really appreciate a “clarification-for-dummies”, so to speak
This could be an issue with my understanding of science or it could be a matter of language barrier, but I’m having problems getting my head around this. Do you mean that by the standards of any other area of science, say biology, evidence of similar quality would be considered scientifically convincing? If so, could you please direct me to the research so I could look it up for myself?
Richard:
yes, it is different standards for different types of claims
so, a normal scientific claims requires a certain level of proof, but a paranormal one requires a higher level
Kuko 4000:
Could you give me an example of a normal scientific claim that in your view offers the same level of proof as the best available evidence for Remote Viewing? This way I could understand the comparison much better.
Richard:
most of psychology!

So let’s look at a psychological effect and contrast it to telepathy.

Normal vs. Extraordinary effect

For a not quite random example let’s take priming. For example, test subjects are given a list of words to read that contains the word carpet. When they are later given the beginning of words they are more like to complete C-A-R to carpet than otherwise.  An other example might be dropping the word ‘yellow’ and then finding that subjects are more likely to mention ‘banana’ when asked about fruits.

That’s not a particularly exciting effect. We already know that humans have memory, that practice helps, etc… We also know that much mental processing is unconscious. Priming is a specific effect of unconscious memory (properly called implicit memory).

Establishing priming is only showing a particular behavior of something that undoubtedly exists. I can’t think of any reason why it should be so but neither of any reason why it shouldn’t. Then again I’m not a psychologist.

In a telepathy test one will have at least two participants between who any normal communication is (supposedly) impossible. Then the experimenter will employ some method to show that communication actually happens. This is where experiments differ.

Parapsychologists argue that if they gain evidence that communication happens while it should be impossible, this must be telepathy. There are problems with that logic but that isn’t the point of this post.

The difference between priming and telepathy should be clear. One has a firm basis, the other, by definition, has none. That’s not even mentioning that people have been trying to establish telepathy for well over a century without managing to convince more than a tiny handful of the validity of the phenomenon.

Bottom line?

I think that Wiseman’s heart is in the right place but the quote is nonsense and those who criticise him for it are justified in doing so. It’s not the only time that he has said something that made little sense to me but it probably happens out of a desire not to call the emperor naked but rather to say something nice.

2 Comments