Recently the OSL prayer group at church was invited to take one of those tests that is supposed to identify what gifts of the Holy Spirit might be operating in your life. A list of about 100 questions covered a variety of activities and you were asked to grade your own interest on a five-point scale.
I’ve taken similar tests several times over the years and this project reminded me of an experiment I conducted once with a famous personality profile test. The boss of the company wanted us employees to take this test. I asked for four separate answer sheets. I went through the questions trying to respond honestly. Twice. That is, I filled out two of my answer sheets while taking the test as straight-forwardly as I could, one right after the other. (Even as I went along, I knew I was giving slightly different responses here and there simply because I felt differently about some of the questions the second time I came to them.)
Taking the test a third time I tried to fill out the answer form the way I thought the boss might do it. (I thought he and I were pretty different in our personalities.)
Then I took my fourth and last answer sheet and, not looking at the questions at all, I filled in every response at random.
When I turned in the four answer sheets to the secretary doing the grading I asked her to “find me.” I figured she had a 50-50 chance since with two of the answer sheets I had tried to play it straight. Surely, I thought, if this test is any good, the secretary knows me from our working together well enough to be able to identify which of my results showed the real me.
When this Gifts of the Holy Spirit evaluation test came along, I decided to repeat that experiment. But instead of asking for multiple answer sheets for the same test, I went online and found some similar Gifts tests provided by different Christian ministries. Each of the tests ran to around 100 questions, although some seemed to be sifting from a larger list of gifts than others. (For example, some of the tests probed for Spiritual Gifts like speaking in tongues but others did not.)
I was curious if all the tests would tend to highlight the same gifts each time for me. Given the slightly different purposes each of those tests had for the originating ministries, I was satisfied that there was a high correlation of results in my case. Certain Spiritual Gifts showed up on several of the tests and others didn’t show up at all. Because after all, this was science.
Except the bottom line was that all that the tests were measuring was my own opinion of myself, my tastes in favorite churchy activities. They might have been more worthwhile if the evaluations had been made by people who knew me well. (“I guess he’s okay when he helps little old ladies across the street – but what a boring teacher!!”)
So in the end the tests probably weren’t so scientific after all, even though we like to think scientific tools and methods can be trusted completely.
Now here comes Glenn Reynolds at InstaPundit with a story about research papers submitted to science journals for publication. The authors took some papers that had already been accepted and published by various science journals, changed the names of authors and schools named in the articles, and sent them back in for publication by the same magazines, 18-32 months after the initial appearance in those magazines. One or two editors recognized the hoax and tossed the articles back. Other editors sent the hoax-articles around to be reviewed again for publication. This time 89% of the judges turned the articles down, citing various objections.
This story comes from 1982.
So, here’s the question. How much can you trust test results? Because, you know.