Folklore #3: Blind Testing
But wait you say. Blind testing has a lot of applications. It is used in medicine. A more controlled test, the AB-X is even used in engineering. It is an accepted method. Surely I’m mistaken? I’d let you be the judge here.
Assuming we want to test the effectiveness of a drug, and have an group of willing guinea pigs (often poor university students). This is a classic double blind experiment:
- Split this group into 2 - the control, and the experimental group
- The control group gets the placebo
- The experimental group gets the drug
- The person who hands out the drugs have no idea if the group is of the control or experimental group
- The person who hands out the drugs have no idea if he is handing out placebo, or the drug
- The poor university student has no idea if he is in the control group, or the experimental.
Now consider a typical audiophile blind test “experiment":
- You have a group of audiophiles
- You have a item to test (e.g. a fancy power cable)
- You introduce a “control” (e.g. a normal power cable)
- You ask the group to correctly identify which cable is which
The latter experiment, if you can even call it that is utter pseudo science nonsense.
Spot the problems yet? We are not even talking about blind tests here. To perform a valid test, you are supposed to expose the subject to only one stimulus, that is the objective of this type of experiment - just one stimulus so there is no bias! Contrast this to the audiophile pseudo science experiment, exposing the subject to both stimulus. This is like administering to every test subject to both the placebo (control) and the drug (object of test).
How can anybody determine the effectiveness of any drug this way? Introducing both stimulus is a form of contamination and any observation made from such an experiment has no bearing or meaning.
The blind testing brigade has cleverly switched the idea of control from the human subjects to the item to be tested. Thereby confusing people that their viewpoint is valid. It’s far from that and their view points are based on pseudo science.
Every time you see the BT brigade cite blind test is used in scientific studies, know this:
- They have no idea what they are talking about
- They are only interested to win an argument
Walk away, do not even attempt and interact with this guy.
Comment from: A fellow audiophile Visitor
Comment from: Agent Kith Member
I don’t buy the argument that if others can’t hear the difference it must be because their equipment, hearing, or appreciation are not up to snuff. If you make the claim you should be willing and able to prove it if challenged.Proponents of blind testing should focus more time and attention to show the world how this discipline is applied in everyday hifi decision making, not just for winning arguments.
In the case of OS, the challenge is already issued. Try out Snakeoil OS and see for yourself if it’s better than what you have right now.
The thing is I cannot <em>talk</em> you or anybody into anything. e.g. my version of "better" is only for me and for me alone. My "better" cannot be forced upon you, all I can do is show to you what my version of "better" is. Whether you accept that is up to you, your comments and opinions is only personal to you. With Snakeoil OS I do encourage people to stray beyond the default settings and explore what their own "better" is. This is why Snakeoil OS is so openly configurable, and not locked down like almost everything out there.
Another reason why I even call this Snakeoil OS - the word itself automatically sets a audiophile up on a bias, trigger a flight or fight response. If you are willing to download the OS, willing to install the OS to test it out, willing to sit down for a minute and listen to the music. Those acts alone will set you apart from most audiophiles I know of. That’s the kind of fight response I’m looking for.
Arguing for blind testing typically on the forums is actually a flight response in my books, only meant to degrade the discussion into "winning" and "losing" and nothing more.
OK, I appreciate the effort you put into this argument but you are wrong. If someone claims, for instance, that one OS sounds better than another OS, then in a proper ABX listening test - where they are given the exact surroundings and all the time they need - I expect them to pick out the OS they said sounds better every time. Otherwise, don’t make the claim.
There are too many non-audio factors at play here to simply take these claims at face value:
I don’t buy the argument that if others can’t hear the difference it must be because their equipment, hearing, or appreciation are not up to snuff. If you make the claim you should be willing and able to prove it if challenged.