How Surveys Mislead Us

We often hear popular newspapers quote, "The risk of _____ disease increases ____ times for people who regularly eat/drink ____". It is not feasible to verify the veracity or futility of such claims. So what should we do? Well, it is safer not to confide in the so called expert findings. Statistical Surveys and Clinical Trials are inherently fragile instruments. While surveys depend on the participant's ability to recall, clinical trials on the other hand are impeded by availability of a good spectrum of human subjects and budget shortages. But what is more dangerous is the tacit manipulation of results to get predetermined outcomes which of cource are commercially motivated. Therefore it becomes very important to know whether the organisations that conducts the survey and compile the results have any personal stakes in the outcome or not. After getting some insider knowledge of surveying and statistics (my main area of interest) I feel a need to expose the ruthlessness with which businesses try to highjack public opinion.

Random sampling is a precursor to ensure a good survey but it is difficult to implement. This is further aggravated by the fact that trial companies deliberately compromise on quality of samples to honor costs and speed. The obvious issue with arbitrary sampling is misrepresentation. It means that the chosen sample fails to apply to the general public. For example, an online study to check whether Facebook is an indispensable part of public life may not give the desired results. Why? It is because not many people have computers with Internet connections (especially in third world countries) and with their absence a significant part of naysayers will be left out of the experiment. This experience may have good relevance in Western Nations where Internet penetration is high (nearly 80%) but it will be foolhardy to take its results on its face value it in Asia (where internet penetration is less than 30%). Similarly, a clinical trial in which all the participants were chosen from hospitals belonging to only New York City may not give results which can be used for patients throughout the US.

Without intending any harm to the genuine players, statistics are highly subject to manipulations. Considering the fact that very few business minded organizations would like to leave any stone unturned to see favorable outcomes, it becomes irresistible to to turn a blind eye to this weapon. There are two approaches in which statistics can be manipulated for personal gains - a priori and a posteriori. In a priori (before collecting data) approach the statisticians may not be involved but a posteriori approach needs statisticians. Let us see how they work. Here are two examples of some common tools under a priori approach adopted by survey companies:

• The Removal of undesirable sample - You may have come across a situation in which your online survey was terminated after a few questions because you didn't qualify for the survey. This is a normal thing and in 99% of the cases done with noble intentions. But in the rest 1% cases surveyors do that deliberately to ensure your exclusion because they identified your potential for negating the favorable result. Unbelievably tactful, isn't it? There's more.
• Use of misleading or confusing questions: The surveyors can choose to manipulate your answer by making the questions complex. Questions like "Do you support military intervention in Libya for peace?". Well, the respondent gets confused because he despises war but likes peace. He is more likely to say yes because he is expected to more so than he wants to.

A priori approach is delicate and hidden therefore more likely to be abused. On the other hand, a posteriori approach is a pure cook-it-up approach and needs all the ingredients before proceeding. It is done after surveys are completed. It can be misused to give the favorable results in the following manner:

• misplaced Causality - We commonly hear the harmful effects of alcohol on our lives. More often than not alcohol is deliberately made the scapegoat by false reporting. Suppose a survey to check the role of alcohol in causing cancer finds that the percentage of alcoholics in cancer patients is higher than general public. From this experiment it can not be concluded that alcohol increases risk to cancer. This is because alcohol is antidepressant and there is a good chance that cancer patient started drinking alcohol to manage the fear/stress of death. So, in actual cancer may be causing alcohol addiction. A way to get better results in this case is to ask cancer patients whether they started drinking alcohol before or after cancer.
• Discarding data -Well, this looks foolish and rude. But when the pressure to produce the desired result is so huge and all other options have been either unavailable or exhausted, this is the sure shot way to milk the Holy Cow.

There are numerous such weapons for surveyors to mislead us which may not be necessary to reproduce here. The message I want to convey is that surveys can be wrong. There are plenty of reasons for surveys to be misleading because of the intentions of surveyors or inherent shortcomings of the process. It would be naïve to take all the studies by their face value. It is more useful to be skeptical than obliging.