Define “Works”

Some say that statistics lie.  Personally, I’ve never met a dishonest statistic.  I have however met dishonest people who misuse statistics.  There are people who misread them.

Unless you want to be the person who is spending $50.00 a month on the latest supplement or therapy program, take some time to learn about controls.  Let’s face it, if all those miracle treatments worked, people wouldn’t have drawers full of fads that did not work.  Same goes for dog training miracle treatments.

Let’s have some fun with statistics by creating some really basic fake study results.

Here is our study:

Scientists told a bunch of people to eat strawberries three times a day.  They want to know if strawberries help people with upset stomach feel better.

You look in a magazine and see that strawberries offer clear benefits to people with upset stomach.  Within the article is a diagram – a graph.  It looks something like this:

Impressive looking results.  Forty percent of people felt better when compared to people who did nothing to change their diet.  Oddly, some people who did nothing felt better too.  Maybe they had the flu and it cleared up on its own.  It certainly looks like we should be eating strawberries with every meal.

Another researcher repeated the same study with one change.  They found similar results, so we have the beginning of a possible trend.

Forty percent of people felt better eating strawberries.  This time the researcher compared junk food eaters to the newly named Strawberry Miracle Diet.

Seriously, how is that revolutionary?  Comparing real food to junk and candy is an absurd idea.  We know junk food is bad.  Comparing to a bad control obviously makes the strawberries look good.  Overall, it doesn’t prove a “Miracle” strawberry diet works.

Researcher number three is skeptical and their study has yet another chart.

This researcher did something interesting.  They created a placebo group.  This means that some of the people ate strawberries.  The other half were given a sugar pill.  A placebo is a treatment with NO medical effect.  Comparing strawberries to a treatment with no benefit helps weed out the placebo effect – when people truly think something is helping but it’s not.  Yes, our quirky human brains are funny that way.

Researcher number three has provided clear evidence that strawberries are no better than sugar pills – a treatment known to do NOTHING.  Suddenly the Strawberry Miracle Diet looks a bit hinky.

Some might say, “Who cares, as long as people feel better?”  In certain cases, certainly this is true.  For example, people in severe pain might be given a placebo if they have reached the maximum dosage on pain medication.

Exceptions aside, an insidious danger lurks.  One more chart to illustrate what could potentially happen.

Another researcher compared a variety of fruits to each other..  Most fruits made 40% of people feel better – except pineapple.  Pineapple stands out, with almost 80% of people feeling better.  That is a significant difference.

Who knows why pineapple is superior.  We need more research.  Perhaps it’s all the digestive enzymes.  Pineapple is better than placebo.  Pineapple is MUCH better than strawberries.  Pineapple actually seems to offer some REAL benefit.  As real as a fake study can be.

For numbers to be meaningful, you need a point of reference for comparison.  For example, a horse is big compared to a mouse.  A horse is small compared to an elephant.  The horse’s size does not change.  However, you only get a real indication of size if you compare to other things.

Similarly, treatments can seem to work.  Only by thoughtfully comparing them to solid reference points can we understand what the numbers really mean.

This is a sticky topic because with our irrational minds, placebos “work.”  You could probably test bracelets made from the hair off a monkey’s butt and some people will swear it “worked.”  Their testimonials will convince other people to wear monkey butt hair bracelets too.

The real tragedy of eating strawberries isn’t that strawberries are bad, immoral or evil.  The tragedy is that it takes people away from pineapples – treatments that work.

You could substitute supplements, holistic remedies, conventional medicine, anxiety treatments or any other “cure” on the planet.  This applies for dog training treatments as much as it does for “Miracle Diets.”  In a world where dog training results are evaluated with owner surveys, you cannot completely avoid the placebo effect.

The fact that something “works” doesn’t mean it works better than placebo.  If you want to know if a protocol, therapy or product works, you need to spend as much, if not more time looking at the control groups.