Top 10 Commandments of Statistical Inference: #5

Well, I am glad on the last post was well received, so now I think it is safe for me to start counting down the last 5 commandments of Statistical Inference. Number five:

Thou shalt not adulterate they model to obtain statistical significance.

Now, when you first look at this you think…Adulterate? But it does make sense. It comes down to some of our previous discussions, and that is make sure you do not knowingly (or unknowingly) allow extraneous variables or inferior ingredients into model. Make sure you take steps to control for the things you can control for. Sometimes, it is as easy as excluding certain people, or certain parameters. Other times you may have to really think practically about what CAN affect your model and control for that. In the marketing world it might be to control the time of day when your sends go out. In an experimental design in the lab you may want to add test at separate times and add a blocking variable. Use common sense and your own knowledge of the situation. This is not a “math” problem per-se. Of course there are some techniques available to help, but this typically requires you to sit down and map out your process and brainstorm about all the things that can affect your design and control the heck out of them when you can. This is always my favorite part of designing experiments. It’s when you can be creative. Next post we will discuss about how to understand and fit your situation into a needed model (rather than the other way around!).