In layman’s terms, statistical significance is the how likely a result is caused by something other than random chance. Essentially, this is how confident you are in the data that random chance didn’t cause winners and losers.
There is a relationship between Statistical Significance and minimum data (which we’ll cover in next week’s column).
For instance, if you flip a coin 4 times, there is a 1/16 chance that heads will show up all 4 times. Yet on the 5th throw, there is still only a 50% chance that you’ll receive another heads. That’s because each time you flip a coin, there’s a 50/50 chance that you will see a heads or a tails. However, on consecutive throws, you need to take in the variables of the previous throws to determine the chance of seeing heads 5 times in a row (which is 1 in 32). This is why we need a certain amount of minimum data before we calculate confidence factors.
Eventually, the odds catch up and after 100 flips, you’ll probably have 47-53 heads assuming it’s a regular coin. If after 100 flips, you had seen heads 90 times, you are either on a very odd trend (and should go to Vegas) as that result is highly improbable or the stats say that your coin isn’t regulation and you are playing with a coin that is not properly weighted.
In fact, if you throw a coin 2,000,000,000,000,000,000,000,000,000 times or 2×1027 * – odds are that you will have 90 heads in a row at some point in time, which is purely due to chance. However, if you were to look at the entire sample set and not any one streak of numbers, you’ll see that heads and tails have each come up 50% of the time.
When you are picking your confidence factors for any one ad test result, you’re really saying, “how confident am I in these results that this result is meaningful and not due to chance?”
As your ads are the only part of your account that searchers see, when you pick winning and losing ads, you want to make sure that you are confident in the results and that you aren’t picking winners due to chance.
We’re often asked about confidence factors and how confident someone should be in their results before they take action; so we’ve made a handy reference chart based upon types of keywords:
|Term Type||Minimum Confidence|
|Long Tail Keywords||90%|
|Mid data terms||90% – 95%|
|3rd Party Brands you Sell||90% (small brands) to 95% (large brands)|
|Top Keywords (the ones you watch daily)||95% – 99%|
|Your Brand Terms||95% (unknown brand) – 99% (well-known brand)|
The overall rule is simple: The more important a word is to your account, the higher you want the confidence factors to be before you take action.
According to expert statisticians, you never want to be less than 90% confident in your results before taking action.
Odds are, you have segmented your account into various campaigns. Some campaigns are branded, others are long tail, and yet others are information terms, ‘hero terms’, competitors, and so forth. Therefore, its useful to just make a note of your minimum confidence level by campaign type.
Now the next time someone wants to discuss confidence factors, there’s just a few rules to keep in mind for the conversation:
- Statistical Significance is how likely an event is caused by something other than chance (in this case, such as the ads being different from each other).
- If your sample size is too small, any result can be due to chance.
- In a large data set, there will be periods of anomalies (throwing heads 90 times in a row), but the overall data will show you the true results.
- Never go below 90% confidence factors.
The other data point that goes hand-in-hand with statistical significance and confidence factors is minimum data. This is the least amount of information that you want to use before determining if your results are significant or not.