The Minimum Data You Should Define by Testing Metric

In our last two column we discussed setting your confidence levels and what you need to know about minimum data.

What we haven’t addressed yet is what data points you should assign minimum data levels to based upon your testing metrics.

As an example, if you are testing by CTR, your conversions don’t matter since CTR doesn’t use conversions data in its calculation.

Most metrics have both a required data point  (as that’s the opportunity) and a data point (as the data is secondary to the opportunity) but used to calculate that metric, which would be an optional data point to define.

For instance, click though rate is the ratio of impressions to clicks. You must have an impression to get a click (no impressions will not receive a click). Thus impressions are mandatory but clicks are optional.

In some cases, you might not want to define the optional metric. For instance, let’s say we’re running two ad tests with this data set:

• Ad 1: Impressions 1000, clicks 100
• Ad 2: Impressions 1000, clicks 10

In this test, we’re confident that ad 1 is the better ad and has achieve over a 90% confidence interval. However, if we defined a minimum of 25 clicks, we’d still be waiting for results since ad 2 hasn’t hit that number yet. When you define the optional data points, you might wait longer to achieve results if one of your tests is significantly below average (in this case a 10% vs 1% CTR).

With minimum data, every ad in the test should hit the minimum data before you look at the information – not the test combined. As there are many ad rotation options, it is common that not all ads within a test have the same opportunity, and thus each ad should meet the minimum requirements before you examine your confidence levels.

As timeframe is highly important to any test, all metrics should be using a timeframe minimum of at least a week; although, using monthly data works just as well.

Here’s the minimum data that you should define by testing metric:

Metric	Impressions	Clicks	Conversions	Timeframe
CTR	Yes	Optional		Yes
CPA			Yes	Yes
Conversion rate		Optional	Yes	Yes
CPI	Yes		Optional	Yes
ROAS			Yes	Yes
RPI			Yes	Yes

For Adalysis users, if you have defined minimum data amounts (which is optional as the system has prebuilt minimums behind the scenes) we automatically only use the minimum data that applies to each metric, so there is no need to tell the system your minimum data by metric.

Next time you’re running an ad test, consider your minimum data, necessary confidence levels, and then make sure you have the minimum data necessary based upon your testing metrics.