You created an ad test, you waited to accumulate some data, and now you have winners and losers.
What do you do next? In reality, you have two simple options:
There are several considerations to take into account before you pick one of these simple options; so we’ll walk through some items to consider.
Many companies list out several hypotheses they want to test over the course of a time period (such as a quarter or a year).
If you are only testing 3 hypotheses per quarter, and you have hit that number, then you’re going to wait for the next quarter to start. However, if you have more hypothesis to test, then you need to consider your testing type.
In single ad group tests, you are examining the results for a single ad group and then making decisions. When you’re testing hypothesis across multiple ad groups; but you are examining the test results within each ad group (such as testing brand messages) then you have two options:
Option 1: Finish the test in all the ad groups first before creating new ads.
The advantage of this method is that all your ad groups are testing the same hypothesis even if they are in different ad groups. This lets you know exactly what is being tested in each ad group and there is less to track. The other advantage of this method is that you are making deliberate decisions with your ads and reaping the benefits of your winning ads as you never know if the next ad testing is going to perform better or worse than your current ads.
The other advantage of this method is that if you are using your ad data in other places (such as on the landing page, in emails, etc) you will know how each message preforms for the various ad groups being tested and can take a moment to really analyze the results per ad group; see what messages are doing best, determine the next steps, and then create all your new tests at once.
The disadvantage of this method is that it will take longer to receive insights since you will have times that you’re not testing. In addition, not all test results will be significant. Sometimes you’ll receive alerts that a test has minimum viable data, has been running for a minimum time frame, and it’s likely that you will not receive a winner or loser from your test. If you always wait until each ad group has a winner, in some cases, it might take a long time to get results.
Option 2: Create the new test immediately
The advantage of creating a new test immediately is that you will receive insights faster across all your ad groups.
The disadvantage is trying to track all the results if you are testing hypotheses. For instance, you might have ten ad groups in test 1, ten other ad groups in test 2, and another five ad groups in test 3. If you are good at tracking results, or using the notes feature, then you can immediately create new ads and track everything. If you are not using this data in other places (such as changing your landing pages); then you don’t need to evaluate the results as closely for use in your other marketing efforts.
The variable
Here’s the main wrinkle: What if you are testing more than two ads in an ad group?
If you are testing 4 ads in an ad group – do you consider your ad tests completed when you have a single loser or when there’s only one ad left in the ad group? Each company has a different viewpoint on this question. For the purposes of this discussion the advantages and disadvantages listed for the options apply when you get down to one ad in each ad group. For instance, if you are using option 1 when you wait until you have one ad left in each ad group to test more ads; then if you start with 4 ads, you’ll keep pausing losers until you have only one ad left. If you are creating new ads immediately, then this isn’t a consideration.
Conclusion for Single Ad Group Hypothesis Testing
Overall, if you are using your testing data in other places or you want to take a moment to really analyze your results for future hypotheses to test; then it’s best to run one hypotheses at a time; get results, analyze the results, refine your hypothesis and start testing again.
If you are risk averse and want to reap the benefits for your ad tests, then it’s best to test down to one ad and then start over.
If you want fast results, you have several hypotheses you want to test, and you aren’t risk averse – then it is best to just keep creating ads in your ad groups. Even with this method; there will be times when you need a testing break or need to sit back and analyze the results; so you might take a break from testing to really analyze the data every few months and then start testing again.
With multi-ad group testing you are running a test across many ad groups at once. This is a great way to test high level information such as shipping methods, calls to action, how keyword insertion does compared to your static lines, etc. This method is also great for testing low volume ad groups since you aggregate the data across all your ad groups at once.
With multi-ad group testing, hypotheses are easier to test than with single ad groups as each ad group can’t be in a different tests. Note: you could have one segment of ads (i.e. brand) in one test and another segment of ads (i.e. services/products) in a different test; this advice is for each segment of tests. This means that there isn’t a tracking consideration of keeping track of which ad groups are in which test.
Therefore, the main advantage of waiting between tests are (and the disadvantages to immediately creating new tests):
With Adalysis, there’s an easy way to pause all losers of a multi-ad group test and create new tests across the entire segment. Therefore, the time period to create the new ads isn’t a consideration if you are using Adalysis. If you are using another system, there may be another consideration: Setup time. If you are testing 1000 ads (or more in many cases) and only creating ads in the AdWords editor, then you need to create the ads, upload them, etc – and you might need this time just to verify all the ad groups are setup properly to test.
The main disadvantages to waiting (and the advantage of immediately creating new tests) is:
Therefore, if you are very deliberate in your testing and follow a pattern such as:
Then you might need to take a break between tests to really think about the information. If you just have several ideas you want to test, and you want fast feedback, then there’s no need for a break in your testing.
Conclusion for Multi Ad Group Hypothesis Testing
The main consideration for when to create new ads for multi-ad group testing is how risk averse you are and how much time you need to analyze the data and use it in other digital marketing efforts.
If you want fast results, there’s no need to take a break in-between your testing.
If you want to analyze the results and really think through your testing; then it can be useful to think about the results first before creating new tests.
Not everyone is overly deliberate in their testing and creates hypothesis to test. Some companies just write the ads, see results, pause losers, create new ads, etc and not a lot of thought is given to each ad tests.
With single ad group testing; this can be OK (outside of brand terms when you really should be thinking about your brand message).
With multi ad group testing, you might not have clearly defined a hypothesis; but there is probably one there. For instance, this was a set of templated description line 2s I saw recently
In this case, while a hypothesis wasn’t defined, what they were really testing is:
Since multi-ad group testing can be done across just a single ad group or millions of ad groups and it can give you a lot of amazing user insight, you should always have a hypothesis or determine why certain ad lines preformed best for you.
For single ad group testing without hypothesis; the main considerations are:
If you live by the mantra of ‘Always be testing’; then you’ll want to create new test results right away.
If you use a 3rd party to write ads; or you don’t always have the time to write ads; but you want the top ads to show the most often; then pause the losing ads (which only takes a moment in Adalysis) and then set aside time to create new ads.
If you are a very deliberate testing company, need to have ads approved by legal or the brand team, or always want to test hypothesis and you use that data in other places, then you’ll take a break between each test to gather information and devise a new test.
There is not a right or wrong answer to when to create new ad tests. It’s a balance of time management, risk aversion, and account improvement.