Master PPC Testing: Here’s How To Design a Paid Search Test for Statistically Significant Results

Anyone managing even one paid search account knows that testing is crucial to ongoing account success, but getting a test approved is difficult. But, if you can set proper expectations about the outcome of a PPC test and then achieve your desired results, you’ll find that receiving approval for future tests will come more easily. 

Here’s a test design template that will help you design a PPC test to ensure statistically significant and actionable results. This structure can be used for any type of paid search test from ad copy to bid strategies.

Every Part of the PPC Testing Template 

Background: Articulate where this test is coming from, why you want to run it, and the benefits this will have on the account. 

Is this a new Beta introduced by your Google team? Have you tested this strategy with other clients in similar verticals and seen success? Is this a best practice that you are not currently following? 

You should have some sort of reason for wanting to run this test that can be backed up with historical results.

Goal: Ask yourself: What are you trying to accomplish during this test? 

his can vary dramatically based on the type of test you are running. If you are testing bid strategies, you may want to understand which bid strategy is more effective at driving the most conversion volume. If you are running an ad copy test, you may want to understand if certain messaging themes outperform others.

Hypothesis: What is your prediction of what your test will find? 

This should be a tentative answer to your testing question. For example, your hypothesis could be: “An automated bid strategy will outperform manual bidding due to the power of machine learnings.”

Methodology: This is arguably the most important part of a test design. This section should outline all the elements that will be used to create a test and EXACTLY how the test will be set up. 

This is a chance for you to list out all the nitty-gritty details for your client or manager’s approval. Methodology considerations should include, but are not limited to:

• Engine – Are you running this test on Google, Microsoft, or both?
• Audience/Audience Size – Information you may have about your audience and/or where the test is running. Who are you targeting? If the audience is going to be split in a certain way, how many people will be in each split? Be sure to include any additional audience information you think may be relevant.
• Test Setup – Articulate how you plan to set the test up. Drafts & Experiments? Pre vs. post? Traffic split? And so on.
• Test Duration – How long do you anticipate this test will take and why? Try to base this estimation using similar tests you have run in the past. Another important consideration is the level of statistical significance needed. We recommend a level of at least 90%. How long will it take to reach that level of significance?
• Expected impact – Here’s one of the most important parts of your methodology. What is the anticipated impact of running this test? This should help you prioritize your tests. If you forecast only a small bump in performance look to test something more impactful first. Be sure to explain how you got to your forecast.

KPIs: After outlining your methodology, list the exact KPIs you will review and monitor throughout the test that will determine success or failure. 

Whenever possible, explain the rationale behind using those KPIs. Also, try not to exceed more than two KPIs. In my experience, having more than two KPIs can often lead to conflicting test results and never-ending tests.

Monitoring: This section should include how you plan to monitor the tests. 

How often will you be checking results? How will you know if your test or the control is winning? What are the performance thresholds, targets, and decision points? I always like to outline performance thresholds in case the test goes horribly. For example, “If our test bid strategy is 25% less efficient than our control bid strategy after two weeks, the test will be paused immediately, and we will revert to the control bid strategy.”

Limitations: Include anything — hopefully outside of your control — that could impact the validity of your test or ability to reach statistical significance. 

A few examples that may be outside of your control could be budget cuts, tracking limitations, or updates to the site experience. Ideally, you should not plan to run a test if you are aware of any of these upcoming limitations. 

It is also important to include any learning or calibration periods you may encounter in your test. If you are testing a bid strategy, it will be important to outline that the bid tool may take a few weeks to calibrate and you will not be analyzing performance during that time.

Additional Information: Here’s your chance for you to list out any additional details and/or relevant information that may not fit in the other sections.

Action Items: What are the steps that need to be taken before you can launch this test? 

Pro tip: Your first action item should be receiving test design approval from your client or manager.

Next Steps: This is an opportunity for you to outline EXACTLY what you are going to do if your test wins. 

This is particularly important if you are running a test in a small subset of your account and plan to roll out the change across your entire account. This should be a stop and pause moment for you because if there is not the next step for your test, you probably do not need to be running this test.