How to write a website testing plan & set expectations

Having a solid and continuous testing strategy is the core of every high-converting post-click marketing program. An ongoing testing program ensures waves of improvement that lift your conversion rates in the short and long-term. Here Jonghee Jo, the multivariate testing & analytics lead at JPMorgan Chase, writes about putting together a testing plan and setting expectations.

If you manage to establish a website testing team and gained senior management approval for your testing endeavor, it’s now time to run an actual test.  Since any kind of website testing requires coordination with different departments and stakeholders, a well-defined test plan is critical for the success of the test.  Without a concrete test plan, you might get astray before the test even starts!

A successful website testing plan should include the following items:

Test Background

Who is requesting the test? What is being tested? Why is it being tested?  These are fundamental questions that need to be answered before you begin.  If you don’t have clear answers for these questions, you better not to start the test.

Hypothesis

It’s always good to start with a hypothesis, rather than a generic test scenario. For example, start with  ”Adding another call-to-action link on the page will produce at least a 1% lift in conversion rate”, instead of “How about adding this button on this page?”. When you start with a hypothesis, you (and your counterparts) will give more serious thought to the test design and possible outcome.

Measure the Outcome

At the end of the test, we need to measure the success of the test. Without a clearly defined test outcome, how can we conclude if the test is successful or not?

For retailers,  one of the major outcomes could be sales per visit (or profit for visit if you can tie cost information into your analysis) .

For non-retailers, it could be one of the various site conversion events such as  number of downloads, number of applications, and number of email subscriptions. You can track multiple outcomes, but I advise you choose one primary outcome to define a winning test because multiple outcomes can bring conflicting numbers.

Detailed Test Variations

In this section, you will need to show test versions/treatments. Showing actual test creative will always help readers understand the test more clearly. If you are running a multivariate test, you can utilize a table format to show the test treatment combinations effectively.

Analysis and Reporting

From the start, be clear on what kind of data will be collected and what kind of analysis and segmentation will be done.  Internal stakeholders may ask for more analysis than you can handle. If you make it clear on what you can do and what you can’t do at the early stage, you will be able to manage your counterparts’ expectations effectively.  One of your analytical mantras should be “Promise less, deliver more.”

Project Timeline

Every website test is a project and every project needs systematic management. List all the detailed steps required for the project and specify the timeline and who’s in charge for each steps. Sometimes you can’t stick to the plan because of various reasons including unexpected low lifts or internal priority changes.   That’s the reality of life, but still you should have an initial timeline when you begin. Then you can negotiate/revise timelines after that.

Roles and Responsibilities

You need to be very clear on roles and responsibilities of your team (and your client) from the start, especially when you initiate your testing efforts in your organization (or your client’s organization). Every member/stakeholder in the project should know what they will do/expect before the test starts. Without clear understanding on roles and responsibilities among project members, there will be a much higher probability of  conflict, finger pointing and even political battles during and after the test.

Anything missing in the plan above? Any other alternatives? Please share your thoughts via comments.

Jonghee Jo

Jonghee is the Web Multivariate Testing Analytics Lead at JPMorgan Chase, one of the largest financial services in the country. His work focuses on analyzing key metrics and providing business-critical insights from numerous web tests he manages. His experience at JPMorgan Chase includes diverse line of businesses such as Credit Cards, Retail Banking and Personal Lending. Prior to joining JPMorgan Chase, Jonghee was a Senior Marketing Analyst at Victoria’s Secret Direct, where he focused on Web Analytics and Testing for the largest intimate fashion e-commerce site in the world. He graduated from the Tepper School of Business at Carnegie Mellon University with an M.B.A., majoring in Marketing and e-Commerce Management.

Website - Twitter - More Posts