Two top executives from VizExplorer focus on the “superpower” of test and control, which gives marketers the ability to truly understand how a marketing campaign is performing
Mastering analytics provides incredible insight into your business. This insight gives you the confidence to act, the ability to measure your results, and brings along with it the self-assured attitude of a superhero… it makes you a Superpowered Marketer.
In previous Superpowered Marketer stories, we explored the superpower of x-ray vision. Now this superpower isn’t simply the ability to peer through walls. Our x-ray vision gives us the ability to see the future, the past, and the present with complete clarity.
We also met Mary the Marketer, whose superpower is her x-ray vision. Mary has intimate, intuitive and immediate understanding of every customer; including their value, their frequency patterns, and how they cluster together into similar groupings.
What is the source of Mary’s amazing power? Simply stated, it’s the scientific method—the ability to test and control her marketing methods.
THE POWER OF SCIENCE
The scientific method is central to science and it is virtually the only way that an idea can be validated. We have worked exhaustively on predictive models to forecast marketing results, and while they had some success, these results do not provide the incredible accuracy and confidence in results that come from testing a marketing program. While the accuracy that comes from test and control is amazing, you must not think of the knowledge gained by it as a single event. Instead, think of it as a knowledge library. In browsing this library, marketers can develop their x-ray vision superpower.
To explain, this x-ray vision ability comes from understanding the customers and thereby being able to influence their behavior. To illustrate, let’s look at Super Stacy at Casino X. Stacy does not literally have superpowers, but she does use test and control. Stacy has been running marketing at a local casino for five years and has been actively following test and control procedures during this time. She has documented each test, and she now has a library of more than 500 marketing experiments completed. One day a new chief operating officer, Boss Barry, starts running the property—and he has a stack of ideas. This places Stacy in an incredible position of being able to say either of the following two power statements to any idea Barry brings up:
Tried before: “Barry, that is something we have tried before, and here was the outcome. Having said that, something might have changed, so let’s test it again to see what happens.”
New idea: “Barry, that idea is novel and we have not tried it before. Let’s set up an experiment to see how it works on our customers.”
Of course, over time the “tried before” ideas that are successful allow for actions that are proven to get the desired results from customers—a kind of superpower, if you will. This special power gives Stacy a very good answer to many questions, backed up with hard scientific experimental results. Furthermore, test and control gives Stacy the ability to grow her knowledge and power over time.
TELLING A DEEPER STORY
Without test and control, marketers not only lack x-ray vision, they also lack the simple ability to truly understand how a marketing campaign is performing.
Let’s take a simplified example of Blah Bob, who does not possess our analytics superpowers. He sends a $10 free play postcard to 10,000 customers. Forty percent of these customers respond, so the cost of the program is $40,000 (4,000 responders at $10 each). However, Bob is very excited to report that each of these customers generated $100,000 of revenue on the day they redeemed an offer. So, our non-superpower marketer concludes that his program has driven $60,000 of profit for his casino ($100,000 of gross gaming revenue, less $40,000 free play expense). A wonderful result! But, unfortunately, this conclusion is completely inaccurate.
In Figure 1, we dig deeper into the story of this $10 postcard. In this version of the story, our marketer is Mary the Marketer, who has x-ray vision. Mary has split her $10 postcard campaign into a test group that received the offer and a control group that did not receive the offer.
The top row of Figure 1 examines the play on the day of the offer, which is October 4. In the first box, we see the test group revenue and expenses on that date. In the second box, we see that the control group generated $45,000 of revenue without receiving any offer at all. So Mary is able to conclude (in the top-right box) that on the day of the offer, her $10 postcard drove $55,000 of incremental gaming revenue with $40,000 of expense, for an incremental profit of $15,000.
But Mary isn’t done yet. She is worried that her postcard program may have stolen visits from other days and just moved that play to October 4 instead. She adjusted her analysis to include “shoulder days” that are near October 4.
In the second row of boxes in Figure 1, we see Mary’s analysis of the $10 postcard over the week surrounding October 4. In this case, the test group has more revenue ($200,000), because now we are looking at a week of data. The control group also has more revenue ($165,000). Now when Mary does her incremental analysis, suddenly the program isn’t profitable anymore. The incremental revenue is in fact $35,000 on the same expense of $40,000, for a $5,000 loss. Mary immediately discontinues the program and saves her casino money. Only with her analytics superpowers is Mary the Marketer able to see what Blah Bob cannot.
BIG RETURNS
One would expect x-ray vision to be an extremely potent superpower, and it is. Armed with knowledge gained from test and control, we can answer some very difficult questions:
• How high should my offers be?
• How many offers should I send, and for what duration?
• What type of offers should I send?
• How deep should I mail?
• How often should I mail?
• I have a crazy idea for a new marketing offer. Will it work?
The answers to these questions have a significant impact on a casino’s bottom line.
Let’s take Mary the Marketer as an example again. Mary works at a casino where the typical test either works and provides a lift of $10,000 per month, or fails and costs the casino $10,000 per month. At first glance, this seems like a coin toss. But, in fact, this is fantastic news for Mary and her casino.
Let’s say Mary tests the impact of increasing free play offers on a segment of customers from $10 to $20. If Mary’s idea is right, proved by test and control, the increased free play will lead to increased revenue, more than offsetting the expense and resulting in a way to make $10,000 more per month for her casino. Mary would then roll out this change, and for the next year, her casino would make $10,000 more than the prior year, for a total annual benefit of $120,000.
If, on the other hand, Mary is wrong and the increased free play expense does not lead to increased revenues, then Mary’s new program would cost the casino $10,000 per month. Yikes! Fortunately, Mary tested her idea first and can simply discontinue the program. In addition, only half her customers were tested with the increased free play offers, so Mary’s test actually only cost her casino $5,000.
So, with test and control, when Mary is right, her new ideas make $120,000, and when she is wrong, costs only $5,000. What a super return on investment, as she only has to be right less than 5 percent of the time to be profitable.
How often a marketer is right can’t be known until after test and control is initiated on a regular basis. However, in our experience, marketers tend to be right about half the time. The old slogan, “I know that half of my advertising is working, I just don’t know which half” is true. With test and control you can know which half is working—and that in itself is a true marketing superpower.
Even with x-ray vision, marketers are not immune to challenges and pitfalls that come along with high-powered analytics. Let’s take some time now to explore some common “kryptonites” of marketing and how to evade them, as well as how to get others on board with your superpowered analytic action plans.
KEEP THE SAUCE A SECRET
[Note: Trade secrets are a specialized area of the law, and the authors are not lawyers. Readers should seek legal advice on the correct procedures for implementing protection under the Uniform Trade Secrets Act.]
The Uniform Trade Secrets Act (UTSA) has been enacted by most states across the U.S., and it provides real legal protection for misuse of trade secrets. This law is very different from patent law, under which ideas are published for all to see while the inventors gain a kind of legal monopoly over their idea for about 20 years. Ideas are protected under UTSA as well, but they instead remain unpublished. The canonical example of the UTSA is the seasoning recipe for Kentucky Fried Chicken. This secret recipe is carefully protected by the UTSA as a trade secret, and if an employee was to steal this recipe and sell it, then KFC would be able to seek legal remedies such as damages and injunctive relief. To see how this impacts the casino industry and our superpowered marketers, let’s return to Super Stacy at Casino X.
If Casino X has required Super Stacy to build up a folder of marketing results (from test and control marketing programs, of course), and if that folder is stamped Trade Secrets on each page and on the front, and if Stacy has an employment agreement that requires her to protect confidential information and trade secrets, then the trade secrets contained in this folder have become the institutional knowledge of Casino X.
This means that Stacy cannot take the secrets within this folder, move to a new job at another casino, and use Casino X’s intellectual property there. We would argue that the knowledge of how Casino X’s customers respond to marketing programs over time becomes one of the most valuable assets of the property, and should therefore be protected. But before you get too excited about locking down every idea in marketing, remember that trade secrets do not stop somebody from building a competing product or their own secret recipe for how to market. Instead think of it as protecting your own specific recipe.
CONTROLLING YOUR SUPER POWERS
Test and control has many statistical challenges to getting the data right. In short, our superpower needs to be controlled! The challenges include:
• Patrons are varied;
• Competition is intense;
• Patrons have choices; and
• The gaming data itself is volatile.
Here we discuss two important concepts when using test and control: balancing and outliers.
Balancing—To understand the importance of balancing, we introduce our brand-new marketing analyst, Johnny the Joke. Johnny has recently joined our casino, gone through his orientation, and is ready to work. This is Johnny’s story:
• Johnny is asked to test reducing the free play expense on a particular customer segment.
• Johnny pulls his segment and randomly splits it into a test group and a control group. The control group gets a typical offer package with $20 free play offers. The test group gets the same package but instead receives $10 free play offers.
• The results come in! Johnny finds that the test group has higher revenues and lower expenses.
• Johnny shares his findings with his boss. “Send lower offers and customers will visit more often and play more in our casino!” he says.
• Johnny gets fired.
So what happened to poor Johnny? His test was unbalanced. This is a concept somewhat unique to gaming, and it is a consequence of the highly volatile gaming data that we deal with on a regular basis.
In Figure 2, we see Johnny’s unbalanced test. Had Johnny been applying the proper testing practice, he would have re-randomized until he got a balanced sample, as seen in Figure 3.
In this simplified example, we are only balancing Theo Win. In practice, though, we should balance multiple metrics, including:
• Days since last play;
• Visits per player; and
• ADT.
Outliers—Once a balanced test has been launched, marketers need to understand the results. Here is where another feature somewhat unique to gaming comes into play: outliers. As an example, let’s watch Lucky Lucy, one of our customers who loves to play at our casino to see how she can wreak havoc with our test and control results.
Lucy is part of our $10 vs. $20 free play test, and she lands in the test group that receives $10 in free play instead of $20. Lucy is a $1 Double Bonus Video Poker player and comes in to gamble on her usual day. Lucy is thrilled when she hits a royal flush for $4,000. Lucy has a gambling budget that she sets aside for her enjoyment each month, so this $4,000 is hers to enjoy on her favorite video poker machine. In fact, she stays later than usual and gambles back $3,000 of the $4,000, then walks away with $1,000 to play at another time.
This is a wonderful outcome for Lucy, and a disaster for your test and control experiment. Now your results are going to include a full $3,000 of theoretical win for Lucy that has absolutely nothing to do with the fact that you tested a $10 free play offer instead of a $20 free play offer.
This situation is quite common in gaming. In fact, in our opinion, approximately 80 percent of all gaming tests are impacted by outliers. Imagine getting reports that are wrong 80 percent of the time—yikes! It is critical, therefore that your test and control process includes steps to manage these outliers and reduce their impact on your analysis.
OVERCOMING OBJECTIONS
Armed with the ability to deeply understand her customers with her x-ray vision, and to be able to change their behavior, Mary the Marketer now has to convince her colleagues that these techniques are the way forward. In short, she needs to be the change agent for her organization, and that is no easy task, even for a superhero. To face the challenge, Mary the Marketer needs to ramp up her superpowers to the next level, becoming not just a superhero, but also an agent of change. With the power of change agent, Mary will become equipped to lead her organization and drive incremental profit in a continuously improving process.
Specifically, with the Change Agent superpower, Mary can:
• Overcome colleagues who invent reasons not to leverage her x-ray vision.
• Get those quick wins necessary to launch her new process into the organization.
• From those quick wins, foster relationships between analysts and operators that provide the best outcome for the casino.
At first, mary may be met with many objections from her co-workers. In particular, colleagues will be tempted to invent reasons not to do test and control. Here are two possible objections that Mary may encounter:
“Customers will get upset if they get different offers.” Here Mary can point out that she is already segmenting customers. They are already being treated differently based on how often they play, how much they play, and the last time they played. And these differences tend to shift from month to month, even for a single customer. In fact, using her superpower of x-ray vision, Mary can run a migration report and quickly identify that the majority of her customers get different offers from month to month anyway.
“It is too difficult or costly to execute test and control.” For this objection, Mary points out that test of control, whereby she can truly understand which offers effectively change customer behavior, can provide hundreds of thousands and often millions of dollars of incremental profit to the casino. This is more than enough to justify the time or expense of bringing test and control into the organization as standard marketing practice.
QUICK WINS
Once she overcomes the initial objections, Mary needs to get a quick win. We have seen thousands of segments tested in our years working in the casino industry. As such, we have some suggestions for where to start. The three tests listed have the highest probability of producing a quick win for Mary’s casino:
Defector Programs—Offer a significant reinvestment to customers who have stopped playing recently to prevent a costly lapse in customer play.
Threshold Management—Set up a test to help answer the question, “At what point should I stop mailing? $15 ADT, $20 ADT, or $25 ADT?”
Offer Type Testing—Food offers should not be “throw-ins”. Sometimes food offers cause customers to visit more often, but sometimes a food offer can cause them to play less.
FOSTER PARTNERSHIPS
By following the process described above, Mary will begin to see that partnerships will naturally develop between her, her team, and the team of casino operators she works with. Mary simply needs to foster these partnerships, work in a collaborative manner, and ensure that everyone continues to move in the same direction to provide the benefits that are best for her company.
EXERCISING YOUR SUPERPOWER
Our metaphor of marketing superpowers was designed to make you stop and think:
• Think about how powerful it is to gain true predictability from marketing programs.
• Think about how you are building up a secret database of ideas that work…and ideas that do not work.
• Think about how you can lead your organization to embrace these powers.
• Think how you can compete with marketers with these super powers, too, because they are out there.
Finally, think about what you can do to gain control over your marketing programs and apply analytics to gain and grow your own superpower-like abilities.
Leave a Comment