Cohort Analysis Done Right

Use a p-chart to properly monitor shifts in customer churn

Let's say you run a Customer Success team and your manager asks you to perform cohort analysis in order to better understand customer churn behaviors. Your customers renew on a monthly basis, and you’re interested in measuring their initial fallout rate. Using data collected over the past year, you count the number of users who canceled their subscriptions after the first 30 days and divide that number by the total number of new users during each month.¹  Your Excel spreadsheet table is shown below:

You then plot the data:

The results concern you. Although the data points vary somewhat, it appears your 30-day churn rate has been increasing all year! Using the trend line function in Excel confirms it—the first month’s churn rate has grown from about 2.2% to 3.2%!!

You share this information with your boss and she’s not happy. Clearly your Customer Success team is not doing the job. She expects immediate improvement. Or else. 

Simple, obvious... and wrong

But hold on a minute—you’ve made a serious mistake. You haven’t managed your team poorly. You’ve analyzed your data improperly!

Your error has to do with statistical sampling. Randomness occurs in all data, and experimenters must be careful to separate the “signal” from the “noise.” Statisticians warn of two types of experimental error: 

• Type 1: False Positive—a result is determined when in fact there’s only randomness 
• Type 2: False Negative—randomness is determined when in fact there’s a valid result

Without using the proper statistical methods, people can easily jump to the wrong conclusions. Overly simplified, standard tools in Excel make the situation worse. As a result, managers tend to over-react, seeing a problem that isn’t there and making changes that can do more harm than good.² Managers can also under-react, missing the cues to make changes when it’s time to do so. Most businesspeople lack a strong analytical background, so statistical errors are quite common, often leading to disastrous consequences.   

Using the p-chart

Fortunately, manufacturers have been using powerful statistical methods for years, and these practices can easily be applied in software companies. Lean Six Sigma, a quality improvement method, incorporates a broad range of tools that ensure teams properly analyze data and reach valid conclusions. You can use these techniques to your advantage. 

The control chart is a handy Lean Six Sigma tool helping managers avoid Type 1 and Type 2 errors. Control charts are used to determine if a process is in a state of statistical control, i.e., if the observed variation is due to randomness and not any particular cause. There are many different types of control charts depending on the type of data involved, but all follow the same basic structure. The p-chart (or proportion chart) uses the binomial distribution. It is the most appropriate control chart to use in this example because there are only two possibilities (the customer cancels or renews) and results are expressed as percentages.³  

The control charting procedure is as follows:

1. Determine rational subgroups. A rational subgroup is a homogeneous set of data that provides a representative sample, such as a batch of parts manufactured during a shift. Many SaaS companies define their subgroups as monthly “cohorts,” the set of new customers who sign up for software during a given month. Usually there’s nothing different about a customer who subscribes in January versus one who subscribes in February, and for the purposes of evaluating churn, it’s more accurate to use sequential, fixed sample sizes instead of varying sample sizes.⁴  In order to calculate the mean with reasonable accuracy, statisticians recommend using around 25 subgroups. In this case, we’ll choose fixed sample sizes with 168 customers in each of 24 subgroups (4032 customers/24 subgroups = 168 sample customers/subgroup).⁵  Below is your new sample set:

2. Plot the dots. As you can see, the shape looks a little different, but it still appears there’s some trending in the data. 

3. Calculate and apply the center line and control limits. 

• Compute p-bar (or average proportion) for the data set. In this example, 114 out of 4,032 customers churned after the first 30 days, a p-bar of 2.8% (114/4032). Draw this line on the chart. 
• Calculate the upper process control limit, UCL. This line shows the mean plus three standard deviations. The line is important because the probability of finding a data point more than three standard deviations away from the mean by chance is less than 1%. The UCL calculation for a p-chart is:

UCL = p-bar + 3 * Square Root {p-bar * (1 – p-bar) / n}

where n is the fixed subgroup sample size

In this example, UCL = 0.028 + 3*SQRT {0.028*(1-0.028)/168} = 0.0662 = 6.6%. Add this line to the chart.⁶  

4. Interpret the results. As you can see, most data points fall in the vicinity of p-bar and none exceed the upper process control limit, UCL. This is your first indication that observed variation is likely due to randomness and not any special causes or shifts in your process. For good measure, add lines at +/-1 and +/-2 standard deviations (probabilities of data points in these ranges 68% and 95% respectively) and add them to the chart. Mark zones “C” between the centerline and 1 standard deviation, “B” between 1 and 2 standard deviations, and “A” between 2 and 3 standard deviations from the mean as shown below.

Then apply three additional statistical tests to rule out any possibility that something is amiss:⁷ 

• Nine points in a row in Zone C or beyond on one side of the center line?—NO 
• Six points in a row steadily increasing or steadily decreasing?—NO 
• Fourteen points in a row alternating up and down?—NO 

Congratulations, you can breathe a sigh of relief! You’ve correctly established that despite appearances, there’s no trending in the data, contradicting what you originally thought. You explain to your boss that nothing has changed and results are due exclusively to randomness. The data shows that your onboarding process is in a state of statistical control and will routinely produce 30-day churn averaging about 2.8%. What’s more, you can accurately predict that 2/3 of the time future churn will be measured between 1.5% and 4%, 1/3 of the time less than 1.5% or over 5%, and only on rare occasions will it ever exceed 6.6%.  

Next steps 

You can continue to add new data points and monitor process stability using the limits you calculated above and by applying the same interpretation procedure. If any of the four statistical test results are positive, take immediate action and determine the underlying cause. You can use p-control charts to investigate customer defections at other periods of time, such as subscriptions after 60 days, 90 days, or upon annual renewal. Several Lean Six Sigma statistical packages are commercially available online (many of which are SaaS), making control charting much easier. 

Once your Customer Success process is stable, it’s possible to systematically improve it. Although keeping things under control is always your first priority, making things better is your end goal. In my next blog, I’ll describe how Lean Six Sigma methods can be used to help you improve customer retention.

Excel-lens is a publication of Service Excellence Partners. We increase customer loyalty and business performance in the cloud computing industry. Contact us today.

Notes:

1. Technically, calculating ratios when denominators vary causes an “average of the average” problem in which comparing percentages between periods introduces significant measurement error. This discussion was deleted for brevity. 
2. Human over-reaction to Type 1 errors is a fascinating subject, one with neurobiological and evolutionary explanations. It’s beyond the scope of this blog, but humans are preconditioned to see patterns and jump to the wrong conclusions, hence the need for robust application of the Scientific Method. 
3. Note that p-control charts can be used with varying sample sizes under certain circumstances (computing average n or using multiple control limits), and that other types of attribute control charts can be used for smaller sample sizes. For simplicity, this discussion was omitted.
4. Homogeneity extends to type of customer, including the market segment and associated value proposition, as described in my previous guest blog. If churn behaviors vary significantly by customer type, you should stratify your data and chart each segment separately.   
5. A rule of thumb is to choose subgroup sample sizes for attribute data such that np>5; in this case, churn is about 3%; n>5/0.03 or n>167
6. Note that in this case, the lower process control limit is negative and can be neglected; this is a “single sided” investigation where the minimum churn is 0.0% and we are interested in detecting a shift upward.
7. Montgomery, D. C. (2005), Introduction to Statistical Quality Control (5 ed.), Hoboken, New Jersey: John Wiley & Sons, ISBN 978-0-471-65631-9, OCLC 56729567

The Best Meeting Agenda

Planning is easy. Execution is hard. How can a monthly meeting keep an organization persistently on track to achieve its goals? 

Nobody likes meetings, including me. People perceive them as time-wasters. Many have gone to “no PowerPoint” talks, got rid of chairs, or eliminated meetings entirely. In my view, these are extreme measures. Meetings serve a purpose. They provide important venues to communicate, learn, and make decisions. Yes, cut back on the number of meetings, but make the ones you keep more productive. 

I think the best meeting agenda is called a Monthly Business Review (MBR). It’s a formal meeting (yes, preparation is required) designed to check progress on key plans, projects, and business performance. What makes it the best? Unlike other meetings, the MBR ensures the organization executes its business plan. The alternative, of course, is to do what most companies do: make a plan, and after a few weeks just go back to what people were doing before the planning session. 

For an MBR, all functional heads and the CEO attend. A scribe and a timekeeper are assigned. Each manager has 15 minutes and may present only three slides. During the meeting, the scribe records issues and ideas that come up on a flip chart, and unless something requires immediate resolution, the topic is parked until the review is complete. The timekeeper also plays an important role. He or she prods managers to stay on topic lest the MBR lose focus and become interminably long. Here’s the agenda: 

• Welcome and Opening Remarks (CEO, 5 minutes) 
• Company-wide Dashboard Review (CEO, 10 minutes) 
• Functional Round-Robin (15 minutes each VP): 
• Dashboard Review
• Updates
• Successes
• Needs
• Hoshin Review (15 minutes for each strategy owner)
• Action Items (15 minutes)
• Other Items/Wrap-Up/Schedule Next Review (10 minutes) 

The first essential element is the dashboard, consisting of the top ten metrics at the enterprise level and for each function. The company-wide dashboard may list things like revenue, expenses and customer satisfaction, whereas the marketing dashboard may include impressions, web visits, and lead conversion rate. Each measure is color-coded: green says everything is going well, yellow indicates things may be going off track, and red means take immediate action. Red signals get special attention; the manager describes the problem’s root causes and the actions being taken. Dashboards show cause-and-effect relationships, allowing managers to view the organization as a living system. The review also keeps managers focused on keeping the fundamentals under control. It provides context for decision making; if a new idea or initiative doesn't positively impact a key metric, it’s probably not that important. 

The second essential element is the Hoshin Review. Hoshin kanri, a Japanese breakthrough improvement method, is an incredibly powerful tool for driving organizational alignment and change. During annual planning, the executive team collaboratively defines a mission-critical, breakthrough objective, along with supporting strategies, owners, and performance targets. An example hoshin objective may be, “Transform the sales process from a direct to a channel marketing and sales model” with goals to “increase annual revenue by 3x and reduce customer acquisition cost by 50%.” The objective is broken down into 3-5 major strategies and goals, such as “Recruit, contract, and implement third-party marketing relationships; goal of 3 by June 1.” Assigned strategy owners then form cross-functional teams and develop implementation plans. During the MBR, the Hoshin Review checks status, surfaces and removes any barriers to execution. 

Order is important: business fundamentals before hoshins. Why? If the basics are out of control, working on more advanced initiatives makes no sense. Make sure the foundation is solid prior to reaching for the breakthrough. 

Towards the end of the session, the scribe reviews any issues or ideas surfaced on the flip chart during the meeting, distilling them into action items, owners, and deadlines. Any outstanding action items from previous meetings not already addressed are also checked to make sure nothing falls through the cracks. 

Obviously some meeting preparation is involved. Functional leaders meet with their teams in advance, rolling up data, interpreting signals, and planning actions for their dashboard metrics. Hoshin strategy leaders check progress on their project plans. The CEO must also prepare. He or she must review the top-level, enterprise-wide metrics and be ready to help executives prioritize next steps. 

Work behind the scenes is also important. If the CEO notices chronic red signals or sluggish execution on hoshin strategies, he or she should meet separately with executives for 1-on-1 coaching sessions. Accountability for progress and results is important to maintain, but CEOs should avoid calling out individual struggles during the actual review. 

So what do you think? If you had to arrange a meeting, wouldn't an MBR be one of the most valuable? If the goal is moving the organization forward, perhaps there’s no better agenda.

CSM Performance Management Done Wrong

Managers often perceive differences in how individuals perform sales and customer service, implementing management practices that make things worse. Faced with the challenge to improve results, many establish numbers-based, “pay-for-performance” systems to reward good employees and punish bad ones. Motivating by comparing, ranking, and paying people according to their individual results just makes sense. After all, prestigious business schools and prominent corporate leaders espouse meritocracies. The “carrot and stick” approach is so widely used that it must be right. But this philosophy is flawed and counterproductive, especially in operations like Customer Success Management. Why? Supervisors wind up focusing on the wrong things and demotivating their employees.

Call centers are a case in point. Managers keep detailed statistics such as call handle time, call quality and conversion rates. Most then rank agents based on their numbers. A pep talk might sound like, “Bill, I see you are near the top on quality, but you’re only about average on your call handle time. Keep up the good work, but you need to go faster!” In addition to coaching, call centers use a variety of bonuses and awards to single out and further motivate agents, believing it’s the answer for achieving high service performance.

How well does it work? The fundamental question is to what extent performance can be attributed to the process (i.e. technology, workflow, customers, training, policies, etc.) or to the people. We studied this question examining average handle times (AHT) and quality scores (QA) for a group of 55 agents over several months. If performance was due exclusively to process factors, individual rankings would be decided by chance alone. If people factors entirely determined results, ranking would always be same, month after month.

Using statistics no more complicated than the probabilities of flipping a coin, we found that individuals played a role, but nowhere near as much as managers expected. In the agent population, only 13% exhibited non-random behavior for QA and 22% for AHT. Combining both metrics, we showed less than 6% of agents had performance that could be attributed to something other than chance. Clearly, rewarding and recognizing individuals for outcomes dominated by systemic variation was a waste of time—and many agents knew it. Management time was much better spent improving the process than motivating the people.

Surprisingly, experts have shown that reward systems can be demotivating, especially when it comes to tasks people enjoy. Psychologists classify motivation as intrinsic (doing something for its inherent satisfaction) or extrinsic (doing something in anticipation of a separable, dependent consequence). Ask people in customer service why they do the job and most will say it’s not the money but because they like helping people. In other words, service people tend to be intrinsically motivated. Years of scientific analysis confirms that virtually every type of reward that is contingent on task performance undermines intrinsic motivation.¹ The flip-side is also true—management threats, deadlines, directives, and competitive pressures also backfire. Quality guru W. Edwards Deming felt so strongly about the obstacles caused by ranking people according to the numbers that he insisted leaders eliminate it in his famous “Fourteen Points for Transformation of Western Management.”²

So does this mean employees shouldn’t be rewarded or held accountable for their work? Absolutely not! In any distribution of people, some will excel and others will struggle. Leaders must help everyone do a better job, and for statistical outliers “manage up or manage out.” But leaders must also understand better results come from better processes, and should make process improvement their primary focus. They should also set aside relative ranking and competitive reward systems, choosing instead to practice leadership and instill teamwork. And not all extrinsic motivation is bad. We found that service employees respond very favorably to team-based goals and incentives that promote cooperation and process innovation—enhancing, not hindering, their natural inclination to serve customers and each other.

And what about CSM functions mixing sales and support? In these cases, small, individual sales commissions may be appropriate, but great care must be taken to balance and clarify job expectations and goals. Many times employees view sales and service to be at odds, and for good reason: customers don't want to be "sold to," but appreciate assistance making a decision. Framing the task as identifying needs when they surface and helping customers decide in a subtle, consultative way tends to be much better received than pressuring CSMs to meet monthly upgrade and renewal quotas at the expense of good service.

Often the road to hell is paved with good intentions when it comes to performance management. With great service increasingly the market differentiator and a primary driver of customer retention in subscription-based revenue models, managers can’t afford missteps. Spending time unproductively improving service performance or lowering front-line employee satisfaction and engagement can be disastrous. Using a more enlightened approach to performance management avoids pitfalls and pays much higher dividends.  

Footnotes:

1.    Ryan, R. and Deci, E. (2000). “Intrinsic and Extrinsic Motivators: Classic Definitions and New Directions,” Contemporary Educational Psychology, issue 25, pp. 54-67.  

2.    Deming, W. (1982). Out of the Crisis, pp. 70-85, MIT Center for Advanced Engineering Study, Cambridge. ISBN 0-911379-01-0