Why OnProcess Uses Control Groups

Editor’s Introduction:

Our sales team is frequently asked why we adhere to a rigorous application of control group methodology in gauging the performance of our various programs in both Asset Retrieval and Customer Experience. The questions tend to fall into a few main categories:

  • What’s a control group?
  • If your methods are proven, why bother testing? Aren’t we wasting opportunities for improvement?
  • Our reports differ from yours – how can we tell if your program is adding value?

To address those questions and more, our VP of Market Research and Product Development, Dan Gettens, has written another in his ongoing “What Gets Measured” series of blog posts. The first one was extremely popular and had high pass-along value, so we suspect you will find this one to be as well. –sk

What gets measured…to show cause and effect


A service provider offers a solution that could replace my current process.  The solution has been adopted by other companies in my industry. How do I demonstrate cause-and-effect improvement before I make a “go / no go” decision on the new solution?

A corporate group is recommending implementation of a solution that would replace my current process.  The solution appears to work well in several Regions. I want a proof point that it will work in my region.  How do I demonstrate cause-and-effect improvement from the new approach at a reasonable cost and under realistic conditions?

I plan to review a successful local pilot process and then implement the solution on a national scale.  How do I objectively demonstrate cause-and-effect improvement to accelerate local adoption of the new process?

The scenario

The change made last year to your Company service / supply chain process cost over $1M and did not result in any detectable improvement in efficiency or effectiveness.   You plan to propose a different approach to the executive team.

Here is your initial assessment:

  • Potential benefit of your proposal to the Company: high
  • Company cost of not making theright change last year: $1M implementation cost plus the cost of not resolving the weak link in the supply chain and the cost of moving the Company’s service / supply chain process in the wrong direction.

You analyzed the current process, interviewed key stakeholders and found one common theme:  The focus of the executive team meeting will be all about demonstrating cause and effect.

To understand how the Company’s service / supply chain process will respond to your proposed change, completing one more step makes sense before you present to the executive team:  to make the change and find out.

Your project manager suggests the approach: “Let’s randomly assign two groups: a stand-still control group using the status-quo process and a managed group using the modified process. We can define a pilot and start a soft-launch in as early 3 weeks.”

What decision factors are in play in successfully managing a controlled pilot?


Consideration: What are the other differences between the control group and the managed group?

The process supporting the managed group should deliver added value (beyond the status quo process) that you can influence, drive, and enhance.

There are other ways that the new process can add value.  The new process should transform the current process. Work in the enhanced process should be done right the first time. The new work should represent value that the customer is willing pay more for.

Consideration: Do you need a controlled pilot or designed experiment for every process change?

No. Selective use makes sense rather than use for every change.  For example, when the root cause of a specific problem seems clear and the measurable improvement appears immediate, it may make sense to “just do it.”   For some changes, the cost of controlled pilots or designed experiments may outweigh the potential benefits.

Consideration: “Control groups are only applicable to manufacturing and scientific environments, not to services or supply chain management.”

If the service delivery process or supply chain process has reasonable volume / throughput and is not highly customized, the approach may be just as applicable to service and supply chain processes as it is to manufacturing.

Consideration: Who manages the controlled pilot, including the business rules for assigning the control group and monitoring results?

Typically, no one cares more about the controlled pilot than the manager driving the proposed change.  That manager’s leadership in driving, monitoring and measuring results for the controlled pilot can make the difference.  The manager could be from the Client’s organization — or could work for the service provider proposing the change, especially if the process is open and auditable to all involved parties.

Consideration: What type of process would you select?

The following are suggested:

  • A process with a consistent, well-defined cycle start time, or start point, as well as a consistent, well-defined cycle end point
  • Operational definitions — those understood universally across the Company.  Here’s one test of an operational definition: If you review the definitions with managers in different functions, do they all quickly buy into the definitions, without triggering a lively debate?
  • Measurements defined up-front, along with one or more leading indicators also defined in advance
  • The start date and end date of the controlled pilot

The timing of the start of the controlled pilot is a key factor.   Once a “go” decision is made, consider launching the pilot with a month, or even better, within 3 weeks.  A typical expectation of the pilot is that results will be demonstrated early. Managers tend to have little patience with delays in seeing initial results.  Also, waiting too long can result in additional changes in the business environment, the business rules, and the requirements for the pilot.

The up-front identification of key input variables, process variables and output measures also makes sense.

Consideration: “Let’s review the results of the controlled pilot then set a target goal based on the capability of our process.“

Maintaining a clear distinction between the Client requirements and the process capability works best.  The “Voice of the Client” goals may include the success rate, the acceptable level of variation daily or weekly, and customer satisfaction, recommender or loyalty ratings.

Consideration:    As a manager, I want to know that the improvement of the managed group using the new process is not due to chance.

The business sponsor wants confidence, based on solid data, to drive actions that lead the Company to implement the new process.

A statistician or Lean Six Sigma Black Belt on the project shares the same goal, maybe in slightly different language:   The null hypothesis is that the change in results from the status quo process is due to chance alone.  The alternative hypothesis, also call the research hypothesis, is that the change is not due to chance alone.   Do you have strong evidence to reject the null hypothesis?

Consideration: Your managers and colleagues may have limited interest in a controlled pilot approach.


To open the discussion, you could share…

  • Case examples provided by your colleagues or your service providers that are relevant
  • Benchmark results within your industry
  • Benchmark results outside your industry —  from our data, one industry typically does not have the monopoly on best supply chain practices


Consideration:  “I implemented a controlled pilot and did not achieve the desired results.”

It is preferable to know the outcome, positive or negative, as early as possible. Not every planned control group will result in the desired outcome.  However, negative results, identified early in the process, are much less costly to a company than learning later after all the implementation costs have been incurred.


Consideration: What are the key implementation steps?

A brief controlled pilot checklist:

  • Is the control group randomly assigned?
  • Is it representative of the population?
  • Is the assignment process open and auditable?

Service / Supply Chain Case Example


Let’s review a case example to tie a few of the concepts together — with data slightly changed to protect the confidentiality of our Client.

Business problem: The Client is experiencing higher rates of remorse returns on new product introductions.

Background: For example, in the U.S., consumers can purchase electronic devices, including PDAs, with the right to return the product with 14 days in some states, and 30 days in other states.   Sometimes, Clients provide a consistent 30-day window for remorse returns across all states.

Methodology: run a controlled pilot program, including a randomly assigned control group and a managed-group, for 4 months

Metrics: Measurements included customer ratings (as a potential leading indicator) and reduction in remorse returns

Improvement Based on a Key Leading Indicator:control-group-illustration


Scope: the scope of the process includes both the supply chain and service chain. The service chain focus includes improving the customer experience with proactive education to address the primary reasons for customers to return product.

Voice of the Client target goals defined up-front include:  12% reduction in remorse returns.

Controlled Pilot results:  25% reduction in remorse returns

–Dan Gettens