Lean Enterprise Institute Logo
  • Contact Us
  • Newsletter Signup
  • Cart (0)
  • Account
  • Search
Lean Enterprise Institute Logo
  • Explore Lean
        • What is Lean?
        • The Lean Transformation Framework
        • A Brief History of Lean
        • Lexicon Terms
        • Topics to explore
          • Operations
          • Lean Product & Process Development
          • Administration & Support
          • Problem-Solving
          • Coaching
          • Executive Leadership
          • Line Management
  • The Lean Post
        • Subscribe to see exclusive content
          • Subscribe
        • Featured posts
          office sign that says - Kaizen Promotion Office

          Ask Art: Why Do I Need a...

          WLEI Podcast with author Paulo Savaget of The Four Workarounds

          Strategies for Tackling Complex Problems: a Conversation...

          • See all Posts
  • Events & Training
        • Custom Coaching and Learning Experiences
        • Co-Learning Partnerships
        • Forms and Templates
        • Featured learning
          • Improvement Kata/Coaching Kata Remotely

            April 10, 2023 | Online Live Course

          • Developing Structured Problem-Solving and Leadership Skills using A3 Thinking: Managing to Learn Remotely

            April 17, 2023 | Online Live Course

          • Hoshin Kanri Remotely: Aligning and Executing on Your Organizational Objectives

            April 21, 2023 | Online Live Course

          • Designing the Future Remotely: A Lean Product Development Immersive Learning Experience

            April 24, 2023 | Online Live Course

          • See all Events
  • Store
        • Shopping Cart
        • Featured books
          Managing to Learn: Using the A3 management process

          Managing to Learn: Using the A3 management process

          Learning to See

          Learning to See

          • See all Books
  • About Us
        • Our people
          • Senior Advisors and Staff
          • Faculty
          • Board of Directors
        • Contact Us
        • Lean Global Network
        • Press Releases
        • In the News
        • Careers
        • About us

The Lean Post / Articles / Just Trying “Stuff” Is Not A Real Experiment

Just Trying “Stuff” Is Not A Real Experiment

Problem Solving

Just Trying “Stuff” Is Not A Real Experiment

By Brent Wahba

August 17, 2016

"Today it seems like you can’t throw a stick of butter without hitting someone who is “running a lean experiment” on one thing or another," writes Brent Wahba. The problem with all that experimentation? Many of these experiments are not real experiments at all. Read more.

FacebookTweetLinkedInEmailPrintComment

I grew up in an, um, interesting household – both my parents were scientists. More specifically, they were food chemists, which led to frequent dinnertime conversations about why Polysorbate 80 was 12% better than Polysorbate 79, and how sweet Betty Crocker was actually Sweaty Larry, who ran a Buick-size industrial mixer in the factory. It’s fun to reminisce about our youth, but as I was writing this article, I was also forced to recall a notable childhood trauma – “The Cheesecake Years.”

At the time, my father was working on a new “recipe” (formula and production process) so that a different Sweaty Larry could mass-produce New York-style cheesecake and subsequently displace your grandmother. Nearly every night, FOR TWO YEARS STRAIGHT, Dad came home with samples for us to try, asked specific questions, and then carefully captured our feedback in his notebook.

While that might be considered child abuse today, I now appreciate that I was also witnessing well-constructed scientific experiments with controlled baselines, carefully managed independent variables (ingredients and process controls), and defined dependent variables like moisture, taste, and texture.

Ironically, this was one of my first exposures to good lean thinking.

Today it seems like you can’t throw a stick of butter without hitting someone who is “running a lean experiment” on one thing or another. As an engineer and scientist myself, that should warm my heart, but in reality the feeling is more often heartburn because frequently they are just trying “stuff” and that rarely leads to good science or optimal improvements. “I thought we needed a new screw machine so I bought one as an experiment” is about the level of experimental sophistication that many companies share with me. Sigh. 

Our organizations need us to do much better – not only because of the results, but also the ongoing improvement of our lean skills. Like all good problem-solving, a real experiment starts with a purpose or valid reason for why this is important and defines the gap we want to bridge. Once we learn more about the situation, we can hypothesize about what is causing the gap and which countermeasures could possibly close it. Our experiments then either confirm our hypotheses or else send us back to the drawing board to try again (see article “Problem? What Problem?”).

Unfortunately, it seems like well-crafted experiments are becoming even fewer and farther between as more and more organizations “test” their solutions to give their favorite ideas legitimacy. Some less-than-savory (but all too common) clues that your organization isn’t running real experiments include:

  1. Inadequate measurement systems and / or missing success and failure criteria: “My new 6S process works great!” Really? How do you know without agreed upon standards and an accurate, repeatable measurement system? You really need to define and validate your measurement system before you run the experiment. 
  2. No control group: Website developers often run A (baseline or control) / B (test case or treatment) comparisons by directing half the traffic to either of two website configurations and monitoring which leads to better conversion (users taking desired action). This takes out time-based, urgency-based, economic-based, and other hard to identify and control variation – leaving only a fair comparison of layout and content factors. If you cannot adequately control confounding variables during your experiment, you really need to run a parallel baseline case.      
  3. Too many variables: If we simultaneously change 27 things in our new lean management system “pilot,” how do we know which were the most critical ones? It would be an inexcusable waste if we spent a lot of effort implementing the 24 things that did nothing or had a negative impact. And conversely, if our pilot failed, how could we ever know if we had a bad solution overall, or just too many confounding variables? As is done in software development, breaking large systems into smaller, testable features makes running valid, controllable experiments much easier than building the whole app or implementing the whole methodology and hoping for the best.
  4. Selection bias: Trying a new tool or methodology with an all-star team, more resources, under ideal/low-variation circumstances, or with extra management attention does not prove that it will work in the real world – especially if the sample size is one. When I facilitate value stream mapping for product development, I always ask the teams to capture information about multiple past projects so they can better understand and plan for “normal variation” when they run their new solution experiments. Engaging the whole problem solving team (not to mention those who will be impacted by the change) in discussions about their own unquestioned assumptions and biases before an experiment is launched helps everyone to design a more robust test.
  5. Confirmation bias: Did we interpret the results in a way that only confirms what we wanted the outcome to be? And related is “expectancy bias” where we create our hypotheses and tests to almost guarantee success like “we are testing to see if people in our office will use the new software.” Many people will probably try something new if you ask them to, but how can you be sure it actually creates an important long-term benefit instead of short-term compliance? Confirmation and expectancy bias can usually be controlled by correctly implementing the previously-mentioned countermeasures.
  6. Not capturing/analyzing/reflecting on unintended consequences (see article “The Chief Cause of Problems Is Solutions”): Complex systems, like organizations and markets, have complex problems and many unforeseeable outcomes. Trading one problem for another is not very lean, so make sure you capture, discuss, and actually do something about any unintended consequences before implementing that new solution. 

There are other forms of scientific thinking that can be useful instead of controlled experiments, however. Many product ideas, like Play-Doh (originally a wallpaper cleaner), were created through leveraging anthropology / ethnography (scientific observation) to uncover situational problems or unexpected benefits. In these cases, it is critical to capture all observations in an unbiased manner while not tampering with the situation. This can lead to insights which can later be leveraged to create testable hypotheses in more conventional experiments – like “do children really enjoy playing with colorful, fake dough” and “are kids really going to try eating large quantities of this stuff?” (Yes and yes, in case you are wondering).

Practicing good science is no doubt tough, but like any skill, practice and application in many different scenarios will eventually lead to it becoming second nature.

I am not suggesting that you should never, ever try something new without a detailed A3 or complex Design of Experiment. Sometimes a broad trial helps us prioritize if we want to try something again with more rigor. “Nope, nothing caught fire and nobody died during that nine-minute stand-up meeting experiment, so maybe we should study the +’s and –‘s further.”

And finally, don’t forget to socialize your experiments to get better and broader input, control more bias, learn faster from each other, and pave the way for implementation. For instance, I am running a long-term experiment right now. My hypothesis is that I can remain happy for the rest of my life without ever eating cheesecake again. Thirty-five years later and my theory is still holding up.

FacebookTweetLinkedInEmailPrintComment

Written by:

Brent Wahba

About Brent Wahba

Brent Wahba is an innovation and transformation executive who teaches & leads strategy, product development, sales & marketing, supply chain, and operations advancement. Placing equal weight on value creation and efficiency improvement, his methodology focuses on helping organizations define, develop, and implement their own solutions rather than forcing a new…

Read more about Brent Wahba

Leave a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Related

WLEI Podcast with author Paulo Savaget of The Four Workarounds

Problem Solving

Strategies for Tackling Complex Problems: a Conversation with Paolo Savaget

Podcast by Tom Ehrenfeld

man at wall putting puzzle together

Problem Solving

Learning to Solve Problems By… Wait for It… Solving Problems

Article by Josh Howell

Animation on Starting a Lean Journey

Problem Solving

Getting Started with Lean Thinking and Practice

Video by James (Jim) Womack, PhD

Related books

The Power of Process – A Story of Innovative Lean Process Development

The Power of Process – A Story of Innovative Lean Process Development

by Eric Ethington and Matt Zayko

Welcome Problems, Find Success – Creating Toyota Cultures Around the World

Welcome Problems, Find Success – Creating Toyota Cultures Around the World

by Nate Furuta

Related events

April 10, 2023 | Online Live Course

Improvement Kata/Coaching Kata Remotely

Learn more

April 17, 2023 | Online Live Course

Developing Structured Problem-Solving and Leadership Skills using A3 Thinking: Managing to Learn Remotely

Learn more

Explore topics

Problem Solving graphic icon Problem Solving
Coaching graphic icon Coaching
Operations graphic icon Operations
Product and Process Development graphic icon Product & Process Development

Subscribe to get the very best of lean thinking delivered right to your inbox

Subscribe
  • Privacy Policy
  • Sitemap
  • LinkedIn
  • Twitter
  • YouTube
  • Instagram
  • Facebook

©Copyright 2000-2023 Lean Enterprise Institute, Inc. All rights reserved.
Lean Enterprise Institute, the leaper image, and stick figure are registered trademarks of Lean Enterprise Institute, Inc.

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Learn More. ACCEPT
Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Non-necessary
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
SAVE & ACCEPT
Share this ArticleLike this article? Email it to a friend!

Email sent!