Randomized controlled trials, once a scientific tool that were not easily understood by anyone, are now taking on a fundamentally different meaning as a primary and widely applicable method.
Michael Luca and Max Bazerman
Business en masse is shifting to new methods of decision-making. In the past, the scientific method and experimentation were the preserve of scientists, but now managers of companies from a wide variety of fields are beginning to massively introduce a culture of experimentation in order to make decisions based on them. For example, about 80% of new Microsoft Bing features go through control group experimentation. EBay saves millions of dollars a year in advertising because they did simple research and found that PPC advertising doesn’t pay off for them. And Google has experimentally found out which interview questions are most correlated with work results, and is now giving those questions more weight.
Huge amounts of user data, new computing power, and the unique simplicity and low cost of randomization created the ideal conditions for a new revolution in business – the so-called revolution of experimentation. Companies are increasingly demanding that their decisions are backed by relevant evidence from experimentation, and MBA programs need to pull up to equip tomorrow’s leaders with the skills they need.
Why experiments are needed
Experiments are very useful – and not only in science. Instead of jerking managers on all issues, even the most insignificant (should the background be yellow or blue? Is it better to perfect basic functions or add new ones? Do employees have the authority and motivation to quickly respond to requests?), You can run an experiment, collect results, and make a decision right away – or at least offer your boss a fact-based option. In addition, such data can be shown to stakeholders as proof of progress and transparency.
Experiments fuel innovation. They can prove the concept and validate the value of new ideas early on, before big risks and scaling up. If the experiment is done correctly, and the data is collected and interpreted objectively, it can also help correct erroneous intuitions, wrong guesses, or overconfidence. The scientific method (on which the experiments are based) is the gold standard that removes cognitive biases and allows questions to be answered objectively.
But now that more and more companies are experimenting, they face a huge talent shortage. Experimenting well is not easy. To do this, you need to understand statistics, be able to clearly formulate the problem and interpret the results. But skills alone are not enough. It is important to act consistently and gradually, to use existing knowledge and contribute to a deeper understanding of the issue, and new experiments should build on old ones. It also raises the question of whether managers will be able to abandon their intuitions if they contradict the data, and whether they will be able to cope with the hierarchy and bureaucracy in order to translate the findings into practice.
Some companies hire armies of advanced graduates to do this (for example, Amazon employs over a hundred Ph.D. economists). This is not surprising, since Ph.D. specialists receive education for several years, and the market for positions in universities with the prospect of full-time work will decrease in the future. therefore, specialists with Ph.D. now in abundance. Other companies teach their own employees using specialized methodologies tailored specifically for their industry. For example, General Mills recently recruited employees for the g-works innovative incubator. The vacancy announcement said they were looking for people who “know how to use entrepreneurial skills and experimentation” and are willing to work “through trial and error and quickly experiment to confirm or disprove hypotheses.” Third-party companies (including Fidelity, LinkedIn, and Aetna) are hiring experimental consultants, such as Irrational Labs, founded by Dan Ariely of Duke University and behavioral economist Kristen Berman.
Thus, companies need versatile people with multiple skills. They must be able to design and conduct experiments; have mathematical literacy to interpret the results and soft skills to put the results into practice. Finally, they must inspire their colleagues to create a culture of experimentation in the company. These are the most essential skills for business school graduates today and need to learn scientific method and experimentation to develop them. But business school programs are built around different principles. How can this be changed?
How to teach experimentation in business schools
Some business schools do teach experimentation, albeit on a small scale. For example, marketing educators often talk about the most basic types of A / B tests: for example, how to find out which advertising has the best effect on sales – regular or social, or how to check if a campaign has paid off. Some educators also use Tomke and Beyersdorfer’s case study of Booking.com, which offers travelers online booking services and has a very strong culture of experimentation, with thousands of employees running them every day. The case not only talks about culture, but also allows you to master some basic concepts and terminology of experimentation (for example, p-value, null hypothesis, and Type I errors). But that’s just one task and one case – and what’s more, the notes to it say that most business school students “have little or no experience of online experimentation.”
One of the authors of this article (Elizabeth) talks about experiments in her Management and Leadership in Organizations course at the PMBA program at the St. David Eccles University of Utah. Each year, she divides the students into groups and asks each group to develop an experiment and then discuss it together. This way students can understand a lot about experiments, although the course is not devoted to their use in decision making. There is only one well-known course of this kind – the Michael Luke course at Harvard Business School, but it is not held every year. Real experiments in business schools are rarely talked about.
Much more can be done in this direction. MBA students have access to university libraries and all published literature from which they can formulate and test hypotheses. In addition, they usually have accounts on Qualtrics or similar platforms so that they can create surveys and use statistical packages to analyze the data. If students were taught methods of business research and taught the basics of experimentation, they could immediately start collaborating with real companies, master experiments in the field, and benefit them.
An MBA-level experimenting course should include some fundamentals of scientific method that students can adapt to the context of a particular company or industry. These are the basics:
How to formulate testable and falsified research questions and use existing insights
How to put ideas into practice and formulate objectives
What you need to know about randomization and what could jeopardize your experiment
How the transparency of the process allows you to control and maintain the purity of the experiment
How to use withdrawal statistics
So, we see three main ways business schools could help their students to lead the experimenting revolution. First, you need to teach them to study the scientific literature and develop the results of past experiments, it is also important to emphasize that this is one of the resources for creating new useful knowledge. Second, you need to teach them the scientific method and rules of effective business experimentation using current best practices. Finally, students should be given the opportunity to practice these methods during their studies so that they are more likely to apply them in their work.
But questions remain. How deep can (and should) integrate experimental methodology into an MBA program? Is one course on this topic enough? What is the best way to conduct experimental internships – in companies or with the participation of a supervisor? Each school should answer these questions in its own way, taking into account its mission, current courses, the composition of students and business partners.
For centuries, experts in a wide variety of industries have used experimentation and scientific method to provide reliable and useful information. But modern business too often relies on intuition to cope with the enormous flow of change – and it pays dearly. It’s time to get back to basics – with the MBA programs.
Elizabeth R. Tenney is an instructor at the School of Business. David Eccles University of Utah. Explores the factors influencing decision-making processes in groups and organizations, examines how excess confidence and other cognitive biases affect social interactions and reputation.
Elaine Costa is a PhD student in the Department of Management at the School of Business. David Eccles University of Utah. Explores social perception and mechanisms for evaluating information in the formation of judgments about others and groups.
Ruchi M. Watson is the managing director of the Center for Strategic Leadership at the University of Utah. Prior to that, she worked for 10 years in the corporate sector at various Fortune 500 companies. She is also a Research Fellow in the Department of Entrepreneurship and Strategy, School of Business. David Eccles.