On May 7, the Digital Data Design Institute at 性视界 hosted Leading with AI: Exploring Business and Technology Frontiers. The conference featured breakout sessions where HBS staff and industry experts discussed specific and specialized topics. The breakout session on The A New Paradigm for Skill Development: A Large-Scale BCG Experiment featured speakers from he Boston Consulting Group (BCG): , Global Director of the BCG Henderson Institute; , Principal; and , Managing Director and Partner of BCG X.
Key Insight: Beyond the Jagged Frontier
鈥淸We wanted] to see whether our consultants [with] no or limited data science capabilities鈥ould actually solve data science issues [using ChatGPT].鈥�
Fran莽ois Candelon
In 2023, Boston Consulting Group (BCG) and the HBS Digital Data Design Institute conducted an experiment on the use of AI in skill development (see the working paper, ). Briefly, this was a scientific study with real employees that looked at how AI can automate some tasks to improve performance. One result was a new term, the 鈥渏agged technological frontier,鈥� where some tasks are easily done by AI, while it cannot do others that are similar in difficulty level.
In a more recent study, BCG wanted to look beyond these results to see if AI could help them up-skill volunteer consultants without backgrounds or experience related to the study tasks. Fran莽ois Candelon, Dan Sack, and Lisa Krayer presented 10-day-old preliminary results of this study.
Key Insight: Experimental Structure and Early Results
鈥淸C]an we give people tooling that helps them get better or more efficient with the tasks they know how to do? 鈥C]an we understand what they can do with something they have no idea what they’re doing?鈥�
Dan Sack
For this study, 525 consultants and 40 data scientists in different age groups from around the world worked on three tasks:
- Predictive analytics (using historical data to predict games in the future)
- Data cleaning (non-programmers assessing the accuracy of data)
- Statistical analysis (identifying when ChatGPT analysis was incorrect)
The data scientists were the baseline, one group of consultants used ChatGPT to work on the tasks, and the other group of consultants did not use ChatGPT to work on the tasks. Both consultant groups received some training on how to use tools available to them:
- Chat GPT group: Prompting best practices, few-shot prompting, chain-of-thought prompting, and Code Interpreter tool
- No ChatGPT group: Stack Overflow, Khan Academy, Python documentation, and search strategies
The study team segmented results based on consultants鈥� previous experience with analytics, data analysis, and learning models to avoid skewing the results. But some early results showed that consultants with no programming experience achieved 17% of what a data scientist could accomplish within the given 90-minute time frame on a data cleaning task. And the team found that with statistical analysis, everyone got a 鈥渂ig boost,鈥� even those without experience. The predictive analytics task was more difficult to measure because there was no 鈥渞ight answer.鈥� But they found that consultants with the least experience had better outcomes on this task, and want to find out why.
Key Insight: Learning and Development
鈥淸W]e want to understand how this works for learning and development as well, not just can people do this, but do they learn something while they’re doing it?鈥�
Dan Sack
After the study, the team tested participants鈥� retention and found similar levels across the segments. They want to understand the learning curve over time, beyond the 90-minute test period. Dan stressed that the study was not designed to teach participants something beyond the one-time tasks. The multistep, iterative nature of the tasks and ChatGPT suggest the need to design longer-term tasks to measure learning and retention. And future tests also must design interactions between teams and systems because AI often does not work 鈥渞ight out of the box in one shot鈥� and must iterate to find answers.
Key Insight: Confidence, Overreliance, and Adoption
Fran莽ois Candelon
鈥淸B]y pushing the adoption, basically you are igniting a positive, virtuous cycle鈥he more you use, the more you trust, and the more you trust鈥he more you use鈥he worst day of AI is day one, so it’s really important to do that and I strongly believe that AI transformations are of a different nature than the others because it has a massive impact on your professional identity, competency, sense of autonomy, sense of belonging, and/or relatedness.鈥�
The team discussed the risk of overconfidence leading to overreliance on AI, and skepticism that makes people and companies avoid it. Companies must understand the jagged frontier and when AI is helpful and when it is not. Increasing engagement with AI can help drive adoption by building confidence in themselves and the tool. Companies should be intentional when rolling out AI tools and clarify why it is valuable. They must consider redesigning their processes and ensure proper supervision to make sure the quality of the output is high.
Frameworks
Change Management
鈥淲hat was interesting in our previous experiment is that when we are asking the consultants whether they were feeling that they were afraid, because maybe strategy or problem-solving will be done, they were saying, we’re not that afraid because we believe there are some other things to do where we are still, for instance, change management becomes very important.鈥�
Fran莽ois Candelon
Adoption of AI involves important work on change management and company culture. For BCG and many companies, AI can be a competitive advantage, but it also involves looking carefully at workforce planning and talent strategies鈥攔ecruiting, interviewing, learning and development, and promotions. Companies must realistically assess these systems鈥� capabilities. Part of the change management process is educating employees on how to work effectively with AI and delegating work to AI for first-draft and peer-editing work.
In Fran莽ois鈥� view, 鈥淎I can help culture鈥e believe that we need to change culture to adapt to AI, but really I would say it’s almost the other way around. AI can help change culture in a positive way鈥� through improved employee professional identity (competence, autonomy, and connection) and morale. As Fran莽ois noted, 鈥淸it鈥檚] not about optimizing the technology, it鈥檚 about optimizing the way humans use this technology.鈥�
Implementation Questions
鈥淚t’s incredibly important to listen to people and not just make decisions. And talk to people about how, because we don’t know how this is going to impact people’s lives and we won’t know until we talk to them and truly take that initiative to listen.鈥�
Lisa Krayer
The session participants and the team surfaced many questions around implementation of AI, including:
- How do you teach discernment around AI results?
- When should you be confident (or not confident) in results?
- How do you supervise AI work when you鈥檙e not trained in AI?
- How do you monitor negative AI information and experiences?
- How should employees be hired or promoted when their work is or is not augmented by AI?
- How will data scientists鈥� jobs change? How will employees鈥� jobs change if AI shifts capabilities and responsibilities?
- When you implement AI, do you need to change workflows and breaks to ensure employees still interact in person? How can employees share in the benefits of productivity gains?
Meet the Speakers
is currently a Partner at Seven2, a French private equity firm. Previously, he was a Managing Director and Senior Partner of Boston Consulting Group and Global Director of the BCG Henderson Institute, BCG鈥檚 strategy think tank. Fran莽ois earned his MSc, Economy from 脡cole Polytechnique, a degree from Mines Paris – PSL, and a DEA from Universit茅 Paris Dauphine – PSL.
is a Principal at Boston Consulting Group, where she has conducted research into the impact of generative AI on individuals and businesses for the BCG Henderson Institute. Lisa obtained her BS in Physics from UC San Diego, and her MSc in Electrical and Electronics Engineering and PhD in Electrical and Electronics Engineering from the University of Maryland.
is a Managing Director and Partner of BCG X, BCG鈥檚 tech build and design division, building digital products based on machine learning and AI. Dan is BCG鈥檚 Global Data Science Chapter lead, and he leads BCG X in the Nordics. Dan earned his MBA from Stanford University Graduate School of Business and his BE in Mechanical Engineering from Dartmouth College.
Additional Resources
Fran莽ois Candelon:
- (research paper)
- (book)
Lisa Krayer:
- (article)
- (article)
Daniel Sack:
- (report)
- (report)