Generative AI in the Real World : Chris Butler on GenAI in Product Management

Format:

Sound recording

Author/Creator:

Lorica, Ben.

Contributor:

Butler, Chris.

Language:

English

Subjects (All):

Artificial intelligence.

Product management.

Physical Description:

1 online resource (1 audio file)

Place of Publication:

O'Reilly Media, Inc. 2025

Summary:

In this episode, Ben Lorica and Chris Butler, director of product operations for GitHub's Synapse team, chat about the experimentation Chris is doing to incorporate generative AI into the product development process--particularly with the goal of reducing toil for cross-functional teams. It isn't just automating busywork (although there's some of that). He and his team have created agents that expose the right information at the right time, use feedback in meetings to develop "straw man" prototypes for the team to react to, and even offer critiques from specific perspectives (a CPO agent?). Very interesting stuff. About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone's agenda. In 2025, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise. Transcript This transcript was created with the help of AI and has been lightly edited for clarity. 00.00: Today we have Chris Butler of GitHub, where he leads a team called the Synapse. Welcome to the podcast, Chris. 00.15: Thank you. Yeah. Synapse is actually part of our product team and what we call EPD operations, which is engineering, product, and design. And our team is mostly engineers. I'm the product lead for it, but we help solve and reduce toil for these cross-functional teams inside of GitHub, mostly building internal tooling, with the focus on process automation and AI. But we also have a speculative part of our practice as well: trying to imagine the future of cross-functional teams working together and how they might do that with agents, for example. 00.45: Actually, you are the first person I've come across who's used the word "toil." Usually "tedium" is what people use, in terms of describing the parts of their job that they would rather automate. So you're actually a big proponent of talking about agents that go beyond coding agents. 01.03: Yeah. That's right. 01.05: And specifically in your context for product people. 01.09: And actually, for just the way that, say, product people work with their cross-functional teams. But I would also include other types of functions, legal privacy and customer support docs, any of these people that are working to actually help build a product; I think there needs to be a transformation of the way we think about these tools. 01.29: GitHub is a very engineering-led organization as well as a very engineering-focused organization. But my role is to really think about "How do we do a better job between all these people that I would call nontechnical--but they are sometimes technical, of course, but the people that are not necessarily there to write code. . . How do we actually work together to build great products?" And so that's what I think about work. 01.48: For people who aren't familiar with product management and product teams, what's toil in the context of product teams? 02.00: So toil is actually something that I stole from a Google SRE from the standpoint of any type of thing that someone has to do that is manual, tactical, repetitive. . . It usually doesn't really add to the value of the product in any way. It's something that as the team gets bigger or the product goes down the SDLC or lifecycle, it scales linearly, with the fact that you're building bigger and bigger things. And so it's usually something that we want to try to cut out, because not only is it potentially a waste of time, but there's also a perception within the team it can cause burnout. 02.35: If I have to constantly be doing toilsome parts of my work, I feel I'm doing things that don't really matter rather than focusing on the things that really matter. And what I would argue is especially for product managers and cross-functional teams, a lot of the time that is processes that they have to use, usually to share information within larger organizations. 02.54: A good example of that is status reporting. Status reporting is one of those things where people will spend anywhere from 30 minutes to hours per week. And sometimes it's in certain parts of the team--technical product managers, product managers, engineering managers, program managers are all dealing with this aspect that they have to in some way summarize the work that the team is doing and then shar[e] that not only with their leadership. . . They want to build trust with their leadership, that they're making the right decisions, that they're making the right calls. They're able to escalate when they need help. But also then to convey information to other teams that are dependent on them or they're dependent on. Again, this is [in] very large organizations, [where] there's a huge cost to communication flows. 03.35: And so that's why I use status reporting as a good example of that. Now with the use of the things like LLMs, especially if we think about our LLMs as a compression engine or a translation engine, we can then start to use these tools inside of these processes around status reporting to make it less toilsome. But there's still aspects of it that we want to keep that are really about humans understanding, making decisions, things like that. 03:59: And this is key. So one of the concerns that people have is about a hollowing out in the following context: If you eliminate toil in general, the problem there is that your most junior or entry-level employees actually learn about the culture of the organization by doing toil. There's some level of toil that becomes part of the onboarding in the acculturation of young employees. But on the other hand, this is a challenge for organizations to just change how they onboard new employees and what kinds of tasks they give them and how they learn more about the culture of the organization. 04.51: I would differentiate between the idea of toil and paying your dues within the organization. In investment banking, there's a whole concern about that: "They just need to sit in the office for 12 hours a day to really get the culture here." And I would differentiate that from. . . 05:04: Or "Get this slide to pitch decks and make sure all the fonts are the right fonts." 05.11: That's right. Yeah, I worked at Facebook Reality Labs, and there were many times where we would do a Zuck review, and getting those slides perfect was a huge task for the team. What I would say is I want to differentiate this from the gaining of expertise. So if we think about Gary Klein, naturalistic decision making, real expertise is actually about being able to see an environment. And that could be a data environment [or] information environment as well. And then as you gain expertise, you're able to discern between important signals and noise. And so what I'm not advocating for is to remove the ability to gain that expertise. But I am saying that toilsome work doesn't necessarily contribute to expertise. 05.49: In the case of status reporting as an example--status reporting is very valuable for a person to be able to understand what is going on with the team, and then, "What actions do I need to take?" And we don't want to remove that. But the idea that a TPM or product manager or EM has to dig through all of the different issues that are inside of a particular repo to look for specific updates and then do their own synthesis of a draft, I think there is a difference there. And so what I would say is that the idea of me reading this information in a way that is very convenient for me to consume and then to be able to shape the signal that I then put out into the organization as a status report, that is still very much a human decision. 06.30: And I think that's where we can start to use tools. Ethan Mollick has talked about this a lot in the way that he's trying to approach including LLMs in, say, the classroom. There's two patterns that I think could come out of this. One is that when I have some type of early draft of something, I should be able to get a lot of early feedback that is very low reputational risk. And what I mean by that is that a bot can tell me "Hey, this is not written in a way with the active voice" or "[This] is not really talking about the impact of this on the organization." And so I can get that super early feedback in a way that is not going to hurt me. If I publish a really bad status report, people may think less of me inside the organization. But using a bot or an agent or just a prompt to even just say, "Hey, these are the ways you can improve this"--that type of early feedback is really, really valuable. That I have a draft and I get critique from a bunch of different viewpoints I think is super valuable and will build expertise. 07.24: And then there's the other side, which is, when we talk about consuming lots of information and then synthesizing or translating it into a draft, I can then critique "Is this actually valuable to the way that I think that this leader t...

Notes:

OCLC-licensed vendor bibliographic record.

OCLC:

1549473330