This episode features an interview with Dustin Pearce, Vice President of Infrastructure at Instacart. Dustin was previously the Director of Service Engineering at Slack, as well as the Head of Platform for Life360. In this episode, Dustin talks about the advances online shopping has made during the pandemic, where startups should place their priorities, the future of infrastructure, and much more.
This episode features an interview with Dustin Pearce, Vice President of Infrastructure at Instacart. Dustin was previously the Director of Service Engineering at Slack, as well as the Head of Platform for Life360.
In this episode, Dustin talks about the advances online shopping has made during the pandemic, where startups should place their priorities, the future of infrastructure, and much more.
--------
How you approach data will define what’s possible for your organization. Data engineers, data scientists, application developers, and a host of other data professionals who depend on the Snowflake Data Cloud continue to thrive thanks to a decade of technology breakthroughs. But that journey is only the beginning.
Attend Snowflake Summit 2023 in Las Vegas June 26-29 to learn how to access, build, and monetize data, tools, models, and applications in ways that were previously unimaginable. Enable seamless alignment and collaboration across these crucial functions in the Data Cloud to transform nearly every aspect of your organization.
Learn more and register at www.snowflake.com/summit
Steve Hamm: [00:00:00] so I want to say, Hey to Dustin piers, our guest today on the podcast. He is the vice president of infrastructure for Instacart, one of the hottest hottest startups in the land in the country and welcome Dustin.
Dustin Pearce: [00:00:16] Thanks for having me pleasure to be here, Steve.
Steve Hamm: [00:00:19] tell us a little bit about your career.
I see that you studied microbiology in college. When and why did you make the leap to information technology?
Dustin Pearce: [00:00:29] No for me, it was interesting. I grew up around computers. My father, uh, was a, uh, was a programmer and, uh, you know, kind of systems architect back in the seventies. And so I grew up sitting in, you know, an IBM three 60 room listening to my father, halt those big mainframe computers. They would make this huge kind of loud klaxon sound.
And I always thought computers were fantastic, but I never thought it would be my. Job. I was very into science. And so after I got towards the end of school, it was 1994. And that year is important because that's pretty much when the internet was hitting the scene, that scape was coming on the scene. And I spent some time at Netscape doing some training.
I was working at Genentech at the time, actually after college and did some training on just webpage development. And I looked at just kind of web pages and the internet. And I said, Holy mackerel, this is what I love. This is what I wanted do. And I marched back to Genentech, talk to an it manager said, Hey, this is what I want to do.
I don't want to do science. And I'll never forget that person had the greatest advice. They said, you know what? You don't want to work here. This is going to be huge. You should go out and just get a job at a consulting firm and just learn as much as you can. So I took us advice and I did, and that really launched my career into kind of it, and primarily at my entire career has been focused around the internet.
Steve Hamm: [00:01:53] You know, it's possible that that are our paths nearly crossed back in 1994. At the time I worked for, um, PC week, PC week side, and, you know, we were learning about the internet and I actually called up those guys down at mosaic communications. You know, than mountain view and said, geez, we, you know, we'd like to learn a little bit more about web browsers.
So they invited us down and, and, and they gave us our first lessons in web browsers. And of course at that time it was eight people Mark Andreus and an eight and seven other engineers. So we got in on the ground floor and it was just, isn't it amazing to think where
we've come to in this in only a few decades?
Dustin Pearce: [00:02:33] It's really remarkable,
you know? And, and, and, and we'll talk about a little bit about how technology's becoming critical infrastructure, right. And more so now than ever. And so I, you know, it's, it's been a really interesting and wild ride. It's, it's, it's wonderful to find something that you love to do.
The old adage of like, if you love it, then it doesn't feel like work. And so, um, some days, you know, I, I, I still love to follow science. Uh, science is getting increasingly more complicated every day. So in some ways I feel like, you know, computers are easier than science, but, um, I've loved it. I love building things.
And you know, for me, my career is kind of. Progressed, it's very typical kind of consulting and then development. And then ultimately at some point all along my career, I'm also a high school water polo coach and have been for a very long time. And so I think a combination of all of that coaching really drew me towards management and kind of supporting teams and doing strategy work.
And so management's always been a large part of my career. Um, and I think in the last, I'd say 10 years, I've been just really focused on like hypergrowth startups or very rapidly growing startups and trying to help them.
Steve Hamm: [00:03:33] Yeah, I see, you know, you've been at a bunch of companies, life three 60 babysitter.com more recently Slack, some really hot companies and fast growing, fast changing, almost chaotic business environments. So you've come through these, you've been a manager. What are the lessons that you've learned about working in the hyper-growth environment, uh, that are guiding you now that you're at Instacart?
Dustin Pearce: [00:03:57] know, I, when I was at Slack, which was the fastest kind of growing company, I think ever, certainly in an enterprise case, um, the, the thing I noticed and noted was that things are accelerated, meaning. It things happen very, very quickly and you have to change direction to be responsive to the two kind of opportunity or problems.
And they in a very rapid pace. And I think it is actually a pace that is accelerating so much that it's a little unsettling for the humans that work there, trying to kind of keep up with that pace. And yeah. It is important for, you know, rapid growing startups to really have a sense that, that speed has a toll on the people who are trying to go with it.
And you have to try to find some kind of sense of sustainability. Um, you know, I think startups and, uh, have to aggressively go after their markets. And I think the most important thing is they have to focus on innovating the product and moving as fast as they possibly can. And, you know, the barriers to entry for business, keep going down and competitors can enter your space so fast that if you are not pedal to the metal, moving as quickly as you can to deliver more value and expand the value that you're offering your customers, somebody's going to blow right past you.
And so I think that what I've learned is that, right, there's this kind of inflection point for every company. It's a little different. Where they have something to protect. They have product market fit. They start to create a certain amount of revenue. And in order to protect that, they feel like they have to siphon some of their attention to their resources away from productivity.
And I think, you know, developing a very creative and very focused strategy on how are you going to do both? How can I build a company around this product, but how can I also keep pushing this product fast, fast, fast to stay away from my competitors is really critical. I think the last thing is, you know, uh, one of the things we said, I used to say when I walked around the halls at Slack is that we don't ship code is that we craft and we care for customer experiences.
And this really goes to the kind of, you've heard other outages of, if you build it, you run it. But development teams are not just kind of trying to make a computer do things is that they're delivering products to customers. And on a global scale, you have to pay attention to production. You have to pay attention to your customers cause they all are having variable experiences.
And so all of these things combined create a very challenging, very demanding environment. I think some of the things that were true in the nineties are still true now, which is when you work in these places, you get thrown into the fire and you learn faster than any place else you would ever kind of work.
So it is very demanding in that regard, but I think the idea of having a rapid impact and learning and iterating and being on the very edge. It's is really fun.
Steve Hamm: [00:06:32] No, I think that's incredible and very important that these times, I mean, you, you arrived as VP for infrastructure at Instacart. I think back in January, just before the COVID crisis hit. And you know, these are truly remarkable times. So you came into this company at that time when it's hyper-growth right.
Took off even more. You're the guy who has to, to make sure that the infrastructure is there powering all the analytics that let them. Respond quickly. So it seems like an incredible job at an incredible time. So how have you, as you come in, what have you had to do? Have you had to react? How are
Dustin Pearce: [00:07:06] Yeah, thanks. Thanks for asking. It is, it has obviously been a very wild ride. I mean, um, Slack was an interest sting, you know, a journey, you know, trying to create a global business communications network with like five nines reliability. Across the globe was a huge challenge and very taxing. And at some level, Instacart was a very fast moving and growing company, but it felt like it was going to be this kind of shift from, uh, this hyper stress of being responsible for business communications to grocery delivery.
And it, it, in my mind's eye, I felt like it was like a stress level down little. Did I know that within three months that we would have total total chaos? You know, I, I, I think that. COVID and the explosion and the demand for digital services in general, but certainly grocery delivery is chief among them was unlike anything we've ever seen before.
So I had been at rapid startups. The other executives of the team had been at rapid startups, but nothing. Even comes close to what we saw in March and April, where you have a company, depending on what metric you're looking at, scaling 10 X in a matter of weeks, you know, we were doubling every couple of days.
And so, yeah. You know, I think that the key thing here is that. At Instacart. A lot of this had to do with the people and the people on our team. And they were very focused on the responsibility. We felt this deep and grave responsibility to ensure that groceries were flowing. People really needed this service and they needed us to be able to scale it.
Um, technically, there are a lot of challenges when, when this happens, uh, you know, databases get stretched, uh, systems get stretched and engineers are working long hours, all that happened. But I think for me, I really this sense of mission and duty that the Instacart engineering team had. And for me as the leader, it's a lot of kind of making sure that people are feeling supported and cheering them on because while we're feeling this crush of growth, which, you know, may feel like success, we also are cooped up at home.
We have kids, you know, all of our people are going through the same exact things and the same kind of amount of uncertainty that everyone else's. And you're still trying to do this very challenging work. And so I found my position as a leader was really trying to make sure that you have this kind of patience and compassion that, you know, Hey, this may not matter.
10 minutes, 10 days, 10 years, you really try to keep things in perspective for people to keep people focused on what matters. Um, a lot of prioritization decisions, a lot of triage of what is the issue. This is that is hurting us right now. And so, you know, for us, we were able to keep the site up through a lot of clever engineering.
I think that, um, A lot of the work that was done, even before I got there at least a year before I got there, there was a lot of investment that was being made in the infrastructure to make things scale that I think if we had not done, we would not have survived. And you know, there are lots of services, I think in that very crunch time, March and April, you saw that they just really couldn't serve the demand.
I mean, at some point the demand felt infinite.
Steve Hamm: [00:10:04] when you came in, did you have to make any strategic shifts with technology or was it more like, Oh, we've already got our right platform. We've got our right strategy. Let's just, let's just, you know, scale it
Dustin Pearce: [00:10:15] Yeah. I mean, at some level there was no time, right. You know, within two months, uh, there, there's not a lot of time for me to make the really big strategic changes as far as our technology approach is concerned. I think for me, the, the key thing and the thing that I was really trying to do, and as an infrastructure team is to really tool and, and, and emphasize service ownership and good decision making at the front lines.
Um, It's a very federated group. We have autonomous teams who own and operate infrastructure on their own. So this isn't this concept of centralization and it allows us to move very quickly. So, you know, I, I, I do think that. Uh, the strategic decisions really came in, like what can we do right now to avoid downtime?
Right? And, and a lot of that is very technical, right on how you use split databases and deal with database technology. Databases is really where you tend to have a choke point. And so, I mean, related to the snowflake services like this, where you have. Set up relationships where it's essentially already somewhat of a bottomless pit.
Um, those are areas you don't have to worry about. And that was an area that, that, that just scaled right along with us. And we had a couple of vendors relationships where it wasn't a lot of engineering on our part to kind of make it happen. We just kind of scaled the service up as we needed it.
Steve Hamm: [00:11:24] I want to explore this, the kind of the crisis management you've been doing a little bit more. Um, so. You know, at the same time, do you have all the, all the things we've discussed? My sense is that the other things happening is there's really a draw for, um, for having delivery the same day. And this company, how has the issues of performance even more? But what I'm really want to get into is, you know, we've talked a bit about infantry sure. And performance of the database, things like that. Talk about the kinds of analytics and the kinds of maybe even, you know, like machine learning, stuff that your data scientists and your business analysts are doing at Instacart.
how are they, what are they doing? How does your infrastructure enable that?
Dustin Pearce: [00:12:07] I think the key thing to understand is that, uh, Instacart is a marketplace, right? It's a place where. You know, buyers and sellers come together for a relationship. So there are people who need groceries and there are people who are willing to deliver those groceries or shop and deliver for those groceries.
And so the key thing here is that you have to be, be able to balance the marketplace. If someone goes to make an order. Your service doesn't work. If there's no one to shop and deliver it. And on the flip side, if there are too many people do shop and deliver it, then there is no work for them to do if there's no demand there.
So from our perspective, you know, one of the very important aspects of data that drives our business are the models that we create that really understands and what we call availability, which is kind of availability of people to kind of take orders and then demand, or, or this idea of. Uh, who needs, you know, what customers are shopping for groceries and you know, .
To me. What's fascinating is obviously this is very geographic ugly, kind of different, you know, there are rural areas that operate very different than urban centers, different geographies, different markets, all have different evolutions or different kind of needs at different times. So if you think of COVID is specifically.
This was not an instant thing that hit everywhere all at once. Right? Different markets were having different kind of experiences. Some were locked down. Some were not, they were locked down at different times. And so we need very good data in order to understand where do we need to apply, you know, our kind of demand increase or supply increase, where do we need to balance the marketplace?
And I think. In order to do all that, you have to crunch it, enormous amount of data. Right? And so for us, those models are trained by real time feeds of data from our data warehouse into training our machine learning models. And then those characterize the marketplace availability. And, you know, there's a really fascinating kind of space because, you know, we.
are becoming more and more accustomed to these digital marketplaces, whether it's Amazon or others, where, you know, sellers and buyers come together at Etsy, , I mean, you know, digital marketplaces is really the new shopping mall and, you know, these things are, are owned and operated on a global scale.
Uh, and, and to, to serve all of those markets and all of those people in ways that are meaningful to them requires a whole lot of data, a whole lot of segmentation and understanding of those markets.
Steve Hamm: [00:14:26] Yeah, yeah, no, I know that Instacart was one of those companies that was born on the cloud. Have you always had the data in the cloud or did you have some kind of on premises databases and
things like that handy for analytics and things like that?
Dustin Pearce: [00:14:40] So the, the last three companies, including Instacart I've been at have had zero on prem. Compute or storage. They are 100% cloud based whether that's their email or their data warehouse. And so what I find is that most, uh, an Instacart is, is no stranger to that. A lot of them start their journey with, you know, a cloud provider hosted service, or a very simple, very large database in the AWS where a lot of people use Redshift.
And I've seen that just about everywhere as kind of the very first step along the journey. And then eventually. You know, they start to kind of run into concurrency problems or scale problems. And they really kind of realize, Oh my goodness, to manage data at the scale, I'm going to have to build a team of people even to use this online service.
And so, you know, I think the needs evolve and that has really seen the rise of some of these higher, what I would call higher order managed services that really abstract away. Some of that complexity.
Steve Hamm: [00:15:35] Yeah. Well, I know that I'm aware that some companies, you know, keep their data in the cloud, but when they do the data science, they pull the data down into, on premises.
Dustin Pearce: [00:15:44] right,
Steve Hamm: [00:15:44] Uh, machines and, and run it there. It's I mean, that seems really inefficient. I mean, what are the advantages and disadvantages of the two approaches and why are you happy?
You're doing all in the cloud.
Dustin Pearce: [00:15:54] think for me as a person who is responsible for security for a very, very large e-commerce marketplace. You know, the access to data and where data goes is critically important, right? And as soon as data starts to leave the cloud, leave the confines of what, what I would call a controlled or, uh, environment where roles are defined and access levels are defined.
And it's put somewhere local, whether that's a protected data center, or even like a, let's say a machine. You know, algorithm, developers, laptop, you lose control and, and the, the chance of breaking trust with your customers just goes up and up and up. And it's just something companies these days can't afford to do.
So from an infrastructure security perspective, I feel like having online workflows for developers to work this idea that. You know, you have to be online in order to kind of develop algorithms is not an unthinkable thing anymore. Right? Like, you know, being online is just part of life. It's, it's, it's a, it's a basic utility.
And so I think that there's some older ideas that like, I need to be offline right. In order to do this. And I, and that just doesn't ring true to me anymore. And certainly we get a lot of advantages by having things in the cloud. We get better collaboration. We get better control. Um, And so, yeah, I find that there is some amount of training or, or bringing people along the journey who maybe are used to older models.
Right. Um, and treat you, teaching them how to leverage cloud-based kind of development practices, but, um, it's certainly, you know, creates better velocity for us as a company.
Steve Hamm: [00:17:25] Now I know that the relationship between Instacart and snowflake predates your time there. I'm not sure exactly when it did start, but it would be great. If you could take us on a little walk down memory lane or down history lane and explain why, why did Instacart first go with snowflake or first consider
Dustin Pearce: [00:17:43] You know, there there's two main kind of components. And I don't remember. No, actually I remember I wasn't there, but I don't know the exact time, but I can tell you that it was simplicity and scale. Right. And so the scale component is they were using. Very typical data warehouse technology from the cloud provider from AWS.
And, you know, as the number of people, if you back up a little bit, uh, Instacart is a very data centric company, right? As measure, measure, measure, measure, measure everything. And so you can imagine the demands on a data warehouse at a company that has that ingrained in its culture ramp up very quickly.
And so the ability for multiple people to start asking lots of questions of this data warehouse, You know, at the same time, very quickly revealed that this monolithic data warehouse was new, going to cut it, this idea that there's a single place and everybody's asking their questions there, they just, things got slower and slower and slower.
And I think in, in order to accommodate that and I've. So I've done this at like three 60. We did it at Slack. You start engineering a lot of complexity to try to kind of continue to provide access and performance for all the people who have these critical business questions. And at the time I think snowflake, you know, the promise was is that, Hey, we can abstract away a lot of that problem for you.
And we can certainly create a bottomless pit, but that's not necessarily the issue is that we can serve all your users simultaneously. We can scale up and provide them with performance queries and answer business questions. And, you know, that's been our core mission all along. Like we don't. We're not super excited about like we're the best kind of open source implementation team in the world.
Cause that doesn't necessarily do anything if it doesn't answer business questions. And so that's my kind of, you know, basic kind of, uh, there's a couple others where I think being sequel based, there's been a lot of. There's been a lot of, uh, you know, discussion and experimentation over the years around how to kind of democratize data.
And I think we find over and over again, that CQL is just a somewhat universal language in that space. And having a SQL based tool was really important. Um, and yeah, there is native support internally as well for, for our events stream. You know, we, we track. So much data about the experience so we can understand it and improve it.
And each of those events has its own little data structure and the ability to just put that event in snowflake and snowflake knows what to do with it. And we can talk to the individual pieces without us having to do extra compute was a huge advantage. So again, it, it just really offloads a huge burden from us.
Steve Hamm: [00:20:07] Yeah. So that's a bit of a history. So how do you use stuff like today? How does it fit into your overall technology strategy?
Dustin Pearce: [00:20:15] So, um, the snowflake is the centerpiece of decision making at Instacart. Right. And I mentioned that data driven decision making is critical in every aspect. So whether it's our planning cycle, where people are trying to discover what are the areas we believe that we can make the biggest impact, whether it's, you know, tracking the execution of that planning, understanding the results.
All of that is, is, is embedded with, um, the Butte, I should say it with data, right? So every goal is, is attached to some form of dashboard that is tracking and measuring the impact of whether we're actually achieving what we're trying to do. You'll see that. Every single product team has embedded data scientists.
I mean, this is how you know, and that's pretty rare. Like I don't typically see that. Usually there's a small group, maybe data science group, that's off to the side of specialist. Right. They, they are not integrated yeah. Every aspect of product development. And so for us being able to scale out our snowflake usage and design warehouses and design complete kind of.
Um, which functional areas, uh, and, and manage all of that capacity and help people understand, uh, what is actually in discovery is, has been a challenge, but it has been central to how we run our business. And, you know, I think more than just doing queries and, uh, you know, more than just doing queries, we do things like invoicing.
Directly out of snowflake. Right? So there's a lot of operational kind of components to, to snowflake and the data that we store there. So it's our, it's our master repository, if you will.
Steve Hamm: [00:21:44] Yeah, it's interesting. I mean, snowflake of course is one of its own major users. I think almost every function in the company is handled through the data warehouse, the data, the data platform. Uh, and it sounds like you do the same, so that's,
Dustin Pearce: [00:21:57] Yeah, I think it's a generation of startups, you know, whether it's Y Combinator has in viewed this kind of, uh, this idea. But my, my, my, my observation is that this is, this is critical to moving fast and, and, and keeping your PR your product innovation moving.
Steve Hamm: [00:22:11] Yeah. Yeah. You know, it would be really helpful. Dustin, if you could kind of walk us through one of your use cases for snowflake, just like, you know, start with, what was the problem? What did you need to accomplish? How does this do it and what kind of results you're getting?
Dustin Pearce: [00:22:24] I think the, one of the more interesting use cases that I thought was relatively novel is that, uh, Instacart is, uh, you know, uses the snowflake for our catalog data. And the catalog for an eCommerce company is obviously a critical component. And for Instacart, with tens of thousands of stores around the country and, you know, hundreds of partners, you can imagine, you know, each of those stores, those grocery stores, if you will have inventory and have products and skews and.
Our job is to try to make that as transparent to the user as possible. And so the ETL or the, the it's called extract transform. You know, and load. So essentially the process for moving data from the stores into Instacart is something that we really had a problem with scaling. Right. And the issue there is not so much just the idea of picking up data from, you know, Point a and putting it in database B it's all the transformation that occurs along the process.
How do you detect duplicates? How do you detect price anomalies? There's very often the data inside. Let's say a stores catalog that they're sending us just has the wrong information for pricing, and that can be really bad in an online experience. And so we are constantly trying to data. Correct. And, and to normalize the data, looking for duplicates, because at the end of the day, the customers.
Don't really understand the complexities of trying to merge 30,000 stores worth of catalogs. They just want to see like, is there Coca-Cola or is there not right? And so for us, you know, moving our catalog into snowflake was certainly, uh, an honor resting storage story on its own, as far as, uh, not using kind of relational databases that you typically would.
And instead using this kind of. Data warehouse service. But for us, the use cases really revolve around this pipeline, this transformation pipeline. And there are features inside of snowflake called user defined functions in Java, the script. And that really allows us to do just SQL transformation's much more complicated and much more sophisticated transformations and checks of the data as it's flowing in.
And the end result is that, you know, our catalog, it, we were able to onboard new stores and new retail retailers in a matter of sometimes hours or even days. And I think this just really. Helps us build trust with our partners because, you know, you can imagine if I'm I'm a grocery store or even a big grocery store chain.
And I realized, Oh my goodness, like I have to be able to sell my stuff online and you're staring down that. Uh, imagine like how long you think it would take to get to that point to be able to deliver and how many people you'd have to hire and how much technology you'd have to master. And then, you know, we can come and partner with you and you can be up and running in a matter of days because of our ability to intake all of your data through this pipeline and normalize it into the marketplace.
So this has been, I think, It really important component, not just to Instacart's growth, but in our ability to be ready to serve America and its time of need and be critical infrastructure, as far as food delivery is concerned. And so it's a novel use case.
Steve Hamm: [00:25:31] Yeah. Well, that's really interesting.
Yeah. But, um, so when somebody shops on Instacart, do they shop at a particular retailer or do they shop for a particular item?
Dustin Pearce: [00:25:42] Good question. So, uh, the mechanics of a Instacart order is you make an order at a particular store. So the, the experiences that you are shopping at. Grocer X, not that I'm looking for a grocery item and show me all the places that's at and let me pick on whatever price reviews. And I think that's a really important.
Aspect. And we've tried to preserve as much this kind of the mentality. Cause what, what we found early on is that people have strong brand affinity. This is my store. This is the store I go to. I trust them. I trust their produce. I trust them. They don't put moldy bread out. And you know, I think that. Based on our understanding of how people actually think about it.
We were able to model our shopping experience, very similar to how people think. And so, you know, when, when you look at Instacart and you see your store, the place that has the butcher that you trust on there, then you are more inclined to engage them. This kind of anonymous warehouse of meat, that where my food comes from, people we'll trust it.
Right. And so that's something we learned early on and it really, I think has paid off.
Steve Hamm: [00:26:48] Yeah, no, that makes total sense though. I think I wonder about in terms of convenience. I mean, obviously I shop at four or five different grocery stores, depending on what I'm looking for. Is there any way you can kind of have a cart and, and have one delivery from several stores or is that just too
Dustin Pearce: [00:27:05] It's not too complicated. Um, and it is possible. Well, I think that our experience is that, you know, creating that experience of going to different stores and creating different orders is not dissimilar to real life. And, you know, one of the things that was really interesting we found during COVID is that we had a large influx of less tech savvy, much older users.
That came onto the platform really quickly. And so we had to do a lot of, kind of very rapid user studies to understand our platform and try to make it as clear as possible. And so I think we're always learning and we're always trying to innovate and make it easier to understand and make it easier to execute.
I think that. To me, it's about trust and it's about a positive experience, less than just like raw, brutal efficiency, right? Like, you know, the, the shopping experience, particularly for groceries is very, it's a very trust, you know, uh, driven relationship, um, versus like, if you're buying a set of headphones, you know, you, you just look for the lowest price, right?
Steve Hamm: [00:28:02] I get that. . Hey, so you know, you are, um, A pretty serious technologist and a pretty serious manager of technology. And I've, you know, just reading about your, your work and your interests, you pay a lot of attention to costs.
You pay a lot of attention to efficiencies. You know, I think, I believe that your field, you practice is called infrastructure engineering. So I really wanted to understand kind of what are. What is that? What's kind of, what's the worldview. How do you go about thinking and how you go about managing teams or managing the work?
Uh, out of that.
Dustin Pearce: [00:28:39] It's infrastructure engineering, you know it maybe. And it's. Kind of worst form is just a rebranding of what people would call ops. Right. You know, like a knock where people go and they carry pagers and they respond to when things are broken. Uh, it is, uh, it is a kind of a holdover of the kind of data center days are some of the older, uh, methodologies for developing software where developers wrote code and operators were in production.
Um, but what I found was that. You know, over the years, since 2000 and the Dawn of agile and certainly the DevOps movement, I think most people realize that the correct canonical way for organizing a software company to move fast, to engage with his customers is a federated group of focused teams who have a lot of local agency to make decisions.
Right. And. Over and over again, I think that people have empirically kind of experienced this as the recipe for success. And so all of these companies are not following that pattern, but what I think they haven't found is that let's say all of a sudden now I have a thousand developers organized in a hundred teams at some level that starts to become very unwieldily.
Right. Either the user experience breaks down or the developer experience breaks down. And so I really see infrastructure in the, in the modern era is trying to solve the problem of how do you scale. these companies without breaking down the agencies and velocity of these teams?
Right? Because the instinct companies have is to centralized, to, to create consistency, to create efficiency. Everybody's going to use the same thing and that's, what's going to save us money and make us go faster. But you need to resist that instinct. That is true in some cases, but for the most part, your primary focus should be velocity of your product teams.
And so infrastructure teams are really, really focused on how do we create. Feedback for those teams to make great decisions. So we hire specialists in security. We hire specialists in reliability and quality performance, and a lot of, especially the early engagements for infrastructure revolve around data.
How do we return data to teens about the security position of the software they're creating or the quality or the reliability, and, you know, SRE falls into that, that model. And there's a lot of people have a lot of different ideas about this site, reliability engineering. Um, but I think that whether those people are using their expertise and embedded on a team to help them improve or they're building tools that are being leveraged by by many teams, the idea is still the same.
That infrastructure is really there to reduce the cost of ownership of service ownership for all of these teams. All at once. And I think there's other disciplines that are also trying to solve this, like in product design of how do you unify a user experience across a hundred teams? That's definitely a hard problem, but for us it's how do we reduce the cost of ownership?
So those engineering teams can continue to move fast and innovate.
Steve Hamm: [00:31:34] So how do you coordinate with the other, with your peers, the people who are running other pieces of, of Instacart's technology. I mean, is there the same kind of dynamic between kind of independence and agency and the need for coordination and how do you manage that?
Dustin Pearce: [00:31:50] You know, what's interesting is that , if you took a survey of infrastructure teams, uh, across software companies and you asked how many of you have product managers? Uh, the answer would probably be zero. Uh, there's, there's very little product management.
And why that's significant is if you ask on the flip side, On their development teams. How many of those have product managers? Like, well, all of them, of course they have to have product managers. And I think that this is a really important kind of. Disconnect. Right? Because the way that we engage is that they are our customers.
And so one of the biggest components of that is not just asking them what they want. It's studying, it's absurd, it's collecting data. There's, there's a qualitative and quantitative aspect to product development. And I think the challenge is that a lot of the people, well who find themselves in infrastructure or senior engineers who deal with issues of scale, they're not, they're not accustomed to a, to providing service.
Right or to doing product development. And that's an area that at Instacart, we're really trying to develop this concept around. How does an infrastructure team. Treat everyone else as customers. And how do we create a product factory internally that engages those customers in a way that is very similar to the way that those teams are engaging, let's say our shoppers or, or, or our users, right.
They're our customers. And, you know, I think that. For me, this is, this is somewhat of a passion. Um, I think that there's a larger burden on us to be transparent and be organized internally, you know, a product team that is autonomous and has high agency. There's not a lot of. Context, they have to share with the rest of the org in order to get their job done.
Right. And so if you looked across, you know, let's say all the directors, the directors that work in my org, the directors that work in my org have a much higher burden for transparency and organization, because they are coordinating across all of these teams. And I think sometimes that is also hard to comprehend that we are different because our customers are internal.
Steve Hamm: [00:33:42] Yeah, no, I don't want to get too far down into the, into the weeds
Dustin Pearce: [00:33:45] Sure. Sure.
Steve Hamm: [00:33:47] our listeners, but I am curious, I mean, you know what, you've talked about, agile, we've talked about dev ops. These are kind of two of the most important trends in software development. And I I'm frankly, gonna admit that I don't really understand the distinction or the complimentary aspects of them.
So there seems to be a lot of overlap. So what do you see as the key elements and differentiators in each of these and which approach do you use
Dustin Pearce: [00:34:14] sure. Um, to me, in my mind, the distinction agile was really about a smarter kind of faster way to develop products and dev ops was a smarter, faster way to own the products that you create. And, you know, I think that over time, It's really fascinating to me because you will see lots of, of different versions of the buzzword DevOps.
You'll see DevSecOps, you see dev fin ops, you'll see dev sec, fin qual ops. Like you see these crazy, the acronyms and, and, and the reason that they keep expanding is that what I try to say is that, you know, service ownership. Is really the, the natural evolution of DevOps, this idea that you're going to create something and then you're going to own and operate it.
That's what cloud computing is all about. And that's what users expect. We no longer sell software on a CD anymore. Right? It's a service that you subscribe to. And so service ownership is a critical component. Of the evolution of dev ops and all of the aspects of ownership, that's financial that's security, that's quality, that's operations, that's development are all integrated in that kind of concept of service ownership.
I think agile itself is. Likely, you know, one of those things that becomes a dirty word over time, it gets overloaded and even some the ceremonies and, and kind of processes associated with it almost form a kind of a negative outlook. If you go to a development team and you're like, Hey, I'm here to implement agile.
I don't think that they're going to be super excited with you. Uh, these days, uh, just because they, they associate, I hate it with a lot of ceremony and process, and I think that there's, there's good parts to agile and there's probably parts that need to kind of. Get back to first principles instead of tools, if that makes sense.
Um, but I think that it is the fact a way to build software. Now I, nobody, I grew up writing what these to call SRS a software requirement specification, and it was a numbered document. And so requirement 10.1 0.2 0.3 0.1. Right. And we used to have to try to eat you. You had to be smart in creating your numbering schemes.
And so this form of software development, where I create a numbered spec, somebody writes the code and then somebody else tests against the spec is just really kind of gone. I rarely see anyone doing that.
Steve Hamm: [00:36:32] Yeah. Yeah. You know, it's interesting to look at agile and, you know, it's, it's, it's been adopted in lots of other industries and lots of other processes out of software, but you know, I'm a journalist. I worked for many years at daily newspapers. I gotta tell ya. Daily newspapers were agile before agile was agile.
I mean, that's, you know, you'd get up. You'd arrive at work at six o'clock in the morning. You'd have a scrum. Everybody race off and, you know, get it done. And then there was the process of, of, of, of the writing, the editing, the, the, the typesetting and the, and the running of the press, all on a mad scramble.
And, uh, and that was the way we did it. So it's kind of fun to kind of see it coming back and all these other
Dustin Pearce: [00:37:06] Yeah, I I'm kind of one person, I always felt agile was an excuse to just kind of like improvise and that wasn't the point, right? The point was get as close to your customer as possible and iterate as fast as you can and do it, don't create any process that detracts from that. But you have to be organized in order to create high quality software.
These are very complex machines we're making. And so going Yolo and just kind of freewheeling it and calling it agile. The outcome is usually always the same, you know, it's kind of like, it's messy, right? It's like, I don't need to do experiments to find out what humans believe about incomplete or bad software.
I already know the outcome. Yeah.
Steve Hamm: [00:37:44] Right. That's funny. Hey, so, um, we always end our podcast by asking people to cast into the future too, to kind of put on there, visionary cap in the air, in your area of expertise. So cast ahead, you know, five years and give us a sense of kind of like where we are now and where we'll be then.
And, and what difference that will make for businesses or even for consumers.
Dustin Pearce: [00:38:06] You know, I think that as I, as I work through these companies and I see some of the same challenges associated with scaling, um, it, it strikes me that the need for us to solve some of these key. Infrastructure components. And, and one of the key areas is the device per experience, right? What is the developer experience like on one hand, it can be so stifling and so centralized and so painful because I have to kind of.
Learn all of these ways that the company forces me to do work, that I just, it stifles creativity. And on the other hand, I have so much agency and I'm creating so much variety across the company that the complexity starts to cave in on itself. And so we haven't found a balance yet. And I think that a lot of what I'm seeing and I mentioned before is that the velocity of just the world, the amount of information, the, the competition is just keeps going.
Faster and faster and faster and faster, you know, Slack was the fastest tool billion ever, but I guarantee that record will be broken almost immediately, right by the next people who come, who will be even fast. And so I think that pressure is, is going to put a lot more demand on finding the balance. How does an infrastructure team create a developer experience that does kind of.
Abstracting away some technical debt that allows teams to be creative and innovate, but is balanced, right? Not, it doesn't create so much complexity that the company kind of implodes on itself and has to go through these, you know, surges, you know, is a very classic term. You'll hear in Silicon Valley of like, we need to do a quality search or reliability surge.
Like those are strong signals that you likely are moving too late. Right. And so as you look forward, I think that, um, It wasn't that long ago we were standing in a data center and, uh, we were pulling cable and we were ripping out drives and replacing them. And we all said, you know, the cloud's cool, but like, no, one's going to use that for real work.
Right. And I think the same thing is somewhat true now in infrastructure where, uh, you look at managed services and people will look at a managed database or a managed kind of streaming process. And they'll say, I mean, That's great. And all for people who don't know what they're doing, but like, Hey man, we know what we're doing.
We're going to run our own Kafka, which is a streaming, opensource streaming service, or we're going to run our own, you know, my SQL databases. And I think the question will continually escalate to like, what's the value in you doing it yourself? And I think the answer to that is increasingly going to be more and more kind of subscriptions or cloud managed services for things that are not differentiating to your business.
And I don't think that concept is new. Yeah. Like buy versus build has been around forever. Forever. I can remember old IBM consultants giving the Sage advice of never build something that doesn't differentiate your business. But I do believe like in today's time, time, you know, the average, uh, software company has something like 300 SAS subscriptions.
Right. So this is, this is accelerating and having the glue and the engineering teams internally to transform all of these tools into a cohesive experience is going to be the next big kind of breakthrough for tech.
Steve Hamm: [00:41:13] Yeah. And that's done at the infrastructure level or is there a kind of, Oh, really? Okay.
Dustin Pearce: [00:41:18] I think the developer experience ultimately needs to fall on the infrastructure team. Right. And you know, that's a combination of. Some, what you would call, let's say low level, like networking and abstracting away, the complexity of running a global network or of some of the it's like we use this term, you know, computering how to computer can be complicated, especially at large scale.
And so how do you abstract that away? And so you do have these kinds of very. Detailed low level kind of abstractions, but you have to go all the way up to the, very, to the workflows and how work, how what's the value stream for each thing. How does a bug it get fixed? How does a new feature gets solved and how are you instrumenting those things and making sure that the it's very lean, very lean product delivery concept and infrastructure of like we're constantly evaluating our value streams.
Steve Hamm: [00:42:12] Okay. Very cool. You know, Dustin, it's been great speaking to you today. Um, you know, I have this vision of you. I feel like, I feel like you're the, you're the guide down on the boiler room. Of the great and fast ship Instacart.
But I feel like we've had a little lesson here or a big lesson here in infrastructure management, infrastructure engineering teams, uh, this great. Ongoing kind of balanced organizations have to have between independence and central control or standardization. And I think it's just, you know, You know, I think a lot of people don't fully appreciate the kind of work that has to be done down in the boiler room to really make these amazing technologies that they, you know, they interact with, uh, work and respond and always be there for them and things like that.
So I think it's been really, uh, instructional is it's been good talking to you today.
Dustin Pearce: [00:43:02] Thank you so much for having me while I have the pulpit, I will just, you know, do a quick pitch to the CEOs out there that effective investment in infrastructure is really your best defense against your next competitor, which is really just two women in a dorm room, starting a company, and they're coming for you.
And so infrastructure is, is the easiest way to kind of keep yourself ahead of the game.
Steve Hamm: [00:43:22] Well said, Dustin. Thank you. Okay.