Microsoft Distinguished Eng: Why And How To Port Your Entire Codebase to Rust | Stephen Toub
In the year 2000, Joel Spolsky (founder of Stack Overflow) wrote a famous article titled “Things You Should Never Do, Part I.” In it, he talks about how rewriting your code from scratch is the single worst strategic mistake any software company can make. However, times have changed.
The cost of writing software is dropping exponentially and that has changed the math entirely on if it is worth rewriting your codebase. For instance, this year Anthropic’s Jared Sumner rewrote Bun in Rust in 11 days leading to many performance and correctness benefits.
Likewise, Stephen Toub (Distinguished Engineer at Microsoft) rewrote the GitHub Copilot runtime (400,000-line codebase) in Rust by himself. I thought it’d be interesting to lay out everything he learned in the process. So I interviewed him about how and why he did it.
I hope this interview can be a helpful resource for others considering this new type of project that has recently become feasible.
Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.
Timestamps
01:06 - Why the Copilot runtime needed a Rust port
11:28 - Discussing rollout strategies
21:57 - How he caught regressions
42:11 - Catching regressions before prod
50:03 - The end benefit and costs
01:03:24 - When to do a full rewrite
01:06:40 - Career learnings on the way to Distinguished
01:16:00 - Top technical book recommendation
01:17:49 - Advice for young engineers
Transcript
01:06 — Why the Copilot runtime needed a Rust port
Ryan:
[01:06] So I wanted to start off by asking you, what was the project and why did you rewrite everything in Rust? What was the problem that you were solving?
Stephen:
[01:15] Yeah, that’s a great question, especially because people sometimes, I think, assume that rewriting things into Rust is kind of all the rage these days. And so people assume, or I’ve had questions that assume, oh, you’re doing this just because, or, like, you can, and so you will. And that is not at all what this was. There are certain shapes the current project could have existed in where rewriting it wouldn’t have been a conversation at all.
[01:45] But the impetus is largely that AI has been this incredibly fast-moving target, as we all know. And over the last few years, the capabilities that it has enabled has skyrocketed. And so across GitHub, across Microsoft, there are all of these projects that have kind of blossomed from the ground up to incorporate AI capabilities into them. And all of them basically started from, oh, we can make these LLM calls.
[02:18] Oh, we have to wrap a little bit of goo around it. We have to add more around it and more around it. And building up effectively what is now called agentic harnesses or agentic loops or agentic runtimes, whatever you want to call them. And we ended up with a lot of them because everyone kind of had their own homegrown implementation. And that’s not a good place to be, especially when you’re trying to have all of them kind of be best of breed and be on the forefront.
[02:48] And in this world of AI, there are new models coming out weekly, if not daily. A couple of weeks ago, there were all these releases from both OpenAI and Anthropic, one-upping each other. And there are new techniques that a month ago were research, and today every harness has to have or they’re immediately out of date. And you don’t want this myriad of different implementations all needing to stay up to date.
[03:15] And so you really want a small number, ideally one, but a small number of extensible harnesses or runtimes that everyone can use and customize and extend to their needs. But the Copilot runtime that was being used for the Copilot CLI and the GitHub app, and Copilot Cloud Agent, like when you’re on GitHub and you ask Copilot to do stuff or code review, that was written in TypeScript using Node.js, using V8, and that has implications for where you can use that as an embeddable library.
[03:58] So as one of the main blockers for that runtime being able to be used everywhere we wanted it to be usable was fundamentally what it was implemented in, the framework it was using, the runtimes it was using, and so on. And that basically, not moving to Rust, but rather moving away from TypeScript and Node to something that was more embeddable, had better efficiency, lower working set, faster throughput, better able to be consumed into any myriad of environments, regardless of what language they’re implemented in, regardless of what frameworks they’re using.
[04:43] That was basically the catalyst for this. And to your point earlier, there are any number of reasons folks might be looking to do these kinds of ports or migrations. They are incredibly common across the industry, whether moving from one deprecated framework to a newer version or from one language to another or from a legacy codebase that has become an unwieldy mess of spaghetti to something cleaner and modern and fresh.
[05:14] These migrations and ports are everywhere. And a lot of times people just don’t take them on because they are too expensive. They take too long. And we would have been in that situation a year or two ago as well. But with modern models and modern harnesses, these projects that were once unaffordable are now quite doable in a very small period of time.
05:38 — Why he chose Rust
Ryan:
[05:38] In the post that you wrote, I saw that there was a line. It said, “I didn’t set out to move to Rust. I set out to move away from Node.js and V8.” And that makes me wonder, if you didn’t need to go to Rust, did you consider other alternatives? And why did you end up going to Rust specifically?
Stephen:
[05:57] So the very first thing I did was basically create a prototype agentically, because you can now create prototypes super fast. Still with TypeScript, but with different runtimes. So, Node versus Bun versus whatever, as well as I think I did Python, Go, C#, Java, and Rust. And created a very small end-to-end sort of steel thread through one of these systems kind of backed by a local chat completion, dummy server, probably 5,000 to 10,000 lines for each of them, and looked at them in terms of end-to-end performance, in terms of working set throughput, what kind of scale that we could get if we threw a lot of load at it.
[06:53] And then also what kind of whether it could actually meet requirements and what it would take for us. We currently have six language SDKs that are thin bindings on top of the Copilot runtime, what it would look like for this to be usable from all of those. And in particular right now, because we don’t want to force, or previously because we didn’t want to force Node.js into the consuming application’s process, the SDK basically spawned another process to host the actual execution of the sessions.
[07:30] But we’d really like, as an option, for it to be in-proc as well, like pretty much every other library in the world. And so basically evaluating all of these languages and runtimes against these performance goals and constraints, and just looking at the data to see what worked and then how well it worked. And for the most part, TypeScript and Python were very quickly out from kind of all of the performance perspective and the embeddability goals.
[08:05] Go had pretty good perf and had a pretty clean codebase, but it’s not really designed for embeddability. It’s not really interop is not its raison d’être, right? It’s, you can do it, but you’re kind of cutting against the grain. And then Java is also not great for that. Java’s perf was okay. The 2 main contenders ended up being C#, in particular C# with Native AOT, so that it could be compiled ahead of time down to assembly.
[08:49] And Rust. And from a perf perspective, they were both very much on par with each other across pretty much all the metrics. In some places, Rust was a little better. In some places, C# was a little better. And so to your question earlier, if the project was already in C#, we wouldn’t be having this conversation. We would have stayed with it. There would have been no reason to rewrite it.
[09:15] But it wasn’t. And so then it came down to these criteria. So from a perf perspective, they both looked good. From an embeddability perspective, they both looked good. C# has very strong interop characteristics as well. And then it came down to primarily softer factors. So there are other projects around GitHub in particular that are built in Rust and very few that are implemented in C#.
[09:39] There are more people around GitHub that know Rust and fewer that know C#. That’s not the case within Microsoft, obviously, but within GitHub. And so factoring in those kinds of criteria, we ended up going with Rust.
Ryan:
[09:53] I know Anders recently did, I think the project started a bit before your project started, but he did a port with the team of the TypeScript compiler from TypeScript being written in TypeScript to being written in Go. I was curious if you compared notes with him.
Stephen:
[10:13] Definitely. Yeah. And this is a great example where different projects can reach different conclusions and be right. There’s no requirement to embed the TypeScript compiler into other applications. And conversely, the shape of the work that is compiling TypeScript looks very different from the shape of the work that is sending requests to an LLM and manipulating prompts and dealing with sandboxes and all that kind of stuff.
[10:52] The nature of the projects was fundamentally different. Although interestingly, in the last few weeks, there have been several agentic ports of which you can, if you search X or Twitter or whatever it’s called now, you can find people that are porting the TypeScript Go compiler into Rust. And doing so very fast with modern models, given the size of the codebase and the cleanliness of the codebase.
[11:24] And so it’s interesting to see.
11:28 — Discussing rollout strategies
Ryan:
[11:28] So once you decided that there was motivation for the project and there’s a clear problem to solve, I wanted to ask you a little bit about the migration strategy because I’ve seen different strategies in different projects. So, for instance, you mentioned Bun earlier. Bun, it looked like they kind of built up a giant, basically a new branch that’s just completely entirely in Rust. Yeah. And then they shipped all-in, all-at-once stable Rust release, and they did it all in one go.
[12:03] Whereas when I was reading what you had done, it looked like a lot more incremental and little by little. How did you think through the different strategies, and when might you select one versus the other?
Stephen:
[12:15] The interesting thing about Bun, first, I loved Jarred, who wrote the Bun post. I loved the post that he wrote. It was really exciting because that post, I wasn’t aware that they were doing that port. And then that post came out about halfway through our migration. And it was really cool to see that this other project of approximately, order of magnitude, same amount of code, implemented in a different language.
[12:38] They were in Zig rather than TypeScript, but they had thought that this was a sensible thing to do and had landed it. And in the post, they talk about it taking I think 11 days for them to get to 98% of their tests passing or something like that, which is really cool. And to your point, they did it sort of in a, en masse, rewrite the world and go. It is interesting.
[13:03] I am not involved in the project, so I don’t know all the details of what’s going on behind the scenes, but it is interesting to me that if you look at their npm page, they were shipping near daily builds. I think a lot of canary builds basically through mid-April, like for a long time, daily builds. And then mid-April it stops. And then they talk about doing the 11 days of porting, which is an impressive feat to get from nothing to this, you know, a 500,000 line codebase ported to Rust in 11 days and have a passing 98 or 99% of the tests.
[13:43] But then they didn’t have another npm release of any kind, at least as far as I could see, for another two and a half months. So there was this window where, yes, the core port took a small period of time, but it seems then like there was some aspect of stabilization and things like that happening such that there was this three-month gap between releases, which interestingly is also approximately the same amount of time that it took Copilot, the other 14 and a half weeks that I spent porting Copilot.
[14:16] So, there are lots of reasons why they differ, but fundamentally it seemed to be about the same amount of time. And we just chose different ways of going about it. In the post, Jarred makes a point about not wanting to do the incremental approach that Copilot did because it would incur lots of temporary interop junk. And he’s right, it did. We just chose that approach knowing full well what we were getting ourselves into because of the constraints that we were under.
[14:51] So, for a JavaScript runtime, obviously fixing bugs and new features, that’s all valuable. But you can pause for 3 months and the world isn’t going to fall over, right? JavaScript 3 months ago isn’t all that different from JavaScript today. And you can do sort of a rewrite-the-world approach, pause everything, and then continue shipping when you’re done. Agentic harnesses in the summer of 2026.
[15:27] If you pause for 3 months, the world you ship into will be very different from the world that you started in. And so trying to pause for a long period, for a period of time, even if it’s a few days, was not something that we were willing to entertain. And even if we had been able to do it in 2 weeks, that would’ve been 2 weeks worth of either blocking the whole team or dealing with a constant influx of changes across a very large code base and having to rationalize all of those changes and merge all those changes in as ports.
[16:10] Instead, we opted for the approach which I think made sense for Copilot and made less sense for Bun, where we just incrementally moved the code base more and more from TypeScript to Rust in pieces, in the main shipping codebase, shipping releases daily. And it’s just that at the beginning, it was 100% TypeScript and 0% Rust. And at the end, it was 100% Rust and 0% TypeScript. And it was just more and more over time.
[16:41] And for the most part, it worked out quite well. It did mean that there was, to the point of the post, of the Bun post, a lot of temporary code that came in and left. Yeah. Right. I think we were using NAPI, NAPI-RS for interop between TypeScript and Rust. And at the peak point, we had several thousand interop signatures for, I think, 2,000 or 3,000 for calling between them. And then by the end, it had dropped to zero because there was no more interop required.
[17:15] So for this project, it made a lot of sense. There are other approaches. You could do a rewrite-the-world. You could have everyone stop what they’re doing and just rewrite in main. You could have a feature branch and allow main to keep going and then constantly be taking ports in. And for the incremental approach, you can do what we did, but rather than do the approach of what I chose, which was sort of atomically swap TypeScript with Rust, you could have sort of an A/B thing going where you have both the TypeScript and the Rust component and you ship with both and you have a feature flag that allows you to switch between them and do sort of flights and experiments of those components side by side.
[17:57] There are various reasons you might choose which, and I think what we chose ended up working out well for us. But all the other options would be valid or probably better for other projects.
Ryan:
[18:10] So it sounds like that A/B approach, you would also still have some temporary testing harness that would be comparing the two legs anyways. Well, in the case where you have a separate branch and then new commits that come in on the old project, or I guess main, need to be continuously ported. I imagine that would be relatively doable with modern models as well, because each of those single commit ports is just probably trivial for a model.
[18:46] Yeah. Did you consider doing that one, or how did the math compare between doing that versus incurring the interop costs?
Stephen:
[18:56] So there’s a couple of questions there. It’s not just about having a separate feature branch and bringing everything over. If we had a separate feature branch that we were maintaining, and at the end we did this swap where we deployed the whole new stack, we would only be testing in the wild everything all at once. And that’s a pretty risky gambit. Whereas one of the big benefits of the approach that we chose was we were effectively able to use production for the last mile of testing and have each deployment be a small diff on top of what the previous one was.
[19:42] So over the course of the port, I think we shipped about 130 releases, something like that, each one of which contains just incrementally more. And so if there were issues, it was pretty straightforward and fast to root cause those issues, even without using a computer at all. Someone reports an issue, you’re like, oh yeah, I know exactly what that is. Right. And then you can very quickly narrow it down.
[20:06] Whereas the whole rewrite the world, it’s harder to do that kind of validation. And there were absolutely regressions that came about from this. A good chunk of those were because of issues with merging in changes from me, because I still had feature branches. They were just much more timeboxed. Instead of being weeks, they were a day for porting a component.
[20:36] But within that day, there might still be 10 or 20 PRs that came in and touched that component. And a non-trivial percentage of the regressions were places where the agents just missed something as part of doing the merge and the port and the merge at the same time. One of the nice things about sort of the atomic replacing from TypeScript to Rust versus having an A/B thing going is the agents were instructed to delete the TypeScript that they were replacing the moment that they ported it.
[21:13] And that, in a good way, created conflicts for those merges from main because it meant that if any code in main touched any of the TypeScript that had been ported, it instantly created a conflict, which sort of raised red flags and pointed the agent’s attention at that code. Whereas if we weren’t deleting it, it would have been very easy for more changes in main to that TypeScript to find their way into the existing untouched TypeScript in the branch.
[21:43] And then the agent would have to reason about all of the code that it had ported, looking for changes to that to bring that in versus it was sort of instantly pointed to those places because we were doing this swap at the time.
21:57 — How he caught regressions
Ryan:
[21:57] Yeah, I could imagine a lot of people, a lot of teams or engineers that want to rewrite their entire codebase. Probably the biggest concern is the correctness concerns, which is there’s so much code that you don’t have the luxury of a human even reading the code to confirm that it’s correct. So how did you discover and fix regressions along the way?
Stephen:
[22:24] So the most important thing was our end-to-end tests, our integration tests, or whatever you want to call them. We basically had two flavors of tests for Copilot. We had unit tests, which were largely testing micro functionality throughout the code base. A lot of them have internals access, so they’re able to test this little function here that’s not exposed or that piece of code there that’s not exposed.
[22:57] And then there are the end-to-end tests either at the runtime level, the SDK level, or the Copilot CLI is in the same codebase. And so its end-to-end tests also effectively exercise the runtime. And this isn’t 100% true, but the Copilot CLI can largely be thought of as sort of just exposing the runtime’s capabilities as sort of the lowest layer possible in a human interface. And so we had all of those end-to-end tests to validate behaviors.
[23:31] And by and large, if not 100%, but the vast majority of regressions that we had were because we didn’t have end-to-end tests covering that functionality. We generally had unit tests covering it, but the problem with unit tests in this situation is they were also implemented in TypeScript testing TypeScript with internals access. And so those, in many situations, those unit tests had to be rewritten at the same time that the code was being ported.
[24:04] Now, because of how we did it, we got the benefit of, if I replace this little piece down here and then the TypeScript— so this is now Rust, and the TypeScript sitting above it was interacting with this through interop. This guy had its own unit tests. And so indirectly, those unit tests are still validating this ported Rust component. So we got benefits from that. But the safeguard was the integration test or the end-to-end tests across everything.
[24:29] And I had written a document for the team, sort of motivating this port and laying out exactly how we would want to go through it and approximate timeframes and approach and strategies and technology we’d use and so on. One of the things that called out was we needed to beef up our end-to-end tests as the very first thing we did. And I did. I think one of my failings, and something I would do differently if I had to do it again, was I would have done it more.
[24:58] Because again, the vast majority, it kind of goes without saying, but if you have tests for something and those tests don’t regress, then the behavior is good as long as you have tests covering all the behaviors you care about. And so the fact that we had any regressions sort of highlights that there were gaps. Now that’s not— Yeah. 100% true. We had cases, the funniest or saddest cases were where we neglected to prevent the agents from rewriting the end-to-end tests at the same time that they were changing the behavior that was being tested.
[25:33] The agents want to get to green; that’s their goal. And so if you let them, they will change the oracle. They will change the tests at the same time as the code to make it pass. Or we had one case where this one didn’t slip through. I noticed this one, but the agent was making changes, and it ended up changing some public APIs. And CI caught this and said, no, no, no, you can’t do that.
[26:01] But we have an escape hatch in CI where if someone really knows what they’re doing, they can put a label on the PR to basically say, no, this particular break is okay. And the agent found that and applied the label to say, oh, this break is okay, even though it wasn’t. And then there are other things that are just really hard for, you could have end-to-end tests, but they’re sort of in the weeds.
[26:26] Things like, you’re sending data out on the wire and making sure that every byte that you send in all possible situations is what’s expected at the end of the wire. That requires an extensive suite. And if you have any gaps, there’s the potential for things to slip through. And then there’s also, TypeScript and Rust have very different threading models and concurrency models.
[26:55] And as much as people like to say that if your Rust compiles, it’s correct and it prevents all these kinds of bugs, that’s true, but it doesn’t prevent all kinds of bugs. And there are plenty of concurrency bugs that slip through. And so a bunch of the issues were lifetime issues, concurrency issues, things that the Rust compiler does not catch. And when you’re coming from a largely serialized, single-threaded execution model and going to one where on purpose we wanted much better scalability, we wanted things to run concurrently.
[27:36] There are plenty of issues that creep in there. So anyhow, to your question, the number one thing is to make sure that all of the behaviors that you care about are encoded in tests. And then let the agents port the code while maintaining, keeping everything passing. And making sure that there are guardrails in place to make sure that they don’t change the thing that you’re using for the validation at the same time that you need that validation to be correct.
Ryan:
[28:08] It still feels like a regression is inevitable because the test coverage is opt-in, kind of, where it relies on a human saying these are the things that are important, but you could always forget something. So I kind of wonder if this type of project, it’s just inevitable that there will be some breakage.
Stephen:
[28:36] I think so. I mean, it’s hard to imagine. Who knows what things will look like in a year? Things are moving really fast. But right now, yes, 100%. There will be regressions. If there aren’t, I would be astounded. Unless we’re talking about a tiny, tiny little codebase, but you’re talking about rewriting hundreds of thousands or millions of lines of code. It’s just law of numbers.
[29:07] If the agents get it right 99.99% of the time, which is probably a 9 or 2 higher than most humans, that still leaves 0.01% of the time where they’re going to get it wrong in some way, shape, or form. And if you’re dealing with a million lines, that 0.1% is a very real small, but real number. You can throw more agents at it, and we did. We just, before merge, doing lots of agentic reviews, and then after merge, running agents to continually look for bugs, look for concurrency issues, look for whatever.
[29:47] And they find things, but they never find—it’s never, at least in my experience, 100% perfect.
Ryan:
[29:55] For such a huge volume of code. I imagine most of it wasn’t human reviewed. How did you handle code review at scale, or did you do any at all?
Stephen:
[30:08] Yeah, so definitely did some. But I did not read every line. That would be impossible. And if I did, it would also be a waste. There’s no, me reading every line would not prevent bugs. In fact, agents reading every line is a lot more effective, not just faster, but more effective in quality than me doing so. I absolutely looked at the code for every PR that I put up, but I was focusing more on big picture stuff, right?
[30:45] Is the direction of this correct? Is the architecture of this correct? Or is the componentization of this correct? Is the approach, are the algorithms that are being used appropriate? Is there anything fishy about, if I, a lot of it is squinting at a 30,000-foot level and looking at the two codebases and does anything look off, right? And then relying on the agents to focus on the line-by-line details.
[31:19] And in many situations, I would sort of do a quick scroll through all the changes to just see if anything popped out. And I might notice a few things here and there, but for the most part, a really good agentic reviewer is going to be way better at sort of the line-level details than certainly me, and I would argue most humans. But at the macro level, there are lots of times when I basically stopped the agents and said, whoa there, this is not the direction to be going.
[31:53] Or the inverse, where the agents would stop and be like, “We’re done.” And I’d be like, no, you’re not. This is not what we’re going for. And so it was a bigger picture than sort of the micro level.
32:08 — Guiding the agents
Ryan:
[32:08] I saw in the blog post that you have this cool bar chart where you basically categorized every single message from you to the agents and what your intent was. Looking over that and reflecting on how you guided the agents, where was most of your time spent, and what were the most impactful times where you would step in and change something?
Stephen:
[32:34] There were two or three even higher-level categories of things that I was doing. There was one or two categories where I think this just speaks to the state of sort of long horizon tasks with models and harnesses today, where it was me being frustrated with their progress in one way, shape, or form, and either telling them to keep going in some fashion when they had stopped or telling them to stop when they were going off down bad paths.
[33:06] But the majority were in a category that was more of there were tradeoffs to be made. And I wanted to make sure, and reasonable people or reasonable agents could choose multiple paths. And I wanted to make sure that we were going down the right one. And a lot of that would be times where I would just have a conversation with the main agent to basically evaluate where we were, the decisions that needed to be made, and kind of talk through, rubber duck through sort of the pros and cons of those choices given the current situation we were in.
[33:54] And some of that, to your earlier questions about sort of the incremental versus pause-the-world or Big Bang rewrite approaches, some of those were less about the end-to-end and more about what I wanted to do for this one slice that we were putting up. So the agents would pause at a particular place saying, “I’m done with this component.”
[34:24] And I’d look at it and say, this is not a good place to leave things in terms of the tendrils that this has into other systems temporarily, or maybe we need to do just a little bit more to get beyond this architectural boundary, and then that’ll clean up a bunch of stuff such that when we merge this, the codebase is in a reasonable place. A lot of times it would sort of declare success for the slice.
[34:54] And if you squint, yeah, okay. But if it would do just a little bit more, or in some case revert and do a little bit less, it would have left the codebase in a better place. Or situations where it actually tried to do more than I wanted it to do. One of the rules that I put in place with instructions that wasn’t always abided by was I wanted these ports to be as strict ports as possible.
[35:23] Not removing functionality, not adding functionality, not making any huge changes in algorithmic approaches, not trying to fix arbitrary bugs that it comes across along the way. I would have it note those for me so I could revisit those separately, but I wanted these to be very strictly as one-to-one functionality ports as possible. And there were times when I would see it violating those rules. And I would basically have it back off from that and go back to what it previously had, or at least talk through what it did to make sure that I was comfortable with that particular change being included.
[36:04] A non-trivial percentage of the regressions that we have had were from places where it violated those rules and I didn’t notice, and it included fixes for things that shouldn’t have actually been done. In fact, there was just one. Yeah, for the most part, we’re not seeing any more regressions being reported from the porting phase. We see regressions all the time. That’s the nature of merging hundreds of PRs a day into a repo.
[36:35] But, you know, it’s just to be expected. And we fix them. Generally before they ship. And it’s just where we see them in main, right? But we had one yesterday or the day before where there was this really corner case thing. And I looked at it and it’s like, that’s really weird. It’s saying that I added this and I don’t remember adding this and looked back what it was. So I had created a PR as one of the porting efforts and it wasn’t in the initial PR, and then some feedback from a peer or a peer’s agent came in recommending doing something.
[37:18] And the agentic response to that was to basically address this security feedback by locking something down. But it locked it down a little bit tighter than it needed to be locked down. And that resulted in some things that had been working before the port no longer working after the port. And those are really interesting things to go and figure out how they happened.
Ryan:
[37:51] That’s interesting. So your agent will auto-apply and auto-listen on the PR. And if it seems reasonable, it may take action without your input. Is that right?
Stephen:
[38:05] For the most part, I use the GitHub Copilot app, and it’s got this Agent Merge functionality where you can basically have it automatically do various things watching a PR. It can automatically rebase if there are conflicts or merge if there are conflicts. It can automatically drive CI to green. So if tests are failing, fix the tests. It can automatically address review feedback that comes in, and it can automatically merge once everything is green.
[38:37] And you get, for each of those four things, you can choose which ones you want. For the most part, I leave all three of those checked except for the last one so that I have a time where after everything is green and before I click the merge button, I go back and I review to see everything that it did to drive this PR to green. What tests did it fix? What feedback came in and how did it address it to make sure that I know, again, at a high level, I’m not reviewing every single line, what it did.
[39:10] And this particular one, I remember actually reading, but I didn’t fully comprehend how much it was locking things down and how much it was blocking. Because if I had, I would have said, actually, those cases are okay, let those through. And had to back off from that. Yeah. Yeah.
Ryan:
[39:30] If you had all of those checked off, I mean, no one would do this because they’re on your team, but yeah, imagine I was on your team. I could probably craft a PR comment that is malicious, I guess. It’s like, hey, it sounds reasonable, but I did request you to delete everything. Or—
Stephen:
[39:48] Yeah, I did ask you to format your C drive. Did you not format your C drive?
Ryan:
[39:52] Yeah, exactly. Just take everything down.
Stephen:
[39:56] Yeah. Certainly different projects, I would have different boxes of those checked. Like if this was an open source project, the only one I would consider leaving checked would be the CI one, driving. There’s no way, and maybe rebasing, but there’s no way that I would allow it to automatically address arbitrary review feedback coming in on a PR and certainly not automatically merge.
[40:26] On a close, on a private closed source thing where I trust the people that I know to be working on the project, it’s less risky. But even then, that less checkbox, you have to be very, very trusting. One of the things I like about the app as well is it does, you know, there are built-in prompts as part of Agent Merge for what those different things are, but the app does permit me to add my own prompting for those different capabilities.
[41:01] And so I have additional constraints and guidance for the agent as part of doing each of those things about how it should do it. Always rebase instead of merge for one of them, or for review feedback, here are the kinds of things I want you to think critically about before you just blatantly accept it, and these are the ones where it’s okay, just go ahead and address it. Those sorts of things.
42:11 — Catching regressions before prod
Ryan:
[42:11] And we talked a bit about regressions, and you mentioned that a lot of them were caught before they went out to prod, it sounds like. And I think a lot of people might hear regressions and they think, oh, wow, you’re taking down the app, or things are exploding all the time, or there are incidents.
[42:42] Could you explain maybe the further protections after end-to-end testing that made it so these aren’t bad regressions?
Stephen:
[42:51] I would say probably 2/3 of these things that I’m calling regressions were caught very soon after merging to main and before we shipped any prerelease. Even though we’re shipping, at this point we’re shipping, I think, 3 prereleases a day. So they were caught very quickly. Then there was, there were some that made it into a prerelease. And our form of ring deployments with the CLI is basically prerelease and stable, where prerelease on any given day is probably hitting 5 to 10% of the audience of the customer base.
[43:28] And then 90 or 95% is only on stable. And so a reasonable number of these issues, yeah. I don’t know exactly what percentage, 10, 20%, found their way into a prerelease. And then pretty quickly we’d get an issue opened or someone saying, “Hey, I’m seeing this difference in behavior. Was this by design or not?” And so a lot of these were things that were different, not necessarily bad, but different enough that someone noticed and then asked us, “Was this on purpose?”
[43:59] And some of those cases were ones where I’d failed to prevent the agent from fixing bugs as part of the port. And so some of them were actually desirable changes, but we hadn’t intended to, they weren’t purposeful in that regard. And then there were a number that made it into stable releases, and the ones that made it into stable releases because we hadn’t noticed them in either main or prerelease, they often ended up staying in and being unnoticed in stable releases for quite a while.
[44:35] And this is mostly the trickle that we have been seeing over the last, well, we don’t really see much anymore, but a month ago would have been the trickle that we were seeing over a few weeks where some super corner case thing just behaved slightly differently. And then we’d get an issue open like, “Hey, this isn’t working the way I expected. Is this on purpose or not?” And oftentimes they weren’t, and we’d fix them as soon as we heard about them.
[45:01] There were also ones that we would see from various telemetry systems monitoring for exceptions, crashes, things like that. And obviously those are unintentional, and we fix those.
45:16 — Which models he used
Ryan:
[45:16] In the post, you shared this graphic of which models you used throughout the course of the project. So you have this nice bar chart that’s showing what portion of each week’s tokens are consumed by either Claude or GPT, and in rare cases, Gemini. At the beginning of the project, it was almost entirely Claude, and then by the end of the project, it was almost entirely GPT. So what made you shift your model of choice in the middle of the project to GPT?
Stephen:
[45:52] For the most part, I was constantly trying different ones. And because over that period, both labs were releasing new models at a very fast pace, I was experimenting. When I would move to one and I found it to work well, I would stick with it until something bothered me, and then I would try a different one. So it doesn’t in any way reflect sort of a long-term preference.
[46:26] It was more of, at that moment, what was working best for me then. And so at the beginning of it, I think I was on mostly Opus 4.6 or Opus 4.8. And it was working great. And then GPT, what was it, 5.5, 5.6, one of those. And that started working even better for these particular workloads. So then I jumped to that. And then Opus 5 came out and I was, oh, this is cool, and jumped to that one. Right. And then recently I’d been having a lot of luck with 6.
[45:49] 1 Sol. And I’ll use Opus 5.5 for code review for the work that I’ve done with 6.1 Sol. And so I’ll sort of split the balance there. Sometimes if I get really frustrated with something, both over the course of the port, both models, both GPT and Claude would exhibit some very frustrating traits at certain points, like a particular point, or if I was working on it for some period of time, one of the models might get stuck and kind of, no matter what I did, it kept kind of refocusing on a particular thing and not stepping out and looking at the forest for the trees.
[47:48] And there are various things I would do in those situations. Sometimes I would keep the code changes but start over with a clear session and try to use that to clear the context window and get it to reset itself, but often it would find itself back in a rut. And so in those situations, I would switch models and then keep going with the new model until it found itself exhibiting some frustrating behavior.
[48:12] And then I would switch to something else. I did experiment using Gemini and Grok as my main drivers, but I would often leave those to be part of a multi-model code review. So a lot of the reviewing that I was doing, I would be using either GPT or Claude as for the main work and then use 3 or 4 different models from different families to do a line-by-line review of the porting that was done.
[48:50] And I would do that very frequently, at least as frequently as every time that I was rebasing to bring in additional functionality from changes that had been coming into main, but often even more frequently than that.
Ryan:
[49:06] Yeah, I did see that you had a prompt that was, I guess, auto-applied on rebase or something like that, where you requested it to be reviewed adversarially, or all of them at the same time. What made you come to that conclusion rather than code review with the same model, but I don’t know, different focuses for each thread?
Stephen:
[49:32] Yeah, it’s basically to get effectively different perspectives. Every little change that you make to a non-deterministic system ends up sending it down different paths and looking at different things. And one of the biggest changes you can make is completely changing the model. And so the best way to get different opinions and focusing on different things that I found is by having different models, especially than the one that I used initially.
50:03 — The end benefit and costs
Ryan:
[50:03] So, after you finished the rewrite, what was the concrete impact and the cost? So if you look at it from a business perspective, what was the benefit and the cost of this project?
Stephen:
[50:19] So the benefits are stark. The benefits are the ability for all these products and services and teams across GitHub and Microsoft to be able to rely on a shared harness that is constantly sort of best of breed and getting better and better every day, and not have to spend their own resources trying to do the same thing. They can focus on their business value and reuse this component.
[50:50] It’s the same as with any library that you want to rely on. And this library is a particularly large library, right? You want to be able to focus on your business value and reuse the constantly evolving work for someone that is spending all their time focusing on that thing. And so from a business value perspective, it’s like incalculable across GitHub and Microsoft being able to leverage this shared harness.
[51:19] Whereas before the port, a lot of that would have been a harder sell and harder to justify. And part of that is because of very real costs, like performance costs. So, yeah. Something that was taking 10 times as much working set, waving my hands, means 10 times less density, which means 10 times as many servers, those sorts of things. And if you can make the number of users that are able to be on the same machine running concurrently, that has impacts on your bottom line.
[52:05] If you can scale the throughput and reduce the latency by 5 or 6x or whatever it is, that has real benefits. One of the biggest impacts from a perf perspective, a lot of these systems, especially services, will do, a request will come in and the agent loop needs to be started to make one more turn, which could be a very short thing, and then the request is done.
[52:36] And so for these things, the startup cost ends up being one of the dominant factors. And the startup of a Node.js application versus the startup of a native Rust or C application is there’s a big difference there. We’re talking hundreds and hundreds of milliseconds difference, or in some cases, seconds. On a per-request basis, that latency can be extremely measurable and extremely impactful for bottom line if you’re even able to adopt it, because you might not be able to, and then you’re back in the previous world that I was talking about.
[53:17] So from a business perspective, it’s worth a lot to do this port. In terms of what it actually cost us, it was about— if I— so I stored all of my session logs or 99% of my session logs from this effort. And from those session logs, I was then able to sort of deduce the token counts for fresh tokens, cache writes, cache reads, output tokens. And all told, it was about 136 billion tokens broken up in such a way that if you were to use the public pricing of those models at those times, it came out to approximately $120,000.
[54:07] This is not factoring any discounts or things that companies might have for what those tokens cost. And this is just sort of what’s on the website of OpenAI and Anthropic, plus my time, which it took about 14, 15 weeks, but I was doing other things at the same time. A lot of these agentic projects, if I look at my Copilot app that I currently have on my screen, I’ve got about 35 sessions right now for things that I’m working on, some of them in the same area, some of them are across different projects.
[54:50] And so I’m not— developers are no longer working on one thing at a time. So if you look at the percentage, just as a rough estimate for my time, if you look at the percentage of PRs that I was submitting over that same period, about a fifth of them, give or take, was to do with this project, and about 80% was for other things. And so if you figure, about 20% of about 15 weeks, you end up with about 3 weeks of my time spread out over 3 months.
[55:25] Plus the token costs. And it’s only getting cheaper. The models are becoming cheaper at an astronomical rate. I think when OpenAI released, I’m not going to remember the exact things, but I think 6.1 Sol is half the price of 6 Sol, and 6 was half the price of 5.6 Sol. It’s just getting cheaper and cheaper and cheaper. And the models are also getting better and better and better, which means that the amount of time that I need to put in to steer the models to do the right thing is also decreasing.
[56:02] And so the costs of these ports, the cost of these migrations is only getting cheaper over time.
Ryan:
[56:08] Wow. So three weeks of developers’ time plus $120,000, you’re able to rewrite the entire codebase. If you were to scope this out pre-AI, let’s say you wanted to undertake this, what would that look like?
Stephen:
[56:29] First, it wouldn’t have happened. And I’m actually serious. For this particular thing, it wouldn’t have happened because by the time we were done, we would’ve ported the wrong thing. This space is moving so fast that we would’ve basically never finished, I think. In fact, you can kind of see that in the charts in the blog post in that for a good portion of the porting effort, the amount of Rust is increasing and the amount of TypeScript is largely remaining the same, but increasing a little bit.
[57:05] And that’s because as the port was happening, there was just a huge amount of new functionality coming into the repo. Right. And at the beginning of the porting effort, because the majority of the codebase was TypeScript, the majority of the new incoming code was also TypeScript. By the end of the project, where the majority of the codebase was Rust, the majority of the new incoming code was Rust.
[57:26] But if you look at the first half of the graph, where we’re generally 50% or less Rust, the amount of TypeScript was doing this even as the amount of Rust was doing that. Even if that wasn’t the case, we’re talking about 450,000 lines of TypeScript, just production source, never mind tests that were ported. I think there were a few hundred thousand lines of unit tests there in TypeScript as well that also needed to be ported to Rust.
[57:59] Just think about the productivity of an individual developer and how much they can reasonably do in a given day if they’re top-notch. And we’re talking a whole team of developers for a year or two to do something like this, right? It is a— if you’re able to do it at all, it would have been a monumental investment. And I struggled to think about even writing that proposal because I know that it would be sort of, I would look bad.
[58:32] Like I would be dismissed immediately as, of course, we’re not doing that. Whereas this was a proposal like, I’m going to take 20% of my time over 3 months. And by the end, this whole thing is going to be moved over. And for the most part, no one else on the team is going to be impacted. And that’s not 100% true, but waving my hands, and we’ll have all these business benefits in a few months.
[59:02] And this will be the bill. That’s a much easier proposal to write. And even if you were paying that actual non-discounted public-facing token rates of $120,000, you compare that to the cost of a senior developer for a year, with all their benefits and their bonus, even that is a good deal.
Ryan:
[59:35] It sounds like at least for this type of work, where what we want is very obvious and encoded somewhere in the existing functionality, this type of work is at least one order of magnitude cheaper, maybe even more. I mean, if you had a team of engineers, and they’re engineers, maybe $200,000 to $400,000 or something like that per year, this is a drop in the bucket, the $120,000. When you were proposing the project, did you have a sense of how much this would cost in tokens before you went and did it?
Stephen:
[01:00:14] Yes. And the bill ended up being larger than I expected, mostly because the line count ended up being larger than I expected. I hadn’t fully factored in the growth rate of how much incoming code there would be.
So this was, there were a few things. I think I did a lot of things right. Yeah. I did a few things wrong, and 20/20 hindsight and all that. One of them I already mentioned: I didn’t spend enough time really fleshing out the end-to-end tests to make sure that they were as comprehensive as they could have been.
[01:00:51] I could have done that better. The other thing I could have done better was really factor in the growth rate of code that would need to be ported. So when I looked at the codebase as part of writing this proposal, it was somewhere on the order of 150,000 lines of production TypeScript, 130, something like that. I don’t remember the exact number. And based on that and some back-of-the-napkin calculations and looking at how long it took to create the prototypes that I’d alluded to earlier for, in the different languages to sort of measure perf and things like that, I think I’d estimated about 6 to 8 weeks.
[01:01:32] to get it done. But when you look at what we actually needed to port, I didn’t factor in a few things. One, that there were tens of developers using agents, very quickly augmenting the codebase with lots of different additional functionality. And this codebase is one that’s— it’s not public open source, but it’s effectively internal open source. So there’s a bunch of folks that come in and add the things that they need, and then the core team is helping to review all that.
[01:02:10] And so there’s this very significant influx of functionality that I hadn’t accounted for. I also hadn’t accounted for the fact that because of how the codebase came about, the Copilot CLI and the runtime were kind of very intertwined, more so than they ideally should have been. And so in order to port just the runtime, and the CLI can be ported separately if you wanted to, but to actually tease them apart to the point where you don’t need the CLI, you don’t need to deploy the CLI as part of your application that’s using the runtime.
[01:02:46] There was this effort of teasing them apart, but then there’s also an effort of moving functionality that really doesn’t need to be in the TUI and that could be shared in the runtime down. And I failed to account for how much of the code that was currently marked as CLI would actually be beneficial to move down. And so over this effort, there was also code that was coming down. So by the end, we ported somewhere between 400,000 and I think 450,000 lines of TypeScript.
[01:03:14] So about 3 times what I’d estimated, which impacted both the end-to-end time that it took to do it. And then also, sort of token counts.
01:03:24 — When to do a full rewrite
Ryan:
[01:03:24] So after having gone through this process, if there was a software engineer out there on some team considering doing some sort of rewrite, what criteria should they look for to decide if it’s worth undertaking or not?
Stephen:
[01:03:41] It really comes down to what is the business value? I mean, there’s the ones you can do for fun where it’s your dime and you get to decide what fun is worth. Just like, I really don’t like going to casinos because I really don’t like what I consider to be losing money gambling, but friends of mine love it and they consider it paying for a night’s worth of entertainment.
[01:04:06] They’re not losing money. They’re paying for fun. Right. And so if you’ve got your own personal development project and you want to spend tokens on porting that to something else, great, have fun. That sounds awesome. But the benefit, you need to understand what the benefits are. The benefits are your enjoyment. When it comes to a business, it’s what is the business value that this is accruing to?
[01:04:30] And then, like anything else, what is it going to cost? And what is the return on that investment? And that needs to be very clear. And there are lots of these migration projects that people all over the world and businesses all over the world are doing even before agents, where they have done that cost assessment and they have decided that the return on investment is very much worthwhile. I saw this a lot with .
[01:04:59] .NET, with people moving from .NET Framework to .NET Core. And because we invested a lot in making .NET Core incredibly fast. And so there would be very real performance benefits that would come from doing that migration. And they could figure out the cost associated with, well, we’re spending X millions of dollars running our services today, and we would save $3 million a year by moving, by doing this port.
[01:05:28] And this port is going to take a year and it’s going to cost us a developer for that year. So they do the math and this is absolutely worthwhile. Even at a more micro level, not even for ports, what features do you invest in? When I was spending a lot of my time working, for example, on the .NET regex engine, I could look to see how much are we spending, we as a company, or how much is this particular service spending on running on regular expressions every year.
[01:06:06] If I invest two weeks in making these changes to the .NET regex engine and it makes the patterns that they’re using four times faster, what is the return on investment for my two weeks and how much they save as a service over the course of the year, right? These kinds of trade-offs are everywhere, and you just need to be very clear about why you’re doing it, what it will cost, and what you get for it. And if you can write all that down and you can justify it, then it’s a good project to do.
[01:06:37] If you can’t justify it, it’s very suspect.
01:06:40 — Career learnings on the way to Distinguished
Ryan:
[01:06:40] When I was doing research for this interview, I looked into your career. Well, on your LinkedIn, there’s no information, but I found an interview you did with— Yeah, I’m very— I basically, I’ve been at Microsoft for 25 years. I love it. I’ve jumped around a lot internally, but I’ve had no interest in going to different companies. And so I haven’t really invested in keeping my LinkedIn profile up to date.
[01:07:03] But so when I did the digging, though, or listened to an interview that you had done about the different roles, your path to Distinguished Engineer is really interesting because you did a bunch of different roles, and it wasn’t just SWE on Team A and then SWE on Team B. You held a variety of ones, starting as almost a forward-deployed engineer, consultant engineer.
Stephen:
[01:07:32] Yep.
Ryan:
[01:07:33] And then you did technical writing at some point, and then you were a technical advisor to a VP. Oh, and you also jumped to a PM at some point. From all those different roles that you tried, which of them would you say was the most you would recommend someone dabble in that direction if they wanted to try something that wasn’t the pure SWE role?
Stephen:
[01:07:57] Yeah, every role that I did taught me something different. And so from that perspective, I have great respect for people that come out of college and want to be developers and become an engineer. And that’s what they do. And they just stay on that path. That’s a very valid path for me. I think I picked up a whole bunch of skills by having more of a circuitous route.
[01:08:19] So, like you mentioned, as a forward deployed engineer, it was more at the time was Microsoft‘s consulting arm. And so I would spend anywhere from three weeks to three months at a particular customer site helping them solve their problems. And I would learn a whole lot about their industry. I would learn a whole lot about their particular breed of problem. I’d learn about Microsoft‘s offerings from the outside and places that it was great and places that it wasn’t so great.
[01:08:48] And learn how our customers, their real struggles and what they were dealing with, and being able to bring that back to then later building solutions for them was incredibly valuable. When I then worked on MSDN Magazine and was doing a bunch of technical writing and technical editing and figuring out what I wanted to share with people and how I wanted to share it, and some amount of development, building things to then talk about.
[01:09:21] I gathered a set of skills around how to think through what was really interesting, what was really relevant, how to communicate it in a clear fashion, what didn’t need to be said, what did need to be said, how to tell a story that would really resonate. Yeah. From a PM perspective, it became, how do I, how do we figure out what actually needs to be built and what are all the pieces that need to come together to really deliver this thing?
[01:09:53] Obviously, from a dev perspective, all the coding and the design and the architectural and all that kind of stuff. The most interesting role for me, I think, which I am incredibly thankful and glad that I did and will never do again, was being sort of this technical advisor for a VP at Microsoft. It was an amazing experience. It was a couple years where I was their right-hand sort of like invisible partner, shadowing everything they were doing and ghostwriting and, after every meeting, having a little review, kind of going through what we heard and what we think we need to do about it and bouncing ideas back and forth.
[01:10:41] It was incredibly insightful into how things actually work, how decisions actually get made. Being a part of things that I wouldn’t otherwise have been a part of, at least not for many, many years later. It was awesome. It was also incredibly draining, incredibly selfless, and it took a lot out of me. And so I cherish those experiences and what I learned from them, and I would highly encourage anyone who has the opportunity to do so to do that kind of thing for a period of time.
Ryan:
[01:11:24] That role is really interesting because it sounds like you’re basically a chief of staff to someone super high up, but technical. So you’re not like an executive assistant. You’re pulled into all the really privileged conversations, and you’re providing that information that they need to make the calls. You mentioned that you learned how things are really done.
[01:11:49] Do you have some examples of stuff that someone might learn if they were in that role or something that was really helpful for you?
Stephen:
[01:11:57] There are lots of times where if you’re sort of a rank-and-file engineer at a company and you see these decisions that get made, an email gets sent out about some new path we’re following or some partnership we’re forming or some investment that we’re making, some acquisition that is being done, winding down a certain product, investing in a new product, spinning up a new team, all of those things look like they just sort of come out of thin air and they’re just sort of handed down as this is the finished decision and it’s fully formed.
[01:12:39] You don’t really get insights into everything that led up to that point and all of the alternatives that were considered and dismissed, and why, and what the trade-offs were that were factored in, and how experienced people in the industry are thinking through those alternatives and those problems, and which things they’re choosing to focus on, which things they consider to be irrelevant.
[01:13:08] All of that is stuff that I didn’t have exposure to. And being able to be a fly on the wall, just soaking in, not to mix metaphors, but soaking in all of that and then being able to have conversations about it. We’d get out of a meeting and he’d basically say, so what did you hear? What do you think? Help reason through it. And sometimes I would have valuable insights and sometimes he would say, no, that’s not how we think about these things.
[01:13:40] And every one of those was a learning experience for me.
Ryan:
[01:13:42] Yeah, I’ve heard engineers say similarly about the value of trying management, even if they ultimately decide they hate it and they don’t want to do it again, is you get to see how the sausage is made a little bit more as a manager. You see all those decisions in managing the org that can be helpful in operating just in your career in general.
Stephen:
[01:14:07] Yeah. And I also, I didn’t mention it before, but I also did that. I thought that what I really wanted to be was a manager. And so I basically set things up where at a certain point I jumped around a lot internally at Microsoft, and I changed to a role where I had sort of handshake agreement that six months later I would be able to sort of shift into a lead position and have a team of people.
[01:14:35] And I arranged all that, and it all worked out. And I became a lead, and 6 months into being a lead, I was like, I’m not sure if this is actually what I thought it was. And 6 months after that, I was like, nope, it’s definitely not. This is not for me. And then it took me about a year after that further to sort of safely work my way back to an IC role where I’d been since. And again, to your point, super grateful for the experience, incredibly grateful to Microsoft for having an individual contributor career track that allows one to sort of progress their whole career and continue to be promoted and whatnot, but still not have to have direct reports.
[01:15:24] For me, what I realized was the thing that I loved about, or that I thought the reason I wanted to be a manager was because I thought it was how I could help others and how I could kind of provide career guidance or help other people solve problems or whatever it was. And that was misguided on my part. And it took becoming a manager to realize that I could do all of that as an individual contributor.
[01:15:54] And the things that I loved about being a manager were things that I didn’t have to be a manager to do.
01:16:00 — Top technical book recommendation
Ryan:
[01:16:00] Do you have any top book recommendation that maybe impacted you?
Stephen:
[01:16:05] Yeah, I mean, it’s old at this point, but I’ve read it several times and I absolutely love it. There’s a book by Charles Petzold called Code. It’s probably from what, 2003 maybe? I don’t know, something in that timeframe, but I love it. And it is one of the things that I really value is being able to explain complicated concepts in a way that doesn’t require the person to know anything about what you’re talking about.
[01:16:36] I think that is a valuable skill, even if you’re talking to people that do know what you’re talking about, because being able to break down complicated concepts into very clear, concise pieces and communicate that clearly, and then at a 30,000-foot level, is a great way of getting everyone onto the same page and sort of arguing your case and pushing things in a direction that you want them to go.
[01:17:06] And Code is a book where Charles basically does that, where he talks about how computers work from the ground up, teaching people that know nothing about computers, that aren’t technically experienced, about binary, about logic gates, about circuits, about programming, about computers, about the Internet, and kind of building things up from basic principles, from nothing to a reasonable understanding of how everything works.
[01:17:37] And I was blown away by the book, and I’ve since reread it on multiple occasions, even though I know everything that’s in there. I just love the way that it communicates things. So if I had one recommendation, that’s it.
01:17:49 — Advice for young engineers
Ryan:
[01:17:49] Looking back on all your experience, you’ve been working in the industry for a while. What professional advice would you give a young engineer just entering the industry today that you believe could be helpful?
Stephen:
[01:18:02] That’s a great question, and it’s one I’ve been struggling with. A couple years ago, I would’ve had a much more clear answer, but with AI, I still don’t know what the right answer is. A lot of the times that I spend working with AI agents, I’m bringing to bear 25 years of experience, for better or worse, in being able to very quickly sniff out things when they’re doing the right thing or they’re doing the wrong thing or how I want to guide them, or an agent says, “I’m 100% confident that the right answer is A.”
[01:18:43] And I’ll say, yeah, but have you also considered B and C and D? Because these trade-offs might not be there, right? And, of course, just those little things, being able to guide them in the right way, requires you to have gone through it. And I’m concerned that because of how AI agents are progressing, they’re taking away the opportunities to have gone through it. Yeah. And I don’t yet know what the answer then is.
[01:19:13] I’m a very firm believer in really learning kind of the basics and building everything up from core principles. And so learning a language, I still think it’s valuable to learn and to start out with a language like C because you’re understanding how computers actually work and you understand what memory management means and you learn about all the various trade-offs between this algorithm and that algorithm, rather than working at a layer where a lot of that is obscured from you, where the language and environment are providing a ton of services to you, but you don’t necessarily understand everything that’s happening.
[01:19:54] And so from that perspective, I still think that there’s value in focusing on those core concepts and really internalizing algorithms and how machines actually work and being able to reason and extrapolate from that. And I suspect that will, in whatever domain you’re in, that kind of core primitive knowledge about distributed systems or about databases, that foundational stuff will help to be successful in a world where the vast, vast, vast majority of the code is actually being written by agents and you’re helping to guide them.
[01:20:38] But I don’t know that for sure. And at the rate that these agents are learning and evolving, I don’t know what the world’s going to look like in 2 or 3 or 4 years. I suspect it’ll look vastly different than it does today. I think we’re in for a wild ride.
Ryan:
[01:20:52] Awesome. Well, yeah, thank you so much for your time today, Stephen. I really appreciate it.
Stephen:
[01:20:57] Pleasure. This has been a fun conversation. Thank you.
© 2026 Ryan Peterman
548 Market Street PMB 72296, San Francisco, CA 94104