Arlo Belshee: What Are the Hidden Foundations of Successful Team-Building?

Arlo: (00:52)
Hi. Thank you for having me.

Richard: (00:53)
You got it. Arlo, will you introduce yourself to our listeners?

Arlo: (00:57)
Yeah, so I am one of those crazy people out there who loves Legacy code. The uglier the code, the better. The more difficult it is to work with, the better. The more important it is to the business, the better. So I really enjoy jumping in with a team, do the hardest bit of work that they have, and immediately showing them more effective ways to work with that, and to eventually master that early Legacy code.

Richard: (01:23)
All right. When you say legacy code, what do you mean?

Arlo: (01:26)
So here I was just using sort of the industry standard definition, code we hate but have to have. When I’m being more nuanced, I actually distinguish between Legacy and indebted.

Arlo: (01:37)
Legacy code is any code that is providing significant value to the business. It’s stuff for which it is more valuable to have it not change than to have it change. So once a product has gotten big enough, if I’ve got a $28 billion product, the incremental value of adding a new feature is $100,000. The incremental value of changing something that my customers are depending on is a risk of perhaps losing 10% of my market. That’s $2.8 billion.

Arlo: (02:06)
So features are worth far less than the code itself is, and that results in very different economics. Naturally you should be more conservative in how you change that code. It’s what’s economically right.

Richard: (02:18)
Yeah. The way you define it, I’m afraid to touch that code now.

Arlo: (02:21)
Right. Then on top of that, there’s indebted code. So indented code is any code that developers legitimately fear to change because the probability is you’re going to screw it up.

Richard: (02:31)
Right.

Arlo: (02:34)
It’s code that’s full of hazards. I also call it hazardous code, so it’s unsafe to go there. The fun part of course is big old projects usually code. It’s both legacy and indebted, and that makes it so that we really fear to change it yet it does need to change.

Richard: (02:51)
Yeah. All right, awesome. So this is the podcast about teams and great teams and the best teams of our lives. When you look back at your whole life, not even your whole career, when you look back at your whole life, what’s the single best team you were ever part of?

Arlo: (03:07)
So the single best team is the one at Silver Platter. It’s the team wherein we first transitioned to extreme programming. It’s fairly common among people who do XP and really do it well that their favorite team is the one that was first transitioned.

Richard: (03:23)
Yeah, it’s my typical example as well. I mean there’s the example of me and my wife, and my other example is that group of people that I did XP with.

Arlo: (03:30)
Yes, yes, absolutely. Me and my wife would be the other one.

Richard: (03:35)
Yeah.

Arlo: (03:36)
That’s the one with the best emotional fluidity and really deep attention to human being. Then my first XP team is the one with the best systems thinking and real alignment to how can we accomplish more effectively together.

Richard: (03:53)
Yeah. What else about that team? What I usually get people to share is a single word to describe that team. Sometimes it’s like we go through a simple guided meditation and we take ourselves back to it and re-experience what it felt like to be together and doing the activity together. Is there a single word you could use for that first XP team or the sensation of working together, being together?

Arlo: (04:16)
Yeah, I absolutely know this sensation. I’m looking for one word that describes that hits that sensation. It’s fundamentally really deep variety. It’s the diversity and inclusiveness are the highly overused buzzwords right now, and I don’t mean them in the current buzzword sense.

Richard: (04:33)
Yeah.

Arlo: (04:34)
It’s that deep diversity where each person is internally diverse. They have many different opinions. They are in contrast with each other. Each person has their dark sides and their shadows as well as the bright happy bits. The dark sides and shadows are fully in play and fully engaged. We had one guy who provided a lot of different pieces of value, but the most valuable thing that he provided was he was the one who would get pissed off. So things that other people are starting to cope with and just yeah whatever, he stands up and throws an eraser at the whiteboard and said, “God damn it, this thing is intolerable.” Then all of us who were coping go-

Richard: (05:15)
Yeah, you’re right. Yeah, it’s hilarious. We get acculturated. I had breakfast with a friend this morning who’s a coach, and he sees people join his company where he’s coaching and they’re full of great ideas. 90 days later nothing’s changed and they’ve lost their spark. They get acculturated. They’re not throwing the eraser at the whiteboard anymore at least. Yeah.

Arlo: (05:41)
Yeah. That’s fairly common, and we all get acculturated. We also all learn to cope. Many of us start looking around for social cues of what is it that you want me to do in this team? We all need that, but we also need to balance it.

Richard: (05:57)
Yeah.

Arlo: (05:59)
It’s the levels of ego development. At the earlier stages, we’re looking for expertise or we’re looking for achievement or accomplishment. Where we look to our peers, we look to our objectives, we look to what is the right or most effective way. Those are where the seat of our ego resides.

Arlo: (06:21)
As long as we’re there, there are certain forces which we are simply subject to. We cannot think about them. We cannot manipulate then, we cannot work with them. It’s only with later levels of development that you can actually take the perspective and see how even those are just part of the little world around and we can manipulate it.

Richard: (06:43)
Yeah. What else about this team? This sounds like it was an incredible, I don’t know. For me, I look back on my analogous team, my homologous team, and it was a life changing experience. What was this like for you? What else about this team?

Arlo: (06:57)
Yeah. So it was a life changing experience. We had everybody. We had the usual combination early on of high pressure. The company was going to die. That’s why we were willing to try something called extreme programming at the extremes.

Richard: (07:11)
Me too.

Arlo: (07:13)
Yes. That was the difference between all the people who started XP before 2003 or so and all the people who started Agile afterwards.

Richard: (07:22)
Yeah.

Arlo: (07:23)
So yeah, we were in that state. That meant that we weren’t willing to fail for long, so we had to really rush to embrace it. This was back in those days when sprints were typically a month long and so on. Pair sessions were many days. That’s why we did promiscuous pairing. We rotated pairs every 60 minutes, then 90 minutes as we grew. We did one week sprints. We had a retrospective that was run by an experimental physicist that we had on the team.

Arlo: (07:55)
So it was plausey experiments. What really matters here and how would we measure that and what could we try and when things sort of feel like they work, but the numbers show they aren’t like, “Okay, throw it overboard. Let’s try something else over that time to keep up with just it feels right. It isn’t working.”

Richard: (08:14)
Yeah.

Arlo: (08:16)
Yet also deep valuing of people’s internal experience. A lot of the things that we were looking were are those qualitatives and how was it feeling? It needed to be both. So this was a team it was very practical, grounded, and very focused on how do we be awesome, how do we be effective and deeply understanding that that is a team activity.

Richard: (08:39)
Yep. So many analogs too. The first two teams that I worked with in an extreme programming style on one of them, well both of them actually, we called them underfunded startups and it was get stuff done or die. On one of them we had a physicist and I didn’t get it at the time. The knowledge that he had, the value that he brought with his background in real science and conducting experiments.

Arlo: (09:06)
Yeah. Yeah. I mean our team had an experimental physicist. It had a pure mathematician. It had a film producer. It had a theater major. It had a guy who had done way too many drugs in the 70s and had no short term memory. It had me. Math, physics, religious studies, sort of general. We had no software engineers, but that wasn’t necessary. Software engineering, it remains a skill without actually a great depth of education and so you can fill that part in.

Richard: (09:37)
Yeah. All right.

Arlo: (09:37)
Through a great deal of learning, but not a great depth of education.

Richard: (09:44)
Yeah. Yeah. Isn’t that interesting? What else about this team? What other subjective sensations? Go into how you know this was the best team of your life or anything subjective or objective? Do you have measurements? Do you have other feelings or?

Arlo: (10:03)
Yeah, bunches on both sides. So feelings, I mean it felt amazing. We would always try new things and the tremendous openness to explore and to try new ideas and read something in a book, bring it, and try it out. Well that didn’t work. Okay, throw it overboard. Or it did, build on it. We did the launch meetings and we’d start that with an emotional check in and we’d do that three times a day.

Richard: (10:28)
Wow.

Arlo: (10:28)
It was separating the period of not working from, we’re working together as a team. We were always pairing. We were always in one room sitting eight of us around one table and in pairs. So there was that transition we got stand up was about a two to three minutes long, most of which was checking in with the emotions of the people and then we’re just like okay, here are the things. Sit at each computer, each computer on the task go.

Arlo: (10:55)
Right. So that was very effective for keeping in that constant fluidity, the human dynamic. Objectively after that team, the company eventually disbanded and the team died as a result. The next job that everybody had I think without exception, pretty close to without exception certainly was their career and dream job.

Richard: (11:17)
Yeah?

Arlo: (11:17)
So most people on that team went from school, to one team, to end of career. Then they just spent the rest of their career doing whatever they wanted to.

Richard: (11:31)
Wait a minute. Their next job was their end of career job. Like what?

Arlo: (11:36)
Like becoming the guy who really loved baseball and was a pure mathematician became the computer guy who was doing the statistician for his favorite baseball team.

Richard: (11:45)
All right.

Arlo: (11:46)
Couple of people decided that they wanted to go into management, so they immediately went into a mid upper level management and go to directors and some retired after that next job and some kept going. I went on to the international speaking. That was my first three, four years. Everything that I brought out was stuff that I invented on that team and was talking a lot about experiences on that team and then the rest of my career has been sort of follow on to the speaking and the coding network of stuff.

Richard: (12:15)
Okay, cool. You shared with us one example of a concrete practice that this team had and actually, okay you alluded to many interesting. XP, right? That’s a lot of concrete practices and you added on this practice of an emotional check-in, which I don’t know, what year was this?

Arlo: (12:33)
2000, 2001, something like that.

Richard: (12:36)
Yeah. Okay. So I think that’s really advanced for bringing in an idea of emotional check into a team, to a technical team. I’m so happy to hear that that was happening back in 2001. What are some other concrete practices that this team had?

Arlo: (12:50)
Well promiscuous pairing. We invented it.

Richard: (12:51)
Yeah. You invented it?

Arlo: (12:53)
Yeah, yeah. We read the book and it said pair and so we said well pair all the time because it sort of alluded that’s the good thing to do.

Richard: (13:02)
Yeah.

Arlo: (13:03)
I didn’t know at the time because I didn’t know any of the rest of the community but I later learned from talking to Ward once I moved to Portland where he lives, that they meant pair all the time in these pairs that would rotate every few days or every sprint or something like that. We paired and then we’d rotate and we noticed that when we rotated more quickly, we would learn more quickly. We found that at the bottom end of time, like when it was really short, everyone was always feeling like they don’t quite get it.

Arlo: (13:32)
Yet we looked at the board and those weeks we would get way more done. We started experimenting again, our physicist, we started experimenting with pair link and we found that with a three to six person team, 60 minutes was the right thing. Once we got to 11, 90 minutes was the right thing and the curve was we had nearly double productivity at that time level to anything greater than about double that. Once it came down to double that time period, it was flat then on. It didn’t really matter how long you swapped between the two swaps.

Richard: (14:05)
You had evidence. This is so cool.

Arlo: (14:07)
We had evidence.

Richard: (14:08)
Yeah.

Arlo: (14:08)
Then we finally found a theory. It took us a year to find a theory, but we had evidence.

Richard: (14:13)
All right, what’s the theory?

Arlo: (14:14)
The theory is beginner’s mind. The theory is that what was happening, because we also experimented with other things like who owns a task? Is it owned by the most qualified or least qualified person? Do people rotate in? Like this task stays with Bob and then he switches partners? We found the most effective thing was that effectively a computer owns a task and that you would-

Richard: (14:34)
A computer owns a task.

Arlo: (14:35)
You would put the two least qualified people on it at first and then you’d rotate one of them out for the next least qualified person and then you’d rotate out whoever had been there longest. So always whoever might know anything about this task, keep them away from it. When you did that and you did that systematically, continuously, a couple of things happen.

Arlo: (14:56)
One, the least qualified grows very, very quickly. So the result was that everyone was damn qualified at everything we did. That meant that differences, again, this is a highly diverse team, differences in ways of being and ways of thinking would shine through because everybody’s a master of the C plus plus programming skills because you learn those fairly quickly when you’re doing these qualified and whatever else is involved. So then the person who’s a visual thinker is able to bring that to bear or the person who’s insanely detail oriented brings that to bear or whatever are the real personality traits because everyone has equal access to bring that in. So that was one big thing that really provided a huge lift. Another was I mentioned beginner’s mind. What Zen Buddhists found long ago and has since been reproduced many times is that there are multiple different brain states you can be in.

Arlo: (15:47)
Flow is one of those, disconnected, and whatever. If you look at those neurologically, flow is a state of extreme practice where you are repeating over and over what you already know and it actually blood flow goes down in the prefrontal cortex. Blood flow goes down in the judgment centers and it goes up in the cerebellum. It’s repetitive, which is wonderful for athletes. It’s wonderful for musicians. You want to do the same thing over and over. How often do you want to invent the same thing over and over?

Richard: (16:16)
Yes. It’s not so good for those of us inventing a new thing every day, which is what we’re doing when we’re writing code.

Arlo: (16:22)
Exactly. So it’s basically the shittiest possible state of mind, least productive possible state of mind for a programmer.

Richard: (16:28)
Yeah.

Arlo: (16:29)
Yet it’s what we seek and we seek it because the only other state of mind that we know is distracted, which is even worse.

Richard: (16:36)
Yeah.

Arlo: (16:37)
Bouncing between a whole bunch of things. Can’t think of anything, and so you can’t use even your short term memory. All the memory registers are flushing. It’s really bad for this. So we see flow as the be all end-all. What we found here is a different mind state, is beginner’s mind.

Richard: (16:51)
Yeah.

Arlo: (16:51)
Beginner’s mind is the one that the brain goes into when you have enough awareness of whatever you’re doing that you feel you should understand it. I really should get this, but not enough that you actually do. Right? This is why board gamers do so well on their first win of a game and their first time playing and then they suck thereafter. In beginner’s mind state.

Richard: (17:15)
We call it beginner’s luck.

Arlo: (17:16)
Yeah, we do. We call it luck. Yet you can measure it. It’s consistent. It happens under laboratory conditions. It’s not luck. You can hook someone brains up to FMRI and you can actually observe. What happens is it turns on the prefrontal cortex, acetylcholine dumps throughout the brain, which amplifies learning. It is the brain’s way to say, “I got this, I got this, I need to learn it.: So it’s highly metabolically intensive. It’s exhausting.

Richard: (17:43)
It’s exhausting.

Arlo: (17:44)
You learn really, really quickly and you put together things you otherwise wouldn’t. It’s optimal for invention and for skill growth. So the brain naturally goes into it if you’re in that state or that situation, that context of I feel I should get it, but I don’t quite. So what we were doing by rotating every hour and rotating off the person who was more experienced, we called it SIF based rotation.

Richard: (18:05)
Call it what?

Arlo: (18:06)
SIF based. There was always one master and one apprentice and soon the apprentice kills the master and becomes the new master. Then stupidly acquires an apprentice. I’m not sure why they do it. They can’t see the system.

Richard: (18:23)
They don’t see the pattern. Yeah,

Arlo: (18:24)
They don’t see the pattern. We did and so we would do that. The result is that you spend an hour brand new on this task and you’re naturally going to be in that state because it’s sort of familiar. I was in this code two days ago. I really should get it, but I don’t get it. It’s changed so much, but okay. Right? That around an hour in, 90 minutes and if you have more people because you’ve been further away from the task, is the point where you start to feel, “Oh, I get it. Now I can be productive.” So it’s time to fire you.

Arlo: (18:53)
So then what we do is we actually have you kill your master, your master goes away, and you become the master and now your job is you think you get it. Well 60 minutes from now your apprentice is going to kill you. So you have to bring that guy up to speed, but not just up speed. Any idea you have that you want to get in the code now that you think you know it, you have to push through that guy’s mind.

Richard: (19:15)
Yeah.

Arlo: (19:16)
So it has some of the results that the strong style pairing that Whelan has as well. So the result is that you’re thrown right back into that beginner’s mind because you’re now not beginner’s mind on the task, you’re beginner’s mind on one level up. How do I help people get this task? How do I make this more effective? You’re just alternating throughout the day, back and forth between beginner’s mind on those two things. You get exhausted when you work a six hour day with some breaks in between and you call it done. In that six hours we all produce weeks worth of work.

Richard: (19:45)
Yeah. So you collected a bunch of data. Did this team figure out that it was beginner’s mind as the theory and how long did it take to figure that the beginner’s mind was the explanation for everything?

Arlo: (19:55)
I don’t remember precisely. It was that team. It was six months to a year, somewhere in that timeframe before we ran across that concept and went, “Oh, that matches,” because we could match the feel of the thing. It’s like, yeah we’re exhausted. It’s this sort of exhausted. Yeah, we don’t quite get it. Then when we do is about the time that we’re supposed to swap and once we had that theory, then as we hired new people, we could explain to them the theory and just show the data and say, so that’s why we want you to do this crazy thing that feels terrible.

Richard: (20:27)
So I know you had data, but what about 60 minutes versus 90 minutes depending on the number of people on the team?

Arlo: (20:32)
Yeah. Our belief as to the reason was that when there were more people on the team, it had been longer since you had been paired with this person or on this part of the code. So it took a little bit longer before you’d lose the beginner’s mind edge.

Richard: (20:46)
Okay. Makes sense.

Arlo: (20:47)
Yeah, that was our guess. Yeah. Yeah, so this was a team that innovated all over the place. We paired differently. We planned differently, our retrospectives were different. I haven’t seen anyone who did retrospectives like we did. Our stand ups were different. Everything.

Richard: (21:04)
Okay. So what about planning and retrospectives?

Arlo: (21:07)
So planning, this was back again, 2001 or so. We did a rolling way of multilevel planning with good feed forward there. At this point it’s fairly common among teams, at least among those who are vertically organized and are in control of their long range planning. At that point it was a little more unusual. So yeah, our planning was there. It was of course, one week sprints.

Richard: (21:35)
Which is still unusual today. People still say, we can’t even get stuff done in three weeks. We can’t do a two weeks sprint.

Arlo: (21:42)
Yeah. Oh, Then there was one other thing which was we measured velocity differently and this was actually important. It resulted in really low variation. We did other things to get rid of the technical variation, but we were still in an environment in which most weeks there are a large number of interrupts of things that could not be foreseen, some technical, lots of business. It’s a startup. There’s this stuff going on. So we looked at velocity and everybody seemed to do it by at the end of the week we count what got done and then we assume that we’ll do that next week.

Arlo: (22:13)
There’s a fundamental problem of knowability because during the week you gain information. So what you’re doing is you’re measuring at the end when you have maximum information and assuming that you have that much information, that planning in your next week.

Richard: (22:28)
Right.

Arlo: (22:28)
You never do. Consistently the part that you don’t know, the unknowns that show up, all add more work. So if you measure velocity at the end and assume you’ll do that next week, you are 100% guaranteeing that you will over plan and then not be able to meet it. So what we did was simply tracked each card and we had multiple categories. There was planned work that got done, unplanned work that’s still valuable that got done. Then there were emergencies, red cards, sorts of things which is on planned work as well. We differentiate between those two because what we wanted was we would then be able to predict we did this much planned work, next week we’ll do that much planned work. We assume the other two categories will show up. They’re just invisible right now.

Richard: (23:10)
Right.

Arlo: (23:10)
So we’d plan that much planned work and it would show up. Now we have exactly the right amount of space for things to show up. Then things would show up into one of two camps. Red cards or unplanned. Unplanned where things that like this is a really good idea, we want to keep this open. This is the stuff that we want to encourage to keep happening.

Richard: (23:31)
Yep.

Arlo: (23:31)
Red cards are the opposite. It’s unplanned work that we would like to discourage. We’d like to systematically eliminate. Like books, right?

Richard: (23:37)
Right.

Arlo: (23:38)
So then when we get to the end of the sprint, we can now look at, okay from our velocity number, we know this is how much we should plan. Also, right now we can see that the unplanned work is coming in two thirds of stuff that we don’t want. One third is the stuff that’s good stuff. Okay. So as we go towards retrospective, what are some systematic solutions we could do to reduce the sources of the bad unplanned work? Or right now actually we’ve got a fair amount of good unplanned work coming in and we’re being responsive to that. What are some things that we could do to enhance systematically the amount of good unplanned work that’s coming in on our ability to respond to change? So we could do either of those looking at exactly what that work was this week and we’re causing it.

Richard: (24:19)
You just said the word systematic twice there and you’re talking about retros and I love that. There are so many teams that are just starting out at this stuff. They don’t have, we’ll say high maturity at retrospectives yet. What you said twice was the word systematic, finding systematic responses or systematic ways to amplify the things that you’re noticing. Systematic responses to minimize the things you don’t like. Systematic.

Arlo: (24:44)
Yes. Yeah because you’re never going to hit that card again.

Richard: (24:47)
Yeah.

Arlo: (24:48)
You are going to hit the things that cause that card. Yeah. Since that team, I’ve refined that all sorts of things and it’s the safeguarding stuff that I currently teach and sell. It’s all about hazard thinking and so on. That also showed up even back then. The way we did retrospectives was a bit different. It ties in both to the emotional fluidity and to the systems thinking. So during the week we had a box, just a shoe box, sort of a thing that’s sitting in the middle of a table and anytime anyone had an emotional response to anything, like emotions are going up. We recognized and agreed and we’d all talked about this. Emotions are the way the brain signals something important happened and it gives you, by which emotion, a little bit of a clue as to why it’s important.

Arlo: (25:32)
So if emotions aren’t coming up, then that means nothing important is going on. That’s just fine. I mean a lot of time we just need to code and we’re just coding through some code and we get an idea and then we put it in the code and whatever else. That’s great. It produces good work. It also means we don’t need to retrospect on that.

Richard: (25:48)
Right.

Arlo: (25:49)
No matter how valuable that work turns out to be, no matter how much of the week it was, it was pleasant. It was whatever. Great.

Richard: (25:56)
It just happens.

Arlo: (25:57)
It just happens. No need for systematic change there. Right. If however, I’m coding along and I feel overjoyed. I feel glad, really glad. Well that tells me that both something positive happened and it was surprising. I did not expect that positive thing.

Richard: (26:13)
Yeah.

Arlo: (26:13)
If I feel glad, I need to write that down and put it in the box. So that was our emotions box and you’d write down, I felt this emotion when this happened and you put it in the box. Then the agreement was that would mean then that having to put that in, we’d work through whatever emotions were there in the present and then we’d come back at the retrospective and talk to them. Every emotion goes in the box. Glad’s there, angry’s there, mad, sad, glad, afraid, and then nuances. Frustrated.

Richard: (26:39)
Yeah.

Arlo: (26:41)
Everyone had something important to tell us. Angry means that something which is valuable to you, some critical resource is under threat. Maybe it’s that you’re feeling disrespected. Maybe it’s that you’re feeling your time is being wasted by something. Maybe it’s that you’re feeling like good ideas aren’t getting out. All of these are something we should talk about, right?

Richard: (27:03)
Yeah.

Arlo: (27:05)
Anger should never be suppressed. It should never explode all over either. It’s that middle way, that way of flow with the anger. See what it has to teach you, write it down, bring it up, talk about it, and make the change to protect the thing that anger is saying needs to be protected. Likewise for sadness and for fear and for everything else. So we would take these emotions, we put them in the box throughout the week, and then that is the data gathering portion of our retrospective. When we get to the retrospective, we just start by pulling those out and we sort through them. The first sort is just the longs and the shorts. It’s short if everyone’s basically agreed there’s not really any systematic change that needs to happen. There might be a little quick, let’s just do this one thing differently or the main thing is it just needs to be heard. So it’s just going to be fairly quick and that short and long is anything else.

Arlo: (27:56)
So we just go through, separate them. Then we did all the shorts in no order and then we’d prioritize the longs that we’d do one of those and we’d do one of those in a systematic way. We did that every single week. So that meant that the little short things that otherwise add up, they just get done at the end of every week. Work through them all quickly. Every time that someone had an emotion, A, it’s felt and they know they’re recognized around the team and it has those human effects and B, the important things got in the box and we worked with them. All the other stuff even we don’t have any recency bias or anything like that. Unimportant stuff on the last two days. Well you didn’t feel emotional enough to put it in the box so it doesn’t count, throw it away. Right?

Richard: (28:33)
Yeah.

Arlo: (28:35)
That important thing for Monday still shows up. So we did that. Then because we were doing one systematic approach every time, then that’s where we did our experiment. We could really take a systematic approach or we’d pause it and then we’d start figuring out what are the multiple causes and contributing factors? What are multiple possible solutions for each of those? How could we run several of these experiments at once? How would we measure which ones are resulting in a change to the underlying thing? Run those over the course of the week or sometimes multiple weeks, and then incorporate those into our system of work.

Richard: (29:09)
Awesome. Genius. You’re doing it every week. So you had a high frequency improvement cycle.

Arlo: (29:16)
Yeah. Very high frequent and very high quality as well.

Richard: (29:20)
Yeah.

Arlo: (29:20)
Good signal.

Richard: (29:21)
You mentioned and you avoided recency bias. The stuff that happened yesterday is the stuff we remember the most, but you remembered everything every day.

Arlo: (29:28)
Right.

Richard: (29:28)
At least the important things that triggered some real emotional response.

Arlo: (29:31)
Yeah. You had a filter that was applied continuously.

Richard: (29:34)
Yeah. Okay. So I know that your job is to teach people all the things that they might want to do to have a more awesome software development experience. What are some fundamental things? What are some foundations? If you could tell people one or two or three things right now, what would those things be to have the best team of their life?

Arlo: (29:53)
Number one, disciplined refactoring. Number two, emotional awareness. Number three, hazard thinking. That’s it.

Richard: (29:59)
All right. Okay. Hold on. We talked about number two enough. Number one was disciplined refactoring and number three was hazard thinking. Can you explain those just a little bit more?

Arlo: (30:09)
Yes, so disciplined refactoring, most important thing. That is how do I make a known change in code structure that I can guarantee is bug for bug compatible? I want to guarantee that the behavior after this thing does not accidentally fix a bug that I don’t know exists. If you’re only thinking about how do I not introduce a bug that I know to look for, that’s a much lower safety bar and there are all sorts of ways to do it and supported by unit tests and unit testing and whatever can mostly hit that bar. It gives you a false sense of security though because it feels like, “Oh, I’m testing so I won’t have bugs.” No, you’ll only not have any bug that you can imagine. You’ll have all the bugs you can’t imagine.

Arlo: (30:52)
What’s even worse if we go back to that Legacy identification if I’m on a product of any real size, any real importance in the world yet, then the most important thing is the value that the product already has. That’s lumped into the behaviors as experienced by the customer, which I guarantee are different than the behaviors that’s experienced by anyone inside the company, which means that you have many bugs that are providing value to your customers that you don’t know about.

Richard: (31:20)
Unintended values.

Arlo: (31:21)
Unintended values. If you were to accidentally fix one of those bugs, it would be really, really problematic.

Richard: (31:30)
Hilarious. We don’t have bugs, we have unintended value.

Arlo: (31:33)
Very much so. Yeah. I was working on a team much, much later at Microsoft wherein I found something screwy in a spec, the O data teams or protocol for data exchange or whatever is something screwy. So I just made it right and simplified it and it was much easier and much more beautiful. Everyone should do it. Then I got that call a couple of weeks later from someone else who had been working in the database for years and he said, “If you change this part of the spec, SQL server will not be able to support Microsoft’s protocol for data exchange.” What? That is there to ensure that we keep having this bug that currently is driving about $2 billion of business because there are a couple of large old companies whose systems depend on that bug and they’ve lost source control. They have no way to have the system work unless we are binary compatible with that bug.

Arlo: (32:32)
It’s more valuable than any feature you could add. So that deep knowledge and experience of that makes you think when you’re refactoring, you’ve got to approach it differently because you cannot accidentally fix a bug you don’t know exists. You need to prove that. No test could ever help with that.

Richard: (32:54)
Multiple levels of naiveness to fix the bug that you don’t know exists.

Arlo: (33:00)
Yes, exactly. No one in the company even knows it exists.

Richard: (33:04)
Right and yet people are counting on that behavior.

Arlo: (33:07)
They really are. So yeah, when you finally really eyes open to that and to understanding that your job is that level of precision and quality, then you have to approach it differently and your factoring can’t be slipshod just edit some code. Even just using Fowler’s recipes that are supported by tests, that’s too risky for a lot of code. That’s great if you’ve got a new product that doesn’t have a lot of customers. It’s great if your product has only been out in the marketplace for a couple of years because you don’t have so much accrued value.

Richard: (33:42)
Right.

Arlo: (33:42)
You can’t use that on something that’s a 15 year old product. You need a higher level of safety. That’s the discipline of refactoring which is focused on using static analysis and language level levels of proof so that you can guarantee yes, according to the rules of this programming language, assuming there was no bug in the compiler, these two are identical. If I’ve gotten it down to the only time I could accidentally screw up a customer is if there’s a compiler bug. Okay. It’ll happen, but really rare.

Richard: (34:09)
Yeah. Yeah. Okay. So that was number one. Number three was hazard thinking.

Arlo: (34:13)
Hazard thinking. Yeah. So this is systems thinking. It’s also the systems thinking with a great degree of humanity thinking around it. So it starts with understanding bugs and where do bugs come from? So every bug is carefully custom, handcrafted by some well-meaning, highly trained, very intelligent, conscientious, careful software developer.

Richard: (34:40)
All right. That’s a beautiful way to say. Really.

Arlo: (34:43)
Yeah.

Richard: (34:44)
We’re all good intentions. Yeah.

Arlo: (34:46)
We’re all good intentions. We’re all very good at our jobs.

Richard: (34:48)
Yeah.

Arlo: (34:49)
We’re all bright, way above average on average. It’s actually true.

Richard: (34:56)
Well yeah. Actually, yeah.

Arlo: (34:58)
We are selected from the top half of the bell curve. So on average, we are above average. Yet we still write a bug and a half a day on average.

Richard: (35:07)
Yeah.

Arlo: (35:08)
Right. Depending on how you count, it could be as many as three, four a day and then those are just the ones that slipped past your initial check to get past your fingers are four and a half a day on average. Which is way too God damn many. So the question is where do they come from? The way I think of it is if we’ve got all these conscious developers, what’s really going on is they’re in extremely hazardous conditions in which they are presented with the opportunity a thousand times a day to custom handcraft a terrible, terrible bug. They’re so good that they’re able to identify 998 and a half of those before it happens and not do that.

Richard: (35:52)
Yep.

Arlo: (35:52)
They’re not good enough to get 100% because no one’s that good.

Richard: (35:56)
Yeah.

Arlo: (35:57)
They’ve got two and a half nines.

Richard: (36:00)
We have data to support that we humans just aren’t that good.

Arlo: (36:03)
Right, right. Honestly if you’re in a system in which you’ve got one thing which is operating at 99.95% effectiveness.

Richard: (36:15)
Yeah.

Arlo: (36:16)
Another thing which is operating at a thousand fuckups a day, where should I optimize this system?

Richard: (36:22)
Let’s think about that for a moment.

Arlo: (36:27)
Yeah, exactly. So that’s hazard thinking.

Richard: (36:30)
Okay.

Arlo: (36:31)
If you ever find a bug and you root cause and you find this person made a mistake and respond with, let’s be more careful, you’re screwing it up. What you want to be thinking about is how could we have been less careful and not have it be a problem? Let’s look for the hazards. Let’s look for the things that made it possible, made it likely that presented the opportunity to the developer to fuck it up. Then let’s do something about those. Safeguarding, the thing that I do combines that with an iterative approach so that on each bug you’re doing a little bit of pay down for each of those hazards. Then over time your bug stream becomes data. Not that you have to go back and analyze, but by responding, sampling it in real time, you effectively are putting your investments to pay down in real time whichever hazards contribute the most to the actual defects that you’re seeing.

Richard: (37:25)
Awesome. All right. Is there anything else that you’d like to share with listeners?

Arlo: (37:30)
No. No. You do all three of those things and you’re awesome.

Richard: (37:32)
All right. If they want to do all three of those things and they want to know how, how could they contact you?

Arlo: (37:40)
Okay, so my company Deep Roots does this. Go to digdeeproots.com or email me Arlo@digdeeproots, or see us on Twitter. We do a few things at the single team scale, but we actually do most of our stuff at the much larger enterprise scale. The company consists of two of us. I am, well me and I know all the technical what needs to happen and how do we get effectiveness. My business partner is an instructional designer. She knows the how. How do you get 2000 people who are a systematic culture this way to simultaneously shift to a different culture without needing to sit down and mentor next to every single one in pair [inaudible 00:38:22] because the pairing will work, it’s just too expensive. So we have built solutions for that. We know precisely what changes need to happen that comes from me, and we know precisely how to make those effectively happen without a lot of resistance, without a lot of cost.

Richard: (38:37)
All right. All right. Awesome. Arlo, thanks so much for joining us today. I was telling somebody this morning. There are two people in the world who I think are the best technical minds. There’s Lou-Ellen and there’s Arlo and I get to talk to Arlo this afternoon. Isn’t that cool?

Arlo: (38:51)
Thank you. Thank you.

Richard: (38:54)
So thanks for joining me. This was awesome fun. Thank you so much.

Arlo: (38:56)
Yeah, this was a lot of fun. I enjoyed it.

Richard: (39:02)
Hi, friends. Thanks again for listening and remember to support this podcast, visit my website kasperowski.com