Charlie Martin, a former startup CTO, senior architect for SUN, and now a consultant for the Flint Hills Group, is a prolific author of articles, books and peer-reviewed journal articles in security and development. He’s been programming for money for almost 50 years starting with punch cards and is currently working with serverless architectures in Python and JavaScript.

 

Thanks for taking the time to go on record with me. In conversation, you mentioned the subject of real Agile and that Agile is a thing of the spirit. Talk to me about what you mean by those things and where you're going.

Sure. Going back as far as 2008, I had an article in CIO that talks about Cargo Cult Agile, which is when you adopt the buzzwords of Agile without actually being Agile. The experience I was talking about was a project at SUN where they wanted to be Agile and have user stories and iterations and such. But they wouldn't go forward until we had a Microsoft Project schedule down to the half-day for the two-year duration of the project and they decided iterations were hard, so we would only iterate every quarter.

 

That's traditional waterfall.

Yes. What it comes down to is they added daily stand up meetings and called the requirements document use cases, but it was a waterfall development. I've seen a lot of companies where I've worked or consulted do that. The core of Agile is not having stand up meetings, which often turn into sit down meetings anyway, but in the recognition that short iterations help you converge on the best match with your customers' needs. Doing that convergence requires you to understand that requirements are volatile, mutable, and to reconsider the requirements at every iteration.

What makes something have the Agile spirit is, maybe somewhat redundantly, is being able to actually be Agile — in other words, being able to change your response to conditions quickly and without fuss.

The other thing that I often see in supposedly Agile projects is people get fascinated with tools. Programmers love to make things complicated. Programmers love tools, and so you'd go from the original ideas of Extreme Programming where your requirements were on index cards, to having gigantic Jira or Rally databases, where you have stories, you have epics, you have complicated Kanban boards, you have fancy tracking back and forth.

Prime example, I just did a project where we were doing eight hours a week of meetings being “Agile.” It was a project with about 50 people and eight hours a week times 50 people is ten staff weeks — 400 staff hours — a week.

 

That's crazy.

Oh, it is. That's my underlying thought with this idea that Agile is a thing of the spirit:  you have to be vigilant against the impulse to do traditional management and traditional waterfall and having everything under control all the time. Agile is like walking on ice. You're not under control all the time. What you have to do is you have to be willing to dance a little to stay upright.

 

I keep a poem up from a Jesuit priest, the basis of which is that you need to be at peace with being in a state of incompleteness.

Yes, exactly. You have to be at peace with being in a state of incompleteness. You have to be comfortable with knowing that you don't know everything, and that change is going to happen whether you like it or not.

 

So it seems to me too that Agile works by walking a fine line between too much process and not enough.

I think that's true. But the common programmer and manager temptation to add process usually lean way too far to the “too much process” side.

I went to graduate school originally intending to be a process weenie. I wanted to learn about software engineering processes. I participated in a research project where we tried to build the user's guide or catalog of all the software methodologies there were. I never found anybody at the end of a project who complained that they hadn't had enough process.

It's fairly easy to define what are the necessary and sufficient conditions for Agile. Necessary is a collection of testable achievements done at the end of the iteration: Sufficient is the number of conditions satisfied at the end of an iteration.

Anything that gets much fancier than that and you're risking spending lots of time processing.

 

Does it get harder for a large organization to scale and be Agile?

Yes. There are a lot of people trying to handle that. There's a new thing I see advertised all the time about scrum at scale.

The thing about this is that the brain can only handle so much complexity. There's the old seven-plus-or-minus-two rule. When you try to scale Agile, you run into the problem of coping with large amounts of information. I'm not saying it's easy, but the more you try to move decision making up the tree, even in an Agile project, the less flexibility you have, the less agility you have. I've done a bunch of stuff over the years on what I call big organizations that act like idiots.

I've worked for IBM, I've worked with the CIA. I've seen a lot of big organizations act like idiots. The people closest to the issue knows the most about it. L. David Marquet wrote a book called, Turn The Ship Around. He was a navy captain put in charge of a submarine where the whole crew was doing very badly in terms of how their evaluations went. He turned them around by basically devolving as much decision-making as possible to the lowest levels. He did that by instituting a policy that people didn't ask him what to do. They told him what they intended to do and unless he face-palmed, they expected that they'd go on and do it. The effect was that it dramatically turned around the performance of the crew from bottom of the rankings to the top in something like 18 months.

The same thing applies to Agile. The second part of big organizations that act like idiots is that everyone has a natural tendency when they report information upward to shade, or slant, it. Now the result of that is if you think about it in a multilevel hierarchy, is that the person at the top who's supposed to be making decisions is least informed about the actual products and processes, and is operating on less than complete information.

When you scale Agile, you're much better off if you scale it by creating more teams with well-defined, clearly defined goals and letting those teams satisfy those goals.

The other side of that, of course, is if you've got a big team, you're spending lots of money, and people want to know it's under control. You have to honestly accept that there are things that aren't in your control, and you need to measure progress towards goals. This is why I'm such a fanatic for making all user stories into testables and measuring progress by a Boolean: does it pass its test or not? None of this 50% done crap.

 

Not measuring things that don't matter.

It's identifying the things that do matter. That's first. The second thing is not doing things that don't matter. I've been thinking a lot about objectives and key results — OKRs — recently, and there is a really good insight there, which is essentially that you establish your objectives, you establish what measurable things show that you're making progress towards the objective, and then you measure them.

 

I had somebody tell me that an assigned architect role is utterly incompatible with the Agile principle.

That's nuts, he said bluntly. It is nuts because — and I've had this argument with Kent Beck, the creator of Extreme Programming, on multiple occasions.

Architecture is driven by the largest and usually nonfunctional requirements; security, reliability, scalability, and so forth. Decide early whether your system will satisfy those requirements.

What the Extreme Programming, and other people who insist that having an architect is useless, never seem to recognize is that the projects they're looking at are projects where the architectural decisions were already made. If you know from the start that you're building a Ruby on Rails project, there aren't many architectural decisions that are up in the air, and at that point, maybe having an architect isn't as useful. But if you do have to make those decisions and you make them wrong, this idea of convergent architecture is a guarantee that if you make those decisions, you have wrong, you will have a lot of sunk cost and rework to try to rethink the decisions.

The real bane of any project is an architect whose technical skills are concentrated in making PowerPoint slides. I've been in projects where the architect came up with an architecture that looked good on PowerPoint slides but wasn't going to work in implementation.

While I do think that having an architect role is important, the architect must spend at least part of their time hands-on building code. Usually, ideally building code that becomes a paradigm, a pattern, for the other implementers.

 

Do you think a large organization can become Agile without changing anything?

No. Trying to become Agile without changing anything is exactly what leads to a Cargo Cult. For instance, the SUN project I mentioned earlier, executives thought the idea of being Agile was great, but the project manager expected that all projects have a Microsoft Project Gantt chart out to the day or more. Trying to do that strangles any chance you have of being Agile.

You can't become Agile by presenting an Agile facade. Change the way you think and what your expectations are to make it work.

 

Do you find companies trying to present a facade of being Agile these days?

I see that a lot more than I see people making the change. It's like an AA first step. The company has to realize that their current situation has become unmanageable, that what you're doing isn't working, and be willing to accept that they have to do something new.

 

And that you've become powerless.

You've become powerless over what you're trying to do. That's right. Now, as a consultant, I usually follow that with the second step, which is turn yourself over to a higher power, namely the lead consultants who you're paying a higher hourly fee.

 

You have ten more steps.

It's a pretty good pattern to make a sincere effort to see the problems. You don't have to go to people and make amends for having been a manager, but you do have to resolve to make it better and recognize making it better means making it different.

 

So what does a truly Agile approach to large scale development look like? And how would a company begin after they've taken that first step? What would be the initial steps that they would take once they've acknowledged problems they can't manage?

The most important first thing you have to do is make it clear that you understand that you can't define the exact requirements, the exact cost, and the exact delivery date all at once because those things are vulnerable to changes in requirements, to the unexpected.

The one project I ever worked on that was 100% waterfall, while it worked great, had this special characteristic that the contract, for various reasons, was fixed. What was to be delivered was fixed. The acceptance test was part of the contract. That allowed us to work towards the acceptance test successfully, and we delivered early and under budget. But the other side of that was that this was a system that I then went overseas with, and I spent the next year and a half implementing all the changes that we didn't allow them to put in because we fixed the requirements in the contract.

One way to think about it is to think about every iteration as being a minimum viable product. Every time you iterate, you are producing something that approximates, however badly, what you want to deliver. And then you accept that every time you deliver an iteration, there are going to be changes that affect the next iteration.

What that means is that you pretty well guarantee that whatever your picture of the system is at the beginning will not be the system that you deliver. The other part is that no system is ever finished, or at least it's not finished until you archive it and delete the files from the disc!

The 25 words or less answer is that you have to realize that you're going to deliver an approximation and accept that you're going to continue building approximations.

The scale issue then comes in. How do you organize the large scale project with that in mind? I could argue, possibly successfully, that there's no such thing as a large scale project. It's just a bunch of small scale projects herded in the same direction.

How would you suggest that IT leaders get on board and support the move toward Agile? What are two things that IT leaders could do to make the biggest difference inside of a larger organization?

No, It's the question of being an interface. On the one side, they have to listen down and recognize the people at lower levels are better informed, and they have to have a picture of the big picture, but they have to realize that they're not painting the whole picture. So they have to listen to what people are saying to them.

The other side of it is that they have to be willing to stand up to their management and say, "We're doing Agile. You said we should do Agile. Agile does not mean micromanagement, and my people are doing what they can do, doing their best, and it's my job to protect them so they can get the work done."

They have to have buy-in on that first step above them. If you say, "I'm going to be Agile on the hill with you," and your boss thinks that GM in the 1950s was the perfect management style, you're not going to be successful.

Is there something that we haven't talked about that you think is important to include here?

The key is to embrace uncertainty and to make sure that you convey the fact that a large scale development, in particular, is an uncertain process to the customers. Nobody likes to hear that, but the fact is that you have to embrace uncertainty because if you don't, uncertainty will embrace you. The uncertainty will be there whether you like it or not; do you accept it and dance with it, or do you try to fight with it?

 

»Read next, on Emerge: