An Introduction to Agile Emergent Architecture – Always Intentional

An Introduction to Agile Emergent Architecture: Always Intentional

Let’s Define Architecture
By definition, architecture is about the major pieces of the system^1, and about satisfying the non functional requirements such as availability, usability, compliance, scalability, security, extensibility , maintainability, and all of the typical “ilities.” It’s important to remember that architecture does not cover anything and everything that can be put into one of the above categories — because they would encompass every last bit of the *entire* system. The architecture should not encompass every last corner of the system. Instead, architecture focuses on the very highest level concerns in each of those categories, and nothing more than that. Further, architecture focuses only on the most basic of building blocks in those categories, and nothing more than that. This leaves as much flexibility as possible for later agility and implementation. So, be sure your architecture is focusing only on the highest of concerns. (The lower concerns might be considered such things as design, code, or implementation)

Big Up Front(BUF) Thinking
Legacy software development approaches put a large emphasis on understanding all the requirements for a system up front. This is also known as BRUF (Big Requirements Up Front). The disproven theory behind this thought is that if one understood all the requirements really well up front, then the most optimum architectures, designs, and software code could be created. As such, BRUF has siblings known as BAUF (Big Architecture Up Front) and BDUF (Big Design Up Front). All of these Big Up Front (BUF) approach theories have been disproven for one overall reason: it takes time to gain this BUF understanding, and by the time you do think you and your team have that understanding, the requirements have changed substantially, which then affects that architecture and design as well. What we have learned over the decades is that the requirements, technologies, users, technical team members, and the market for the product change quicker than we can gain that understanding, and certainly quicker than we can implement an architecture or design. In addition, customers often have trouble describing exactly what they want — they tend to have a much better understanding of what they want only after they see the working software. Further, with BUF thinking, usually the people that did the up front thinking are way different than the people implementing that thinking. So downstream from requirements, when changes to the architecture and design are truly needed, because of all of the previous history, BUF thinking, and totally different people involved, changes are extraordinarily costly *and* complex to retrofit. As such, Big Up Front thinking is a model that has been disproven, so it’s unacceptable, and indeed now.. passe.

No Up Front(NUF) Thinking
However, the other end of the spectrum, NUF (No Up Front Thinking), is also unacceptable. You can’t create a cohesive architecture, that is financially viable, without some good upfront thinking. Without at least some good upfront thinking, the architecture turns out haphazard, almost accidental in nature. Some people have called this unintentional architecture, and the name probably fits.

Signs you MIght be a Victim of NUF
One possible sign of NUF is a system rewrite. A system rewrite is almost always a sign of failure ^2, and the two biggest reasons for that failure are inattention to the user marketplace and inattention to continuous technical excellence via architecture, design, testing and coding practices. Inattention to continuous technical excellence creates what is known as technical debt, which is the other tell tale sign of NUF. Massive technical debt generally presents itself as a highly unacceptable amount of bugs, new functionality that takes way longer than it should, or systems that get jettisoned or re-written. Massive technical debt is pretty much a guaranteed outcome of No Up Front Thinking. As such, since massive technical debt has so many bad outcomes, No Up Front Thinking is also unacceptable.

Big Up Front thinking can lead to No Up Front Thinking
Now, before we move on, let me also address another relevant point. BUF thinking can actually lead to NUF. If a BUF architecture is not kept up to date, is not shepherded, or is extremely inflexible to change, people will avoid thinking about and making architectural changes. This results in NUF and massive technical debt. So, a single system can be a victim of both BUF and NUF. It would be really great if there was a way to find that right balance between BUF and NUF… right?

The Better Way: Agile Emergent Architecture
Enter “Emergent Architecture”, a term suggested by Agile thought leaders. One can think of this as Little Up Front(LUF) Architecture, combined with continuous attention to technical excellence. You could even think of it as “Continuous Architecture” if you so desired. With Emergent Architecture, you do just enough, just in time, at the “last responsible moment.” It’s also important to note that Emergent Architecture is also 100% intentional architecture. Architecture doesn’t just “magically appear”.

The Benefits of Emergent Architecture
By architecting at the last responsible moment, you are minimizing the requirement churn damage that accompanies BUF. By architecting at the last responsible moment, you are taking advantage of the latest and greatest technology knowledge. Finally, by architecting at the last responsible moment, you are very sure that the people collaborating on that architecture are the people that are about to implement that architecture. All of the sudden, the ingredients for success are all in the right place at the right time!

Technical Excellence
Technical Excellence can refer to architecture, design, testing, coding, and probably other practices too. Regardless of whether you do BUF, NUF, or Emergent Architecture, the ability to quickly and cheaply extend or change your software architecture at any given moment is directly proportional to your practices around technical excellence. The higher your technical excellence, the more quickly and cheaply you can change direction (<– indeed, this is the definition of agility). Having said that, in BUF approaches, since people make the mistake of thinking that they can “lock down all the requirements and architecture Up Front” , they rarely put in the technical excellence needed for rapid change. Examples of technical excellence are paired programming, collective code ownership, continuous integration, continuous automated testing, build automation and build pipelines, Test Driven Development, Unit Testing, lightweight code reviews, YAGNI, and many other practices. Agile approaches harness change for the benefit of the customer, so don’t forget Agile Manifesto Principle #9: “Continuous attention to technical excellence and good design enhances agility.”

Architectural Runway and the “Last Responsible Moment”
Now, let’s be careful with the “last responsible moment”. Note the word “responsible” in “last responsible moment.” For different architectural pieces, that last responsible moment will be at different times. Some architectural pieces will require a longer runway, and other pieces can do well on a shorter runway. In summary the more complexity and learning in the requirements, people, and technology, the longer the runway needed. If the complexity factors are low, then less runway is needed. Figure out that needed runway length (length of time before that architectural piece needs to be in place), and work your way back to when that runway needs to begin being built.

Let’s look at some examples.

Architectural Runway for a Deployment Platform
For example, determining the deployment platform(the initial UI, logic, and other tiers) for the Minimum Viable Product for a new product probably needs to happen before the first Sprint of developing that product. Having said that, let’s not regress to BAUF, but let’s do execute some upfront thinking and have that deployment platform pretty well figured out before that first Sprint. Since every Sprint has to include at least some small amount of user/business valuable functionality, it’s going to be hard to create some releasable software functionality *and* create your initial deployment platform all in Sprint 1. It is theoretically possible, but not likely.

Architectural Runway for an Open Source Logging Framework (Complexity is low, should be a fairly quick decision)
Another example of “last responsible moment” might be choosing an open source framework to do logging in your system. If logging is forecasted to be in Sprint 23, there is no reason to choose that framework in Sprint 3. You can probably wait until a few sprints before to begin working on that decision.

Architectural Runway for a 3rd Party Processing Component/Framework (Complexity is high, will be a long duration before a decision can be made)
Let’s assume the same as the above — we plan to begin using this component in Sprint 23, and we are currently in Sprint 3. We need to choose a large 3rd party proprietary component that does a large amount of processing or functionality(think accounting, medical, legal, or aerospace). Since it comes from a 3rd party, there is likely going to be a long runway needed to have that choice in place before functionality can be built on top of it. In this example, it might be perfectly fine to start the architectural discussions about which 3rd party framework to use and purchase in Sprint 3. Remember that you’re going to have to leave time for product evaluation, purchasing, legal, training/learning, and maybe even some technical investigation and technical spikes. It could actually take 20 sprints to execute that particular architectural runway and have the architecture first able to support business functionality.

Architectural Runway for logging standards across multiple teams(Complexity is medium, requires time for buy-in from multiple teams)
Remember that logging framework implemented in Sprint 23? Well, we’re now in Sprint 27, and now that the logging framework has been implemented on a couple of teams, the teams realize a need for standardization of use because the framework is beginning to be used widely across the product. At first people didn’t think this was a “high enough concern” to be considered architecture, but then then they realized that the teams were using widely different “logging levels”, that debug logging often crossed module/system boundaries, and that debugging across those modules was getting difficult. The teams finally realized that there would be some major value in ensuring some light standards were followed across the teams. Three might even be a nice wrapper around the chosen logging framework to make it easier for teams to use in a more consistent way. So, in Sprint 27, they began working on an architecture effort to solve those needs. By Sprint 31, the first teams were using the new wrapper and light standards. One thing to note here is that, back in Sprint 20, when they were originally choosing a logging framework, they could have chose to do all of that standardization up front — but that would have been BUF thinking, and they could not have predicted the ways the framework would be used. It’s very possible that no other team would ever pick it up and use it — as happened with most of their open source frameworks. In other words, they would have been heavily speculating needs rather than having a practical, experienced understanding of the real needs. Now, with the experience of a couple of different teams actually using it, in production, they can make better decisions about how to lightly standardize. In this way, they chose to do Emergent Architecture over BUF thinking.

Architectural Runway for Performance and Scalability(Complexity is highly variable here, depends on the product and the target stakeholders for the release)
Hopefully, if you have a new product, you have a Minimum Viable Product(MVP) release that doesn’t require supporting a zillion users concurrently. However, every product is different, so act accordingly. An MVP will certainly need to perform and scale in a way that meets the needs of the initial users. Get this wrong, and you will pay a very heavy price in terms of customer satisfaction and product adoption. As such, for an initial release, this topic is extremely important. The non functional requirements around performance and scalability will often be ordered high in your backlog, as they are both high value and high risk for an initial release. Having said that, sometimes premature optimization is a source of waste. The runway here will also depend on the tools and environments you need to do proper load testing. Choosing and setting those up can often take serious time, so that would extend the runway needed. So, the runway answer here is probably — “it depends.”

Architectural Runway for Government Regulatory Compliance(Complexity is highly variable here, depends on the weight/complexity of compliance, the agility of the agency, etc.)
The guidance for this piece of the architecture is almost identical to the factors involved in dealing with scalability and performance requirements. They are “must haves” that will often be ordered very high in the product backlog due to their risk and value. Note here that we are mainly discussing compliance needed by the product, not the process. (On a related subject, sometimes software process compliance ends up on your Definition of Done, which may or may not affect the product architecture or product backlog directly.)

The above runway discussions are just some examples to think about. The architectural runway for each of these needed pieces of the architecture will be different. As such, the “last responsible moment” to build that runway will require some serious thought as you refine your product backlog. Further, in order to deal with the unknown unknowns, you’d better put a time buffer into that runway since you can’t possible know the unknowable!

In Summary
Hopefully the above has given you a picture of what Emergent Architecture looks like. With each vital part of the architecture, you will need to consider the runway that gives you the “last responsible moment” and some buffer for the unknowns. Because the requirements will be emerging over time, so too will each of the architectural pieces. The architecture will emerge, or evolve, over time, but with intentional forethought to the necessary runway. Remember, due to the number and churn of moving parts (requirements, people, technology, etc) inherent in almost all software development, the *only* way to keep an optimum architecture over time is to give continuous attention to technical excellence. Finally, in order to avoid the gigantic trappings of BUF and NUF, Emergent Architecture is really the only sane choice.

(If you have several teams on the same product or in the same org who need to keep the Architecture of systems excellent, see:  An Agile Architecture Community of Practice


Notes:

^1 The focus of this article is Agile Emergent Architecture, so we give only brief attention to a definition of architecture here. It’s worth noting that many thought leaders have multiple credible definitions of architecture. Further, Martin Fowler and Molly Dishman have mentioned that some think the term “architect” is not even the correct metaphor — that maybe “city planner” is the better metaphor.

^2 It’s worth noting that there are some very rare scenarios where a system re-write might not be a sign of failure. For instance, if the system re-write is an early pivot in a product’s lifecycle, it might make great sense and not be due to the above mentioned causes. Having said that, I’ve never personally seen a re-write that was due to a good reason.

A Response to Mike Cohns Comments on 64% of Software Features Rarely or Never Used

I have a saying.  “Scrum Trainers usually agree on 99% of Scrum, but they spend a lot of time debating the other 1%.”

Let me say this first.  I’m a huge fan of Mike Cohn.  I teach Scrum and Agile classes all over the country at Fortune 50 companies, and it is very rare for a class to go by that I won’t mention at least one of his awesome books on Scrum.  I also recommend him on my list of favorite Agile resources on one of our web sites.  In addition to all of this, I’ve had numerous personal interactions with Mike one on one, and he’s always been extremely nice to me, traded professional practice opinions/advice, and he even offered to let me attend one of his classes at a “trainer courtesy” discount one time.  Great guy!  In summary, I like the guy a lot personally, and I highly respect him professionally.  He’s done a ton for the software and Agile industry, and no one should forget that.

So, with that said, let’s get back to that 1% debate.  🙂

In his recent blog post, Mike reveals some little known details about the oft cited 64% of features that are rarely or never used in software systems.  His information is factual and likely true.  I’m ok with all of that.

What I don’t understand is, why bother broadcasting this?

This is one of the most credible studies available on the subject.  If you think hard about this data for a minute, you’ll realize why it is incredibly difficult to obtain… No company wants to admit that there is a TON of bloat in their software!  But, what percentage of Microsoft Excel/PowerPoint/Word features do you use and benefit from?  What percentage of Rally features do you actually use and benefit from?  Bloat bloat bloat, negative value, negative value, negative value.   In my recent articles on the New New Product Owner, I’ve talked about the need for the New New Product Owner to be a marketplace expert, so that they can maximize the value and profits from software development for their company.

Now, the value equation is way more complicated than “rarely or never used”, but still, I think we all know that there is a TON of negative ROI functionality in any non trivially sized application, and there is a TON of software teams with far too little focus on value and profits.  Anyone who has worked on the front lines of software development knows that.  The oft cited study just helps confirm some of our suspicions.  One of our Agile Metrics consulting services at AgileSoftwareTraining.com is helping to give company leaders even more transparency into how to extract more profits and cost savings out of all of their software development efforts, whether they be internal or external systems.  Give us a call if you’re interested.

What makes that limited study useful as a teaching tool is it gets people to think about value, and think about low value, low ROI features, and realize that value delivery is important, far too important to ignore.

Newer Study confirms the same idea!

Further, Standish again re-iterated this point in 2014, ostensibly from wider data: https://www.standishgroup.com/sample_research_files/Exceeding%20Value_Layout.pdf — “Standish Group research shows that 80% of features and functions have low to no value.”

There are other “studies” cited in our industry that are totally bogus, software leprechauns if you will, and I’m totally against relying on those.  Things like the “Cone of Uncertainty” and the so-called “Weinberg study” on task switching have shown to be totally made up.  However, the Standish Group study is real, with real data, and it is highly credible, even if somewhat limited in its scope.

So, Mike wants us to stop citing the study, or for us to caveat it with “in the weeds” details.  Of course, that will just confuse those new to Scrum and the teaching value would be lost.  And people would focus less on software value and profits.  I don’t think that’s good.  I’m totally open to hearing about a more credible public study, but I’m unaware of one. 

With all due respect to a friendly colleague, and one of the best Scrum trainers on the planet, I think ignoring or caveating the 64% study is bad for the industry.  Let’s just put this in the 1% bucket that we as Scrum trainers will agree to disagree on.  🙂

If you’d like to disagree with my contrarian view, feel free to sound off in the comments below!

The New New Product Owner: The Product Value Maximizer in Scrum.

Preface:  In the last 4 years, the Scrum Guide has had two very significant updates, including updates to the Product Owner role that have far reaching implications. In this article and the series that follows, I attempt to describe “The New New Product Owner” role in Scrum.

In a series of upcoming articles, I will detail the different focus areas that the modern Product Owner needs to concentrate on in order to fulfill their duties on a Scrum team.
POFocusAreas_NewNewPOThe New New Product Owner understands that she will likely need to execute these activities at different times, and that she might need to delegate to others in order to effectively produce software that maximizes ROI for the software development effort.

The first and most important focus area is for the Product Owner to be the “Product Value Maximizer.”  There are lots of way to do this, but 3 key steps are involved:

  1. Order the Product Backlog by (estimated) value.
  2. Work with the Development team to get small increments of value to market quickly, for feedback.  Release to market = in production, available to end users.
  3. Rapidly assess value delivery feedback from the market, and then start over again at step #1.

These three steps will ensure that the New New Product Owner delivers “the right thing.”

Note that your market might be internal (IT software) or external (Saas, Consumer software).  Either way, it’s important to get features into production quickly, and to assess whether we have “hit the target” with respect to value… or not.  This quick release to production, every few weeks, is a key aspect of Agile software development.

If you haven’t already signed up for our blog, be sure and sign up (upper left hand corner) so you will get the future articles on this topic!

In future articles, I will detail the remaining six focus areas for the New New Product Owner:

For a high quality class that focuses exclusively on the Product Owner role(send your business stakeholders too!), see our Professional Scrum Product Owner class and contact us if you’re interested in one.  We teach all over the USA.

User Stories – Focusing on Conversations instead of Writing – Gojko Adzic’s New User Story Book

In my recent article on telling user stories instead of writing user stories, I mentioned that many Scrum Teams focus way too much on documentation and way too little on good collaborations.

More support for this concept comes from the first chapter in Gojko Adzic’s new User Story book, Fifty Quick Ideas to Improve your User Stories.

User stories imply a completely different model: requirements by collaboration. Hand-overs are replaced by frequent involvement and discussions…. If requirements are just written down and handed over, this discussion does not happen. Even when such documents are called stories, by the time a team receives them, all the important decisions have already been made…. Try telling stories instead of writing down details. Use physical story cards, electronic ticketing systems and backlog management tools just as reminders for conversations…Engage business stakeholders and delivery team members in a discussion, look at a story from different perspectives and explore options. That’s the way to unlock the real benefits of working with user stories.

Gojko has been nice enough to publish the “Tell stories, don’t write them” chapter available completely free here!  It is also important to note, that this chapter is tip #1 in his book, as it really sets the stage for the best use of the User Story practice.

The User Story practice was always intended as a very close, verbal collaboration between the Dev Team and the PO/Customer. In modern times, you can achieve this very easily with good Product Backlog Refinement practices.

Anyway, it’s totally worth another five minutes of your time to read Gojko’s free chapter, and be sure to share it with your teams and organizations too!

To maximize your Scrum and User Stories practice, bring us into your company to deliver coaching or our User Stories Class.

Focus on Telling User Stories, NOT Writing User Stories

Ebin Poovathany has written a wonderful article on how we should focus more on the verbal conversation aspects of User Stories rather than focusing too much attention on “writing” User Stories. I myself have written an article about this as well (See Trap #’s 1, 8, 10,and 13). It’s great to see that this topic is starting to get more attention in the industry.

As Ebin points out, using so called “User Story Templates” (“As a user, I want..”, “In order to…I want…”, etc) causes people to backslide into older waterfall habits, and creating the same old kinds of documents that we used to create in waterfall, along with the same old problems. He said this is sad, and as a User Story proponent, I agree. It’s a horrible misunderstanding, but it’s rampant in our industry. The User Story practice was always intended as a very close, verbal collaboration between the Dev Team and the PO/Customer. In modern times, you can achieve this very easily with good Product Backlog Refinement practices.

Anyway, it’s totally worth your five minutes to read Ebin’s article, and be sure to share it with your teams and organizations too!

To learn how to avoid User Story Traps and maximize your User Stories practice, see more info about our User Stories Class.

New and Improved User Story Lifeycle Diagram — Free Creative Commons PDF download!

I had a designer friend update my User Story Lifecycle diagram, and she did a fantastic job!  You can download the PDF here:  http://www.scrumcrazy.com/lifecycle

New and Improved Diagram:

UserStoryLifeCycle_final_lg

The Older Diagram(also still available at the above link):

UserStoryLifecyclexm

Other Good User Story Links

Bradley Bug Chart Referenced in Awesome New Scrum Book

The Bradley Bug Chart was referenced in an awesome new Scrum Book that is now on the market.

Background

I got an email from this guy named Rich one day who wanted to talk by phone and ask me some questions about the Bradley Bug Chart. A great collaboration ensued, and I incorporated some of his suggestions into my chart. A couple of weeks later, he asked me to review Chapter 1 of this book he was writing. I have some of the highest standards around, and I’m not easily impressed, but that first chapter was extraordinary! Through that one chapter, and the conversations with Rich, I realized this guy knew Scrum inside and out. I then reviewed and commented on a few more chapters. I get asked to do unpaid reviews often, but I only spend my time on things that I think are excellent for the Agile community. I’ve spent some time reviewing another book that hasn’t been published yet, and it too, is a great one for the industry in my opinion. More on that one when it gets published.

The Bradley Bug Chart Gets Called Out

A little while later, Rich asked me if he could refer to my Bradley Bug Chart in his book(see page 277), in a section on how to handle bugs in Scrum. I was ecstatic, in part just having my work get recognition, but it was also recognition from someone I highly respected myself. It was great. I also re-wrote some of the praise that I gave Rich on that Chapter 1 review as a quote for his book. Rich also ended up quoting me a few times in other parts of the book, particularly in the chapter on “Continuous Improvement.” I’m not trying to brag here, just sharing the satisfaction of seeing some of my hard work and learning as being objectively recognized in the community.

About the Book

Here is the quote I wrote for his book:

  • “Finally a book about Scrum from the Development Team’s point of view. Richard’s description of the best and worst ways to implement Scrum is priceless. The first chapter alone is one of the best descriptions of ‘Scrum done well’ that I’ve ever seen.” — Charles Bradley

I highly recommend his book for anyone working with Scrum and Microsoft Visual Studio/.Net technologies.

Here is the book:

PSDBook_Small

Small Caveat: Rich is and was a trainer with Scrum.org, and since our initial collaboration, Rich inspired me to become a trainer for Scrum.org as well. So, while none of this was true at the time of the above events, I wanted to mention that we are now both trainers for Scrum.org. I have no financial interests in his book, whatsoever, and my journey to become a Scrum.org trainer came after my review of his book. Thanks Rich!

%d bloggers like this: