An Agile Architecture Community of Practice

Note:  The format of this article did not transfer over well from the previous site, so this pdf version might be much easier to read and distribute.

Background

We often coach organizations on scaling Scrum, where 4-12 Scrum Teams are working on the same Product or a set of closely related Products (Applications, Systems, etc). We often get asked how things like Architecture and other multi-team related concerns are handled in a scaled Scrum approach. In Agile practice, handling these multi-team concerns is usually handled via a mechanism called a “Community of Practice”. Below is an example of a moderately mature community, based on a compilation of ideas that we have seen work well in the field. Note also that a TeamSet should always have more than one “Community of Practice”,and new ones should be formed and dissolved as needed.

For more info on Communities of Practice, see An Agile Community of Practice Starter Kit

Caveats about the example below:

1. We use the term “TeamSet” below to refer to a set of teams, the 4-12 mentioned above, working on a Product or set of closely related Products. Obviously the more high impact the architectual initiatives, the more formality of process you will likely need. So, if your set of closely related Products is not that closely related, then this community should be less formal and less broadly applicable in its decisions. The converse of this is also likely true.
2. The below example is that this shows more of an “ideal end state” — your org will likely have to take a few steps of organizational change before you can get here. But try to get as far as you can on step 1 — you might surprise yourself.
3. We have probably added more formality and more documentation below than would be typical in a real life, highly Agile, community of practice. We do so here primarily for illustrative learning purposes, to give you more ideas than are truly needed (i.e. so you can pick and choose what work in your context). In real life, the community would likely be less formal.
4. This example is for Architecture, but this same kind of approach easily fits other types of CoP’s: Agile CoP, Scrum CoP, Scrum Master CoP, Product Owner CoP, DevOps CoP, Programming CoP, Automated Testing CoP, UX CoP, SAFe CoP, RTE CoP, Nexus CoP, LeSS CoP, etc. [ONE MORE REMINDER: Your CoP should have the minimum amount of formality necessary, and to the extent possible, should operate bottom up.]

---

The Aegis Architecture Community of Practice

Our Charter Statement: We all work on a set of closely related products, collectively known as Aegis. This community organized around ensuring that the highest Architectural concerns that have a high impact on Aegis are efficiently and effectively addressed. Please note that this community focuses only on the highest Architectural concerns.

In Scope: The set of teams (TeamSet) that this community encompasses are: Stingray, 49’ers, Falcon, Journey, Explorers, Red October, Phoenix, and Hawking. The main “in scope” topic is high Architectural concerns, though the dividing line between “high” and “not high” is not always black and white. As you look below, hopefully the mechanisms we have in place will give you the idea of where that line is usually drawn.

Out of Scope: Practices related to Programming, Design and other concerns are generally left to others (Other communities and/or the Scrum teams themselves to self organize and solve). Test and Build automation is left to others. The Director of Software Dev hires and chooses the LA (Lead Architect), so that is out of scope for our community. Our Arch CoP only covers the teams in the Aegis TeamSet, so for Arch issues that are decided at the corporate level, you will need to talk to the EAG (Enterprise Architecture Group, not a CoP yet. 😦 ). Our Lead Architect has good communication with the EAG, but anyone should feel free to go the EAG for the appropriate services(just loop the Lead Architect in as well). Typically we form a working group from this CoP to go and talk to the EAG for any requests.

---

(In the sections below, note that we have specifically named them “Individuals”, “Interactions”, “Processes”, “Tools” to relate to the Agile Manifesto, valuing “Individuals and Interactions over Processes and Tools”.)

Individuals

Community Leadership			Home Scrum Team
Community Coordinators (CC)	Jill Hutchins, Henry Nguyen		Stingray, Falcon
Agile Coach	Jeff Schwaber (Liaison to Scrum Master CoP)		Falcon
Management Contact	Ellie Swanson		n/a, Director of Software Dev
Lead Architect(LA) 90% allocated to community 10% allocated to Scrum Team	Ian Robison		Journey
Community Architects(CA) 50% allocated to community 50% allocated to Scrum Team
	Phillip Smith		Explorers
	Brijesh Singh		Hawking
	Jill Hutchins		Stringray
	Henry Nguyen (Liaison to Tech Excellence COP)		Falcon
Community Members
Architecture Team Reps(ATR) 10% allocated to community 90% allocated to Scrum Team
	Sheila Hill		Stingray
	Carlos Diego		49’ers
	Rachel Story		Falcon
	Chris Bradford		Journey
	(open) (interim- Phillip Smith)		Explorers
	Adam Gideon		Red October
	Adrienne Maxhouse		Phoenix
	Sandesh Mokkarala		Hawking
General Member (GM)
	All people above are also considered GM’s
	Anyone interested!
	Melissa Hayden (Liaison to Testing CoP)

Role Definitions
Community Coordinators	This role is to be a servant leader to guide the community on what initiatives and other efforts to focus on. This role also helps provide leadership on which decisions are truly of high enough concern to warrant formality and process via the community. The only requirements for this role are that the person has the appropriate architecture knowledge and must have 3 months of experience in the role of ATR and/or CA prior to serving. The Lead Architect cannot be a coordinator (this helps prevent command and control hierarchical leadership — and respects self organization of the community). The coordinators are elected every 6 months.
Agile Coach	This role is to be a servant leader to guide the community on how to respect the Agile Manifesto, the Scrum Guide, and in general, the “Community of Practice” approach to self organization. This person is expected to be an experienced senior Scrum Master or Agile Coach. The requirements for this role are: 1 year of experience as a Scrum Master or Agile Coach, 2 Agile/Scrum certifications, and at least 3 months of participation as a General Member in this community prior to serving. This person will also need to be able to spend ~20% of their time playing this role. The Agile Coach is elected every 6 months.
Management Contact	This is senior management role from the Dev Org, cannot be a first line dev manager. Currently the Director of Software Dev plays this role. This contact is used to secure funding, facilities, and other logistical approvals needed for the community to hold its events. This person does not usually spend much time interacting with the community.
Lead Architect	This person is hired by the Director of Software Dev to be the Lead Architect for the TeamSet and for the community. This person participates heavily in the community and has some decision making power (see “Process” below). The requirements for this role are determined by the Director of Software Dev. Cannot be the Community Coordinator (see that role description for more info)
Community Architect	This person spends around 50% of their time on community initiatives. They are expected to be good communicators, accessible, and have the appropriate architecture knowledge. The only requirements for this role are that the person has the appropriate architecture knowledge and must have 3 months of experience in the role of ATR and/or CA prior to serving.
Architectural Team Rep	This person spends around 10% of their time on community initiatives. This usually revolves around being a communication radiator for their Home Scrum Team and ensuring that all relevant community communication gets shared with their home Scrum Team. This person will also often help with architectural initiatives that their Scrum Team is sponsoring. This should never be considered a gate or bottleneck role — i.e. anyone on any Scrum Team can interact with the community without having to go through (or get approval from) their ATR.
General Member	Anyone who has an interest in architecture can participate in the public activities, meetings, and communication mechanisms of this community. In order to vote for the approval of CWA’s or in approval meetings, the person should have significant knowledge of the subject at hand, and have materially participated in 3 months worth of immediately prior community activities. It is strongly recommended that you recuse yourself from votes any time you do not meet these pre-requisites, OR any time you don’t have a strong opinion on the thing being voted on.
Liaison	The community has various volunteer liaisons to other parts of the organization, generally to other CoP’s. These people help us identify synnergies and conflicts of scope between this community and the other communities. This is a pretty informal position, and any General Member is free to provide the same kind of information — it’s just that these people volunteer to definitely keep an ear to the ground between the two communities.

CWA’s re: Individuals

(CWA’s = Community Working Agreements)

1. In everything we do, we try to honor Scrum(as defined in the Scrum Guide) and Agile (as defined in the Agile Manifesto). Because we have not yet chosen a scaled Scrum approach, we simply extend many of the ideas of the Scrum Guide and Agile Manifesto to our entire Community and TeamSet.
2. The table and information above includes information that is also effectively CWA’s.
3. Because we believe in the Agile value statement of valuing “Individuals and Interactions over Processes and Tools” and “Responding to Change over Following a Plan”, don’t ever be afraid to get some architects together in an ad hoc way to solve an architectural challenge — we can always retrofit those actions to our processes and tools later.
4. Unless otherwise indicated, the use of the term “architect” refers to the LA, CA’s, and ATR’s (i.e. does not refer to GM’s).
5. The architect titles above are completely independent from your job title and career path. The above titles are bestowed by the community (except for the LA, who is hired by the Director of Software Dev). This community makes no claim over the management or career path domains of the company. We are simply a self organizing community that co-exists with the rest of the organization.
6. Every architect belongs to and works on a Scrum Team. In their Scrum team, they hold no special authority or title on their team other than “Scrum Dev Team member, ” regardless of their title in this community.
7. Being a GM of the community is voluntary, with one exception: each Scrum team must choose and have an ATR with the appropriate skills, who is not an LA or CA.
8. All new CWA’s(re: Individuals or any other topic) must be approved by a “fist of 3” by the entire community. For all new CWA’s that have a heavy impact beyond the community, a forum must be held where all Scrum team members from the TeamSet can give feedback prior to approval.
9. There is always a Scrum team sponsor for each initiative, even if there is only a subset of the team working on the initiative.
10. We strongly prefer bottom up initiation of all architectural initiatives, coming from the teams. Initiatives need not come from the architects, but the ATR for that sponsoring team must be involved and be highly informed of the initiative.
11. The general time allocation expected of architects is listed in the chart above. Of course, there are exceptions at times. At any one given moment, an architect might have to choose between focusing on helping their team or helping their community. In that moment, the architect is expected to consult with others on which focus yields the most value for the entire TeamSet. Sometimes this means focusing on community efforts, and sometimes it means focusing on team efforts. Try to choose wisely in that moment.
12. Coordinators can fulfill the role of either a CA or an ATR while also fulfilling the coordinator role, but this is not a requirement.
    1. Note above that an ATR cannot fulfill the role of ATR AND [CA or LA] at the same time, but an ATR could be an ATR and a coordinator at the same time.

---

Interactions

Communication Mechanisms
	Ad hoc conversations and informal meetings — we encourage these the most! We encourage involving your ATR or CA when needed.
	This Wiki
	“Roughly” Monthly Public Meetings
	Occasional ad hoc public meetings when needed (includes educational meetings, approval meetings, etc)
	Group Chat (HipChat)
	Email List
	Occasional Private Meetings (Primarily only open to the LA, CA’s, ATR’s, and specially invited guests)
	ATR’s are responsible for communicating the important outcomes of all of the above to their respective teams.
	Architectural elements of the TeamSet Definition of Done
	Scrum Team “Code Based Tools” pages
Communication Contact	To get connected to our communication mechanisms, ask your ATR who to talk to.

CWA’s re: Interactions

1. In everything we do, we try to honor Scrum(as defined in the Scrum Guide) and Agile (as defined in the Agile Manifesto). Because we have not yet chosen a scaled Scrum approach, we simply extend many of the ideas of the Scrum Guide and Agile Manifesto to our entire Community and TeamSet.
2. The table above includes information that is also effectively CWA’s.
3. Because we believe in the Agile value statement of valuing “Individuals and Interactions over Processes and Tools” and “Responding to Change over Following a Plan”, don’t ever be afraid to get some architects together in an ad hoc way to solve an architectural challenge — we can always retrofit those actions to our processes and tools later.
4. In all of our architecture discussions and initiatives, we agree to use Agile Emergent Architecture.
5. Our community maintains a wiki for any communication where light documentation seems like a good communication mechanism. This page is just one of our pages. See our home page for lots more stuff.
6. Any person in the TeamSet can contact their ATR or a CA for help, collaboration, mentoring, or whatever is needed. It is NOT required to go through your ATR for every architecture interaction. We generally discourage direct communication with the LA unless you are working on an initiative with that person. The person is VERY busy. Your ATR or CA will involve the LA if that is needed.
7. Each Scrum Team must keep an updated “Code Based Tools”(CBT) page connected to their team wiki. Only include tools that your team regularly uses and/or has significant experience with. The purpose of this CBT page is to spread knowledge to the entire TeamSet about which tools are in use, and which teams have knowledge of those tools. On the CBT page, the team must include 2 categories of tools and info:
   1. CBT’s that they use that are expressly approved by the Arch Community Tool Matrix.
      1. For each 3rd party library, please specify the exact library and versions in use, why the library is being used(it’s purpose), known scope of use(product, module, class, etc), and how widespread is its use (low, medium, high).
   2. CBT’s that they have not been expressly approved by the Arch Community Tool Matrix. (Note that this is not considered bad usage — not all tools are in the scope of our community)
      1. please specify the exact tool and versions in use, why the library is being used(it’s purpose), known scope of use(product, module, class, etc) and how widespread is its use (low, medium, high).
8. In all of our communication mechanisms, we try very hard to be specific about topics of discussion and whether they are they “in scope for the community”– or not?
   1. For instance, using our communication mechanisms to just get general ad hoc architectural or even design/implementation/technical help is perfectly fine, but say something like “This is really more in scope for just our team, but we could really use some help on — who can help us with that?”
   2. If you’re not sure whether a topic is in scope for the community, just ask the community for help in determining that!
   3. Obviously, if you realize that a topic is in scope for a different community, by all means, please use that community’s communication mechanisms instead of ours.

---

Processes

Processes
	Architecture Tools Approval Process (ATAP) (tools, frameworks, arch approaches, etc)
	Election of Community Leaders
	Quarterly Retrospectives

CWA’s re: Processes

1. In everything we do, we try to honor Scrum(as defined in the Scrum Guide) and Agile (as defined in the Agile Manifesto). Because we have not yet chosen a scaled Scrum approach, we simply extend many of the ideas of the Scrum Guide and Agile Manifesto to our entire Community and TeamSet.
2. The table above includes information that is also effectively CWA’s.
3. Because we believe in the Agile value statement of valuing “Individuals and Interactions over Processes and Tools” and “Responding to Change over Following a Plan”, don’t ever be afraid to get some architects together in an ad hoc way to solve an architectural challenge — we can always retrofit those actions to our processes and tools later.
4. We use the term “Tools” fairly broadly, to include essentially all architectural initiatives that require community approval or coordination.
5. Community retrospectives are held at least once each quarter, and at least within the 2 weeks prior to a new community leadership election. At this time, we often review our CWA’s and Charter Statement to ensure we are in alignment.
6. Every 6 months, an election is held to select the CC’s, CA’s, and Agile Coach.
7. All architectural initiatives must include an “independent usage plan” that describes how future users of the initiative can be quickly educated on the tool/approach such that they will not be heavily dependent on tribal knowledge by a small number of people. This often includes light documentation as well as video recordings of education sessions for the initiative. Decreasing this type of “key person” risk enhances our Agility and ability to respond to change in the future.
8. The ATAP is documented in detail elsewhere, but here is a summary:
   1. A Scrum Team suggests sponsoring an initiative to be approved as an experiment or as a tool, initiative, or decision that is approved for widespread community use.
      1. We encourage the teams, as much as possible, to sponsor initiatives of their own choosing. I..e we prefer they initiate.
         1. In rarer cases, sometimes the CA’s or LA will ask a team to sponsor, but the decision is up to the Scrum Team.
   2. An approval meeting is scheduled (giving the team time to be prepared). Sometimes this is done in regular monthly meetings, sometimes scheduled ad hoc.
   3. The Scrum Team makes review material available 1 week prior to the approval meeting for voting members to review prior to the approval meeting.
   4. The Scrum Team presents to the community.
   5. The community votes with a fist of five, where at least a fist of 3 is required of all approved voters. If a fist of 3 cannot be obtained, a “unity group” is formed to discuss further and/or come up with a compromise within 2 weeks, including those strongly in favor, as well as any that are a fist of 2 or lower(the dissenters). If the unity group can agree with in 2 weeks, then the voted is considered approved. If they cannot agree, then the LA makes the decision to approve or disapprove as a last resort.
   6. If approved, the community then documents the new tool as “approved for experiment” or “approved for use” and is added to the tool matrix. (see below)

---

Tools

Tools Matrix

Tool/Initiative Name		Being Proposed	Approved For Experiment	Approved For Use	Deprecated	Sunsetted
Logging Framework – SLF4J (v3.4, v4.0.1)				X
Programming Language: Java (23.4 or above)				X
Dependency Injection Framework: InjectorSpace 3.2 or above (Home Grown)				X
Dependency Injection Framework: InjectorSpace 3.1					X
Dependency Injection Framework: InjectorSpace 3.0 or below						X
Architectural Pattern/Domain Logic: Domain Model				X
Architectural Pattern/Domain Logic: Service Layer				X
Database: Oracle 23i or above				X
Deployment Platforms: ???
Other various open source libraries: Must be GPL-3.0 or EPL-1.0 license.				X
Programming Language: Scala (12 or above)			X
Architectural Pattern: Microservices (will likely be limited in scope, as only valuable in certain contexts)		X
Persistence Framework: Hibernate (54 or above)				X
Architectural Pattern: Monolith applications				X
Peer Review: Scrum Team must have documented procedures for peer reviews.				X
… (author note — there would likely be many more items in a real CoP — these are just examples)

Note that the TeamSet Definition of Done requires that all 3rd party tools that are code based (Libraries, code frameworks, Development Environments, etc) be represented on the above Tool Matrix.

The items below have at one time been considered out of scope for the Architectural Community.

Out of Scope Matrix

>

CWA’s re: Tools

1. In everything we do, we try to honor Scrum(as defined in the Scrum Guide) and Agile (as defined in the Agile Manifesto). Because we have not yet chosen a scaled Scrum approach, we simply extend many of the ideas of the Scrum Guide and Agile Manifesto to our entire Community and TeamSet.
2. The table above includes information that is also effectively CWA’s.
3. Because we believe in the Agile value statement of valuing “Individuals and Interactions over Processes and Tools” and “Responding to Change over Following a Plan”, don’t ever be afraid to get some architects together in an ad hoc way to solve an architectural challenge — we can always retrofit those actions to our processes and tools later.
4. We use the term “Tools” fairly broadly, to include essentially all architectural initiatives that require community approval or coordination.
5. We don’t yet have any more special CWA’s on the Tools topic — most of what we record here is in the tool Matrix above.
6. These tools have been considered to be “out of scope” for this community: Agile ALM Tool, Wiki Tool Choice, Test Driven Development, Peer review procedures, Test Automation techniques, Build Automation techniques, process compliance.

 

[TODO: Make Tables appear better]

An Introduction to Agile Emergent Architecture – Always Intentional

An Introduction to Agile Emergent Architecture: Always Intentional

Let’s Define Architecture
By definition, architecture is about the major pieces of the system^1, and about satisfying the non functional requirements such as availability, usability, compliance, scalability, security, extensibility , maintainability, and all of the typical “ilities.” It’s important to remember that architecture does not cover anything and everything that can be put into one of the above categories — because they would encompass every last bit of the *entire* system. The architecture should not encompass every last corner of the system. Instead, architecture focuses on the very highest level concerns in each of those categories, and nothing more than that. Further, architecture focuses only on the most basic of building blocks in those categories, and nothing more than that. This leaves as much flexibility as possible for later agility and implementation. So, be sure your architecture is focusing only on the highest of concerns. (The lower concerns might be considered such things as design, code, or implementation)

Big Up Front(BUF) Thinking
Legacy software development approaches put a large emphasis on understanding all the requirements for a system up front. This is also known as BRUF (Big Requirements Up Front). The disproven theory behind this thought is that if one understood all the requirements really well up front, then the most optimum architectures, designs, and software code could be created. As such, BRUF has siblings known as BAUF (Big Architecture Up Front) and BDUF (Big Design Up Front). All of these Big Up Front (BUF) approach theories have been disproven for one overall reason: it takes time to gain this BUF understanding, and by the time you do think you and your team have that understanding, the requirements have changed substantially, which then affects that architecture and design as well. What we have learned over the decades is that the requirements, technologies, users, technical team members, and the market for the product change quicker than we can gain that understanding, and certainly quicker than we can implement an architecture or design. In addition, customers often have trouble describing exactly what they want — they tend to have a much better understanding of what they want only after they see the working software. Further, with BUF thinking, usually the people that did the up front thinking are way different than the people implementing that thinking. So downstream from requirements, when changes to the architecture and design are truly needed, because of all of the previous history, BUF thinking, and totally different people involved, changes are extraordinarily costly *and* complex to retrofit. As such, Big Up Front thinking is a model that has been disproven, so it’s unacceptable, and indeed now.. passe.

No Up Front(NUF) Thinking
However, the other end of the spectrum, NUF (No Up Front Thinking), is also unacceptable. You can’t create a cohesive architecture, that is financially viable, without some good upfront thinking. Without at least some good upfront thinking, the architecture turns out haphazard, almost accidental in nature. Some people have called this unintentional architecture, and the name probably fits.

Signs you MIght be a Victim of NUF
One possible sign of NUF is a system rewrite. A system rewrite is almost always a sign of failure ^2, and the two biggest reasons for that failure are inattention to the user marketplace and inattention to continuous technical excellence via architecture, design, testing and coding practices. Inattention to continuous technical excellence creates what is known as technical debt, which is the other tell tale sign of NUF. Massive technical debt generally presents itself as a highly unacceptable amount of bugs, new functionality that takes way longer than it should, or systems that get jettisoned or re-written. Massive technical debt is pretty much a guaranteed outcome of No Up Front Thinking. As such, since massive technical debt has so many bad outcomes, No Up Front Thinking is also unacceptable.

Big Up Front thinking can lead to No Up Front Thinking
Now, before we move on, let me also address another relevant point. BUF thinking can actually lead to NUF. If a BUF architecture is not kept up to date, is not shepherded, or is extremely inflexible to change, people will avoid thinking about and making architectural changes. This results in NUF and massive technical debt. So, a single system can be a victim of both BUF and NUF. It would be really great if there was a way to find that right balance between BUF and NUF… right?

The Better Way: Agile Emergent Architecture
Enter “Emergent Architecture”, a term suggested by Agile thought leaders. One can think of this as Little Up Front(LUF) Architecture, combined with continuous attention to technical excellence. You could even think of it as “Continuous Architecture” if you so desired. With Emergent Architecture, you do just enough, just in time, at the “last responsible moment.” It’s also important to note that Emergent Architecture is also 100% intentional architecture. Architecture doesn’t just “magically appear”.

The Benefits of Emergent Architecture
By architecting at the last responsible moment, you are minimizing the requirement churn damage that accompanies BUF. By architecting at the last responsible moment, you are taking advantage of the latest and greatest technology knowledge. Finally, by architecting at the last responsible moment, you are very sure that the people collaborating on that architecture are the people that are about to implement that architecture. All of the sudden, the ingredients for success are all in the right place at the right time!

Technical Excellence
Technical Excellence can refer to architecture, design, testing, coding, and probably other practices too. Regardless of whether you do BUF, NUF, or Emergent Architecture, the ability to quickly and cheaply extend or change your software architecture at any given moment is directly proportional to your practices around technical excellence. The higher your technical excellence, the more quickly and cheaply you can change direction (<– indeed, this is the definition of agility). Having said that, in BUF approaches, since people make the mistake of thinking that they can “lock down all the requirements and architecture Up Front” , they rarely put in the technical excellence needed for rapid change. Examples of technical excellence are paired programming, collective code ownership, continuous integration, continuous automated testing, build automation and build pipelines, Test Driven Development, Unit Testing, lightweight code reviews, YAGNI, and many other practices. Agile approaches harness change for the benefit of the customer, so don’t forget Agile Manifesto Principle #9: “Continuous attention to technical excellence and good design enhances agility.”

Architectural Runway and the “Last Responsible Moment”
Now, let’s be careful with the “last responsible moment”. Note the word “responsible” in “last responsible moment.” For different architectural pieces, that last responsible moment will be at different times. Some architectural pieces will require a longer runway, and other pieces can do well on a shorter runway. In summary the more complexity and learning in the requirements, people, and technology, the longer the runway needed. If the complexity factors are low, then less runway is needed. Figure out that needed runway length (length of time before that architectural piece needs to be in place), and work your way back to when that runway needs to begin being built.

Let’s look at some examples.

Architectural Runway for a Deployment Platform
For example, determining the deployment platform(the initial UI, logic, and other tiers) for the Minimum Viable Product for a new product probably needs to happen before the first Sprint of developing that product. Having said that, let’s not regress to BAUF, but let’s do execute some upfront thinking and have that deployment platform pretty well figured out before that first Sprint. Since every Sprint has to include at least some small amount of user/business valuable functionality, it’s going to be hard to create some releasable software functionality *and* create your initial deployment platform all in Sprint 1. It is theoretically possible, but not likely.

Architectural Runway for an Open Source Logging Framework (Complexity is low, should be a fairly quick decision)
Another example of “last responsible moment” might be choosing an open source framework to do logging in your system. If logging is forecasted to be in Sprint 23, there is no reason to choose that framework in Sprint 3. You can probably wait until a few sprints before to begin working on that decision.

Architectural Runway for a 3rd Party Processing Component/Framework (Complexity is high, will be a long duration before a decision can be made)
Let’s assume the same as the above — we plan to begin using this component in Sprint 23, and we are currently in Sprint 3. We need to choose a large 3rd party proprietary component that does a large amount of processing or functionality(think accounting, medical, legal, or aerospace). Since it comes from a 3rd party, there is likely going to be a long runway needed to have that choice in place before functionality can be built on top of it. In this example, it might be perfectly fine to start the architectural discussions about which 3rd party framework to use and purchase in Sprint 3. Remember that you’re going to have to leave time for product evaluation, purchasing, legal, training/learning, and maybe even some technical investigation and technical spikes. It could actually take 20 sprints to execute that particular architectural runway and have the architecture first able to support business functionality.

Architectural Runway for logging standards across multiple teams(Complexity is medium, requires time for buy-in from multiple teams)
Remember that logging framework implemented in Sprint 23? Well, we’re now in Sprint 27, and now that the logging framework has been implemented on a couple of teams, the teams realize a need for standardization of use because the framework is beginning to be used widely across the product. At first people didn’t think this was a “high enough concern” to be considered architecture, but then then they realized that the teams were using widely different “logging levels”, that debug logging often crossed module/system boundaries, and that debugging across those modules was getting difficult. The teams finally realized that there would be some major value in ensuring some light standards were followed across the teams. Three might even be a nice wrapper around the chosen logging framework to make it easier for teams to use in a more consistent way. So, in Sprint 27, they began working on an architecture effort to solve those needs. By Sprint 31, the first teams were using the new wrapper and light standards. One thing to note here is that, back in Sprint 20, when they were originally choosing a logging framework, they could have chose to do all of that standardization up front — but that would have been BUF thinking, and they could not have predicted the ways the framework would be used. It’s very possible that no other team would ever pick it up and use it — as happened with most of their open source frameworks. In other words, they would have been heavily speculating needs rather than having a practical, experienced understanding of the real needs. Now, with the experience of a couple of different teams actually using it, in production, they can make better decisions about how to lightly standardize. In this way, they chose to do Emergent Architecture over BUF thinking.

Architectural Runway for Performance and Scalability(Complexity is highly variable here, depends on the product and the target stakeholders for the release)
Hopefully, if you have a new product, you have a Minimum Viable Product(MVP) release that doesn’t require supporting a zillion users concurrently. However, every product is different, so act accordingly. An MVP will certainly need to perform and scale in a way that meets the needs of the initial users. Get this wrong, and you will pay a very heavy price in terms of customer satisfaction and product adoption. As such, for an initial release, this topic is extremely important. The non functional requirements around performance and scalability will often be ordered high in your backlog, as they are both high value and high risk for an initial release. Having said that, sometimes premature optimization is a source of waste. The runway here will also depend on the tools and environments you need to do proper load testing. Choosing and setting those up can often take serious time, so that would extend the runway needed. So, the runway answer here is probably — “it depends.”

Architectural Runway for Government Regulatory Compliance(Complexity is highly variable here, depends on the weight/complexity of compliance, the agility of the agency, etc.)
The guidance for this piece of the architecture is almost identical to the factors involved in dealing with scalability and performance requirements. They are “must haves” that will often be ordered very high in the product backlog due to their risk and value. Note here that we are mainly discussing compliance needed by the product, not the process. (On a related subject, sometimes software process compliance ends up on your Definition of Done, which may or may not affect the product architecture or product backlog directly.)

The above runway discussions are just some examples to think about. The architectural runway for each of these needed pieces of the architecture will be different. As such, the “last responsible moment” to build that runway will require some serious thought as you refine your product backlog. Further, in order to deal with the unknown unknowns, you’d better put a time buffer into that runway since you can’t possible know the unknowable!

In Summary
Hopefully the above has given you a picture of what Emergent Architecture looks like. With each vital part of the architecture, you will need to consider the runway that gives you the “last responsible moment” and some buffer for the unknowns. Because the requirements will be emerging over time, so too will each of the architectural pieces. The architecture will emerge, or evolve, over time, but with intentional forethought to the necessary runway. Remember, due to the number and churn of moving parts (requirements, people, technology, etc) inherent in almost all software development, the *only* way to keep an optimum architecture over time is to give continuous attention to technical excellence. Finally, in order to avoid the gigantic trappings of BUF and NUF, Emergent Architecture is really the only sane choice.

(If you have several teams on the same product or in the same org who need to keep the Architecture of systems excellent, see:  An Agile Architecture Community of Practice

---

Notes:

^1 The focus of this article is Agile Emergent Architecture, so we give only brief attention to a definition of architecture here. It’s worth noting that many thought leaders have multiple credible definitions of architecture. Further, Martin Fowler and Molly Dishman have mentioned that some think the term “architect” is not even the correct metaphor — that maybe “city planner” is the better metaphor.

^2 It’s worth noting that there are some very rare scenarios where a system re-write might not be a sign of failure. For instance, if the system re-write is an early pivot in a product’s lifecycle, it might make great sense and not be due to the above mentioned causes. Having said that, I’ve never personally seen a re-write that was due to a good reason.

The New Definition of Software Success

In a previous post on why large scale Agile and Scrum is 6X more successful than waterfall, I explain how Agile projects/products are so much more successful than waterfall based approaches like the Rational Unified Process.

What is also equally important to note, is that the Standish Group, an industry leader in the software project management survey field, has changed their definition of success.  According to Jennifer Lynch from the Standish Group:

The Standish Group has redefined project success as onTime, onBudget with a satisfactory result…we have seen many projects that have met the Triple Constraints [schedule/scope/cost]and did not return value to the organization or the users and executive sponsor were unsatisfied.

It’s important to note that this definition changed in 2015, and the new definition is applied equally across waterfall and Agile projects at The Standish Group.

A parallel development also captures this trend.  Scrum.org is an industry leading organization focused on improving the profession of software delivery through Agile approaches like the wildly popular Scrum approach (some 90% of software teams use Scrum). Scrum.org has now publicly released a new software success metrics model that they call “Evidence Based Management for Software Organizations(tm)“.  The approach is free to the public, and the metrics apply to both Agile and waterfall based software delivery.  Just knowing the basics of the approach can help any organization improve.  In it, Scrum.org has identified around 23 key software metrics that can be used to trend whether your software investment ROI is heading in the right direction or not.  What’s important to note is that these key metrics are all just more detailed derivative metrics of, you guessed it: schedule, scope, and customer/user satisfaction.

So, the clear trend from the above two developments from industry thought leaders is that we in the software industry should take notice on how we evaluate the success of software projects.  The data strongly indicates that we have been focused on the wrong success metrics for the past 50 years in our industry.  It appears that schedule/scope/cost is now passe in the industry. It’s time to begin focusing on value delivery via the key metrics of schedule/satisfaction/cost.

Large Scale Agile and Scrum vs. Waterfall: Agile is 6X More Successful, 1/4 the Cost, and 10X Faster Payback!

A pair of recent findings from the Standish Group confirm the astonishing success and cost savings of Agile approaches over waterfall.

In the “Chaos Report 2015“, Standish found that large Agile projects are 6X more successful than waterfall projects.  While Standish doesn’t get specific on what is a “large” project, it’s worth noting that “Large” is the biggest size category for projects and their data encompasses over 10,000 software projects.

In a new report called  The Money Pit, The Standish Group studied two very similar large software projects, done at two very similarly sized, mature companies.  One project was done with Agile, and one with Waterfall.  The astounding results they found:

• The Agile project was 4X cheaper than the cost of the equivalent waterfall project, AND
• The Agile project was “delivered with high user satisfaction,” while the waterfall project “had a watered-down critical function and the high-value feature was not part of the delivered application.”, AND
• The Agile Project’s payback was “At the end of two years the application costs were fully paid back and the users were highly proficient” while the waterfall organization estimated the system payback would “break even in 20 years”

 

* Note that the activities depicted on the graph were done sequentially via waterfall, but iteratively via Agile.

It should also be noted that a survey from Forrester Research showed that of all companies attempting Agile, some 90% are using Scrum.

Just to re-iterate — 6X more successful, cost payback 10X faster, and 4X cheaper.  How is that for Better, Faster, Cheaper?

At AgileSoftwareTraining.com, we provide coaching, consulting, and training solutions to help your company achieve the astounding success of Agile and Scrum approaches.  Contact us today for a free consultation.

So, if you’re an organization that is doing waterfall or struggling with Agile, what are you waiting for?  The research is overwhelmingly in favor of Agile and Scrum approaches.  Get on the road to millions more in profit and cost savings.  No seriously, what are you waiting for?

The New New Product Owner: The Product Visionary

In my previous post, I discussed the New New Product Owner as The Product Marketplace expert.  In this post, we explore the “Product Visionary” focus area.

The Product Owner is the chief product visionary. The PO should be able to clearly articulate the product vision to the Scrum Team and key stakeholders, and how that vision aims to maximize the value of the product and of the work the Scrum Team performs.
The PO should communicate and re-iterate this vision early and often, reminding all involved of how to help maximize value.

For a high quality class that focuses exclusively on the Product Owner role(the course is also great for key stakeholders!), see our Professional Scrum Product Owner class and contact us if you’re interested in one.

Utilizing the underlying empirical product planning features of Scrum, the PO should also be ready to make strategic pivots for the product vision whenever there are significant changes in how value can or likely be captured. This vision is brought to life in a more tactical way, via the Product Backlog and iterating towards that vision every Sprint.

In my next post, we’ll discuss the New New Product Owner as the Release Decision Maker.

A Response to Mike Cohns Comments on 64% of Software Features Rarely or Never Used

I have a saying.  “Scrum Trainers usually agree on 99% of Scrum, but they spend a lot of time debating the other 1%.”

Let me say this first.  I’m a huge fan of Mike Cohn.  I teach Scrum and Agile classes all over the country at Fortune 50 companies, and it is very rare for a class to go by that I won’t mention at least one of his awesome books on Scrum.  I also recommend him on my list of favorite Agile resources on one of our web sites.  In addition to all of this, I’ve had numerous personal interactions with Mike one on one, and he’s always been extremely nice to me, traded professional practice opinions/advice, and he even offered to let me attend one of his classes at a “trainer courtesy” discount one time.  Great guy!  In summary, I like the guy a lot personally, and I highly respect him professionally.  He’s done a ton for the software and Agile industry, and no one should forget that.

So, with that said, let’s get back to that 1% debate.  🙂

In his recent blog post, Mike reveals some little known details about the oft cited 64% of features that are rarely or never used in software systems.  His information is factual and likely true.  I’m ok with all of that.

What I don’t understand is, why bother broadcasting this?

This is one of the most credible studies available on the subject.  If you think hard about this data for a minute, you’ll realize why it is incredibly difficult to obtain… No company wants to admit that there is a TON of bloat in their software!  But, what percentage of Microsoft Excel/PowerPoint/Word features do you use and benefit from?  What percentage of Rally features do you actually use and benefit from?  Bloat bloat bloat, negative value, negative value, negative value.   In my recent articles on the New New Product Owner, I’ve talked about the need for the New New Product Owner to be a marketplace expert, so that they can maximize the value and profits from software development for their company.

Now, the value equation is way more complicated than “rarely or never used”, but still, I think we all know that there is a TON of negative ROI functionality in any non trivially sized application, and there is a TON of software teams with far too little focus on value and profits.  Anyone who has worked on the front lines of software development knows that.  The oft cited study just helps confirm some of our suspicions.  One of our Agile Metrics consulting services at AgileSoftwareTraining.com is helping to give company leaders even more transparency into how to extract more profits and cost savings out of all of their software development efforts, whether they be internal or external systems.  Give us a call if you’re interested.

What makes that limited study useful as a teaching tool is it gets people to think about value, and think about low value, low ROI features, and realize that value delivery is important, far too important to ignore.

Newer Study confirms the same idea!

Further, Standish again re-iterated this point in 2014, ostensibly from wider data: https://www.standishgroup.com/sample_research_files/Exceeding%20Value_Layout.pdf — “Standish Group research shows that 80% of features and functions have low to no value.”

There are other “studies” cited in our industry that are totally bogus, software leprechauns if you will, and I’m totally against relying on those.  Things like the “Cone of Uncertainty” and the so-called “Weinberg study” on task switching have shown to be totally made up.  However, the Standish Group study is real, with real data, and it is highly credible, even if somewhat limited in its scope.

So, Mike wants us to stop citing the study, or for us to caveat it with “in the weeds” details.  Of course, that will just confuse those new to Scrum and the teaching value would be lost.  And people would focus less on software value and profits.  I don’t think that’s good.  I’m totally open to hearing about a more credible public study, but I’m unaware of one. 

With all due respect to a friendly colleague, and one of the best Scrum trainers on the planet, I think ignoring or caveating the 64% study is bad for the industry.  Let’s just put this in the 1% bucket that we as Scrum trainers will agree to disagree on.  🙂

If you’d like to disagree with my contrarian view, feel free to sound off in the comments below!

Scaling Agile and Scrum to multiple teams: Great Overview of LeSS and Large Scale Scrum

If you’re like many companies out there, you’re trying to figure out a way to effectively get multiple Agile teams to work together to deliver more with less.  Well… Let me introduce you to LeSS — Large Scale Scrum.  Large Scale Scrum has been in practice since 2005.  The co-creators of LeSS, Certified Scrum Trainers in their own right, have just released a kick arse chapter from their new book on LeSS.  Their chapter, in my opinion, is the best introduction to LeSS for those who are not familiar with it.  I highly recommend you give it a read!

If you’re interested in learning more about Large Scale Scrum (LeSS), check out the LeSS class being held in Denver in late September. (Includes certification, SEU’s, etc)

Here is the link for the chapter introducing LeSS:  http://less.works/less/framework/introduction.html

The New New Scrum Master: Two Main Focus Areas

I’m going to take a break from talking about the New New Product Owner to talk about the New New Scrum Master now.

Preface:  In the last 4 years, the Scrum Guide has had two very significant updates, including updates to the Scrum Master Role. In this article and the series that follows, I attempt to describe “The New New Scrum Master” role in Scrum.

The Scrum Master role has not changed as much in recent Scrum Guide updates as much as the Product Owner.  In many ways, however, what has changed, is the number and higher frequency of misconceptions about the Scrum Master role.  This is, in my opinion, due to late adopters to Scrum who don’t take the time or money to attend proper and professional Scrum training.  Yes, this appears to be a completely self serving statement since I’m a Scrum Trainer.  However, the bigger, and more important reason this statement is true, is because Scrum is a much deeper topic than people think.  We often use the metaphor of the difference between a Chess player who knows how the different chess pieces move, and a Chess player that has extensive experience and knowledge about how to be excellent at the strategy and techniques of Chess.  There is a vast difference between those two ends of the spectrum, between knowing the rules and being able to excel at winning.  With Scrum, people and organizations vastly underestimate just how long that spectrum is.  All you have to do to confirm this is to witness or hear about a Scrum adoption that is horribly painful or not working.    So, my hope is that we can clear up some misconceptions for the New New Scrum Master and help reduce some pain and increase some profits!

The two main focus areas for the New New Scrum Master are:

• Teaching and coaching the organization on how to achieve the benefits of Scrum, and
• Removing impediments that are beyond the reach of the Development Team.

For brevity’s sake, we will shorten these to “teach/coach” and “remove impediments.”

All of the other Scrum Master focus and duties derive from the above two focus areas.

For a high quality class that focuses exclusively on the Scrum Master role(the course is also great for management), see our Professional Scrum Master class and contact us if you’re interested in one.

Teach/Coach

One of the main focus areas for the New New Scrum Master is teaching and coaching the organization on how to achieve the benefits of Scrum.  Let’s talk about how this might be done.

The New New Scrum Master knows the “Why’s” behind Scrum.
In my experience, Scrum Masters would do well to understand the benefits of Scrum on several levels, before worrying about specific teaching or coaching techniques.

First and probably foremost, the Scrum Master should understand and *be able to succinctly communicate* the *business* benefits of Scrum to the organization. It is important to to be able to communicate these benefits succinctly because, in the wider organization, the Scrum Master will very often be given 5 minutes or less to convey them.

Each Scrum Master should have their own such list of benefits memorized, but certainly some of them should be:

• Faster time to market
• Quicker ability to pivot to market opportunities and competitor threats.
• Higher Customer Satisfaction
• Higher Productivity
• Higher Transparency
• Better Predictability
• Better alignment between the business and software teams
• And the list goes on…

A good study of the 11 key metrics in “Evidence Based Management” will help you to be aware of some of the business benefits, but come up with some of your own.  Make it yours!

Being able to rattle a good handful of these off, and then being able to *explain succinctly* how different sub parts of Scrum support these goals should certainly be in the New New Scrum Master’s toolbox.  When facing an organization that has not been to proper Scrum Training, be sure not to use too much Scrum speak — keep it very simple and mostly devoid of Scrum terminology.  Also, definitely focus on the business benefits of Scrum that align with organizational desires and goals.  The more you can point to those higher organizational desires the more buy-in you’ll likely get from those higher in the organization!

The New New Scrum Master should also be able to communicate the benefits that will appeal more to those on and around the team, such as:

• Benefits to Development Team members
• Benefits to Business Stakeholders
• Benefits to Product Owners

Again, the same advice as above, be ready to rattle off a list, be ready to explain how different parts of Scrum support each of the list items, and be ready to avoid Scrum terminology for those who haven’t had proper training.

Knowing the “Why’s” behind Scrum will help convince others of “why” to pay attention to your teaching and coaching.  Notice I said “help convince” — it will likely take much more than that, but knowing these benefits is a “must have” for those conversations.

Knowing the “Why’s” behind Scrum is just one aspect of “teach/coach.”  Don’t misunderstand me, teaching and coaching is actually more than just teaching and coaching — using those two terms is simply a way to oversimplify this focus area.  You might also want to mentor, advise, and facilitate at times.  But even those things can fall under “teach/coach.”  We’ll explore this more in future posts.

Removing Impediments

The second New New Scrum Master focus area is removing impediments that are beyond the reach of the Development Team.

There is a common misconception that the Scrum Master is responsible for removing *all* impediments for the Development Team.  While the Scrum Guide is a little vague on this subject, it is somewhat hard to articulate the fine balance between the Scrum Master’s duty to remove impediments, and the Development Team’s responsibility to self-organize. This misconception drives a lot of failure in Scrum implementations.  We’ll explore this more in future posts.

The metaphor I like to use here is “Give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime.”  When respecting and coaching on the Development Team’s obligation to self organize, it is important for the New New Scrum Master to realize when is the right to time to give the fish, and when is the right time to teach to fish.  The answer is usually the latter, but sometimes the former.

Remember:  Teach/Coach, and Remove Impediments

I have thought about this a lot, and have even conferred with my fellow Scrum coaches on this.  Pretty much all of the responsibilities of the New New Scrum Master boil down to the two focus areas of “teach/coach” and “remove impediments.”

In future articles, I will go deeper on each of these two focus areas.

In the meantime, do you think any Scrum Master duties cannot effectively fall under those two focus areas?  If so, what are they?  Sound off in the comments below!

The New New Product Owner: The Product Marketplace Expert

In my previous post, I discussed the 7 focus areas for the New New Product Owner, as well as the “Product Value Maximizer” focus area.  In this post, we explore the “Product Marketplace Expert” focus area.

The New New Product Owner should be expertly aware of the marketplace for the product. They should constantly be gathering and re-gathering information and data regarding the marketplace, so that the product value is maximized. Getting out of touch with the marketplace can be a recipe for product disaster. Note here that the New New Product Owner may or may not be the one doing the legwork of gathering the marketplace data(they might delegate), but they should certainly be aware of the market research.  Often times the New New Product Owner will delegate to others or to automation to aid them in obtaining the market data.
Note that your market might be internal (IT software) or external (Saas, Consumer software).  Either way, it’s important to gather the marketplace data.

With IT software, the market data will often include how the LOB(Line of Business) uses the software, as well as understanding the business function that the LOB executes on.  Heavy interaction with those in the LOB will usually yield this data, or again, the New New Product Owner might delegate or rely on someone in the LOB to supply her with that data.  A good starting point for gathering this data is understanding who your key stakeholders are.

Additionally, the New New Product Owner should never be afraid to change the vision or tactics based on marketplace changes. Being able to strategically re-pivot and capture value in new and different ways is one of the key benefits of an Agile mindset.

For a high quality class that focuses exclusively on the Product Owner role(the course is also great for key stakeholders!), see our Professional Scrum Product Owner class and contact us if you’re interested in one.

The New New Product Owner communicates all of this marketplace knowledge to the Scrum Team through daily ad hoc interactions as well as Product Backlog Management, Product Backlog Refinement, and in Sprint Reviews.

Knowing the market for your product can help you fulfill another key focus area, being The Product Visionary.

The New New Product Owner: The Product Value Maximizer in Scrum.

Preface:  In the last 4 years, the Scrum Guide has had two very significant updates, including updates to the Product Owner role that have far reaching implications. In this article and the series that follows, I attempt to describe “The New New Product Owner” role in Scrum.

In a series of upcoming articles, I will detail the different focus areas that the modern Product Owner needs to concentrate on in order to fulfill their duties on a Scrum team.
The New New Product Owner understands that she will likely need to execute these activities at different times, and that she might need to delegate to others in order to effectively produce software that maximizes ROI for the software development effort.

The first and most important focus area is for the Product Owner to be the “Product Value Maximizer.”  There are lots of way to do this, but 3 key steps are involved:

1. Order the Product Backlog by (estimated) value.
2. Work with the Development team to get small increments of value to market quickly, for feedback.  Release to market = in production, available to end users.
3. Rapidly assess value delivery feedback from the market, and then start over again at step #1.

These three steps will ensure that the New New Product Owner delivers “the right thing.”

Note that your market might be internal (IT software) or external (Saas, Consumer software).  Either way, it’s important to get features into production quickly, and to assess whether we have “hit the target” with respect to value… or not.  This quick release to production, every few weeks, is a key aspect of Agile software development.

If you haven’t already signed up for our blog, be sure and sign up (upper left hand corner) so you will get the future articles on this topic!

In future articles, I will detail the remaining six focus areas for the New New Product Owner:

• Product Marketplace Expert
• Product Visionary
  
• Product Release Decision Maker
  
• Product Backlog Management Leader
  
• Lead Facilitator of Key Stakeholder Involvement
  
• Effective and Active Scrum Team Collaborator

For a high quality class that focuses exclusively on the Product Owner role(send your business stakeholders too!), see our Professional Scrum Product Owner class and contact us if you’re interested in one.  We teach all over the USA.