Scrum For System Integration & Analytics Data Warehousing Teams

 

–This article was co-authored by Professional Scrum

Trainers Charles Bradley and Mark Noneman.

.

Things to Think About When Using Scrum for System Integration Efforts

Vendor Platform: Think of something like SAP or SalesForce App Cloud Platform. The platform might have some out of the box functionality, but really, to get the major value out of the platform you need to hook in all of your company’s data sources (usually with the need to do data ETL type work on the incoming data to transform it into the target data model of the platform). You will also need to “configure” (tailor/enhance) the out-of-the-box functionality to your specific organization. For example, using your organizations language for labels, implementing workflows, identifying roles and permissions, adding functionality through scripts or software, etc. Some Business Intelligence platforms (Informatica, Business Objects, etc) also fall into this category (at least in how they affect Scrum).

Since Scrum’s earliest adopters were “vanilla software teams”, often doing full stack development where software programming is the primary activity, System Integration Scrum Teams tend to be late adopters. As such, they suffer from several typical problems that even vanilla software teams suffer from as late adopters to Scrum. See more about that in the Top 10 Challenges to Scrum Adoption. In addition, certain challenges present themselves to vanilla software teams regardless of whether they are late adopters to Scrum or not. Analogous challenges happen in System Integration Scrum Teams, so it might be a good idea to familiarize yourself with the Top 10 Challenges That Scrum Teams Face.

Caveat: The term “vanilla software team” is just a term used for this article to distinguish System Integration Scrum Teams from your basic full stack development software team. Vanilla is not in any way meant to imply that “vanilla software” is easier or harder, only that it’s the more generic kind of software development. Having said that, System Integration, BI/DW, straight web development, and lots of other forms of software development are also extremely valid forms of software development. They’re just other flavors of software development that have some uniqueness about them.

When doing System Integration (SI), many teams like to use Scrum for this effort. Many SI Scrum Teams think that the challenges that they face when doing Scrum are unique to SI, but most of those challenges are not unique to SI at all. Having said that, there are some challenges that seem unique, or at least more accentuated, in SI efforts.

Challenges not Unique to System Integration

Here are some challenges that SIVP (System Integration of Vendor Product) Scrum Teams perceive as being unique to them, but they really aren’t unique to SIVP Scrum teams at all.

1. Due to the proprietary unknowns about how the Vendor Platorm(VP) works, SI Scrum teams feel the need to do more spikes, analysis and design when requirement gathering.
   1. Most VP’s, whether due to feature bloat or having to deal with such a wide variety of clients, have several ways to accomplish a particular business need, which prompts teams to do more analysis or design in order to come up with system requirements that describe application behavior. This is understandable. However, when compared to vanilla software dev, the # of ways to accomplish a business goal is infinite, so this should make the number of ways to accomplish a goal in SI teams fewer than in vanilla teams, right?
   2. Due to the vendor product proprietary bits that are hidden from the Scrum Team(think of the VP as a “black box” component), it might take more time for a SI team to figure out good paths forward(analysis/design/architecture). Hopefully this can be mitigated by good platform documentation, team experience with the VP, and/or support/professional services from the vendor.
   3. All of the aforementioned challenges happen in vanilla software teams too. See “Extensive Discovery/Market Research”, “3rd Party Component Analysis/Design,” and “Complex Business Domains” in the Top 10 Challenges That Scrum Teams Face..
   4. In SI teams you tend to have a higher occurrence of situations where the dev work is small, but the number of User Story acceptance criteria (aka acceptance tests) is very large. The User Storry or effort can be said to be “lopsided.” This is sometimes due to the data heavy nature of SI, and sometimes due to the fact that once the data is in the VP, the VP GUI has numerous ways of viewing that same or similar sets of data. (Of course, the converse can also be a true — a whole lot of data manipulation, but it only shows up in one place in the GUI — that’s just lopsided in the opposite direction)
   5. VP’s usually require extensive configuration that will need to be tested against. This is really no different then finding the right balance of testing as mentioned in the points just above. This also happens in vanilla software teams when the system under development offers a ton of configuration choices. It’s another case where you might end up with lopsided stories.
   6. All of the above happens often in vanilla software teams too — see “Lopsided User Stories” in the Top 10 Challenges That Scrum Teams Face..
   7. Much more frequent traffic (real time, daily feeds) and volume in those data interfaces, and
   8. Much less knowledge about and control over those data interfaces. (which increases analysis/requirements time/complexity as mentioned in #1 above)
   9. Much higher number and impact from external dependencies on 3rd parties in development coordination, release deployment coordination, and in production support coordination. This cannot be emphasized enough. This is a heavy complexity multiplier that results in less predictability of software schedules.
      1. Even MORE extra complexity and delays here when the 3rd parties are using waterfall.
   10. The upstream/downstream data source challenge happens often in vanilla software teams too. See “Upstream/Downstream Systems”, “Waterfall Organizational Design by Application(WODA), and “External Dependencies”in the Top 10 Challenges That Scrum Teams Face..
2. SI Scrum teams tend to have difficulties around writing poor User Story acceptance criteria
   1. 1. This happens often in vanilla software teams too — see “Poor Acceptance Criteria” in the Top 10 Challenges That Scrum Teams Face.
3. SI Scrum teams tend to have discomfort around not having “stories” that account for the time spent in refinement, understanding requirements, analysis, and just enough of the design to create proper User Stories with the INVEST attributes.
   1. This happens often in vanilla software teams too — see “Massive Unknowns” in the Top 10 Challenges That Scrum Teams Face..

Challenges that are Mostly Unique to System Integration

Now, finally, here are the few challenges that the authors have seen that seem to be mostly unique to SIPV Scrum Teams.

1. Test automation at the GUI level might be challenging since you may not have as easy access to the undercarriage or GUI of the VP GUI.
2. SI Scrum teams tend to struggle with incorporating Vendor professional services personnel into their Scrum implementation
   1. This is especially difficult when the vendor is either not Agile friendly or when they want to matrix specialists in and out of the team rapidly like special forces (In Agile we call this “specialist silos”, and it’s waterfall thinking — we want T-shaped skills and long lived team members instead)
   2. Scrum Teams should get to interview and help choose new dev team members, but vendors usually don’t allow for this.
   3. Waterfall thinking in terms of fixed scope/fixed date vendor professional services contracts can misalign with Agile principles.
3. There is an extremely low amount of “on point” Agile/Scrum literature about how to use Agile/Scrum practices for system integration type work. As we have shown above, many of the Scrum concepts transfer readily, but the literature almost always gives examples from vanilla software development. This can lead to further “snowflake”/ScrumBut thinking (see “The Snowflake Objection” in the Top 10 Challenges to Scrum Adoption.)
4. SIVP teams often feel that their tool is the “Golden Hammer” for all the work related to their domain. For instance, they might be able to solve a particular problem using 15 transformation steps, when they could have written 15 lines of custom code instead, possibly reducing a 6 week story into a 1 week story. This is not a direct threat to Scrum per se, but it can impact a team’s creativity and cause a dysfunction where stories are constantly carrying over from sprint to sprint. Also, it might be a sign that the team is not terribly cross functional, which can cause other complications with respect to Scrum. Last but not least, it could save 5 weeks of dev time and money! –Submitted by Professional Scrum Trainer Jesse Houwing