Wednesday, March 30, 2011

Work in Progress and Done

One finds that when SCRUM teams start with development, the team takes on a bunch of tasks and starts marking them as Work in Progress ( WIP ), so you find yourself seeing a sharp spike upwards on the WIP status which results in an interesting situation as the burndown occurs.

You might think that this is correct practice, getting many things fired off at once and then converging down to "done done". Well, based on my experience and articles out there, that's not a good thing. Its far better to get to done items quicker in the sprint by having teams focus on getting the tasks done vs the "spray gun" approach.  What can happen is that  you can mark that 80% of work is done but only 20% of your features get done, there is a great example of this when google was working on the the adwords project ( I also experienced similar behaviors with teams when I first implemented SCRUM at my previous companies).

The best technique is to put a couple of engineers to work on the particular feature and tasks in a more serial fashion so that the features get "done done" and move to the next item in the list. You drive quality and reduce the "work in progress" gaps

Here is a chart ( borrowed from Ken's talk at google, that represents the idea). Too  many tasks, marked in Yellow, represent tasks which are started but in complete, White represents the Burned tasks("done") and Red are the not started tasks. The goal is to make the burndown more lean and reduce the amount of parallel tasks which are started and increase the amount of burndown tasks which are "done". The bottom line is that the higher the white in the chart below the higher the quality and features complete and the better chance you have in finishing, versus, the higher the yellow in the chart the more risk of having a slower velocity in completion.



thoughts and comments are welcome.

Thursday, March 24, 2011

Using Waterfall and SCRUM together?

Can it possible to even think of the idea that Waterfall and SCRUM can me combined? Well, I was talking to a CTO of an established company, 350 employees, 70 million dollar run rate-- they have over 100 engineers and established product and they just moved to SCRUM about a year ago. It was an interesting conversation and I would love to get opinions on this as I have some of my own.

Here are some snippets from the conversation...
- Waterfall is a great way to communicate releases and dates
- Scrum can do the same but its a continuous process with no seeming end-date ( at least thats how some people perceive it)
- Execs are used to Waterfall where concepts of GA releases have a ship date

In a recent project at Oracle I was able to use the strengths of both approaches in helping deliver software. I am curious if there are others sharing similar or different thoughts on this. One could argue that if we needed both then perhaps SCRUM wasn't followed that strictly since one can group stories and backlog items into a release.

comments welcome..

Wednesday, March 23, 2011

Engineering operations and SCRUM

So, now you have decided to take the plunge and start working with SCRUM. That's great...BUT, it isn't enough to get to you cross the finish line. There are some key engineering processes, in my mind, having lived through the horror stories of not having these in place, that need to compliment the SCRUM process which teams may or may not be aware of which are listed below.  In my experience, implementing these would get you on the right track to success.

Note: The list items below are not exhaustive (comments are welcome :-) ) but in my experience having proven this out in the field, its a solid start.


Key Practice #1:  Implement a proper Source code management system
Not having the right tool in place spells disaster for the team and for quality which in many organizations isn't such a negotiable quantity. I know this sounds trivial and its assumed most of the time.  There are many tools out in the market such as subversion(svn), CVS, Git, clearcase(which has a very high overhead), perforce(uses change sets, an interesting concept). Pick a tool that meets engineering goals of branching so you can have branching capabilities to run current and future releases. If you have a distributed team, pick a tool that supports that easily.


Key Practice #2: Implement Design and Breakdown into manageable parts
Letting the team go off and start coding in parallel is a recipe for potential disaster and will impact your project. Implement the ability to create simple designs so that the teams can understand dependencies up front, be able to have a conceptual, high level approach to coding the feature set. I think there are key areas that should be looked at first like the data model or Domain Object Model( DOM) and together with re-factoring (Key Practice # 4), be able to make changes along the way. The KEY here is to get agreement within the team on the broad concepts. This allows everyone to speak in the same terms.

Key Practice #3: Continuous Integration and Builds
The pre-requisite here is Key Practice #1 since Continuous Integration(CI) assumes a good SCM behind it for check-ins and versioning. I have personally used Cruise Control(with ANT,MAVEN)  and Hudson and there are other tools available. I cannot stress how important this is, since the CI system will also run Unit test suites which help with regression testing and help push quality up the chain and early in the cycle.
I have seen in previous projects where the Quality Assurance teams have saved up to 30% time on cycles as a result of implementing this. This also means the engineering team needs to stay committed to building and checking in Unit tests frequently.

Key Practice #4: Re-factoring
One cannot expect to know all the design and coding issues up front since we are not implementing waterfall and even in waterfall, you cannot possibly anticipate all the issues up front.  I think that re-factoring must be embraced as standard practice in order to get the quality you need for a winning product. Re-factoring basically means that the functional behavior of the module is the same yet, the code behind it has changed. It has changed because we want to close the technical gap, assumptions,error handling or other aspects , which if not done, create a quality problem. The added benefit is that engineers will get a better sense of how things work as the project moves forward and sometimes assumptions made early are invalid may impact the project. This allows  engineers to re-look at code and reduce the impact of change for later modification.  Let me clarify what I mean, which is, that if refactoring is ignored, then one can have technical limitations within the code which later on, makes it much more time consuming to change since additions feature/function has been added.

Key Practice #5: Automation/Automated Testing Suite
You have implemented a CI system already, having reached this point, so why not implement the the ability to run Unit tests, BIT (Build Integration) tests, or "Smoke" tests in order to ensure builds get to a stability point early. The point of reaching the best quality, is to push Quality Assurance(QA) up the chain to all levels. I have seen from working on product releases, that one saves a great deal of time within the QA organization from having to run functional tests when there is automation in place. You find that with time, your regression test suites mature, more tests are added, freeing up QA to focus on finding the hard bugs.  Ask any QA engineer and they will tell you that functional testing takes a long time so anything that can be done to help your team mates is a good thing.


In addition to the 5 practices I mention above, the final one, which is not to be ignored, but I assume it to hold true is that there needs to be Ownership and Accountability in the team. If there is an problem and we need to resolve it, its important to implement a culture of collective ownership of the code. I was in a situation where there was one key engineer owning key areas and it created a bottleneck for progress, so make sure you build a redundancy model where more that one person knows the particular code line to be able fix issues.  This is less of an engineering operation issue, but more a Team issue and a culture within the team which should be instilled early on.


I strongly recommend you implement these techniques and spend time on each of them so that you are able to get the best results in the end. If there are more details you wish to know, please let me know.

Absorption of SCRUM - valuable lessons learnt

If you are software professional like me and have experience with SCRUM as well as waterfall , yet inherit a team that hasn't had much experience with SCRUM. What should you do as critical success factors to get going and move the team in the right direction. I am reflecting on some recent "real life" experiences that I hope will be useful to you and the community out there. These experiences come from a recent implementation of SCRUM at a fortune 100 software company.

Here is a typical real life situation:
  1. you have a release to push out and you're responsible for it
  2. you have a new team, which has little or no SCRUM experience
  3. you're in charge of figuring out what process is best while not hampering the team and yet instilling a sense of discipline with a new process and there isn't too much time for training

Faced with this challenge I decided that having the right SCRUM Toolkit was an important first step.  There are a couple of very good options out there and here are some examples:
  • SCRUMWORKS by Danube (http://www.danube.com/scrumworks) - there is a free version and a license version. The free version allows you to do basic management like manage sprints, add backlog items to releases, allow a team definition and show basic burndown charts. It works over the internet and its free and a great tool to get started. The paid version has better charting, integration with JIRA and other useful features.
  • RALLY by Rally Software(www.rallydev.com) - this is a paid for monthly scrum tool, more advanced than SCRUMWORKS, but it depends if you have the budget to cover costs. its free for up to 10 users. For more details on pricing go to the company website. Rally has many good features on tracking and is very comprehensive.

There is always the "excel" version to use which means that an excel sheet is used to look at burndown, tasks left, backlog etc. I am not a big fan of this approach because its hard to coordinate among the team members and if you are managing a larger team, the updates can get time consuming.
Note: There are many other tools in the market, so make sure you use the best tool for your level of maturity in SCRUM and team culture.

Second, I spent time providing my team with basic training on principles of SCRUM, I focused on a couple of key aspects:

1) SCRUM is bottom up - team members should be accountable for hours and there is always a "goto" person for a task to be completed

2) SCRUM collaboration - this really means, you need to ensure your team is open about communications and having the daily scrum. Ensure a team lead, or manager allocates a SCRUM master and ensure daily SCRUMS occur.  Ensure this is also monitored and ask only 3 questions:
  • What did you work on yesterday ?
  • What are you working on today ?
  • What is preventing you from reaching your goal ?
The KEY is to not let the SCRUM turn to a long meeting where people try to resolve the impediments facing them. I can't stress this enough, as I find with new team members and people adopting SCRUM, they have a natural tendency to resolve the issues in the SCRUM itself

3) Plan , Plan and Plan - you have to spend time planning and estimating your sprints and how many you will have. Its worth the extra time to do this up front and spend time with the team on this.

4) Burning up, Burndown - the team naturally seems to understand the concept of burndown, but seems to resist the concept of burnup or they don't understand it. It takes time to understand and internalize these concepts. Spend time explaining and guide the team to ensure that its "ok" to burnup if there is legitimate reasons to do so.

5) Backlog management - features go on the backlog, organize them in a release which allows people to frame key features for the release you are currently working on. Explain the difference between tasks and the backlog.  Assign level of effort points to items. I typically use Fibonacci series ( 1, 2, 3, 5, 8 ...). For a great article on this read: http://kenschwaber.wordpress.com/2011/03/11/planning-poker/

6) Team updates - its critical that the team update impediments if they arise, close them out and update hours ( burndown ) on tasks. People can think its a "hassle" or unimportant - but its the essence on getting metrics.  As a spent a lot of time ensuring updates where timely and accurate.

7) Definition of "Done" - Get proper agreed definitions of what the definition of "Done" means for the team so that when a team member marks a item as Done, there is no ambiguity about it.  This is impact to quality further down your cycle if this is not agreed to and followed by everyone.

Thirdly, I was guiding the team and monitoring the process on a frequent basis.  What I mean is to facilitate the team in grasping and internalizing SCRUM concepts like daily SCRUM meeting, updates, logging impediments, burning down etc. I think this is key to success when you start, since the team is still absorbing the concepts, working them out between themselves and trying to satisfy software release goals. The amount of monitoring, which is why I like more automated tools, which allow you to manage a "dashboard" is key in real time. This is even more important when you have a distributed team in various time zones ( which was my case).

Finally the last part of the puzzle, is to do a retrospective. I had some spot checks during my release but spent the "downtime" after my release going through completed sprints and what worked and what didn't. We looked at sprints that we planned well and others that we had to invalidate and start over . We also had a frank open discussion about it. As I expected, there were still lingering questions on how to handle certain situations.

SCRUM Maturity

SCRUM maturity is an interesting concept since its closely related to the complexity of what needs to be done and the number of people involved.   When one looks at small teams and new implementation of SCRUM, its important to get to a "beating rhythm" first before attempting more mature concepts like Scrum of Scrums or release road maps.
I like to call the beating rhythm or basic flow, the first step to success and getting it done in a predictable fashion. What I mean is that the team needs to get used to the concepts which are basic and implement these in a predictable fashion and get product out. As teams become better at it, this becomes faster and more reliable.
Basic concepts and tools for starting teams to flow is depicted in the table below:

Documents Product Backlog (includes features and other log items) Sprint Backlog
Burndown Chart
Ceremonies Sprint Planning Daily meetings
Sprint Review
Roles Product Owner Scrum Master
Team

The next level of maturity involves taking the natural flow and doing more complex upfront planning at various levels and moving the concept of Agile up the chain. Moving it up the chain, means making Agile part of the CxO suite and be able to make decisions quickly so that the entire team and company is able to make decisions quickly. This is all about making the product a winner in the market.
There are many companies out there using SCRUM and lots of technology built, but those companies that are able to respond to change quickly and marry the ability to change product to meet market needs are truly the ones creating a winning company with a winning product.
In order to do this the table above changes to add some additional aspects as follows:

Documents Release Roadmap Product Backlog (includes features and other log items)
Sprint Backlog
Burndown Chart
Ceremonies Multi-level planning across departments Daily meetings
Sprint Review
Scrum of Scrums ( including cross functional disciplines)
Roles Chief Product Owner/Officer Product Owners
Chief Scrum Master(“Uber” Scrum Master)
Scrum Masters
Team

Agile development at this level involves a high level of interaction between product owners and the chief product officer, as well as scrum masters and the chief scrum master.