Process Performance Professionals
Your Subtitle text
 PPP's SCAMPI Approach

SCAMPI: Beyond the MDD

The SCAMPI Method Definition Document MDD is about 300 pages long and there is a whole appendix on tailoring options.  Yet there are many activities within the SCAMPI that can be performed in many different manners.  These activities are particularly important if one views the appraisal itself as a process and business improvement activity (as PPP does) and not just as a measure of process improvement activities.  The lead appraiser’s planned implementation of these activities should be useful in making a choice of a lead appraiser.  

The activities are listed below (in approximate chronological order).  Following the list are PPP’s approach to each of them.

  • Understanding the company
  • Focusing on business improvement
  • Planning the appraisal
  • Deciding the purpose of the Opening Meeting
  • Determining the scope of the team training
  • Including  “opportunities for improvement”
  • Providing model interpretation
  • Determining the degree of team members’ Involvement
  • Determining the need for technical knowledge of the lead appraiser
  • Using the appraisal  as a learning experience
  • Determining how much emphasis should be placed on  interviews
  • Determining the format of the Final Findings

Understanding the Company

 It is essential for an effective appraisal plan to have some familiarity with the structure of the appraised organization.  Understanding this structure is arguably more important than understanding the purpose of the software whose development is being appraised.  A good knowledge of the structure is particularly needed for determining the data collection strategy and planning the interviews.  Some of the questions that may need to be asked up front are:

  • To what extent are requirements provided by outside groups?   For example in a typical IT environment, requirements at some level are provided by the business people outside of IT.
  • What external constraints are there on policies, procedures, and methodologies?  For example the appraised organization may be part of a larger and/or external organization which imposes some of these items.
  • Will there be a mix of project types, such as development, infrastructure, and maintenance?
  • Will any projects using agile methods be included?
  • Do any of the projects produce both software and hardware?
  • Are suppliers chosen or are there long-standing service agreements in place?
  • What central offices exist?  For example there may be a central supplier management group, an architecture group, a QA group, a Program Management Office (PMO). A central QA or test group, a measurement group, a training group, a process group, a risk management group, etc.  It is important not only to know the existence of these groups, but the extent of their involvement with the projects being appraised.

It is obviously important that the sponsor provides the lead appraiser with the right people to provide the answers to these questions prior to any in-depth planning.

Focusing on Business Improvement

What is the objective of an appraisal?  Is it just to provide evidence of compliance to the model?  Or should it be a means of providing process improvement?  The answer to that question is – and always has been – right in front of our eyes.  It’s not explicitly inside the MDD, but outside on the cover - the name of the method – SCAMPI, Standard CMMI Appraisal Method for Process Improvement.  Let us elaborate with an equation:

[An Appraisal (SCAMPI)] – [Process Improvement (PI)] = SCAM

Of course PPP is exaggerating to make a point.  But let’s go all the way back to the CMMI (nee CMM) roots.  The objective was to produce better software by improving the development process.  The model was developed by examining the best practices of successful companies.  The thought was that being compliant to those practices was a means of achieving the end of achieving better (e.g. more error-proof and more efficient) development practices.

So process improvement is the end – or is it?  This is really a cost/benefit decision.  If the cost to improve a process is below the benefit, is the change the right business decision.  Maybe – maybe not.  In any event shouldn’t the end be business improvement, not process improvement.  Maybe the wrong emphasis leads to undue cost and the shelving of the improvement initiatives.  Maybe SCAMPI should really be SCAMBI!

Planning the Appraisal

Obviously understanding the company and the broad objective of the appraisal are critical to its effective planning.  There are other factors to be considered:

  • The insights of the sponsor – it is important to understand the sponsor’s concerns so the appraisal can focus on the model elements related to that concern.  Of course the coverage rules imply that all relevant model components are adequately examined, but the rules do not preclude spending extra time on a few areas.
  • Logistics – this goes beyond the geographical location of the onsite activities.  The adequacy of the team and interview rooms, computers, teleconferencing mechanics, and even food need to be considered.
  • Team composition – the number of team members; their expertise, experience, and model knowledge must be understood.  Having team members whose collective knowledge sheds light on the intricacies of the company is extremely useful.  But, arguably,  the most important criteria for an internal team member is that they are highly respected by both management and project personnel.  If not, the end result of business improvement may be seriously compromised.
  • Project Mix – this primarily depends on getting the answers to the questions raised in the “Understanding the Company” section above.  This in turn should make it relatively easy to define the sampling factors introduced in MDD V1.3.  Note that it is very likely – and completely acceptable - that these factors differ greatly from the suggested factors in the MDD.  After that, secondary factors such as project size, criticality, personnel mix, etc. should be considered to get a good mix and still meet financial constraints.

Deciding the Purpose of the Opening Meeting

One of the keys to a successful appraisal is the openness and honesty of the participants.  The right tone of the Opening Meeting is a simple and effective way of fostering these qualities.  In particular the meeting should make the participants feel comfortable with the lead appraiser and be convinced he there to help the business, not to “get them.”  A short direct presentation with few slides helps to achieve this.

Determining the Scope of the Team Training

The CMMI Institute’s Team Training slides are designed for a two-and-one-half training session.  Thank goodness they are not required.  PPP prefers a short training that outlines the appraisal, leaving most of the mechanics to be handled with a “just in time” approach.  But the groundwork for the basic skills of note-taking, interviewing, and getting coverage must be laid out up front.  In particular the team must understand the importance elements of effective interviews.  The following should be stressed: getting SCAMPI compliant coverage, having the proper listening skills, noting body language, and asking relevant (and generally unplanned) follow-up questions.

Determining the Scope of the Team Training

The CMMI Institute’s Team Training slides are designed for a two-and-one-half training session.  Thank goodness they are not required.  PPP prefers a short training that outlines the appraisal, leaving most of the mechanics to be handled with a “just in time” approach.  But the groundwork for the basic skills of note-taking, interviewing, and getting coverage must be laid out up front.  In particular the team must understand the importance elements of effective interviews.  The following should be stressed: getting SCAMPI compliant coverage, having the proper listening skills, noting body language, and asking relevant (and generally unplanned) follow-up questions.

Including  “Opportunities for Improvement

The MDD requires strengths and weaknesses, where a weakness is basically a non-compliance to a CMMI practice.  It does not require “opportunities for improvement,” which are practices which are arguably compliant, but are possible candidates for improvement.  The MDD simply refers to these as “notes,”  which seems to understate their significance.  PPP’s experience is that most reasonably mature companies will have few weaknesses, but many of these improvement opportunities.  PPP makes appoint of including these in the Final Findings and advising the company that they might provide action plans to address about half of them.  It is (arguably) the results of these action plans which provide real business improvement and helps the appraisal have a positive ROI.

Providing Model Interpretation

Many team members have reasonably strong management and/or technical experience, but have little CMMI knowledge beyond the mandatory “Intro” course.  Our experience is that the lead appraisers must informally fill the “model gap” and relate the model to the business.  This can be done by reviewing the key relevant principles before each interview session.  It is also important to review the model when coverage or ratings become an issue.

Determining the Degree of Team Members’ Involvement

It is important that team members be involved with interviews and findings presentations.  Generally the lead appraiser is a more effective interviewer and presenter, based on his/her experience (there are exceptions, of course).  However, there is also value in the participants seeing the internal team members doing more than just taking notes.  This underscores the fact that all findings represent the consensus of the full team.  Possibly more important is the fact that the internal team members will probably be the major players in implementing any actions from the appraisal, and their deeper involvement should make them more effective.

Determining the Need for Technical Knowledge of the Lead Appraiser

The credibility of the lead appraiser and the appraisal process itself is essential; without it the rank and file will likely negatively view the appraisal results.  PPP’s belief is that the lead appraiser must be able to stand toe-to-toe technically with the participants to help establish this credibility.  This does not imply that the lead appraiser is an expert in all aspects of software development.  It doe simply, however, that the lead appraiser has broad knowledge and can exhibit this knowledge to the team and the participants, and can elaborate with personal examples.

Using the Appraisal as a Learning Experience

As mentioned above, the team members will most likely be instrumental in implementing agreed-to actions from the appraisal, so any additional training (by example, not by formal presentations) are bound to be useful.  But “training” goes beyond the team members. The participants can also benefit from the appraisal and be trained in the broad sense of the word; i.e. they can learn ways to improve their work.  The depth and the wording of the interview questions can actually serve this purpose.  

To illustrate, here’s a simple example of a question that could be asked to the people who perform the PPQA evaluations: “When you speak to project personnel to ascertain compliance to some required process, do you also ask them if they have any ideas for the improvement of that process?”  Note that the CMMI does not require such an approach, but the question itself should cause them to stop and consider the possibility of doing that in the future. 

Determining How Much Emphasis Should Be Placed on  Interviews

We have argued that: (1) business improvement is more important than process improvement and compliance checking; (2) opportunities for improvement must be included with strengths and weaknesses; (3) team members must be overtly involved; and (4) the appraisal should be a learning experience.  If one agrees with those opinions then one must also recognize the importance of interviews.

In fact, based on the 30+ appraisals with which Mike D’Ambrosa has been involved it is our opinion that in some cases interviews (we don’t like the term affirmations because it sets a minimal goal) provide better opportunities to achieve the objective of better development practices (and thus better software) than document review.  It is related to the fact that for many situations mere compliance to the model is insufficient to achieve meaningful business improvement.

Here is an example from the Requirements Management Process Area (SP 1.2 - prioritization of requirements):  The word prioritization is stated in the “long form” of the SP and in Subpractice 2.  The document review and affirmation can end if some prioritization was done and recorded.  But in traditional development as well as agile development this prioritization is often extremely critical.  For example, if a delivery date is sacrosanct (which is common) then the prioritization helps to define the minimal acceptable functionality and, in turn, strongly influences the sequencing of the implementation activities.   The interview process is arguably the best mechanism to determine if these concerns were adequately considered. 

Determining the Format of the Final Findings

The MDD (V1.3) has a summary of the appraisal process, strengths, and weaknesses as the only required contents.  PPP always include the opportunities for improvement and some high-level recommendations (generally a list of things to consider, as opposed to things to do).  Often the interviewees are asked some specific question relevant to their experience and the answers are included in the presentation.  A detailed Final Findings (which is developed in real time during the entire on-site period) makes additional reports unnecessary.  Generally all participants and any other personnel the sponsor identifies are invited, although the sponsor is free to do otherwise.  As stated in the MDD the strengths and weaknesses are read word-for-word.  PPP does the same for the opportunities for improvement, but there might be some follow-up discussion in conjunction with summary thoughts and recommendations.

Website Builder