Pixel

Journals
Author
Volume
Issue
Publication Year
Article Type
Keyword

Is Agile Scrum Improving IT Project Outcomes?

0
Original scientific paper

Citation Download PDF

International Journal of Management Science and Business Administration

Volume 10, Issue 2, January 2024, Pages 41-55


Is Agile Scrum Improving IT Project Outcomes?

DOI: 10.18775/ijmsba.1849-5664-5419.2014.102.1004  
URL: https://doi.org/10.18775/ijmsba.1849-5664-5419.2014.102.1004 

George Mihailides, William Young
.

School of Business, Law and Entrepreneurship, Swinburne University of Technology, Melbourne, Australia

Abstract: The publication of the Agile Manifesto in 2001 gave rise to a range of Agile project management methodologies with Agile Scrum particularly prevalent. Agile’s incremental and iterative nature appearing to serve many project contexts and needs over the more traditional, planned project management approach. Agile’s more visceral insight of customer needs, exposing and developing understanding along the journey, appeals to many projects where scope development is not easily articulated or necessarily clear. On the surface it looks distinctly beneficial in helping customers achieve end results. The following outlines a study undertaken of IT professionals to understand if Agile Scrum does deliver the benefits its system alludes to. This study surveyed 88 Agile Scrum IT professionals to determine their views on the efficacy of using such a system, particularly over previous traditional approaches. The survey data collected was statistically analysed using SPSS, and critically assessed. The limitations of its findings are noted along with recommendations for further research. The study findings suggest that across the project measures of cost control, schedule management and scope handling, no demonstrative benefit was derived from the adoption of Agile Scrum. However, the findings did indicate moderate gains in the measures of quality control, benefits realization, stakeholder connection and as a result overall project satisfaction. Notwithstanding these benefits, the implication was they were modest and not the across-the-board step improvement anticipated. It suggests a more forensic search for the drivers of project success needs to be undertaken of system development to achieve a wholistic outcome. 

Keywords: Agile scrum, Agile project management

1. Introduction

Many factors contribute to an IT project’s success or failure. A non-exclusive list includes competence of the project manager, capability of the project team, senior management buy-in, adequate project control systems, having clearly defined goals and requirements, the project’s complexity, volatility of the environment the project is operating in, corporate and team cultures, risk management, and the level of user engagement. The focus of this study pertains more specifically to project methodology, following along the lines of Tiwana and Keil (2004). They surveyed 720 IT projects and found that project methodology was the single biggest risk driver of IT projects.

The term “IT Project” can represent a broad range and variety of information technology projects. This research focuses on the category of projects conducted by organisations, underpinned by a business case, approved by a governance body, and empowered to make capital investment decisions on behalf of the organisation. Typical business cases identify the project’s scope of activities, estimated cost, duration and expected benefits to be derived.

The term ‘outcomes’ as it pertains to IT projects can be ambiguous and is inextricably connected to the concept of project success. On this matter, the academic literature identifies two concepts. De Wit (1988) makes a distinction between ‘project management success’ and ‘project success’. Project management success refers to competing project dimensions of time, cost, and scope (‘iron triangle’). Project success is more difficult to quantify and is often an amalgamation of the perceived benefits of a project where such benefits may be assessed differently by different observers or stakeholders. The term ‘outcome’ has been deliberately selected because projects underpinned by business cases are seeking benefits from those projects, delivered within the parameters of time, cost and scope. In other words, use of the term ‘outcome’ within the context of this study is a conflation of project management success and project success. Hence, the specific questions this study seeks to answer are:

  1. Does the use of Scrum improve the likelihood of an IT project completing within budget? Henceforth described as cost control.
  2. Does the use of Scrum improve the likelihood of an IT project completing on time/schedule? Henceforth described as schedule control.
  3. Does the use of Scrum improve the likelihood of an IT project delivering the scope of outputs identified in a business case, and does it reduce the likelihood of scope creep from occurring? Henceforth described as scope control.
  4. Does the use of Scrum improve the quality of the outputs of an IT project? Henceforth described as quality control.
  5. Does the use of Scrum improve the likelihood of an IT project delivering the benefits nominated in the business case? Henceforth described as benefits derivation.
  6. Are IT project stakeholders (users and sponsors) more satisfied with the outcomes of projects that are run with Scrum? Henceforth described as stakeholder satisfaction.
  7. Does the use of Scrum improve overall project outcomes? Henceforth described as overall project outcomes.

Answers to these questions are important as they may help inform the direction and nature of IT project management methodologies in the future and ultimately assist organisations investing in technology uplift to improve their investments.

The research study method follows along with its survey execution and statistical analysis of the data collected. A critical assessment of the analysis resulted in a range of findings, along with discussion, implications and concluding recommendations.

2. Literature Review

Giving consideration to the objectives of this research as indicated by the research model (Figure 1 below), a number of approaches were considered including, seeking the expert opinion of IT project participants or alternatively, undertaking a project-by-project analysis, comparing each project’s original objectives with its eventual outcomes.

Noting that the wholesale adoption of Agile methodologies commenced with the publication of the Agile Manifesto in 2001, and continues today, it was reasonably expected that within the community of practicing IT professionals, many participants may have experienced a transition to Agile methodologies (including Scrum in particular), during the conduct of their career. Inevitably, those that have transitioned will at some point leave the workforce, which will extinguish any opportunity that might exist today to harness their experiences. In designing this research therefore capturing the transition knowledge, it was decided to target IT project professionals as respondents to a quantitative survey instrument, as opposed to conducting a project-by-project analysis. The ambition was to recruit respondents with experiences in a broad range of project methodologies, that included Agile Scrum, in the expectation that such experiences would enrichen responses. A survey instrument was regarded as an acceptable means of gathering the quantitative data being sought (Edmondson and McManus 2007, p. 1160).

According to Edmondson and McManus (2007, p. 1158), field research exists on a continuum, from nascent to mature. The nature of this research and the emerging field of Agile Project Management could be described as in the nascent phase. There are still too many gaps and limitations in this study and the extant literature to claim otherwise.

2.1 Conceptual Model

The conceptual design for this study is depicted in Figure 1.

Figure 1: Research Model

This conceptual model encapsulates the specific matters seeking to investigate with this study; the hypotheses follow:

Table 1: Hypotheses

 
H1 Agile Scrum increases the likelihood that a project will stay within budget (cost control).
H2 Agile Scrum increases the likelihood that a project will complete to schedule (schedulecontrol).
H3 Agile Scrum increases the likelihood that a project will deliver intended scope and reduce the  likelihood of scope creep from occurring (scope control).
H4 Agile Scrum increases the likelihood that a project will improve the quality of outputs (qualitycontrol).
H5 Agile Scrum increases the likelihood that a project will deliver intended benefits (benefitsrealization).
H6 Agile Scrum increases the likelihood that a project improves stakeholder satisfaction (forusers and sponsors).
H7 Agile Scrum increases the likelihood that a project improves overall project outcomes

It was hypothesized that IT project practitioners who had experiences with both Agile and non-Agile methodologies may not hold the same view of the efficacy of Agile methodologies as those who’s experience is limited to Agile methodologies alone. Therefore, the conceptual model includes IT project practitioners’ experiences with Agile Scrum and non-Agile Scrum methodologies as moderators. The Project Practitioners’ age was also regarded as a moderator purely to understand whether more senior IT project professionals, who are arguably better placed to compare project outcomes of today with those prior to the introduction of Agile methodologies, hold a different view to less senior IT project professionals.

2.1.1 Survey Instrument

A survey questionnaire was devised for distribution which included questions to support the conceptual model. The research relied on the subjective assessment of IT project professionals with respect to the efficacy of Agile Scrum across seven measures, as outlined. According to Podsakoff, MacKenzie and Lee (2003), studies such as this are exposed to the risk of common method bias, which can have a deleterious impact on findings. A means of minimizing this risk was to seek anonymised responses. Considering this and other privacy requirements, an internet-based survey using the Qualtrics survey tool, was distributed via email incorporating an anonymised link to the survey. Respondents were made aware of this within the preamble of the survey itself. The survey was pre-tested with a small sample of respondents and feedback obtained prior to incorporating into the final version.

2.1.2 Definitions

Within the survey preamble, respondents were provided with definitions of “Agile Scrum” and “IT Projects” to contextualize their responses.

A project was to be considered an Agile Scrum project by respondents if it exhibited “most or all” of the following characteristics:

  1. Planned Project outputs of the project were contained within a list, commonly referred to as a product backlog.
  2. A product owner had overall responsibility for the product backlog.
  3. The outputs defined in the product backlog could change at the discretion of the product owner.
  4. Project outputs were delivered in an iterative manner, commonly referred to as series of short-term sprints (usually 2-4 weeks in duration).
  5. The planned outputs of any given sprint were established during a sprint planning session, at which time planned outputs from the product backlog were selected for inclusion in a sprint.
  6. Sprint teams were multi-skilled and self-organising.
  7. Day to day activities of sprint teams were facilitated by a Scrum Master.
  8. Sprint teams met briefly daily (often described as a stand-up).
  9. At these meetings, team members discussed what they had completed, what they were planning to complete and what challenges they may have been experiencing.
  10. At the end of each sprint, a sprint review was held in which the sprint outputs were presented to customer representatives.
  11. Feedback from customer representatives could result in the product owner changing the product backlog.

Similarly, participants were provided the following definition of an IT project when considering their responses:

  • The primary deliverables of the project were software and/or technology infrastructure.
  • The implementation of pre-developed software packages and/or Software as a Service (SaaS) applications were included within this definition.
  • A business case would have been prepared to establish the project’s parameters. As a minimum, the business case would have specified budget, schedule, scope, and benefits.

2.1.3 Survey Questions

Survey questions sought a binary comparison of Agile Scrum to all other project methodologies from respondents (a matter discussed within this study’s conclusions). Each question was a non-optional, multiple-choice questions permitting a single response and followed the pattern provided as per this example question:

“Has Agile Scrum improved the prospects of a project keeping to the budget specified in the business case?” Answer options:

  1. Yes, made things better
  2. No, made things worse
  3. Made no significant difference
  4. Not sure

This style of questioning was applied across all seven measures of project outcomes under consideration. To provide further granularity in the data being collected, three of the seven measures (scope control, stakeholder satisfaction and benefits realization) were each separated into two parts to ensure participants considered the differing dimensions of these measures when responding. These specific dimensions are provided in Table 2. Responses from each pair of questions were aggregated for subsequent analysis.

Table 2: Outcome Measures Consolidation

Scope Control Ability of Scrum to deliver the scope identified in the business case
Ability of Scrum to prevent scope creep from occurring
Stakeholder Satisfaction Ability of Scrum to enhance user satisfaction
Ability of Scrum to enhance sponsor satisfaction
Benefits Realization Ability of Scrum to deliver the benefits identified in the business case
Ability of Scrum to deliver benefits not identified in the business case

3. Results

The research recruited respondents through the membership of project management peak bodies, social media platform LinkedIn and employees of consulting organizations who provide IT resources to private and public sector organizations. The study was disrupted by the COVID-19 pandemic reducing the number of possible respondents. 88 respondents took part in the survey, quantitative data was extracted from Qualtrics and uploaded into SPSS for statistical analysis. Given one query from a respondent with respect to ambiguity re the question on scope creep, and noting others had already commenced the survey, the question was withdrawn from the analysis.

3.1 Respondent Demographics

Demographic data of the respondent pool is provided in the following diagrams. Tabulated data is included in the appendices.

Figure 2: Respondent job function

Respondents were asked to provide their most senior job qualification. 39% of respondents came from project leadership roles (project manager,program manager and program director). A further 19% came from senior IT leadership ranks (Head of project/program delivery, General Manager and CIO), while the balance of 41% were project team members (Figure 2). Figures 3 and 4 identify respondents’ experiences workingwith Agile Scrum and non-Agile Scrum methodologies. Of significance to this research is that only three respondents had no prior experience withAgile

Scrum, while only two had no prior experience with Non-Agile Scrum methodologies. The implication being that most respondents had relevant professional experiences across Agile Scrum and non-Agile Scrum methodologies upon which to base an informed comparison.

         Figure 3: Agile Scrum Experience                                   Figure 4: Non-Agile Scrum Experience

Respondents between the ages of 40 and 60   dominated the respondent pool which would seem consistent with the substantive representation ofproject leaders and senior IT leaders (Figure 5).

Figure 5: Age Group Distribution

Figure 6: Educational Background

The respondent pool was by and large highly trained with 85% of respondents holding a Bachelors’ Degree or higher.

Other observations of the respondent pool are as follows:

  1. 36 respondents held formal Agile qualifications including 25 Scrum Masters and six Scaled
  2. Of the 88 respondents, 85 lived in Australia, two in the UK and one in
  3. Eight respondents held AIPM RegPM certification, 18 held PMP Professional certification and 19 held Prince2 Practitioner
  4. There were seven AIPM members, 15 PMI members and six IIBA

3.2 Preliminary Results Analysis

Analysis commenced by examining the data with respect to overall results and ignoring the potential impact of moderators. The raw response counts, before any aggregation or consolidation are presented in Table 3.

Table 3: Raw respondent data

 

 

Project Measure

Yes, made

things better

Made

significantdifference

No, made

thing worse

 

 

Not sure

 

 

Total

Cost Control 22 34 21 11 88
Schedule Control 35 25 24 4 88
Business Case Scope 34 25 23 6 88
Scope Creep 7 11 24 4 46
Quality Control 44 28 8 8 88
Benefits Realization -Planned 35 34 6 13 88
Benefits Realization -Unplanned 41 21 4 22 88
User Satisfaction 43 27 8 10 88
Sponsor Satisfaction 40 21 13 14 88
Overall Outcomes 45 26 13 4 88

 

The following aggregations were then performed to align the data across the seven measures being investigated by this study:

  • Scope Control = Business Case Scope + Scope Creep
  • Benefits Realization = Benefits Realization Planned + Benefits Realization Unplanned
  • Stakeholder Satisfaction = User Satisfaction + Sponsor Satisfaction

The aggregated data expressed in percentages is presented in Table 5 and Figure 7.

Table 4: Agile Scrum Efficacy

 

 

Project Measure

Yes, made

things better

Made significantdifference No, made

things worse

 

 

Not sure

 

 

Total

Cost Control 25% 39% 24% 13% 100%
Schedule Control 40% 28% 27% 5% 100%
Scope Control 31% 27% 35% 7% 100%
Quality Control 50% 32% 9% 9% 100%
Benefits Realization 43% 31% 6% 20% 100%
Stakeholder Satisfaction 47% 27% 12% 14% 100%
Overall Outcomes 51% 30% 15% 5% 100%

 

Figure 7: Agile Scrum Efficacy

Noting the objectives of this research, which were to determine whether the use of Agile Scrum was improving IT project outcomes, “No, madethings worse” responses were consolidated with “made no significant difference” responses to facilitate the intended binary analysis. The consolidated data is presented in Table 5 and Figure 8.

Table 5: Agile Scrum Efficacy (consolidation of “no significant difference” and “made things worse”)

 

Project Measure

Yes, made

things better

No Significant difference or worse  

Not sure

 

Total

Cost Control 25% 63% 12% 100%
Schedule Control 40% 56% 4% 100%
Scope Control 31% 62% 7% 100%
Quality Control 50% 41% 9% 100%
Benefits Realization 43% 37% 20% 100%
Stakeholder Satisfaction 47% 39% 14% 100%
Overall Outcomes 51% 44% 5% 100%

 

Figure 8: Agile Scrum Efficacy (consolidation of “no significant difference” and “made things worse”)

The data indicates that most respondents held the view that Agile Scrum did not improve project outcomes across the measures of cost control,schedule control and scope control. However, the reverse is true across the measures of quality control, benefits realization, stakeholder satisfactionand overall outcomes.

Given the dichotomous outcomes, a univariate binomial test was utilized to determine statistical significance. The test examined whether there was a statistically significant deviation from an expected outcome in data that is dichotomous (Van Den Berg 2020). With respect to this study, theexpected outcome is somewhat subjective. What does ‘improving IT project outcomes’ mean, statistically? For the hypotheses to be found true,Agile Scrum must show some level of improvement in outcomes, that is, it is not appropriate to describe Agile Scrum as improving outcomes when50% of respondents believe it does while the other 50% doesn’t agree. Further, a characteristic of a univariate binomial test being applied to thedichotomous data contained within this study means that the probability of any of the seven hypotheses failing will increase with an elevation in the proportion of respondents who are expected to believe Agile Scrum is improving outcomes. This is akin to the elevation of the bar in a sportinghigh jump competition.

To avoid adjudication on what an improvement in outcomes might mean, three tests were conducted to understand the sensitivity of the data to the setting of an expectation level. The analysis considered proportions of 55% versus 45%, 60% versus 40% and 65% versus 35%. In practical terms this means that at the 55% versus 45% proportion, Agile Scrum must be identified to improve outcomes by 55% of respondent’s vs 45% of respondents (statistically significant) for the hypotheses to be found true. Likewise, for 60% versus 40% and 65% versus 35%. The analysis only included answers ‘Yes, made things better’ along with ‘No, made things worse’ and ‘No Significant Difference’ (which were aggregated together as previously discussed). ‘Not Sure’ responses were excluded from all three analyses. The results of the three binomial tests are provided inTable 6. Colour coding has been adopted to assist readability. Red has been used to indicate a failed hypothesis, green to indicate a passedhypothesis and amber as a threshold indicator (pass or fail).

Table 6: Univariate Binomial Tests

 

Binomial Test

Test 1 Test 2 Test 3
 

N

ObservedProportion TestProportion Sign.1-tail TestProportion Sign.1-tail TestProportion Sign.1-tail
Cost Control(H1) No betteror worse 55 0.71 0.45 0.000 0.40 0.000 0.35 0.000
Better 22 0.29 0.55 0.60 0.65
77 1.00
 ScheduleControl (H2) No betteror worse 49 0.58 0.45 0.010 0.40 0.001 0.35 0.000
Better 35 0.42 0.55 0.60 0.65
84 1.00
ScopeControl (H3) No betteror worse 61 0.73 0.45 0.000 0.40 0.000 0.35 0.000
Better 22 0.27 0.55 0.60 0.65
83 1.00
QualityControl (44) No betteror worse 36 0.45 0.45 0.546 0.40 0.211 0.35 0.041
Better 44 0.55 0.55 0.60 0.65
80 1.00
BenefitsRealization(H5) No betteror Worse 65 0.46 0.45 0.459 0.40 0.094 0.35 0.005
Better 77 0.54 0.55 0.60 0.65
142 1.00
StakeholderSatisfaction(H6) No betteror worse 69 0.45 0.45 0.492 0.40 0.102 0.35 0.005
Better 83 0.55 0.55 0.60 0.65
152 1.00
OverallOutcomes(H7) No betteror worse 39 0.46 0.45 0.438 0.40 0.138 0.35 0.020
Better 45 0.54 0.55 0.60 0.65
84 1.00

The propensity for any hypotheses to fail as the expected proportion of respondents who believe Agile Scrum affords advantages over othermethodologies is demonstrated in these results.

3.3 Results Discussion

3.3.1 Test 1

At a 55% vs 45% proportion, the statistical significance for the measures of cost control (H1), schedule control (H2) and scope control (H3) are all well below 0.05, therefore the hypotheses fail for these measures. Conversely, the statistical significance for the measures of quality control (H4), benefits realization (H5), stakeholder satisfaction (H6) and overall outcomes (H7) are all well over 0.05, therefore the hypotheses is found true for these measures.

3.3.2 Test 2

The results of Test 2 were similar to Test 1. At a 60% vs 40% proportion, the statistical significance for the measures of cost control (H1), schedulecontrol (H2) and scope control (H3) remain well below 0.05, therefore the hypotheses fail for these measures. Conversely, the statisticalsignificance for the measures of quality control (H4), benefits realization (H5), stakeholder satisfaction (H6) and overall outcomes (H7) remainabove 0.05, albeit substantively reduced when compared with the results of Test 1. The hypotheses remain true for these measures. It’s worthnoting that the project success measures of benefits realization (H5), stakeholder satisfaction (H6) and overall outcomes (H7) are only marginallyabove being statistically significant (circa 0. 1 vs 0.05 for significance).

3.3.3 Test 3

Finally, at a 65% vs 35% proportion, the statistical significance of all seven measures fell below 0.05, therefore the hypotheses fail across all seven measures.

Overall results of the three tests are summarized in Table 7.

Table 7: Sensitivity Analysis

 

Hypothesis

Test 1

55% vs 45%

Test 2

60% vs 40%

Test 3

65% vs 35%

Hypothesis Pass / Fail
Cost Control (H1) Fail Fail Fail
Schedule Control (H2) Fail Fail Fail
Scope Control (H3) Fail Fail Fail
Quality Control (H4) Pass Pass Fail
Benefits Realization (H5) Pass Pass Fail
Stakeholder Satisfaction (H6) Pass Pass Fail
Overall Outcomes (H7) Pass Pass Fail

3.4 Benefits Realization – Further Examination

One final analysis was undertaken to examine the impact of aggregating the results of ‘Benefits – Planned’ with ‘Benefits – Unplanned’ to assess H5 (benefits realization). To undertake this assessment, ‘Benefits – Unplanned’ was removed from the data and the binomial analysis was re-ran.The results are presented in Table 8.

Table 8: Binomial Test: Benefits Planned

 

Binomial Test

 

N

ObservedProportion Test

Proportion

ExactSignificance

(1-tailed)

 

Benefits Planned   (H5)

Group 1 No better or worse 40 0.53 0.45 0.091
Group 2 Better 35 0.47 0.55
Total 75 1.00
Benefits Planned(H5) Group 1 No better or worse 40 0.53 0.40 0.013
Group 2 Better 35 0.47 0.60
Total 75 1.00

 

After taking away the effect Agile Scrum has on the delivery of unplanned benefits, H5 is left perilously close to failing at the lowest level of proportion of 55% versus 45%, and then fails at a proportion level of 60% vs 40%. The conclusion thus is that respondents did not feel particularlyconvinced of Agile Scrum’s ability to deliver the benefits nominated in the business case relative to other methodologies, leaving the measure of benefits realization close to the measures of cost control, schedule control and scope control.

3.4.1 The Effect of the Theorized Moderators

The theorized impact of three moderators on the statistical significance of these findings were considered:

  1. The experience of the respondent with Agile methodologies and
  2. The experience of the respondent with non-Agile methodologies
  3. The age of the respondent

In plain language, is an IT project professional with substantive Agile Scrum experience likely to consider Agile Scrum more effective? Is an ITproject professional having substantive experience with Non-Agile Scrum methodologies likely to consider Agile Scrum more effective? Finally, do less experienced professionals consider Agile Scrum to be more effective?

Noting that each of the seven hypotheses have a dichotomous outcome, a binary logistic regression was selected to test the effect of the proposedmoderators. This type of statistical analysis is designed to predict the relationship between a dependent variable, that can have 1 of 2 outcomes, based on two or more co-variates (Dodson, Hammett, and Klerx 2014, p. 205). The co-variates included in this analysis are provided in Table 9.

Table 9: Binary Logistics Analysis Covariates

1.  IT project professionals with more than 5 years Agile Scrum Experience
2.  IT project professionals with more than 5 years non-Agile Scrum experience
3.  IT project professionals under the age of 50 years of age.

 

Accuracy of the analysis requires that co-variates to not be correlated. To confirm this, the three intended co-variates were checked forcorrelation. The results of the correlation check are provided in Table 10.

Table 10: Covariate correlation check

 

 

Under 50

5 or more

years of

Scrum exp.

5 or more years ofnon-

Scrum exp.

 

Under 50 years ofage

Pearson Correlation 1 0.025 -0.196
Sig. (2-tailed) 0.818 0.067
N 88 88 88
 

5 or more years ofScrum exp.

Pearson Correlation 0.025 1 0.227
Sig. (2-tailed) 0.818 0.034
N 88 88 88
 

5 or more years ofnon-Scrum exp.

Pearson Correlation -0.196 0.227 1
Sig. (2-tailed) 0.067 0.034
N 88 88 88

 

As there were no significant correlations between the co-variates (> 0.5), a binary logistic regression was undertaken with all three co-variatesconcurrently across all the project success measures incorporated within the research data. The results are presented in Table 11.

Table 11: Binary Logistic Regression per Moderator

B S.E. Wald Sign.
 

Cost Control

5 or more years of Scrum 0.826 0.575 2.060 0.151
5 or more years of Non-Scrum -0.596 0.746 0.639 0.424
Under the Age of 50 0.265 0.528 0.251 0.616
ScheduleControl 5 or more years of Scrum 0.707 0.499 2.004 0.157
5 or more years of Non-Scrum -1.316 0.667 3.890 0.049*
Under the Age of 50 0.330 0.465 0.505 0.478
ScopeControl 5 or more years of Scrum 0.257 0.502 0.262 0.608
5 or more years of Non-Scrum -1.328 0.697 3.636 0.057
Under the Age of 50 0.763 0.474 2.592 0.107
 

Scope Creep

5 or more years of Scrum 1.252 1.272 0.969 0.325
5 or more years of Non-Scrum -3.657 1.308 7.812 0.005*
Under the Age of 50 -0.480 1.117 0.185 0.667
QualityControl 5 or more years of Scrum 0.768 0.484 2.513 0.113
5 or more years of Non-Scrum -0.535 0.668 0.642 0.423
Under the Age of 50 0.254 0.465 0.298 0.585
BenefitsPlanned 5 or more years of Scrum 0.903 0.541 2.791 0.095
5 or more years of Non-Scrum -1.065 0.756 1.989 0.158
Under the Age of 50 0.674 0.489 1.900 0.168
BenefitsUnplanned 5 or more years of Scrum -0.103 0.552 0.035 0.852
5 or more years of Non-Scrum 0.243 0.771 0.099 0.753
Under the Age of 50 0.263 0.526 0.250 0.617
UserSatisfaction 5 or more years of Scrum 0.770 0.482 2.553 0.110
5 or more years of Non-Scrum 0.186 0.677 0.075 0.784
Under the Age of 50 0.662 0.478 1.916 0.166
StakeholderSatisfaction 5 or more years of Scrum 0.691 0.500 1.913 0.167
5 or more years of Non-Scrum -0.344 0.758 0.206 0.650
Under the Age of 50 0.235 0.474 0.245 0.620
OverallProjectOutcomes 5 or more years of Scrum 0.895 0.489 3.352 0.067
5 or more years of Non-Scrum -1.576 0.755 4.362 0.037*
Under the Age of 50 0.242 0.461 0.276 0.599

 

Explanatory notes:

B” enumerates the ability of the dependent variable to predict the independent variable. “S.E.” is the standard errors associated with the coefficients. “Wald” provides the Wald chi-square value and “Sign.” is a 2-tailed p-value used to test whether the hypothesis is statistically significant.

3.4.2 Binary Logistics Regression Analysis Summary

A statistically significant result (Sign. < 0.05) was found in just three instances, all pertaining to respondents who had more than five years non-Agile Scrum experience. In comparison to the univariate binomial analysis discussed previously, those with more than 5 years non-Agileexperience are negatively and statistically significant associated with the positive perceptions of schedule control, scope creep and overall projectoutcomes. In plain terms, this cohort’s perceptions harmonize with the findings of the univariate binomial analysis concerning schedule control and scope creep but diverge with respect to overall project outcomes.

The results here are somewhat confounding, given the cohort of respondents with 5 years or more of non- Agile experience believed that Agile Scrum afforded no benefits for schedule and scope measures. Why would the same cohort have not felt similar regarding cost, particularly given their negative perception toward overall project outcomes. Further, the lack of other statistically significant outcomes across these tests more generally is also curious.

An explanation could be that the examined moderators by and large play no role in the dependant variables (being the seven measures of project outcomes). A further possible explanation may lie in the limited sample size (88). There is evidence within the extant literature indicating thatbinary logistic regression analyses can derive misleading results with ‘small’ sample sizes (Jewell 1984; Genell et al 2009). Therefore, it may be inappropriate to draw conclusions concerning the moderators examined herein, hence no further comment will be made regarding their existence in this study.

3.5 Analysis and Findings

With respect to the measures of cost control, schedule control and scope control, this study found that Agile Scrum offers no benefit, even at modest levels of comparison to alternate methodologies. Across the measures of quality control, benefits realization, stakeholder satisfaction and overalloutcomes, the respondent cohort perceived Agile Scrum extends an improvement in outcomes, but by a thin margin. Importantly, no evidence was found that Agile Scrum had adversely impacted project outcomes in any way Figure 08).

Empirical research undertaken in other studies have found that Agile methodologies confer benefits to traditional project management success measures (time, cost and scope), but a substantive proportion of these studies are potentially impacted by unintended bias as they targeted self-identifying Agile practitioners and members of Agile special interest groups as respondents (Ghani and Bello 2015, Kapitsaki and Christou 2014, Papatheocharous and Andreou 2013, Senapathi and Srinivasan 2014, Tripp 2012). Other studies finding benefits of Agile delivery methodologies do not provide sufficient data to clearly identify whether or not that study was likewise potentially impacted (Lei et al 2017, Rodriguez et al 2012, Yli-Huumo, Taipale and Smolander 2014, Serrador and Pinto 2015).

Just a single study, with a sample size of 7, linked benefits to traditional project management success measures from the adoption of agile methodologies in which the respondent pool seems to include practitioners possessing both agile and non-agile delivery methodology experience (Kamei et al 2017). The results of this study stand unambiguously in contrast to the studies noted. There are several possible explanations for this disparity. Firstly, this study is more recent than those cited above and time/experience with Agile methods, including Scrum may be shifting opinions. Secondly, unintended bias may have impacted the results of the aforementioned studies, wherein this study has mitigated against that possibility to the extent possible by the very nature of the respondent pool, which was dominated by practitioners having experience with both agile and non-agile methodologies (see Figures 3 and 4).

The findings of this study do not suggest that organizations who have transitioned their IT delivery methodology to Agile Scrum have wasted their time and money as they can expect some gains across the measures of quality control, benefits realization, stakeholder satisfaction and overall outcomes; but they may be disappointed with gains across the traditional ‘iron triangle’ measures of project success.

These results suggest that the benefits of adopting Agile Scrum would be modest. A transition to Agile Scrum requires a substantive investment in time, cost and change management to complete, particularly in large complex organizations that are operating globally. Scrum’s successful implementation in general requires structural change to be implemented not just at the project delivery level, but more broadly across the project management function, including governance forums that approve capital funding and Project Management Offices who oversee project portfolio health. Moreover, Agile Scrum mandates increased engagement with the stakeholders for whom the projects are being delivered on behalf of. The smooth and successful conduct of Agile Scrum projects therefore requires stakeholder groups to be familiar with, culturally adapted to and trained in the methodology. Indeed, organizational cultural change may be the most important factor in the successful implementation of Agile methodologies (Cooke, 2012, p. 125). The magnitude of the investment for an organization to successfully transition its IT project delivery methodology from a traditional project management approach to Agile Scrum therefore needs to be weighed up against the gains that may be achieved. Particularly where such gains do not include the more reliable execution of IT projects across their key measures of time, cost and scope.

Its perhaps of little surprise that Agile Scrum has provided no meaningful improvement in scope control when embedded within its methodology is the ability for the Product Owner, who represents the interests of stakeholders (users/sponsors), to modify a project’s backlog. Further, if a measure of discipline isn’t applied to the management of the product backlog, it is not difficult to foresee a consequential impact on both schedule and cost. The nexus that has historically existed between time, cost and scope is transformed in Agile Scrum projects. However, the positive impacts of Agile Scrum on the measures of stakeholder satisfaction are an outcome of scope being left untethered; stakeholders are getting more of what they want/need at the expense of time and cost. This study’s findings suggest evidence of an inverse relationship between the measures of scope control and stakeholder satisfaction. Perhaps an intentional shift by the framers of the Agile Manifesto, seeking to better balance project outcomes in favour of the people for whom the systems are being built.

While an improvement in measures such as quality of outputs and stakeholder satisfaction are not to be minimized, some 50 years since technology uplift has emerged as a critical organizational capability, a means by which IT projects can be relied upon to complete their ascribed scope within their approved time and cost parameters continues to be elusive. This remains a substantive concern, as ultimately, organizations seeking to benefit from their capital investments make decisions and expect outcomes that are underpinned by fiscal measures.

The importance of fiscal stewardship cannot be overstated. Many organizations annually set aside capital funding pools from which projects are prioritized. Decisions often pivot from one project to another based upon a project’s estimated time, cost, and scope parameters, along with an expected return on investment. Boardrooms making these decisions are effectively operating blindly. One is left to ponder whether projects that run 200% over budget, as was found by Flyvbjerg and Budzier (2011) to occur routinely, would have been approved for execution, at the expense of other projects, had their costs been accurately reflected in the business cases relied upon for their approval.

Agile methodologies may confer a range of benefits on technology projects but their apparent inability to address key project traditional iron triangle measures is a clear limitation. This study is not suggesting that a return to plan-based methodologies will solve all project woes; that would be too simplistic. Nor does it profess to having identified a remedy from this investigation. These findings suggest however that further research into the practice of IT project delivery would be well served by focussing on understanding the naggingly persistent nexus between project estimates and expectations, and eventual outcomes.

4. Limitations

The decision to specifically target Agile Scrum as a methodology for analysis, as opposed to all and/or other Agile methodologies has some limitations. The question arises what were respondents comparing Agile Scrum to? Included within the gamut of possibilities are other Agile methodologies (e.g., Kanban), a continuous spectrum of hybrid methodologies and purely plan-based methodologies such as Waterfall. Agile Scrum’s efficacy, observing that it appears the predominant Agile methodology, should not necessarily be considered an indication of the benefits of other Agile delivery methodologies.

Further, this research while quantitative in nature, relied on the opinions of 88 practicing IT project professionals. The relationship between opinions and reality may not be a reliable one and should be taken into consideration when interpreting these results. Additionally, the limited respondent pool has the effect of increasing statistical uncertainty within the binomial analyses in determining statistical significance. A larger respondent pool would have had the effect of reducing error bars.

5. Conclusion

This study has undertaken an investigation into the impact Agile Scrum has had on the practice of IT project delivery. The study findings suggest that Agile Scrum has not afforded improvement in the project outcome measures of cost control, schedule control and scope control but has resulted in a moderate improvement in the measures of quality control, benefits realization, stakeholder satisfaction, leading to overall project outcomes.

While Agile Scrum has conferred benefits in important dimensions of project outcomes, the need for projects to reliably deliver on the traditional iron triangle of measures (cost, schedule, and scope) remains of vital importance to those vested with the responsibility of apportioning capital expenditure. Gains in these metrics will remain elusive until the true nature of the factors that result in IT projects diverging from their baselined parameters are fully understood.

References

  • Cooke, J. Everything You Want to Know About Agile: How to Get Agile Results in a Less-than-agile Organization. Ely, Cambridgeshire: IT Governance Publishing, 2012, Ebook Central (ProQuest)
  • De Wit, A. Measurement of project success, International Journal of Project Management, 1988 vol. 6, no. 3, pp. 164-170 CrossRef
  • Dodson, B, Hammett, PC & Klerx, R. Binary Logistic Regression, John Wiley & Sons, Ltd, Chichester, UK, 2014, pp. 202-224.
  • Edmondson, AC & McManus, SE. Methodological Fit in Management Field Research, The Academy of Management Review, 2007, vol. 32, no. 4, pp. 1155-1179. CrossRef
  • Flyvbjerg, B & Budzier, A. Why Your IT Project May Be Riskier Than You Think’, Harvard Business Review, 2011, vol. 89, no. 9, pp. 23-25. CrossRef
  • Genell, A, Jonasson, J, Nemes, S & Steineck, G. Bias in odds ratios by logistic regression modelling and sample size, BMC Medical Research Methodology, 2009, vol. 9, no. 1, p. 56. CrossRef
  • Ghani, I & Bello, M. Agile adoption in IT organizations, KSII Transactions on Internet and Information Systems, 2015, vol. 9, no. 8, pp. 3231-3248. CrossRef
  • Jewell, NP. Small-sample bias of point estimators of the odds ratio from matched sets, Biometrics, 1984, vol. 40, no. 2, p. 421. CrossRef
  • Kamei, F, Pinto, G, Cartaxo, B & Vasconcelos, A. On the Benefits/Limitations of Agile Software Development: An Interview Study with Brazilian Companies, Proceedings of the 21st International Conference on evaluation and assessment in software engineering, 2017, vol. 128635, pp. 154-159. CrossRef
  • Kapitsaki, GM & Christou M. Where is Scrum in the current Agile World, ENASE 2014 – Proceedings of the 9th International Conference on Evaluation of Novel Approaches to Software Engineering, IEEE Xplore, 28-30 April 2014, Lisbon Portugal, pp. 101-108.
  • Lei, H, Ganjeizadeh, F, Jayachandran, PK & Ozcan, P. A statistical analysis of the effects of Scrum and Kanban on software development projects, Robotics and Computer Integrated Manufacturing, 2017, vol. 43, pp. 59-67. CrossRef
  • Papatheocharous, E & Andreou, AS. Evidence of Agile Adoption in Software Organizations: An Empirical Survey, 20th European Conference, EuroSPI 2013, Dundalk, Ireland, 25-27 June 2013, vol. 364, pp. 237-246.
  • Podsakoff, PM, Mackenzie, SB, Lee, J-Y & Podsakoff, NP. Common Method Biases in Behavioral Research: A Critical Review of the Literature and Recommended Remedies, Journal of Applied Psychology, 2003, vol. 88, no. 5, pp. 879-903. CrossRef
  • Rodriguez, P, Markkula, J, Oivo, M & Turula, K. Survey on agile and lean usage in Finnish software industry, Proceedings of the 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, 20-21 September 2012, Lund, Sweden, 39-148. CrossRef
  • Senapathi, M & Srinivasan, A. An empirical investigation of the factors affecting agile usage, EASE ’14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, Association for Computing Machinery, May 2014, London England United Kingdom, pp. 1-10. CrossRef
  • Serrador, P & Pinto, JK. Does Agile work? — A quantitative analysis of agile project success’, International Journal of Project Management, 2015, vol. 33, no. 5, pp. 1040-1051. CrossRef
  • Tiwana, A & Keil, M. The one-minute risk assessment tool, Communications of the ACM, 2004, vol. 47, no. 11, pp. 73-77. CrossRef
  • Tripp, JF. The impacts of agile development methodology use on project success’, Michigan State University (Doctoral Dissertation, 2012.
  • Van Den Berg, RG Binomial Test Simple Tutorial, SPSS Tutorials, https://www.spss- tutorials.com/binomial-test/ 2020.
  • Yli-Huumo, J, Taipale, O & Smolander, K. Software Development Methods and Quality Assurance: Special Focus on South Korea, 21st European Conference EuroSPI 2014, Communications in Computer and Information Science, 25-27 June 2014, Luxembourg, Berlin, Heidelberg, vol. 425, pp. 159-169. CrossRef
Share.

Comments are closed.