Audit of Information Technology Project Management

June 2, 2015

Agriculture and Agri-Food Canada (AAFC) Office of Audit and Evaluation

 

Executive summary

AAFC has established a standard Departmental Project Management Framework (DPMF), with the last major revisions in 2013, that encompasses the structure within which projects are initiated, planned, executed, controlled and closed. The Treasury Board Secretariat (TBS) Organizational Project Management Capacity Assessment (OPMCA), which measures the capacity for departments to deliver on their investments, rated AAFC at a level 3 (Evolutionary), on a scale from 0 to 4.

There is a Departmental Information Management and Information Technology (IM/IT) Portfolio Management Framework (the FrameworkFootnote 1) that aligns to the overall AAFC DPMF. The Framework defines a project as "a temporary endeavour of an activity or series of activities that has a defined beginning and an end. A project is required to produce defined outputs and realize specific outcomes in support of public policy objectives, with a clear schedule and budget. A project is undertaken within specific time, cost, and performance parameters. The temporary nature of projects stands in contrast to operations, which are repetitive and on-going work. Projects are undertaken in addition to operations to assist the department in implementing the strategic priorities."

The objective of the Information Technology (IT) Project Management audit was to provide assurance on the adequacy and effectiveness of the IT Project Management processes in place including benefit realization and outcomes.

The scope of the internal audit included an assessment of the overall IT project management governance processes and framework implemented by AAFC against Treasury Board requirements and leading practices. A sample of IT projects that were recently completed or currently in progress were selected for further examination to determine the extent to which the projects adhered to these requirements and leading practices.

As identified throughout the report, AAFC has taken a number of positive steps related to IT project management. The Framework that has been implemented for IT projects is comprehensive in nature and includes a consideration of all major aspects of project management. The audit noted that projects are completing the deliverables as outlined in the Framework. Furthermore, there is a well-defined governance structure that reviews and approves project deliverables at each project management gate as defined by the Framework.

In addition to the positive practices noted above, the audit found that there are opportunities for improvements in the current IT project management framework. The opportunities for improvement for AAFC's project management practices are related to: ensuring a portfolio approach is taken to the management of IT projects; ensuring the IT project management framework is flexible enough to ensure both the effective and efficient governance of IT projects, including those of varying complexity; and, formalizing the process to identify and measure the benefits realized from completed projects.

1.0 Introduction

1.1 Background

  • 1.1.1 AAFC has a standard Departmental Project Management Framework (DPMF), with the last major revisions in 2013, that encompasses the structure within which projects are initiated, planned, executed, controlled and closed. The Framework consists of five fundamental phases:
    • Phase 1 – Pre-Project Phase
    • Phase 2 – Initiation Phase
    • Phase 3 – Planning Phase
    • Phase 4 – Execution, Monitoring and Control Phase
    • Phase 5 – Close-out Phase
  • 1.1.2 The Treasury Board Secretariat (TBS) Organizational Project Management Capacity Assessment (OPMCA), which measures the capacity for departments to deliver on their investments, rated AAFC at a level 3 (Evolutionary) on a scale from 0 to 4Footnote 2. At an OPMCA of level 3, AAFC has the authority to initiate projects that are $1 million or greater whose individual complexity and risk are assessed through the Treasury Board Secretariat Project Complexity and Risk Assessment (PCRA) tool at levels of 3 (Evolutionary), 2 (Tactical), or 1 (Sustaining). However, for these projects, AAFC must still submit the PCRA to TBS for their review and acknowledgement.
  • 1.1.3 There is a Departmental IM/IT Portfolio Management Framework (the Framework) that aligns to the overall AAFC DPMF; however, due to the nature of the projects, the templates and tools utilized for asset-related and IT projects differ, but the deliverables, such as costs, schedules, etc. are intended to be equivalent. Each phase of the Framework includes specific deliverables and the end of each phase is marked by a gate, for which approval to move onto the next phase must be obtained through governance.
  • 1.1.4 The Framework defines a project as "a temporary endeavour of an activity or series of activities that has a defined beginning and an end. A project is required to produce defined outputs and realize specific outcomes in support of public policy objectives, with a clear schedule and budget. A project is undertaken within specific time, cost, and performance parameters. The temporary nature of projects stands in contrast to operations, which are repetitive and on-going work. Projects are undertaken in addition to operations to assist the department in implementing the strategic priorities".
  • 1.1.5 Based on AAFC's Investment Plan for Fiscal Years 2013-14 to 2017-18, supporting AAFC IM/IT products and services requirements necessitates an average annual investment of $18.2 millionFootnote 3. This funding is targeted for the development of business applications, as well as supporting an adequate and sustainable information technology end-user infrastructure essential for the delivery of the Department's programs and services.
  • 1.1.6 The Framework is supported within the Department by a formal project governance structure. At the time of the audit, the decision making body for all projects (both assets and IM/IT) was the Horizontal Management Committee (HMC), chaired by the Associate Deputy Minister and co-chaired by the Deputy Minister. Effective 1 April 2015, the Departmental Management Committee (DMC), chaired by the Deputy Minister will assume the role of decision making body for the investment plan. The Investment Planning Committee (IPC) was a Director General (DG)-level sub-committee of the HMC, co-chaired by the Chief Financial Officer (CFO) and the Chief Information Officer (CIO). The Project Review Committee (PRC) was a governance body specific for IM/IT projects, and chaired by rotating the Information Systems Branch (ISB) Director General with representation from key ISB service areas.
  • 1.1.7 Generally, governance includes review and approval at each gate from both the IPC and the PRC, although a tiered model for project governance has been implemented based on project cost. For those projects of lesser cost, there are fewer governance approval requirements, and the rigour of the deliverables is intended to be 'lighter'.
  • 1.1.8 While AAFC does not have a department-wide enterprise Project Management Office (PMO), the Information Systems Branch (ISB) has its own PMO that provides advice and support to IT projects within the Department. The PMO consists of one resource and assists projects in their adherence to the Framework, as well as collates and follows-up on project status reporting.

1.2 Audit objective

  • 1.2.1 The audit of IT Project Management was included for 2014-15, as part of the 2014-17 Risk-Based Audit Plan (RBAP). The objective of the IT Project Management audit was to provide assurance on the adequacy and effectiveness of the IT Project Management processes in place including benefit realization and outcomes.

1.3 Audit scope

  • 1.3.1 The scope of the internal audit included an assessment of the overall IT project management governance processes and framework implemented by AAFC against Treasury Board requirements (e.g. Treasury Board Secretariat Policy on the Management of Projects) and leading practices.
  • 1.3.2 A sample of IT projects recently completed or currently in progress was selected for further examination, to determine the extent to which the projects adhered to these requirements and leading practices. Projects were selected to ensure a representative sample across branches and across each of the project phases. The projects selected were:
    • Science and Technology Branch
      • Science Management Systems (SMS) - Monitoring & Reporting Data Collection Pilot & Deployment (MRDC) – At the time of the audit, this project was complete
      • Science Management Systems (SMS) - Monitoring & Reporting System (MRS) Integration – At the time of the audit, this project was not complete and in Phase 4 – Execution, Monitoring and Control Phase
    • Programs Branch
      • Agri-Invest Portal – At the time of the audit, this project was complete
      • Renewal of Advance Payments Program (APP) Electronic Delivery System (RAPPEDS) – At the time of the audit, this project was complete
      • Advance Payments Program (APP) Enhancements – At the time of the audit, this project was not complete and in Phase 4 – Execution, Monitoring and Control Phase
    • Information Systems Branch
      • Knowledge Workspace Collaboration Rollout (Knowledge Workspace) – At the time of the audit, this project was complete

1.4 Audit approach

  • 1.4.1 The overall IT project management governance process and framework implemented by AAFC was assessed against Treasury Board requirements (e.g. Treasury Board Secretariat Policy on the Management of Projects) as well as leading practicesFootnote 4. This was done through document review as well as interviews with ISB management, as well as those throughout the department in a project management governance or Project Management Office function involved with IT projects.
  • 1.4.2 The sample of IT projects that had been recently completed or currently in progress that were selected for further examination were assessed against the extent to which the projects adhered to requirements and leading practices. This was done through a review of project-specific documentation, as well as interviews with key project stakeholders.
  • 1.4.3 The audit approach and methodology was risk-based and consistent with the International Standards for the Professional Practice of Internal Auditing and the Internal Auditing Standards for the Government of Canada, as required under the Treasury Board Secretariat's Policy on Internal Audit. These standards require that the audit be planned and performed in such a way as to obtain reasonable assurance that the audit objective is achieved. The audit was conducted in accordance with an audit program that defines audit tasks to assess each audit criterion.
  • 1.4.4 Audit evidence was gathered through various methods including interviews, observations, walkthroughs, documentation review and analysis.

1.5 Conclusion

  • 1.5.1 The AAFC Office of Audit and Evaluation (OAE) concluded that AAFC has implemented a number of key controls related to IT project management, including the development of a comprehensive framework with oversight by governance structures and supported by a Project Management Office. The Framework supports effective requirements identification and engagement with the relevant business units, and includes requirements for project design, planning, and scheduling. Audit work noted that the Framework supports an effective post-implementation transition of projects to ongoing operations.
  • 1.5.2 In addition to the positive practices noted above, the audit found that there are opportunities for improvements in the current IT project management framework. The opportunities for improvement for AAFC's project management practices are related to: ensuring a portfolio approach is taken to the management of IT projects; ensuring the IT project management framework is flexible enough to ensure both the effective and efficient governance of IT projects, including those of varying complexity; and, formalizing the process to identify and measure the benefits realized from completed projects. These opportunities for improvement are presented in Section 2.0 of the report.

1.6 Statement of conformation

  • 1.6.1 In the professional opinion of the Chief Audit Executive, sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the conclusion provided and contained in this report. The conclusion is based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria that were agreed on with management. The conclusion is applicable only to the entities examined.
  • 1.6.2 This audit conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program.

2.0 Detailed observations, recommendations and management responses

  • 2.0.1 This section presents the key observations, based on the evidence and analysis associated with the audit, and provides recommendations for improvement.
  • 2.0.2 Management responses are included and provide:
    • An action plan to address each recommendation;
    • A lead responsible for implementation of the action plan; and,
    • A target date for completion of the implementation of the action plan.

2.1 Portfolio management approach to projects

  • 2.1.1 The audit expected that AAFC has adopted a portfolio based approach for project management with formally defined roles and responsibilities, which included ensuring there is the appropriate authority to ensure the consistent application of project management within an overall project portfolio framework.
  • 2.1.2 AAFC has implemented the foundational elements to allow for a portfolio based approach for project management. These foundation elements or positive practices include the implementation of a standard DPMF that encompasses the structure within which projects are initiated, planned, executed, controlled and closed. There is a Departmental IM/IT Portfolio Management Framework (the Framework) that aligns to the overall DPMF. The Project Management Office within the Information Systems Branch (ISB) provides advice and support related to adherence to the Framework. The PMO assists projects in ensuring they are completing the required templates and adhering to the Framework, vetting project documents for governance committees and senior management, as well as collecting monthly status reports from projects that are provided to the PRC for review. The formal status report template is comprehensive in nature, and outlines the status of the project budget and schedule, as well as project critical path milestones and project risks and issues.
  • 2.1.3 The PMO has limited formal authority, and consists of only one resource. The roles and responsibilities of the PMO are not formally outlined in the Framework. It was noted that some Project Managers work closely with the PMO and assign the PMO more of a project oversight role; however, not all Project Managers choose to work closely with the PMO and generally those projects have had more difficulty with following the Framework.
  • 2.1.4 Although the roles and responsibilities of the PMO are not formally outlined in the Framework, the Framework indicates that all Phase 3 project artifacts and deliverables must be submitted to the Project Front Office (PFO) for review prior to requesting governance approval. The PFO is composed of the PMO Manager and six manager-level representatives from various areas of the IM/IT domain. Similar requirements are not outlined for other phases.
  • 2.1.5 Given resource constraints and the lack of authority, there is limited ability for the Project Management Office to follow up with projects related to items such as late or incomplete status reports. It was noted that obtaining accurate status reports from projects in a timely fashion has been difficult. In a review of the status reports for the sample of projects selected for the audit, several discrepancies or potential missing information was noted, for instance:
    • The Science Management Systems (SMS) - Monitoring & Reporting System (MRS) Project – Manual overrides (i.e. changing status from yellow to green) were used on the status of items such as project budget and project issues; however, there was no evidence that these manual overrides were reviewed by project governance to ensure the ongoing justification of their use.
    • The Agri-Invest Portal Project – Manual overrides were used however, explanations of their use or changes in status were not included as part of the status report.
    • The Knowledge Workspace Collaboration Rollout (Knowledge Workspace) Project – 'Project issues' was assigned a status of yellow on the status report, and at the time there were delays in the project schedule and the project was over budget; however, there were no current issues listed for the project on the status report.
  • 2.1.6 Status reporting for all projects is a manual process and done through Microsoft Excel, and determining the correct version of the status report, or being able to easily summarize and analyze the results across projects to gain a portfolio view can be difficult.
  • 2.1.7 There is not a formal, centrally managed process within the ISB, through the PMO or other mechanisms, related to the assignment of Project Managers to IM/IT projects.
  • 2.1.8 Project management training was last provided to ISB management in 2011 in order to build awareness and a common understanding of project management. Interviewees did note that additional training and awareness on Project Management principals would be very useful, especially to those internal resources that are acting as a Project Manager or playing other key supporting roles in projects. Following the Framework and completing all required deliverables can be very difficult for inexperienced Project Managers, even though the PMO was identified as a very helpful and responsive resource, it was recognized that although the PMO provides assistance to any project that asks, there is a limited amount of time available for each project.
  • 2.1.9 AAFC has a Project Management Working Group co-chaired by ISB and the Science and Technology Branch (STB) that brings together Project Management staff from across the Department, with a mandate of knowledge sharing. It was indicated that many branches do not regularly have a representative attend.
  • 2.1.10 ISB resource requirements are planned on a quarterly basis. The initial IM/IT Plan, approved in April of each year, is reviewed and re-baselined every quarter taking into account new requirements and those ready to start based on ISB capacity, funding availability and client readiness. This approach is meant to ensure that the approved Plan is based on ISB's resource capacity to deliver. Even with this approach, projects may experience difficulty in resourcing specific skillsets and these resource issues are intended to be identified through the monthly project status reports. As outlined above, given resource constraints and the lack of authority, there is limited ability for the Project Management Office to follow up with projects related to items such as late or incomplete status reports. For example, the SMS MRS Project, noted that a project resource was not available for certain portions of the project due to competing requirements. It was identified by those interviewed, and through a review of project documentation, that ISB projects often compete for resources, and there can be many interdependencies related to resource requirements. Further to this, interdependencies between risks, common risks across projects, or the cumulative effect different risks may have on the portfolio of projects are also not formally tracked or monitored.
  • 2.1.11 Although lessons learned are developed at the close out phase by projects, it was noted that there are no formal mechanisms for discussing these lessons learned with projects or briefing projects on lessons learned from similar projects. Based on interviews, Project Managers were not aware of lessons learned experienced by similar projects, and therefore were not able to take advantage of these corporate learnings.
  • 2.1.12 The lack of a strong Project Management Office function increases the risk that projects will not be consistently managed or reported upon, and may lead to issues with project outcomes (budget and schedule) and scope. This is often the case if a portfolio approach to resource allocation and risk management is not implemented. This can lead to resource and skill gaps and/or delays related to project deliverables. Of the sample of the six projects reviewed, of which four have been complete, there were scheduling delays reported for three of these four completed projects, and budget overruns reported on two of the four projects.

  • 2.1.13 Recommendations

    Recommendation 1 – It is recommended that the Associate Deputy Minister (ADM), ISB consider moving towards a portfolio based approach for project management, for example, considering project resources, risks and lessons learned across the portfolio of IT projects. This includes ensuring more consistent and accurate project reporting, as well as additional oversight and training of Project Managers.

    Management Response: Agrees

    Action Plan:

    • 1a) ADM, ISB will move toward a portfolio based approach to project management by:

      1. Documenting the roles and responsibilities of the Portfolio Management function.

        Target Date for Completion: July 31, 2015 - Completed

      2. Compiling portfolio based observations/recommendations and presenting them along with the monthly portfolio report to ISB senior management (i.e. Project Review Committee) to enable portfolio based decisions and actions.

        Target Date for Completion: September 30, 2015 - Completed

    Responsible Leads: ADM, ISB Director General (DG), Strategic Management Directorate (SMD)

    • 1b) ADM, ISB will undertake more consistent and accurate reporting by:

      1. Including 'timely, accurate and consistent monthly project reporting' into ISB, Director General performance agreements.

        Target Date for Completion: June 30, 2015 - Completed

      2. Conducting information sessions and/or walkthroughs with Project Managers and Directors to convey and set expectations of monthly project dashboard reporting.

        Target Date for Completion: October 31, 2015 - Completed

    Responsible Leads: ADM, ISB DG, SMD

    • 1c) ADM, ISB will undertake additional oversight and training of Project Managers by:

      1. Documenting the PM oversight roles/responsibilities and communicating them to the PRC.

        Target Date for Completion: September 30, 2015 - Completed

      2. Providing a recommended project management training/skillset development path.

        Target Date for Completion: March 31, 2016

    Responsible Leads: ADM, ISB DG, SMD

    Recommendation 2 – It is recommended that the ADM, ISB ensure the mandate of the PMO is formally defined, including its roles and responsibilities, and that this is communicated throughout the Department.

    Management Response: Agrees

    Action Plan:

    • 2a) ADM, ISB to ensure that the mandate of the PMO is formally defined to be an efficient challenge authority by:

      1. Defining and documenting the mandate and associated operational capacity requirements of the PMO.

        Target Date for Completion: October 31, 2015 - Completed

      2. Communicating the PMO mandate to the IPC and within ISB (DG's, Directors, and Project Managers) as well as posting the documented mandate on the internal Departmental web site.

        Target Date for Completion: February 28, 2016

    Responsible Leads: ADM, ISB DG, SMD

2.2 Information management/information technology project management framework and governance

  • 2.2.1 The audit expected that AAFC's Departmental information management/information technology (IM/IT) Portfolio Management Framework (the Framework) was sufficiently flexible to ensure project governance was appropriate and timely based on the complexity of the IT project. Furthermore, it was expected that the Framework ensured factors such as the need to engage Shared Services Canada (SSC) were formally considered during the early phases of the project.
  • 2.2.2 The audit noted positive practices, including that the Framework is comprehensive in nature and includes a consideration of all major aspects of project management. This includes effective requirements identification and engagement with the relevant business units during project initiation, as well as requirements for project design, planning, and scheduling. The audit noted that projects are completing the deliverables as outlined in the Framework. Furthermore, the Framework supports an effective post-implementation transition of projects to ongoing operations. There is a well-defined governance structure in, and generally, governance includes review and approval at each gate as defined by the Framework for both the IPC and the Project Review Committee (PRC), although a tiered model for project governance has recently been implemented based on project cost. For those projects under $400,000 there are 'lighter' versions of some project artefacts that are required.
  • 2.2.3 In addition to the positive practices outlined above, there were also some noted drawbacks to the current governance structure, including the workload (the significant amount of project documentation that may need to be reviewed) and timing of governance committee meetings. Despite modification to the Framework to make it more flexible, the current project tiering approach remains based solely on the dollar value of the projects.
  • 2.2.4 Although decisions can be made secretarially, the fixed schedule (monthly) of the committee meetings could be made more responsive, such as organising meetings based on project needs. If the project plan has not appropriately considered the timing for project gating approval, projects may need to wait until the next committee meeting in order to obtain approval. The time required for gating approval requires not only the time for governance approval but the upfront time required to develop the deliverables and gain the input and approval of the business. For a project such as the Renewal of Advance Payments Program and Electronic Delivery Systems, which was being driven by legislative requirements that govern the terms of financial loans for agriculture producers, the project indicated it was hard to line up the completion of deliverables and governance approval dates, and consequently approval was sometimes obtained for certain deliverables after the fact. The risk of delayed approval of gating requirement deliverables has been included in the risk registers of some of the sample of projects reviewed, for instance the Payment Program and Electronic Delivery Systems Project.
  • 2.2.5 Based on a review of the records of decision examined for the sample of projects reviewed, projects appear to be rarely challenged at each project gate, and are not often sent back to further develop or modify the project planning, scope or assumptions that are presented, despite several of these projects ultimately experiencing delivery issues. Furthermore, although projects are required to provide monthly status reports, Project Managers indicated that there is little formal communication back to projects in relation to their status. For instance, project status reports indicate project issues that had due dates assigned to project resources that had subsequently passed prior to presentation to the committee, yet there was no evidence that additional action or follow-up was taken.
  • 2.2.6 The current gating requirements, especially related to the deliverables required and estimated project cost estimates, are not always aligned with the realities of contracting within the Department. For example, to proceed past the Planning Phase, detailed project planning with cost estimates are required, and these estimates are expected to be within 10% of the final project costs. It was noted that in some cases (2 out of 6 projects selected in the sampling) the internal resources or expertise to do such planning is not available, and therefore without engaging external resources, developing these cost estimates can be difficult. The funds to hire the external resources; however, are not available until the Planning Phase is approved.
  • 2.2.7 The Framework reviewed during the audit conduct phase included as part of Gate 2 requirements for projects to consult SSC on the infrastructure costs; and Gate 5 requirements included projects considering the impact of SSC on project schedules and timelines. A number of those projects included in the scope of the audit indicated risks and issues related to delays caused by SSC. For the SMS MRS Project, based on status reports and discussion with the Project Manager, there was an unforeseen delay due to SSC taking longer to perform a request related to a network change than expected. The risk of such delay was not taken into consideration at the onset of the project. For the Knowledge Workspace Project, key items had to be scoped out of the project due to unforeseen delays. Given recent revisions to the Framework, there is now a more formal intake process between SSC and AAFC, and a consideration of the impact of SSC requirements on AAFC project costs and timing.

  • 2.2.8 Recommendations

    Recommendation 3 – It is recommended that the ADM, ISB enhance the current IT project management Framework to:

    1. ensure the level of governance responds to the project risk level and provides flexibility based on project requirements; and,
    2. ensure that clients and partners (including SSC) are more engaged throughout all phases of project development and delivery.

    Management Response: Agrees

    Action Plan:

    • 3a) ADM, ISB will enhance the IT DPMF to ensure that the level of governance is mapped to the risk level of the project by:

      1. Creating and implementing a risk analysis assessment to be completed during the Pre-Project phase which will be used in recommending governance requirements to PRC for approval.

        Target Date for Completion: September 30, 2015 - Completed

    • 3b) ADM, ISB will ensure that clients and partners are more engaged by:

      1. Updating the IM/IT DPMF and associated gate checklists to include recommended client and partner (e.g. SSC) engagement touch points

        Target Date for Completion: August 31, 2015 - Completed

    Responsible Leads: ADM, ISB DG, SMD

2.3 Benefit realization

  • 2.3.1 The audit expected that tangible and measurable benefits of IT projects were identified during project initiation, revisited throughout the life of the project, and a formal process was established to ensure the actual benefits realized as a result of the completion of the project were determined, even for those benefits with a longer time horizon to be realized.
  • 2.3.2 The audit noted positive practices including the requirement for projects to consider benefits realization through the Departmental IM/IT Portfolio Management Framework (the Framework). The Business Case template includes a discussion of benefits including the cost-benefit analysis, financial benefits and non-financial benefits. The Project Charter template has a section for Goals, Objectives, and Business Outcomes and this section also contains guidance on benefit identification. A review of the sample of projects included in the scope of the audit indicated that projects are completing these sections of the template documents. In addition, project status reporting has been changed so that projects that have been moved to production but have not been properly closed out remain on the monthly status report that is maintained by the PMO and provided to the governance committees. The health of these projects is indicated as 'yellow' until they are properly closed out. This is intended to provide visibility to the governance committees on projects that have not been properly closed out. The above practices represent the foundational elements required for AAFC to further determine the value for money related to IT projects.
  • 2.3.3 Issues related to the timely close out of IT projects were observed for three of the four completed projects that were included in the scope of the audit. Two of the projects had significant delays between project completion and the formal close-out process; specifically, the Agri-Invest Portal Project was completed in October 2013, but was not formally closed out until November 2014. A third project, the Science Management Systems (SMS) - Monitoring & Reporting Data Collection Pilot & Deployment (MRDC) Project was completed in June 2014 but had yet to be formally closed out as of February 2015.
  • 2.3.4 Although projects included in the scope of the audit had identified benefits during the initiation and planning phases; they were not consistent in their approach. The expected benefits for a project are not always fully articulated during the initiation and planning phases of a project. This includes translating benefits into tangible and measurable goals, making it difficult to determine if projects are achieving value for money.
  • 2.3.5 As part of project close out, projects are reporting on the results of the performance measures that were identified at project initiation; however, some of the performance measures are not tangible or easily measurable, resulting in less effective measurement. Furthermore, based on discussion with Project Managers, benefits are only being measured at project close out; despite some projects having indicated the realization of benefits would be over a much longer time period, for example, more than a year after the project was closed. There is no evidence of formal planning or engagement with the business on ensuring that the measurement of benefits would be ongoing after project close out, and reported back to the IPC and appropriate governance bodies.
  • 2.3.6 A summary of the benefits identified and subsequently measured for the four completed projects is provided in the subsequent paragraphs.
  • 2.3.7 The Science Management Systems (SMS) Monitoring & Reporting Data Collection (MRDC) Project defined benefits/outcomes at a high level, such as "Reduced administrative burden" and "Increased reporting coverage & quality"; however, these were not tied to tangible or easily measurable goals, making the determination of benefits realization difficult. Furthermore, although the project has been completed since June 2014, per the Project Manager, it has yet to be formally closed out or the benefits/outcomes revisited.
  • 2.3.8 The Agri-Invest Portal Project defined benefits/outcomes in project documentation that was tied to tangible and measurable goals that included a detailed performance measurement method. This included benefits such as "Reduction in requests for duplicate copies of statements" and "Reduce paper burden for program participants" that were tied to potential cost reductions in the running of the program. At project close out only one of the benefits was revisited, related to the project enabling all hours access to program information for program participants, and this was measured by reviewing the number of after hour portal visits by participants. The other benefits identified were indicated as having a realization time of 24 months. Based on discussions with the Project Manager, there is no formal plan to revisit these benefits at the 24 month mark or report results back through the project governance structure.
  • 2.3.9 The Renewal of Advance Payments Program (APP) Electronic Delivery System (RAPPEDS) Project defined benefits/outcomes in project documentation, this included ensuring business process changes were made to the system that were required given legislative changes, simplifying business processes and rules where possible, and increasing the timeliness and accuracy of data. These benefits were revisited at project closeout, through reporting that user satisfaction had increased; however, the benefits were not further quantified.
  • 2.3.10 The Knowledge Workspace Collaboration Rollout (Knowledge Workspace) Project identified outcomes/benefits in project documentation; however, it was noted that many of these were outcomes related to training and not specifically related to benefits achieved through the implementation of Knowledge Workspace. At project close out, only an initial measurement of the originally identified benefits was conducted given many of the measurements were related to functionality that although anticipated during project initiation and planning it was not implemented through the project execution phase. The Records Centre (for records management functionality) and other related functionality was scoped out of the project.
  • 2.3.11 The Knowledge Workspace project was intended to be rolled out to all employees but it was decided to focus on 'horizontal committees' instead. This impacted the benefits measurement at project close out. For instance, an initial benefit defined was 'Increased Productivity', but at project close out the project indicated that with only 350 users as of mid-February, 2014Footnote 5, an employee survey at that time would not bear accurate results of productivity gains. It was indicated that "an employee survey should be initiated at the mid-way point of FY 14/15 when there will be approximately 2,000 users on Knowledge Workspace."Footnote 6 Through discussions with IM Services, such a survey or other performance measurement is not currently occurring, and a benefits measurement plan is being drafted but is not yet in place. AAFC currently is only measuring the total number of employees that are a member of at least one SharePoint site (1,700 out of a total population of 5,000) and the number of documents in SharePoint (a total of 39,000)Footnote 7. AAFC is currently not measuring what employees are doing on SharePoint (if they are active, how and what documents may be posted, etc.).
  • 2.3.12 The Knowledge Workspace end-date was extended from March 31, 2013 to December 31, 2013 and the project budget was $3.2 million, rebase lined to $3.4 million and the final actual budget was $3.6 million.Footnote 8, Footnote 9. Despite the increase in project length and budget there were a number of items scoped out, such as the Records Centre. Furthermore, given changes to the scope of the project, it is difficult to determine the overall value for money for the project, as some of the originally planned functionality and rollout is now being done as part of separate projects as well as ongoing activities. The Records Centre is now part of the Content Lifecycle Management Project which is expected to be done by September 2016 at a cost of $1.6 million. This is an expanded scope from that which was going to be done under the original project, as this now involves linking SharePoint to GCDocs as the Records Centre. The rollout of SharePoint was supposed to be done by project close, but given this was not achieved, this rollout was extended as an operational activity to be completed by end of March 2015; however, as of February 2015 it has only been deployed to 1,700 out of 5,000 employeesFootnote 10. IM Services is working on a deployment plan but could not confirm when all employees would have access.

  • 2.3.13 Recommendations

    Recommendation 4 – It is recommended that the ADM, ISB enhance the current governance and approval framework, to ensure measurable benefits are defined and approved by Project Sponsors during project initiation and planning phases, including defining the anticipated return on investment for each project. Mechanisms should be established to ensure Project Sponsors have considered the ongoing tracking and reporting of longer-term benefits after project completion. Furthermore, additional measures should be implemented to ensure project close out is completed in a more timely fashion.

    Management Response: Agrees

    Action Plan:

    • 4a) ADM, ISB will ensure that projects are closed out in a timely manner by:

      1. Including in ISB, DG performance agreements a commitment that project closeout reports will be provided for approval to IPC within two months of the completion of the Execution phase.

        Target Date for Completion: June 30, 2015 - Completed

    • 4b) ADM, ISB will ensure measurable benefits are defined and approved by:

      1. Creating a guidance document to assist Project Sponsors in identifying and establishing measurable benefits in a consistent manner and reference this document in the IM/IT DPMF.

        Target Date for Completion: September 30, 2015 - Completed

    • 4c) ADM, ISB will undertake additional oversight and training of Project Managers by:

      1. Modifying the project close-out report to include a section that will identify for the IPC forward agenda when post project benefits realization will be measured and reported on by the Project Sponsor.

        Target Date for Completion: November 30, 2015 - Completed

    Responsible Leads: ADM, ISB DG, SMD

Annex A: Audit criteria

Line of Enquiry 1

  • 1.1 There are project governance structures to ensure that informed decisions are made, in the correct timeframes, by the appropriate individuals or groups, and to ensure the success of projects. This includes project benefits being clearly documented and an effective approach is in place to track against the realization of these benefits.
    • 1.1.1 The project management framework supports an appropriate approach for project governance and oversight based on the size and complexity of the project, including governance bodies providing useful and timely direction and approval.
    • 1.1.2 The project management framework includes an appropriate project gating methodology in line with Treasury Board Secretariat requirements and industry standards.
    • 1.1.3 The project management framework supports an appropriate approach for identifying, evaluating and measuring the benefits of a project to the organization.

Line of Enquiry 2

  • 2.1 Appropriate project management processes and controls are implemented to ensure projects can deliver against their objectives, timelines and budgets through project plans and schedules. This includes a framework in place to ensure relevant business units provide adequate support to applicable projects to support its effective delivery.
    • 2.1.1 The project management framework ensures projects have documented an appropriate project design, plan and schedule. Projects are assessed against approved plans and schedules and corrective action taken as required.
    • 2.1.2 The project management framework supports effective requirements identification and engagement with the relevant business units and other stakeholders, including appropriate change management and awareness.
    • 2.1.3 The project management framework supports an effective post-implementation transition of the project to ongoing operations.

Line of Enquiry 3

  • 3.1 Roles and responsibilities for projects are clearly defined within AAFC, and supported by a defined resource model including the appropriate use of contractors, which includes an integrated plan for the project and use of resources across the enterprise.
    • 3.1.1 The project management framework supports appropriate project resource planning/selection to ensure project resources have adequate skills and experience based on the nature of the project.
    • 3.1.2 The project management framework supports process and ensures project resources are appropriately deployed to the project, including those from business units.
    • 3.1.3 The project management framework defines accountability, including roles and responsibilities for those resources involved in the project, include the role of the Project Manager, contracted resources, and other project staff, including those from the business.

Line of Enquiry 4

  • 4.1 There is a framework in place to ensure project risks are appropriately identified, captured, reported, and mitigation plans developed. This includes project assumptions being robustly validated and, where required, captured as project risks with appropriate mitigation measures.
    • 4.1.1 The project management framework includes an approach to identifying, mitigating and managing project risks and issues.
    • 4.1.2 The project management framework supports the development of robust and validated assumptions that are revisited and updated throughout the life of the project.

Annex B: List of Acronyms

AAFC
Agriculture and Agri-Food Canada
ADM
Assistant Deputy Minister
APP
Advance Payments Program
DG
Director General
DPMF
Departmental Project Management Framework
CMB
Corporate Management Branch
COBIT
Control Objectives for Information and Related Technology
HMC
Horizontal Management Committee
IMIT
Information Management and Information Technology
IPC
Investment Planning Committee
ISB
Information Systems Branch
IT
Information Technology
MRDC
Monitoring & Reporting Data Collection Pilot & Deployment
MRS
Monitoring & Reporting System
OAE
Office of Audit and Evaluation
OPMCA
Organizational Project Management Capacity Assessment
PCRA
Project Complexity and Risk Assessment
PFO
Project Front Office
PMBOK
Project Management Body of Knowledge
PMO
Project Management Office
PRC
Project Review Committee
RAPPEDS
Renewal of Advance Payments Program Electronic Delivery System
RBAP
Risk-Based Audit Plan
SMS
Science Management Systems
SSC
Shared Services Canada
TBS
Treasury Board Secretariat
Date modified: