Internal Audit Report of IT Systems under Development - Phase 2
Advance Commercial Information - Electronic Data Interface Reporting for Air

October 2007

Table of Contents

• Executive Summary
• Introduction
  • Background
  • Objectives and Scope
  • Audit Criteria
  • Approach and Methodology
  • Results from Phase 1
  • Purpose
• Overview of ACI-EDI Reporting for Air
• Audit Findings
  • Technical Solution Development
  • Business Transformation
  • Authority, Responsibility and Accountability
  • Project Management Framework
  • Project Risk Management
  • Security Assessment
• Appendix A - Audit Criteria
• Appendix B - List of Acronyms

Executive Summary

The audit of Canada Border Services Agency (CBSA) systems under development was identified as a priority in the CBSA’s Risk-Based Multi-Year Audit Plan. This Plan was approved by the Internal Audit and Evaluation Committee on March 17, 2005.

The principal objective of this audit was to provide the necessary assurances to the CBSA that its integrated project management and system development practices and procedures for automated business systems under development are adhering to the internal policies and procedures that have been established by the Agency. A further objective of this audit was to identify any opportunities for improvement to the CBSA’s existing internal policies and procedures for managing its systems under development.

The audit was divided into three phases. Phase 1, reported in February 2007, provided an assessment of the adequacy and effectiveness of the CBSA’s current management control framework over systems under development. Phases 2 and 3 provide an assessment of the degree to which two selected system development projects are effectively and appropriately applying the control framework. This report presents the findings from Phase 2 of the audit that reviewed the management framework for the development of a selected system development project: Advance Commercial Information (ACI)-Electronic Data Interface (EDI) Reporting for Air. The audit was conducted by Interis Consulting and audit fieldwork was conducted between July and October 2006.

ACI-EDI Reporting for Air is one component of the ACI initiative, which is a multi-year multi‑million dollar initiative launched under the Customs Action Plan in 2000 prior to the formation of the CBSA in December 2003. ACI provides the CBSA with pre-arrival data regarding commercial shipments entering Canada to identify high-risk commercial cargo and shipments requiring further scrutiny on arrival. A multi-modal phased approach was taken for the ACI development. ACI has been operational in the marine mode since April 2004 and became operational in the air mode in December 2005 (with a phased-in implementation to July 2006). Implementation of the final modes, highway and rail, is ongoing through the eManifest project.

The audit objective for Phase 2 was to assess the development of the electronic reporting of advance information for the air mode. The project was developed by the Innovation, Science and Technology Branch (ISTB) with sponsorship from the Enforcement, Admissibility and Operations branches.

Audit Findings

Based on the audit work conducted in Phase 2, it is concluded that ACI-EDI Reporting for Air was developed in accordance with the CBSA policies, procedures and methodology in place at the time and the functionality of the application is working as intended. Many of the observations made in Phase 1 of the audit were supported by the audit work in Phase 2. In addition, the audit noted that since the time of the ACI-EDI Reporting for Air development in 2004 and 2005, many new processes and governance practices have been developed and introduced for more recent projects.

The audit noted a number of strengths for the ACI-EDI Reporting for Air project including the use of a sound system development approach that included the following activities:

• The feasibility and viability of the solution were assessed.
• User requirements were defined with involvement from stakeholders.
• Testing was appropriately performed.
• Implementation of the ACI-EDI Reporting for Air was well managed. The carrier community was routinely informed of progress and provided with information and tools to ensure they were ready to electronically transmit their data.
• A formal structure and processes were put in place to support the external carriers.

Although many of the findings from the audit of ACI-EDI Reporting for Air are the same as those reported in the Phase 1 audit report, the following additional areas of control were identified in Phase 2 that could be strengthened at the project level: 

• Authority, Responsibility and Accountability -- These should be clearly defined at the outset of every system development project, and they were not clear for this project. The commitment from sponsors was not well understood, secured and documented at the project outset.
• Project Management Framework -- The Phase 2 audit identified that:
  • Planning and scheduling was not sufficiently integrated across all directorates of the ISTB to ensure that interdependencies and key dates were clearly understood and planned to meet scheduled release dates and improve efficiencies;
  • The change management process had not been fully defined and communicated to sponsors, including the expectations and authorities for approvals; and
  • The issues management process had not always been in place to consistently document, track and rank development issues for effective and efficient resolution.
• Security Assessment -- The role of the IT Security Group in the ISTB and the Departmental Security Office (DSO) should be clarified for assessing security requirements for individual projects. At the time of the system development, the IT Security Group in the ISTB was being established and the audit could not determine who in the CBSA was responsible for assessing, reviewing and approving the security requirements.

Management’s Response

1. Introduction

1.1 Background

The audit of CBSA systems under development was identified as a priority in the CBSA’s Risk‑Based Multi-Year Audit Plan. This Plan was approved by the Internal Audit and Evaluation Committee on March 17, 2005.

1.2 Objectives and Scope

The principal objective of this audit was to provide the necessary assurances to the CBSA that its integrated project management and system development practices and procedures for automated business systems under development are adhering to the internal policies and procedures that have been established by the Agency.

A further objective of this audit was to identify any opportunities for improvement to the CBSA’s existing internal policies and procedures for managing its systems under development.

To achieve these objectives, the internal audit plan was divided into the following three phases:

• Phase 1 -- A review of the CBSA’s management control framework (MCF) for the development of automated business systems.
• Phase 2 -- Audit of a selected system under development project.
• Phase 3 -- Audit of a second selected system under development project.

The audit deliverable for Phase 1 was the following:

• Provide an assessment of the adequacy and effectiveness of the CBSA’s current MCF; and
• Select two of the CBSA’s current system development projects for a more comprehensive audit review of their application of current controls and governance processes. The following two systems under development were selected:
  • Phase 2 -- Advance Commercial Information/Electronic Data Interface Reporting for Air (ACI-EDI)
  • Phase 3 -- Advance Passenger Information/Passenger Name Record (API/PNR) Risk Scoring

The audit deliverable for Phases 2 and 3 were the following:

• Provide an assessment of the degree to which the two selected system development projects are effectively and appropriately applying CBSA policies, procedures and methodology for systems development projects; and
• Provide recommendations for improving CBSA policies and procedures for future system development projects.

The scope of this audit was to assess the development of the ACI-EDI Reporting for Air and not the effectiveness of the application itself.

1.3 Audit Criteria

The audit criteria used for Phase 2 were developed in Phase 1 of the audit and were based on risk elements inherent to system development projects, as well as on public sector trends for the effective governance of system development project lifecycles. A detailed set of audit criteria were established, factoring in the primary influences that can significantly impact system development projects in the Government of Canada (GC): management structure and processes, system development lifecycle policies and standards, as well as planning and acquisition processes and procedures. These detailed audit criteria were used to assess CBSA practices.

It was noted during the initial audit planning that business system development projects at the CBSA were inherently exposed to risk as:

• The CBSA has undergone significant reorganization;
• The Agency has had to adopt and adapt separate system development processes, including separate information technology (IT) infrastructures, that were established in the legacy departments (primarily the former Canada Customs and Revenue Agency and Ciizenship and Immigration Canada); and
• There are aggressive delivery timelines imposed on the CBSA from outside sources.

Key control areas addressing inherent risks were identified based on the Treasury Board of Canada Secretariat’s (TBS) Enhanced Management Framework and the Control Objectives for Information and related Technology (CobiT) Framework issued by the IT Governance Institute. The audit criteria developed to assess the CBSA’s overall business practices, general controls and governance processes for its system under development projects were organized into the following six categories:

• Governance
• Business Benefits
• User Requirements
• Project Management
• Technical Solution
• Business Transformation

The audit criteria are presented in Appendix A.

1.4 Approach and Methodology

The approach and methodology were risk-based and compliant with the applicable TBS Internal Audit Policy. The audit was conducted in accordance with an audit program that defined the audit tasks to assess each criterion. Through interviews and documentation review, it assessed the current CBSA practices against the criteria and formally assessed the effectiveness of each practice. Interviews were conducted with various representatives of the different organizations within the Innovation, Science and Technology Branch (ISTB) and with representatives of the project sponsors, the Enforcement, Admissibility, Operations and Comptrollership branches.

The audit fieldwork for Phase 2 was conducted between July and October 2006.

1.5 Results from Phase 1

The results of the audit from Phase 1 were reported in February 2007. A number of strengths were noted in the controls over systems under development. These can provide the foundation for building a strong MCF over systems under development, namely a governance structure and a project management framework. Given the newness of the agency reorganization, many of the processes were still in early development and had not yet reached a mature state in the organization. New committees were being formed, roles and responsibilities were being clarified and specific deliverables were being developed. These activities are indicators that the Agency is making progress toward enhancing its control over systems under development.

The following key areas of control that should be strengthened were noted in the Phase 1 audit report:

• Project Management Framework -- This was not fully defined and did not include processes such as the following:
  • Gate approvals, where predefined gates are established for management to formally evaluate the continued realization of benefits and provide a “go” or “no‑go” decision;
  • Project status reporting on project cost tracking against budget; and
  • Ways of consolidating individual project schedules into one master schedule.
• Project Prioritization -- The process of realigning priorities and resources in-year for the portfolio of projects when new initiatives are introduced was insufficient given the Agency’s capacity limitations. The prioritization of projects continues to be a particular challenge for the CBSA that does not have full control over its priorities to drop or realign existing priorities given the external influences and pressures. 
• Sponsor Involvement -- Sponsors were not always sufficiently engaged in the system development process, particularly in the acceptance of the functional design, final testing and approvals of the system before it was put into production, as well as the formal review and acceptance of scope changes.

Management reviewed the audit report, provided management action plans to address its recommendations and expected completion dates for each action.

1.6 Purpose

The purpose of this document is to present the findings from Phase 2 of the audit.

2. Overview of ACI-EDI Reporting for Air

ACI-EDI Reporting for Air is one component of the ACI initiative, which is a multi‑year multi‑million dollar initiative launched under the Customs Action Plan in 2000 prior to the formation of the CBSA in December 2003. The Plan introduced a new, comprehensive risk‑management model of program delivery based on the principles of advance information and pre-approval. The Plan consisted of 17 initiatives, with proposed implementation over five years. The ACI initiative (formerly known as “Carrier Re-engineering”) was launched in 2000.

ACI is intended to provide the CBSA with more effective risk-management processes and the tools to more effectively manage commercial clients and their commercial shipments. The initiative provides CBSA officers with pre-arrival data regarding commercial shipments entering Canada. Advance access permits officers to analyze the information, with the assistance of an automated targeting and risk-assessment tool, to identify high-risk commercial cargo and shipments requiring further scrutiny on arrival.

ACI development was separated into two concurrent but distinct components:

1. To develop the electronic reporting of advance information.
2. To develop the risk-assessment tool and process.

The audit looked at the system development project that addressed the second component of ACI, specifically for the air mode. ACI has been operational in the marine mode since April 2004 and became operational in the air mode in December 2005 (with a phased-in implementation to July 2006). Implementation of the final modes, highway and rail, is ongoing through the eManifest [ 1 ] project.

The organizational structure of the ISTB [ 2 ] at the time that ACI-EDI Reporting for Air was being developed is presented on the next page.

CBSA / Innovation Science and Technology Organizational Structure

Under this structure, the ACI-EDI Reporting for Air project was managed by one project manager from the MPDD Directorate and by one IT project manager from the Border Systems Directorate. The project was sponsored by the Enforcement, Admissibility and Operations branches. Throughout the development of ACI-EDI Reporting for Air, there were personnel and organizational changes to the ISTB team, as well as in the branches, creating challenges in the development activities.

At the time of the development, there were many pressures on the CBSA and the ISTB. The operations of the Agency are highly dependent on information systems and technology, which are developed, implemented and managed by the ISTB. The scope of system development activity at CBSA is significant, with annual spending on development and maintenance of systems, infrastructure and related technology estimated to be in the $150-250 million range over the next few years. In response to increased North American security concerns following the terrorist attacks in September 2001, the GC introduced a considerable number of security-related initiatives under the umbrella of the Public Security and Anti-terrorism (PSAT) initiative and the Shared Border Declaration (SBD), resulting in the expedited development of initiatives. This put pressure on a number of system development projects including ACI-EDI Reporting for Air.

Although development work began on the ACI initiative prior to the formation of the Agency, the initial development of the scope for the ACI-EDI Reporting for Air project began in early 2004. At that time, the project scope included both air and rail modes. The rail mode was later scoped out and included in the eManifest project which is currently in development. Initial project planning activity for ACI-EDI Reporting for Air largely began after the marine mode was put into production in April 2004. [ 3 ] The initial planned implementation date (for both the air and rail modes) was May 9, 2005, [ 4 ] which was later revised to December 5, 2005. [ 5 ] The systems for ACI electronic reporting for air mode were put into production on December 12, 2005. [ 6 ] As a phased‑in approach was used, air carriers and freight forwarders that were not fully prepared to report electronically to the CBSA on December 12, 2005, were given until June 26, 2006, to fully comply with ACI reporting requirements.

The ACI initiative was funded from the PSAT and Manley-Ridge funding envelopes, as well as through internal departmental allocations. Although a budget was not officially established for the ACI-EDI Reporting for Air project, development costs were estimated to have been approximately $4.7 million. [ 7 ] This cost estimate includes all CBSA costs related to the development of ACI-EDI Reporting for Air from 2005-2006 and 2006-2007, with some costs related to development work on the marine mode, which could not be separated out for presentation.

3. Audit Findings

Based on the audit work conducted in Phase 2, the audit concluded that ACI-EDI Reporting for Air was developed in accordance with the CBSA policies, procedures and methodology in place at the time. Furthermore, the application is working as intended. However, opportunities exist to strengthen the system development processes used in ACI-EDI Reporting for Air to ensure adequate governance, risk management and control over future system under development projects.

Since the time of ACI-EDI Reporting for Air development in 2004 and 2005, many new processes and governance practices have been developed and introduced for more recent projects. In addition, many of the observations made in Phase 1 were supported by this Phase 2 audit. Management action plans for Phase 1 audit recommendations indicated that many audit areas needing strengthening have recently been addressed or are in the process of being addressed.

The audit findings are described in detail below.

3.1 Technical Solution Development

The audit found that the CBSA system development methodology was used across all phases of the project. The following key activities were carried out:

• The feasibility and viability of the solution were assessed.
• User requirements were defined with involvement from stakeholders.
• Testing was appropriately performed.

This project was developed using an industry-recognized methodology, the Rational Unified Process (RUP), which is an iterative process to achieve the final design and build [ 8 ]. The audit examined various documents that demonstrated good development practices for the technical solution, as expected in any system development process. The system design was described in a project scope document, an initiation phase process document and a high-level business requirements document. User requirements were clearly described and documented in business use cases. The requirements were then translated into a system component design document and system use cases for use by the system developers. The technical design was documented in an abbreviated technology design document and technical specifications document. Many of the documents prepared for the ACI-EDI Reporting for Air development were developed using templates that have since evolved and been replaced by new templates, but the key objectives of the documents remain the same. Key individuals from the ISTB were involved in the initial design and development phases. The Enterprise Architecture Group interfaced with the Business Architecture Group in the MPDD Direcorate and sponsor representatives to define the requirements and review the feasibility of the technical solution.

User requirements were defined with involvement from key stakeholders. The MPDD Directorate took a large role in determining the high-level requirements and priorities; however, joint application development sessions were held with sponsor representatives to develop the business use cases. The Contraband Program in Enforcement Branch provided functional guidance and subject matter expertise to the ACI-EDI Reporting for Air project. The Electronic Commerce Unit in the MPDD Directorate brought its experience in working with the carriers to the system developers. The audit found evidence of ongoing refinement of the business requirements, largely through e-mail communications and updates to project documents. To improve its process of developing user requirements, the ISTB has begun using “usability architects” who look at the interface and how people use the system. User requirements can then be further enhanced to make the system more "usable" for the users.

Testing of ACI-EDI Reporting for Air was performed in a number of different test environments before implementation. Development testing was performed by the IT group and included data and database integrity testing, function testing, systems integration testing, business cycle testing and user interface testing. Test cases were mapped out, schedules were defined, resources were assigned and test results were documented and summarized. After the development testing was completed, the MPDD Directorate conducted user acceptance (client) testing and operational testing. Individuals outside the directorate were invited to participate in the user acceptance testing.

Typically, once the development and implementation are complete, the system application is moved from development to production and is transferred to the Systems Operations Group in the MPDD Directorate for ongoing support. The move is well controlled through release management procedures. At the time of the audit, the ACI-EDI Reporting for Air project had not yet been transferred to Systems Operations. It is a normal practice within the ISTB to keep a system development project under the control of the development group for about a year after implementation to monitor initial operational problems before it gets moved to Systems Operations.

3.2 Business Transformation

Regular communication and consultation with the carrier community occurred. Implementation plans were communicated to the carrier community through periodic customs notices. Various working groups were established with the carrier community to share information on the implementation. A formal process was in place for carriers to first plan their implementation and then test their interface before they would be permitted to send data electronically. Testing packages were sent to the carriers for implementation. Conferences were held with the industry to inform it of the process. Implementation status and transmission start dates were regularly monitored by account managers in the MPDD Directorate assigned to liaise with specific carriers.

A formal structure and processes were put in place to support the external carriers. More specifically, carriers reporting electronically for air shipments are supported by the Electronic Commerce Unit (ECU), already in place in the MPDD Directorate. The unit is the first line of support for data transmission problems and if required, it escalates problems to the technology services group responsible for monitoring the external interface. A call centre approach is used to document and resolve problems from carriers. Carriers are provided with toll-free support during regular business hours. Support is critical since the ACI is deemed to be a high-availability application, which requires support outside the normal hours of service.

3.3 Authority, Responsibility and Accountability

A project charter was not developed to clearly identify the authorities and accountabilities of the different directorates in the ISTB and the project sponsors. The official sponsor was not clearly identified at the outset of the project. Project managers identified three different branches to the audit team as unofficial project sponsors: the Enforcement, Admissibility and Operations branches.This created an environment where there were many different masters. ACI was a broad Agency initiative that crossed a number of internal organizational boundaries and, as such, the MPDD Directorate took a lead role.

Sponsorship and engagement of the business units is seen as an ongoing challenge. As with all development projects, the ISTB and the sponsors have authorities and responsibilities to make decisions affecting the development project. For ACI-EDI Reporting for Air, the delineation of authorities and responsibilities was not clear. This may, at least in part, be caused by the historical structure whereby the MPDD Directorate was part of, and represented, the business group and held overall sponsor authority for all major system development projects. Since sponsors were not officially engaged at the project outset, interviewees indicated that the MPDD Directorate, out of necessity, assumed authority for many development decisions. Senior managers from the sponsor branches interviewed indicated that they did not sufficiently involve themselves early in the development largely due to competing priorities and other operational commitments. Directors general (DGs) from the sponsor branches were informed of the status of commercial system projects through established committee meetings, but their attendance was not constant. As the project evolved, sponsor branches became more involved in providing policy direction. They became particularly engaged late in the development when questions of policy were raised over when and where targeting would happen -- a fundamental component of ACI. This was a significant policy decision requiring extensive consultation that was not made early in the development, as was needed. A decision was made, just prior to implementation, to pilot two approaches to targeting, which will be analyzed later and a final policy decision made. For ACI-EDI Reporting for Air, roles and responsibilities at the development level in the ISTB were well understood. However, within the ISTB, accountability for successful delivery of the project was not clearly defined at the senior levels. There was no single senior manager responsible and answerable for all aspects of project delivery for the project. During this project, there were two DGs with shared accountability for project delivery:

• One in the MPDD Directorate responsible for all planning and project management for all system projects; and
• One in the Border Systems Directorate (IT) responsible for application development for all system projects.

Despite the lack of clarity around authorities and accountabilities, traditional roles and individual efforts ensured that the system was developed effectively. With the reorganization, IT was combined with the MPDD Directorate so that one DG was responsible for all aspects of project development and delivery for all commercial system projects. This should improve the accountability for system development delivery for future projects.

The audit noted that governance processes are evolving and improving in the CBSA. For more recent projects such as eManifest, the audit team was informed that a DG steering committee was established at the project outset to include senior management from all affected branches, including the regions. In addition, authorities, responsibilities and accountabilities are clearly defined in a project charter for the project. These practices are consistent with the new Major Project Governance Framework being developed. The Framework explains the authority, processes and oversight for all phases of a project including definitions of the roles and responsibilities of sponsors, senior management and managers, as well as organizational committees established to govern project activities.

Clearly defined authorities and accountabilities at the project outset can help to ensure that sponsors are sufficiently engaged in the project to define and monitor their requirements to obtain a project that meets their needs. Similarly, accountabilities for successful development will be clear such that a directorate is held accountable for the successful development of the project (i.e. on time, within budget and meets user requirements).

Recommendation:

1. The ISTB, in collaboration with the Enforcement, Admissibility, Operations and Comptrollership branches, should ensure that authorities, responsibilities and accountabilities are clearly defined at the outset of every system development project.

3.4 Project Management Framework

As noted in the Phase 1 audit report, a project management framework (or lifecycle) to manage systems under development is in development and is being refined with supporting tools and templates for implementation. This framework was not in place at the time of the ACI-EDI Reporting for Air development project. The project did however follow a development methodology with clear phases closely aligned to the framework being developed.

The Phase 1 audit report noted areas where the project management framework could be enhanced or improved. The audit work done in Phase 2 supports those earlier conclusions. The following observations were made on the project management practices for the development of the ACI-EDI Reporting for Air project:

• Documentation of Gate Approvals -- The audit noted that formal approval decisions at key milestones were not well documented. The audit did not find documentation supporting approvals at key milestones. Interviewees indicated that although a formal sign-off process was not in place at the time of the development, consensus for proceeding was obtained within the development team. Although there were several forums that discussed the progress of all projects, there was no steering committee to oversee project progress and make decisions specific to this project including approvals to proceed at predefined gates. This process weakness may be a result of the authorities, responsibilities and accountabilities for projects not being well defined. An effective oversight body that formally documents and communicates approvals will ensure a common understanding and transparency to the development process and ensure that approvals are received when needed.
• Project Status Reporting -- This audit found that project status was formally reported at a highly summarized level to executive management through the quarterly executive dashboard; however, standard project reports on progress (against schedule, budget and scope) below the executive level were not available. The audit examined five detailed status reports prepared for this project, but all varied in format and were not produced regularly. As time pressures increased and status reports to senior executives evolved, the more detailed status reports for this project were replaced with the one high-level report, the dashboard. Interviewees indicated that more information on project progress was provided orally at various informal and formal meetings. Oral updates on project status were also regularly provided to the release management team once the project was added to the release schedule. Standard project reporting that includes information on budgets, schedules and scope will ensure that management has the information needed for effective decision making.  
• Cost Monitoring -- A single budget estimate for this project could not be determined since allocations for ACI-EDI Reporting for Air were highly distributed and not easily separated from other ACI sub-projects and accumulated. The CBSA’s cost accounting system tracks costs by cost centre, which was designed to track costs by funding source not by project. As such, costs associated with the ACI-EDI Reporting for Air project were incurred through many different cost centres and not accumulated or reported by project. Within the cost centres, regular budgeting and cost comparisons were prepared to show monthly progress against the plan, which provides a rough indicator of whether budgets are being achieved. As was the practice for the IT directorates for many years, all CBSA employees are now required to capture their time by project code, as of April 1, 2007. This should permit the accumulation and tracking of project costs thereby enabling appropriate oversight of project spending.
• Integrated Planning and Scheduling -- The audit found that project plans and schedules existed within the main development groups, but were not integrated to coordinate all the various development tasks and interdependencies of the project. A review of project schedules maintained separately by the MPDD Directorate and IT noted that the schedules were not linked nor aligned for key dates such as the completion of the analysis and design phase. This project involved many individuals including project managers, architects, data management, release management and Customs Electronic Platform teams and ongoing oral communications between various teams were considered by the project teams to be an effective way to coordinate activities. However, there was uncertainty around key cut-off dates for changes to requirements. This may have been better communicated through an integrated project plan and schedule. An integrated plan and schedule will ensure that interdependencies are clearly understood and key dates are planned to meet scheduled release dates and improve efficiencies. The ISTB reorganization that consolidated the IT developers with the MPDD Directorate project managers should improve integration issues.
• Benefits Realization -- The high-level business benefits for the overall ACI initiative were understood as defined in the Customs Action Plan and the original Treasury Board submission from 1999. It is recognized that the real benefit of EDI Reporting for Air will accrue from the broader risk-assessment component of ACI to improve targeting. Nevertheless, the detailed operational benefits and results of the EDI Reporting for Air project were not defined to fully assess the realization and contribution to the overall ACI program benefits. There was no business case that identified measures of success, such as improvements to efficiency and accuracy of information, with the introduction of EDI Reporting for Air.
• Change Management -- The audit of the ACI-EDI Reporting for Air project noted that although the project scope was well defined, the process and authorities for approving change requests were not. The audit found that approvals from the sponsors of change requests were demonstrated through an iterative exchange of e-mails to discuss the rationale for the change request, but it was difficult to assess whether the appropriate person approved the change. Several sponsor representatives interviewed also indicated that the required turnaround time to fully review the change or involve more senior staff was sometimes short. Once the sponsor group approved the change, there was a formal change control process in place within IT to review the change and add it to the release schedule. For change requests, IT did its own estimate of the costs and prioritized the changes for inclusion in the release schedule and change requests were tracked by several different groups. As such, the labelling of change requests was different, making it difficult to trace the individual change requests. Change management is a critical component of project management to ensure sufficient rigour is in place to avoid scope creep. Sponsors would value a clearer understanding of the change management process and the expectations and authorities for approving changes. This process weakness could have been the result of the authorities and accountabilities for projects not being well defined. 
• Issue Management -- The audit noted that various logs were in place to track issues. These logs were different in format, issues identified and numbering systems. There was no comprehensive log in place and there was no ranking of the severity of issues, timing for resolution identified or dependencies noted. A clear process for documenting, tracking and ranking of issues in the development process would strengthen management’s assurance that all issues are effectively and efficiently resolved.

In the Phase 1 audit report, it was recommended that the ISTB should continue with its plans to implement a process to ensure that gates were clearly defined and benefit realization is reviewed and documented at each gate. Management had indicated that such processes would be implemented. It was also recommended that a project status reporting regime and a business case process be developed and management had indicated that such processes would be developed.

Recommendation:

2. The ISTB, in collaboration with the Enforcement, Admissibility, Operations and Comptrollership branches, should further enhance its project management framework to include the following:
   • A formal process to integrate planning and scheduling within individual projects across all directorates.
   • Identification and communication with sponsors of the change management process and the expectations and authorities for approving changes.
   • A process to document, track and rank development issues consistently across the  branch for their effective and efficient resolution.

3.5 Project Risk Management

A formal project risk management process involves the identification, assessment, treatment and monitoring of risks that could jeopardize the success of the project with respect to product quality, schedule and cost. The potential consequences associated with risks and the tolerance for risk exposure would be discussed to determine the formal risk response (avoid, mitigate, assume or transfer/escalate). The project team would determine whether the risks are acceptable and what can and could be done to remedy or better manage the risk. A formal escalation process would ensure that senior management is engaged in the risk discussions at the appropriate times.

For the ACI-EDI Reporting for Air project, the audit found that risks were informally addressed on a day-to-day basis, but they were not well documented. Interviewees indicated that when risks were identified, they were reported to the relevant project manager during regular meetings. Minutes of meetings were not always kept for the regular project meetings to support the documentation of the risks and actions taken. A standard risk log was not maintained for the project to monitor the risk exposure. Since the time of the project development, a risk log template has been developed to document project risks.

In the Phase 1 audit report, it was recommended that the ISTB should continue with its efforts to implement a risk management strategy to support the Major Project Governance and Project Lifecycle Framework. Management had indicated that such a strategy would be implemented.

3.6 Security Assessment

A security assessment for the ACI initiative was prepared in the appropriate template to ensure key aspects of IT security are consistently addressed for all projects. From the documentation, it was unclear who reviewed and approved the security assessment. Through interviews, it was noted that the process for conducting security assessments was being reviewed at the CBSA by the IT Security Group in the ISTB and the Departmental Security Office (DSO) in the Comptrollership Branch.

Recommendation:

3. The Comptrollership Branch, in coordination with the ISTB, should continue its review of the processes to conduct security assessments and to clarify the role of the IT Security Group and the DSO in assessing security requirements for individual projects, including the technical, information and physical security requirements.

Appendix A - Audit Criteria

The audit criteria used for Phase 2 of the audit were:

Appendix B - List of Acronyms

ACI

Advance Commercial Information

CBSA

Canada Border Services Agency

CFIA

Canadian Food Inspection Agency

CIC

Citizenship and Immigration Canada

CRA

Canada Revenue Agency

DSO

Departmental Security Office

ECU

Electronic Commerce Unit

EDI

Electronic Data Interface

ISTB

Innovation, Science and Technology Branch

MCF

Management Control Framework

MPDD

Major Projects Design and Development

PSAT

Public Security and Anti-Terrorism

RUP

Rational Unified Process

SBD

Shared Border Declaration

SOS

Statement of Sensitivity

TRA

Threat and Risk Assessment