Expected Condition

1. Timely process to allow early resolution to be attempted at the earliest opportunity

Expectations

We expected to find that the intake function was resulting in quick flow of complaints to ER investigators so that the complaints could be resolved at the earliest opportunity.

Actual Practice

Average complaint processing time during the period of June 08 to March 09 was as follows:

ER and CRC files (measured in calendar days):

• 22 days to receive, validate, and register complaints in IIA
• 19 days to issue a summary of the complaint and request documents from departments
• 90 days to receive documents from departments and prepare files for investigators

ER files only:

• 18 days from time records received and the file set up to assign complaints to an ER investigator and for the investigation to begin
• 16 days to investigate and develop a finding
• 11 days to approve an ER finding

Note: Because the intake unit prepares files for both ER and CRC, averages were calculated including both ER and CRC files for the 1st three steps noted above; but only ER files were included for the last three steps because the scope of the audit excluded CRC investigations.

Management also reported that the transfer of general mail duties to the Intake Unit has resulted in a significant slowdown in non-complaint related mail delivery within the Office overall.

Although Management had not set processing time targets, in the course of conducting this audit Management did note that the above averages, as calculated by the audit team, were not acceptable. Note that it takes, on average, over 40 days to receive a complaint and request documents from a department. It then takes, on average, 90 days to receive documents from departments (which is significantly higher than OIC’s expectation of a 10 day turnaround). Thus, it takes over four months, on average, before complaints can be assigned to an investigator. Management has recently established an escalation process to deal with such delays, but this new process has not been implemented for a long enough period of time to assess its effectiveness.

Once the file is assigned to an investigator, the timelines appear more reasonable. It should be noted, however, that expected standards for processing time have not been established so there is no benchmark against which the audit team was able to compare these processing times.

Based on a review of the intake process through walkthroughs and from interviews, potential causes for the lengthy processing times include:

• There are a significant number of hand-offs between the administrative staff in the Intake Unit, the ER Chief, the Client Liaison Officer and the ER Investigators. In total, the file is transferred a total of 6 times between the various staff members before being assigned to an investigator. This causes delays in the process. As per interviews conducted, the purpose of the majority of hand-offs is to ensure that assigned roles are being respected (i.e. that administrative staff are performing administrative functions freeing investigators to focus their time on activities requiring their skills).
• Departmental delays may be caused by some of the same issues that the OIC has faced, namely, significant increases in the volume of ATIP requests, and lack of experienced ATIP officers. In addition, some CRC investigators noted that the correspondence sent to departments to request documents may not be clear enough, particularly given that ATIP officers may be new and inexperienced. The CRC investigators have identified and communicated opportunities for improvement via meetings between Chiefs, but are not aware of the outcome of these suggestions.
• Through interviews, it was reported that other responsibilities assigned to the Client Liaison Officer such as producing reports from the investigation system, as well as the lack of back-up for the position, has had a significant impact on registering complaints in a timely manner.
• Through interviews conducted, the audit team perceived that there is ongoing confusion about the new complaints processing system overall, and a lack of clarity of individual roles, responsibilities and expectations. The audit team did not see a clear and singular focus on improving complaints process efficiency within the IERU process which we expected based on the importance of avoiding a new backlog for the OIC.

Recommendations

1.1 Management should continue to engage all staff in the imperative of not creating a new backlog and drive the need for a processing efficiency culture. This may be accomplished through such activities as setting targets, communicating results, considering individual and team incentives to motivate the desired behaviour, etc.

1.2 Management should establish clear targets for average processing time that, based on data gathered from the IERU pilot to date, are achievable. Management should ensure that these targets are well communicated, ensure there is an opportunity for open dialog about the targets, and ultimately, that there is buy-in from staff in achieving the targets.

1.3 In reviewing the above results, Management should consider whether there are some aspects of the process (such as multiple hand-offs, confusion on roles, etc.) that can be quickly improved as well as develop strategies to address the steps of the process that take the longest – specifically, the length of time to obtain required documents from departments.

1.4 Given that complaints processing efficiency is a priority, Management should re-consider the assignment of reporting responsibilities to the Client Liaison Officer given that this duty limits his ability to perform his other intake duties.

1.5 Management should monitor results to ensure continuous improvement.

Expected Condition

2. Complaints prioritized, assigned and investigated on a priority basis

Expectations

2a) We expected to find a clearly communicated and understood purpose and criteria for prioritizing complaints.

2b) We expected to see the criteria applied to incoming complaints by a qualified staff member.

2c) For the highest priority complaints, we expected to see a “fast-track” intake process to ensure these moved to the investigation stage as soon as possible. Ultimately, we expected to find that high priority complaints on average were processed faster than lower-priority complaints.

Actual Practice

2a) We expected to find a clearly communicated and understood purpose and criteria for prioritizing complaints.

With respect to the purpose of prioritizing complaints, we did not find this clearly stated and understood within the IERU process and team. Based on announcements related to the pilot of IERU and resulting news coverage, the purpose of introducing prioritization seems to be to move away from a first-come, first-served system in favour of an alternative approach.

At a high-level, the OIC prioritization approach considers a large number of factors such as client-focused needs (e.g. legal or academic deadlines), macro-level interests of parliamentarians, public, and the OIC, ease of resolution, and age of the complaint (i.e. first-come, first-served), amongst others, to make this determination. In all, more than 20 considerations are taken into account in determining priority which is a complex prioritization system that requires significant judgement. In addition, internal documentation includes maximum points that can be assigned in one category (urgency), but no direction is provided in terms of total points that can be assigned in other categories. Under all of these conditions, it can be difficult to achieve consistency across files and over time.

Criteria for prioritizing complaints are documented in three different places: on the OIC web-site, in internal procedures documentation, and within the investigations system. These are all fairly consistent, but some differences were noted which cause confusion. As an example, in the internal procedures, the age of the complaint is duplicated appearing in two categories.

2b) We expected to see the criteria applied to incoming complaints by a qualified staff member.

When IERU was initially established, prioritization was done by the Client Liaison Officer. During the course of the audit, the responsibility for prioritization was moved to an ER investigator who is appropriately qualified to do this work, therefore, our expectation was met.

2c) For the highest priority complaints, we expected to see a “fast-track” intake process to ensure these moved to the investigation stage as soon as possible.

Based on procedures documentation, high priority complaints are those having a priority rating of greater than 70. Through discussions with Management, however, the threshold number being used is actually 150. Looking at files received by ER and CRC during the period of June 01, 2008 to March 31, 2009, we found that the average processing time from receipt of a complaint to the assignment of the complaint to an investigator was marginally faster for files with a priority score above 150 vs. all complaints (109 days vs. 123 days respectively). In the population of files analysed above, 14% had a score of 150 or greater.

In addition to the data analysis above, interviews with IERU staff noted that the prioritization of files only affects its placement within the queue of files to be assigned to investigators. The prioritization has no affect on the procedures undertaken to get documents from departments or the order in which an investigator works on assigned files.

In February 2009, a new process and system functionality was implemented to bring forward complaints with a high prioritization level in order to escalate the file for further attention, particularly with respect to departments who have not provided requested documents within the expected 10 day period. As noted previously, and based on interviews, the planned escalation procedures have not yet been put in place, so we are unable to comment on the effectiveness of this new process.

Recommendations

2.1 Management should consider simplifying the number of prioritization considerations taken into account. In doing so, Management should take into account:

• The relative importance of client urgency versus other non-client factors and weight these accordingly;
• Factors that are very similar or duplicative (e.g. nature of complaint category duplicates in part how files are assigned to ER vs. CRC; age of complaints duplicated in internal documentation);
• The future expectation that all administrative complaints be resolved within 90 days and whether this expectation eliminates the need to further prioritize these complaints

2.2 Management should ensure that external and internal prioritization criteria and business rules are consistent and that implementation of the criteria in the investigations system is consistent.

2.3 Management should provide more information to departments about priority complaints, and implement escalation procedures in order to obtain documents faster.

2.4 Management should clearly differentiate the full high priority “fast-track” process from the general process and ensure that this is documented and communicated to all staff involved.

Expected Condition

3. Adequate volume of complaints being resolved.

Expectations

Given that ER was launched at the beginning of June 2008, we expected to find that the number of complaints closed by investigators was higher during the period of June 08 to March 09 than the equivalent period the year before, and that the volume of complaints being closed by IERU was in line with expectations.

Actual Practice

Notes:

• In analyzing processing efficiency, complaints that are assigned to the Strategic Case Management Team (SCMT) are not included as this was a special purpose temporary team formed to deal with backlog complaints. The audit is focused on IERU which was formed to deal with new complaints and not backlog of complaints (the cut-off point being April 01, 2008).
• Similarly, in analyzing the efficiency of the complaints process, files that are “cancelled” or “discontinued” have not been included in our analysis of the number of “closed” files. The rationale for their exclusion focuses on the fact that, often cancelled or discontinued files result from a significant passage of time since the complaint has been filed. Thus, closing these files is not, generally speaking, as a result of OIC processes.

The total number of complaints closed (excluding cancelled and discontinued files) is approximately 25% lower in the current ten-month period than for the same period in the previous year. Although there was a bulk closure of files last year which may explain some of the variation in this year’s results, even after factoring that adjustment in, the number of files closed is still less this year than last year which is not what was expected given the introduction of the new process.

Additionally, given that OIC reports have typically included cancelled/discontinued files in their analysis of closed files, this decrease may not be evident to Management or other users of the reports. When cancelled/discontinued files are included, there is almost no difference between the number of complaints closed between the two periods. Given the reasons stated previously, this is not an accurate depiction of processing effectiveness/efficiency within OIC.

In addition to the above analysis, a comparison of the number of files that are required to be closed by ER as compared to the future expectation built into the recently completed A-Base review process highlights that for the current fiscal year, ER closed about 67% of what will be expected next year (667 files closed vs. approximately 1000 targeted). It is recognized that additional resources are planned for next year, but even taking this into account, this will be a significant challenge. (Note – because the A-base review included discontinued and cancelled files, these have also been included in the number provided above; as noted previously, however, this is not recommended going forward).

Based on interviews conducted, potential causes for the lower than desired volume of closed complaints (once cancelled/discontinued files are excluded) may include:

• There was an extended amount of time required to get IERU “up and running” to full capacity which may have lowered processing volumes during that time;
• A number of investigators have left the OIC resulting in fewer investigators available to work on complaints
• Lack of clarity between units (IERU and CRC) with respect to roles and responsibilities, as well as a lack of opportunity for CRC officers to provide process feedback to the IERU unit may have caused some delays in the process and resulting volumes of files closed;
• Investigators were informed of complaint closure targets at a Branch retreat held in February, 2009 – this may incent behaviours to drive increased number of file closures.

In conducting this analysis, it should be noted that our audit team was challenged in dealing with the volume of different types of reports and the variety of business rules used to create reports – especially related to how closed complaints are counted. Business rules for this key statistic are not documented or well understood beyond the single CLO position. As such, Management is limited in its ability to provide guidance to the CLO in producing this statistic (and related reports). This condition also results in a challenge for Management to be able to effectively review reports for accuracy and consistency over time.

A variety of examples of issues in this regard were observed during the audit such as:

• cut-off rules for closures (e.g. investigation close date or complaint close date);
• inclusion/exclusion of Commissioner-initiated complaints;
• inclusion/exclusion of bulk closures;
• inclusion/exclusion of complaints dealt with outside the OIC’s main source system; and,
• system conversion adjustments required due to data transfer issues.

These are a few examples of the types of issues encountered. Without a deep understanding of these types of nuances, it is a challenge to effectively review and analyze resulting reports. As a result, the statistics provided in this audit report should be viewed as directional only (having said that, they have been validated and agreed with Management on this basis).

Recommendations

3.1 Management should ensure clear targets for files closures are communicated regularly along with actual results to identify whether progress is being made on this front.

3.2 Management should establish a comprehensive set of business rules related to the key statistics that need to be reported on an ongoing basis that should be documented, consistently applied, understood by management, and clearly disclosed on any resulting reports for clarity.

Expected Condition

4. Reporting capacity and capability that informs Management about the risk of a new backlog developing and provides reliable information for decision-making including establishing service/performance standards.

Expectations

We expected to find that resources were available and qualified to produce reports and that the data used to produce the reports was reliable. Given the focus on preventing a new backlog, we anticipated that Management would need reports (and therefore data) that showed the number of complaints in the system at any given time, and how long complaints were in various stages by team.

Dates or processing stages such as those below were anticipated:

• Complaint received
• Complaint deemed to be valid
• Acknowledgement sent
• Documents requested from department
• Documents received from department
• Complaint file prepared and assigned to ER or CRC
• Investigation completed
• Findings approved
• Findings communicated
• File closed

We expected controls to be in place to ensure the integrity of the data.

Actual Practice

We found that adequate resources were not available to produce reports. The Client Liaison Officer in the Intake Unit has been assigned this responsibility. Although he has the capability to perform the tasks, the volume of report requests has been significant taking him away from his intake duties. Furthermore, because a back-up for the Client Liaison Officer has not been designated (a long-standing un-addressed problem), when he is away, both processing efficiency and reporting are delayed.

As noted previously, the audit was challenged to deal with the volume and variety of different reports that are produced. Based on review and observation, Management is also challenged to have a consistent and clear understanding of the volume of data and variety of views being presented such that the reports are sometimes challenging to rely on for decision-making purposes. On this front, it is important to also note that Management has not identified what set of data (performance measures) is key to determining if objectives are being achieved or if the Office is on track to achieve future objectives.

With respect to the reliability of the reports, we found the following issues that Management should be aware of in using reports:

• Although April 01, 2008 was established as the cut-off point, this was not strictly adhered to. As a result, IERU and CRC were not purely focused on closing files that were assigned to them based on the envisioned intake process.
• Integrity of statistics regarding the number of files closed via each team/process (i.e. early resolution, normal investigations, or strategic case management) is impaired by the fact that in the investigations system, the team assigned to a file is derived based on the team that an investigator belongs to. When an investigator changes team or leaves the OIC, the data becomes incorrect. In the latter case (an investigator leaves the OIC), the data is changed to “Other” and for reporting purposes, this is assumed to be CRC. There are currently 188 files assigned to “Other”. This practice becomes difficult to manage from a reporting perspective because files reported as being closed by ER in a previous reporting period may be reported under CRC when an investigator changes teams. Management has been manually tracking and correcting this in reports, but this increases the risk of errors, is costly, and is not sustainable in the longer term.
• Given that the longest stage of the complaint process is obtaining documents from departments, the date of the request for documents from the department and the date of receipt of documents from the department should be, but are not, clearly tracked in the system. A date of when intake is completed is captured, but because intake completion includes receiving the files from the department, as well as then preparing the files for an investigator, it has been used to determine the average time to get documents from departments.

Recommendations

4.1 Management should assign the responsibility for reporting in a manner that does not impact the critical path of processing complaints.

4.2 Management should identify the set of data, reporting criteria (e.g. including or excluding discontinued and cancelled), and the report format that is key to determining if objectives are being achieved and if the Office is on track to achieve future objectives.

4.3 Management should be aware of the data integrity issues noted above and address the issues where possible. One example for management consideration is to use an existing field in the IIA system to identify the team that is handling a complaint rather than deriving this information from the team that an investigator belongs to which as reported above, can change.

Expected Condition

5. Improved ability to handle corporate information needs related to complaints

Expectations

We expected a clear process to be in place (even if in its early stages of development) in the Intake Unit to identify complaints of strategic importance and a process to promptly communicate this upwards. We expected to find that resources were available and qualified for identifying complaints of strategic importance and/or that guidance would have been provided to support the task assigned.

We expected to find that resources were available and qualified to produce reports and that the data used to produce the reports was reliable. (See Expected Condition 4 above).

Actual Practice

While activities to review incoming general enquiries and new complaints by the ER Chief are being undertaken, the desired objective of early communication of strategically important complaints to Senior Management is not effective. There is a lack of understanding and guidance of what types of complaints are strategically important and how these should then be communicated to Senior Management.

Also the review of new complaints and general enquiries creates an additional handoff in the process which is increasing new complaint processing time.

Recommendations

5.1 Management should ensure that resources assigned to the task of identifying incoming complaints of strategic importance have the requisite competencies and guidance to perform the task. Furthermore, Management should clarify the process to be used to communicate these to Senior Management on a timely basis.

5.2 Management should ensure that the review of incoming enquiries and complaints is done in such a way that the activity does not extend the overall processing time of complaints.

Expected Condition

6. Information that would support the OIC in informing both complainants and institutions to increase efficiency in handling complaints and prevent future backlogs

Expectations

Through their daily work with departments and complainants, we expected to see IERU gathering intelligence about what the Office could develop and publish for departments and/or complainants that would improve efficiency and prevent future backlogs.

Actual Practice

Initial information notices were sent to ATIP offices with respect to the establishment of the IERU pilot project and follow-up meetings were held with the ATIP community to explain the pilot. Information about the IERU pilot was posted on the website.

General enquiries (both written and verbal) are handled by IERU, but no specific roles or responsibilities for analyzing these enquires and developing materials that could be broadly distributed for the purposes of increasing efficiency in handling complaints and prevent future backlog complaints have been assigned.

Recommendations

6.1 Given that the longest stage of the complaints process is obtaining documents from departments, it is recommended that management consider this as a first priority with respect to determining what the Office could provide or develop for departments that would improve efficiency and prevent future backlogs. Based on interviews with CRC investigators, opportunities exist to help departments respond to OIC requests better and faster.