2022 Evaluation of the Investigations Program Final Report

Report

Table of Content

• Executive Summary
• Evaluation Findings and Conclusions
• Recommendations
• 1 Introduction
• 2 Overview of the Investigations Program
• 3 Evaluation Methodology
• 4 Evaluation Findings and Conclusions
  • 4.1 The Investigation Program Has Contributed to Change in Institutions
  • 4.2 Roles and Responsibilities Are Well Defined
  • 4.3 Mixed Views on the Clarity of the Investigation Processes
  • 4.4 Complainants Are Believed to Have Confidence in the Investigation Process
  • 4.5 Concerns over Institutions’ Confidence in the Investigation Process
  • 4.6 Systemic Investigations Are Viewed Positively
  • 4.7 Ready for Future Growth
• Investigations Program Operations
  • 4.8 Decision Control Points Are Suitable
  • 4.9 Internal Support to the Program
• 5 Conclusion
• 6 OIC Management Response
• Appendix A – Logic Model
• Appendix B - Interviewees
  • Internal Interviewees

Executive Summary

This report sets out the findings, conclusions, and recommendations from the evaluation of the Office of the Information Commissioner of Canada’s (OIC) Investigations Program. Investigations is the OIC’s only program. This project was the second part of a multi-phased evaluation of the OIC’s Investigations Program. It is assessing the investigation program from the perspective of institutions, and the senior investigations officers implementing it. It was conducted between April and September 2022.

The OIC investigates complaints from requesters who believe federal institutions did not respect their rights under the Access to Information Act. The Program officers report to a Senior Director of Investigations, and the overall program reports to the Deputy Commissioner, Investigations and Governance.

The focus of the evaluation was on the effectiveness of the Investigation Program processes to achieve its intended results from the perspective of the institutions and the senior investigations officers. The evaluation assessed the investigation process from the time a file is assigned to an investigator to the end of the investigation. It is not intended to examine the decisions resulting from the Program. The evaluation relied on the findings from interviews with Program personnel and external institutions that OIC works with. This evaluation did not include interviews with complainants because their experience and participation in the process is very different from institutions. The plan is to conduct a separate and focus consultation with complainants to better understand and respond to their specific perspective on the effectiveness of the Program.

Evaluation Findings and Conclusions

Representatives from institutions indicated that, as a result of the Investigation Program, their institutions had made sustained changes leading to improved compliance with the Act. This is a positive outcome for the OIC, particularly given that departments also indicated that they understood the Act well.

Interviewees indicated that the roles and responsibilities within the Investigations process were well defined. Some noted that these were defined in legislation, and others indicated that OIC’s clear directions within an investigation transaction helped in this regard.

The evaluation indicated that more could be done to clarify the Program process. The investigation process per se is seen as uneven and not sufficiently transparent to institutions. The Program’s frequent contact with institutions is beneficial and appreciated as a mitigation of this perceived weakness in process clarity. However, additional communications tools would help to explain process and decision-making criteria to key stakeholders.

The institutions’ concerns over the clarity of the Program process are affecting their confidence in the process. Institutions indicated they had confidence in the process for the more straightforward files. However, some had less confidence in how the process handled more complex files. Respondents from institutions and the Program recognized that the increasing number of complaints created a significant workload burden on investigators. Some institution representatives indicated that this may affect the OIC’s capacity to address complex files, which in turn adversely affects their institution’s confidence in the process. This situation appears to be mitigated where there is frequent communication between institutions and the Investigations Program.

Overall, the evaluation’s interviews suggest that complainants have confidence in the Investigation process and find the reports clear and useful. As indicated above, this question will be examined further in a later exercise.

The evaluation findings suggested that institutions have confidence in the systemic investigation process. According to interviewees, the systemic investigation resulted in an increase in the institution’s understanding of the rights and obligations set out in the Act. Therefore, systemic investigations are contributing positively to the Program’s intended outcomes.

The findings also suggest that, from a process perspective, the Program is performing well within the current volume and workload conditions, and that the Program is ready for future growth. In general, interviewees believe the processes in place are sound and can sustain an increase in volume. However, to address future challenges, the Investigations Program will need to maintain its efforts to train a cadre of Investigators.

Finally, the Investigation Program appears to be well supported by key stakeholders and internal services.

Recommendations:

1. The Investigations Program should develop and communicate information to explain the Program processes to institutions and other stakeholders.
2. The Investigation Program should formalize approaches leading to frequent contact between investigations personnel and institutions.

This report sets out the findings, conclusions and recommendations from the evaluation of the Office of the Information Commissioner of Canada’s (OIC) Investigations Program.

1 Introduction

The OIC supports the Commissioner in her advisory role to Parliament and parliamentary committees on all matters pertaining to access to information. It also actively makes the case for greater freedom of information in Canada through targeted initiatives such as Right to Know Week and ongoing dialogue with Canadians, Parliament and government institutions.

The Information Commissioner is supported by a staff of approximately 135 employees led by three deputy commissioners responsible for investigations and governance, legal services and public affairs, and corporate services, strategic planning and transformation services.

This evaluation project was the second part of a multi-phased evaluation to address, as per the Policy on Results, the relevance and performance of the OIC’s Investigations Program. The first phase of the evaluation, conducted in 2021, focused on the OIC’s Registry function. This second phase evaluates the overall Investigations Program from the perspective of the institutions and the senior investigations officers. Investigations is the OIC’s only program, and the Risked-based Audit and Evaluation Plan recommends it be evaluated every five years.

This phase of the evaluation of the OIC’s Investigations Program was conducted between April and September 2022.

2 Overview of the Investigations Program

The OIC investigates complaints from requesters who believe federal institutions did not respect their rights under the Access to Information Act. There are four types of complaints:

• Administrative complaints generally relate to institutions extending or delaying timelines for responses to requesters. Complainants have not yet received a response to their request.
• Refusal complaints relate to institutions applying exemptions or exclusions under the Act to refuse disclosure of information. Complainants are not satisfied with the information they receive.
• Exclusion complaints relate to situations where an individual does not agree with the institution that the records requested are not covered by the Act.
• Miscellaneous complaints involve any other matter related to requesting or obtaining access to records under the Act.

The Program also undertakes systemic investigations and reviews follow-up information provided by institutions. During a systemic investigation, OIC officials conduct interviews and review documents in an institution to better understand how it responds to access to information requests. Based on this information, the Commissioner identifies issues, shares findings, and presents recommendations to the responsible Minister.

The Program officers report to a Senior Director of Investigations, and the overall program reports to the Deputy Commissioner, Investigations and Governance.

As of March 31, 2022, the OIC had registered 6,945 complaints. In fiscal year 2020–2021, the OIC registered 4,068 complaints and in 2019–2020, 6,172 complaints.

In 2021-22, 6,787 investigations were completed, and two systemic investigations were completed.

It is noteworthy that the OIC’s resource base has grown considerably over the past few years. In 2018-19, the OIC had a budget of $11.3 million and 93 Full-Time-Equivalent personnel. During that time, it completed a total of 2,608 investigations. In 2021-22, the OIC’s budget was A-base funding of $11.3M and $5.5 million in temporary funding for a total of $16.8 million, and approximately 135 Full-Time-Equivalent personnel. With the reallocation and in-year funding, the OIC was able to close a total of 6,787 complaints during that period while being permanently funded for 4,414.

The logic model for the overall Program is included as Appendix A.

3 Evaluation Methodology

The evaluation methodology was developed during a planning phase that included the review of documentation and internal interviews. A logic model was developed to clarify the activities, outputs, and intended outcomes of the Investigations Program. Interviews provided the information to define the evaluation issues and questions, identify interviewees, and define the parameters for applying the methodologies. An Evaluation Workplan was developed and approved by the Project Authority.

The focus of the evaluation was on the effectiveness of the Investigation Program processes to achieve its intended results. The evaluation is assessing the investigation process from the time a file is assigned to an investigator to the end of the investigation. It was not intended to examine the decisions resulting from the Program.

The Investigations Program processes involve considerable interaction with institutions subject to the Act. As a result, findings relied heavily on the findings from interviews with Program personnel and external institutions that OIC works with. The interviews included open-ended questions to enable respondents to provide a full range of views on the Program.

This evaluation did not include interviews with complainants. The complainants’ perspective will be the object of the further work in the coming years.

Interviews with Representatives from Institutions

The evaluation included interviews with key informants from external institutions that worked with the Investigations Program. The methodology included several institutions that had been involved in the earlier Evaluation of the OIC Registry Function, since they were already familiar with the OIC’s evaluation exercise. The following criteria were used to select the institutions for interview:

• The largest-volume external institutions were included -- Immigration, Refugees and Citizenship Canada and the Royal Canadian Mounted Police
• Other higher-volume institutions were included to reflect a diversity of sizes, and to include some that are “operational” and “not operational.” The following institutions were included: Canada Border Services Agency, Canadian Broadcasting Corporation, Transport Canada, Department of Finance Canada, Justice Canada, and the Privy Council Office.

These institutions know the Investigations Program well and were in a position to provide useful information on the evaluation questions. The OIC works with almost all federal institutions. However, it was determined that these core external institutions were best positioned to provide knowledgeable and relevant information to support the evaluation.

Interviews with OIC Personnel

Interviews were held with OIC senior management and senior Investigations Program officers.

The list of internal interviewees is included as Appendix B.

Validity of the Evaluation Methodology

Overall, from an evaluation perspective, the methodology is sufficient to provide a high level of confidence in the accuracy of the findings and conclusions. However, the findings from this evaluation will be strengthened under a separate future exercise that will include research involving a sample of complainants.

4 Evaluation Findings and Conclusions

The following section presents the findings and conclusions from the evaluation research. The first sub-sections present the findings that involved interviews with representatives from institutions. The last two sub-sections discuss matters pertaining to internal Program operations.

4.1 The Investigation Program Has Contributed to Change in Institutions

Representatives from institutions indicated that, as a result of the Investigation Program, their institutions had made sustained changes leading to improved compliance with the Act.

Five of the six representatives from institutions indicated that the complaint process had contributed to changes in how their department made decisions about information disclosure. Respondents indicated that they looked at precedent, and made changes to their procedures. They also indicated that they included the “thought processes” from the OIC into their training of analysts. Some institutions post information more pro-actively, and post more detailed information in cases where there is public interest.

This is a positive outcome for the OIC, particularly given that departments also indicated that they understand the Act well. The evaluation asked whether the Investigations Program helped their institution increase its understanding of the rights and obligations set out in the Act. Four of the six interviewees indicated that they knew the Act very well, and that the Program didn’t increase their understanding in this regard. The remaining two interviewees indicated that working with the Program does help their senior management, management, and the officer-level increase their understanding.

As a result, even though institutions know the Act, the Program has contributed to improving the alignment of their information disclosure practices with the requirements of the Act.

4.2 Roles and Responsibilities Are Well Defined

All institutions’ representatives indicated that the roles and responsibilities within the Investigations process were well defined. All respondents provided their answer quickly, indicating that they were confident that they knew what were the OIC’s and their institution’s roles and responsibilities. Some noted that the roles and responsibilities were defined in legislation, and others indicated that OIC’s clear directions within an investigation transaction helped in this regard.

OIC’s senior Investigation officers and senior management also indicated that roles and responsibilities were well defined, and that much work had been done in recent years to improve these.

4.3 Mixed Views on the Clarity of the Investigation Processes

When asked whether the Investigations Program processes were clearly defined, sufficiently documented, and well communicated, half of the interviewed institutions indicated that more could be done.

Three of the six interviewed institutions indicated that the processes were clear, well-defined and communicated, noting that frequent discussions with Investigations personnel were an important contributor to achieving this result.

However, the other three commented that they lacked the information needed to sufficiently understand the OIC’s process. These respondents indicated that they didn’t clearly understand how the OIC conducted its investigations, and would like more information from OIC to understand the nature of an individual complaint and how the OIC proceeds through its decision processes. As an example, interviewees indicated that they didn’t understand how the OIC decided on whether to proceed with a resolution versus a letter to the Minister, and were not able to predict what would happen and why. Some indicated that the OIC had removed useful information from its website that had previously contributed to transparency, and that this posted information on investigation tests had been a useful tool for explaining files and decisions to their management.

As another illustration, two of the respondents indicated opposing views about OIC’s application of the process. One indicated that some investigations decisions seemed inconsistent, citing as an example that the OIC will sometimes accept a complaint after the defined lapse period and sometimes is doesn’t. The other interviewee believed that the OIC adheres to the process too much without applying discretion. They believed that this approach turns an investigation into a paper exercise.

When asked whether they have seen improvements in the Investigations process over recent years, four of the six respondents highlighted the increase in contact and dialogue between themselves and Investigations personnel. They indicated the OIC is open to discussion about improvements, and that the meetings were very helpful and provided useful feedback. They indicated that the OIC is willing to listen to suggestions. The other two interviewees indicated that they hadn’t seen any improvement in the investigation process. All these views suggest there is a need for more clarity, and that the increased communication is contributing to clarifying the process for institutions.

Senior Investigations officers commented on improvements in clarity achieved in recent years because procedures and guidelines were formalized. This is an important accomplishment. However, it may not yet have fully translated into improved process clarity for all institutions.

These findings suggest that the investigation process per se is seen as uneven and not sufficiently transparent to institutions. The Investigations Program’s frequent contact with institutions is beneficial and appreciated as a mitigation of this perceived weakness in process clarity. However, this evaluation finding suggests that more could be done to put in place additional communications tools that explain the Investigations Program process and decision-making criteria to key stakeholders.

Recommendation:

The Investigations Program should develop and communicate information to explain the Program processes to institutions and other stakeholders.

4.4 Complainants Are Believed to Have Confidence in the Investigation Process

Representatives from the interviewed institutions believe that complainants generally have confidence in the Investigations process, and that complainants find the reports useful. It is noted that this question will be researched further by the OIC in the coming years to assess the perspective of the complainants.

Only one of these interviewees indicated that they didn’t have a view on this question. Also, all interviewees indicated that they couldn’t answer definitively on the part of complainants, and that some would have confidence in the process and some would not. However, those with a view all believed that complainants generally have confidence in the process.

These respondents believed that the long time delays adversely affect complainants’ confidence. Also, they recognized that complainants may not be satisfied with the outcome from a complaint. However, they felt that, even so, complainants viewed the process as objective and independent.

Senior Investigations officers held the same view that complainants generally had confidence in the process. They also emphasized that “frequent” complainants who understood the role and processes of the OIC would have more confidence in the Investigations process due to their familiarity. Infrequent complainants might have less confidence, since they might be unfamiliar with the mandate of the Commissioner, and the processes.

With respect to the reports, respondents from institutions provided mixed views on whether they thought complainants found the reports clear and useful. Four of the institution representatives had a view on this, indicating that they thought the reports were clear and generally provided useful information to complainants. However, they also added a mix of views. They indicated that the reports provide the complainant with the information they are looking for. Some also indicated that the reports are lengthy and technical, and others indicated that providing more information would be beneficial. Even though they provided these varied suggestions, they indicted that, generally, they believed the reports were clear and useful to complainants.

Senior Investigations officers noted that the reports had improved significantly in terms of completeness and clarity.

Therefore, overall, representatives from institutions as well as senior Investigations officers believe that complainants have confidence in the Investigation process and find the reports clear and useful. As indicated, this question will be examined further in a later evaluation exercise.

4.5 Concerns over Institutions’ Confidence in the Investigation Process

The institutions’ concerns over the clarity of the Investigations Program process are affecting their confidence in the process.

When asked whether their institution had confidence in the complaint treatment and request processes, all respondents from institutions indicated they had confidence in the process for the more straight-forward files.

However, some had less confidence in how the process handled more complex files. The institution representatives who met frequently with Investigations personnel indicated they appreciated the opportunity to discuss and resolve complex files. However, representatives from the other institutions indicated a level of frustration with the process. On one hand, they fully expected there would be disagreements with the OIC on some complex issues. However, they believed that the OIC process was not well adapted to more complex situations, and that they didn’t understand how the OIC reached some decisions. Interviewees provided examples of cases where they weren’t able to understand the rationale for investigations decisions, which reduces their confidence in the process. In some cases, they suggested that the investigator didn’t fully understand the institution’s operating environment or the full complexity of the case, which further contributed to a loss in confidence.

Conversely, senior Investigations officers all believed that institution’s confidence in the process has increased. They indicated this is due to more-defined processes and improved training.

When asked whether there were are any inherent biases in the Investigations Program, all respondents from institutions felt there was a bias in favour of the complainant. This view is not unexpected when discussing the OIC’s Investigations Program. However, this also contributes to the concern over the lower confidence in the process on the part of some.

It should be noted that there was no concern from any interviewee over any bias in the process related to equity, diversity and inclusion.

All respondents from institutions and the Program recognized that the increasing number of complaints created a significant workload burden on investigators. Some institution representatives indicated that this may affect the OIC’s capacity to address complex files, which in turn adversely affects their institution’s confidence in the process.

It is noteworthy that the institutions that met frequently with Investigations personnel had more confidence in the process. Further, senior Investigations officers cited frequent contact as a contributor to their perception of institutions’ confidence in the process. This suggests that frequent contact, in addition to the earlier-mentioned communications materials on processes, would help to increase institutions’ confidence in the Investigations process.

In summary, institutions have confidence in the process’s handling of straightforward complaints. However, some institutions are not as confident in the Investigation process’s handling of more complex files, and the root cause of this is unclear. This situation appears to be mitigated where there is frequent communication between institutions and the Investigations Program.

Recommendation:

The Investigation Program should formalize approaches leading to frequent contact between investigations personnel and institutions.

4.6 Systemic Investigations Are Viewed Positively

Two of the six interviewed institutions had views on the OIC’s systemic investigation of their organizations, and these views were positive.

In both cases, the representative indicated without reservation that the institution had confidence in the systemic investigation process. Also, interviewees indicated the systemic investigation resulted in an increase in the institution’s understanding of the rights and obligations set out in the Act. The exercise had helped to confirm that that institution processes were effective, and provided support to modernize operations and receive increased resources. The interviewed OIC senior Investigation officer who was familiar with systemic investigations indicated that, overall, OIC has seen positive results from the exercises.

As a result, the evaluation findings suggest that systemic investigations are contributing positively to the Program’s intended outcomes.

4.7 Ready for Future Growth

The four institutions that indicated they work collaboratively with Investigations personnel believe that the Program is ready for future growth. In general, they believe the processes in place are sound and can sustain an increase in volume. However, they caution that the significant challenge will be to increase the number of knowledgeable personnel to deal with a growth in volume. Of the other two interviewees, one did not have a view, and the other believed that investigators did not have sufficient experience to address larger volumes.

OIC’s senior Investigations officers believed that the processes and training methods are in place to address future growth internally. However, they expect that staffing and training will be an ongoing challenge for institutions as well as the OIC.

This finding suggests that, to address future challenges, the Investigations Program will need to maintain its efforts to train a cadre of Investigators.

Investigations Program Operations

4.8 Decision Control Points Are Suitable

An important contributor to Program efficiency and effectiveness is the timing and delegation of decision-making on investigations files. These key control points ensure that proper decisions are made on a file, and that these decisions are made without causing delays in the process.

All internal interviewees believed that the Investigations Program control points were suitable and contributed to program performance. They indicated that delegations were appropriate and that internal responses within the Program were prompt. This suggests that the current risk-based approach is effective for the Program.

Furthermore, internal respondents indicated that, in their opinion, the processes did not contribute to any undesired biases. This further supports the view that the Program’s decision control points are suitable.

All internal interviewees viewed the Investigations Program’s operations positively, highlighting the improvements that have been introduced in recent years. Interviewees believed that the current approach is effective for achieving results and did not suggest changes.

All these findings suggest that, from a process perspective, internal interviewees are confident that the Program is performing well within the current volume and workload conditions. As indicated earlier in this report, they also believe that staffing and training will be an ongoing challenge.

4.9 Internal Support to the Program

The Investigations Program is supported by three internal services: the Registry, Legal Services, and Corporate Services

Interviews with senior OIC management and senior Investigations officers indicated that they are very well supported by the Registry.

Interviewees also indicated that services from the Communications function met their needs well. Further, they indicated they were currently well supported by the finance and human resources functions, recognizing the shortages in available resources as well as the future challenges in those two areas.

Some interviewees highlighted that information technology (IT) was a key internal service and that Intrac was an important tool that would be under continuous development. Interviewees were very satisfied with the more routine IT support services, such as the service desk. However, there was concern expressed over the OIC’s ability to progress on larger IT innovations. Specifically, some senior internal interviewees noted that OIC’s innovation projects were often delayed, and that these delays had an adverse impact on the Program’s ability to modernize. They recognized the complexity of these projects, and the expertise required to move them forward. However, there was some frustration that the Program’s progress would be held back if these projects continued to be delayed.

With respect to Legal Services, senior Investigations officers indicated that, when required, they worked to involve Legal Services in a file as soon as possible. This was done with a view to maintaining timelines on files, and in recognition of the high workload volume of Legal Services. All interviewees recognized the resource constraints facing this expert area. Interviewees indicated that, given the workload, service levels were generally acceptable to them. For some, the turnaround was satisfactory and for others it was slower than expected.

In summary, the Investigation Program appears to be well supported by key stakeholders and internal services.

5 Conclusion

In conclusion, the evaluation determined that the Investigations Program is achieving important intended results. Institutions have made sustained changes leading to improved compliance with the Act, and systemic investigations are contributing positively to the Program’s intended results. Furthermore, roles and responsibilities are well defined, and complainants are believed, generally, to have confidence in the Program processes and to find the reports useful.

Institutions indicated there is a need for the OIC to provide more clarity on the Program’s processes. This, in turn, would contribute to increasing institutions’ confidence in the Program processes. Improved information on processes and frequent contact between the Program and institutions would help mitigate this area of concern.

Within the Program, decision-making processes are seen to be suitable and working well. Support provided by the Registry, Legal Services, and the communications, finance, human resources, and routine IT functions are seen to be meeting the Program’s current needs. There is a concern over OIC’s ability to progress on larger IT innovations supporting the program delivery.

Interviewees generally believed that the Investigations Program is ready for future growth. In particular, the processes in place are viewed as sound and can sustain an increase in volume. However, there is a recognition that the Investigations Program will need to maintain its efforts to train a cadre of Investigators.

6 OIC Management Response

The Office of the Information Commissioner of Canada (OIC) would like to thank Samson and Associates (“the evaluator”) for the independent evaluation that was conducted on the Investigation program. The focus of the evaluation was on the effectiveness of the Investigation Program processes to achieve its intended results from the perspective of institutions and senior investigations officers. The evaluation assessed the investigation process from the time a file is assigned to an investigator to the end of an investigation. The evaluation relied on the findings from interviews with Program personnel and external institutions that the OIC works with. This evaluation did not include interviews with complainants. There is a plan to conduct additional work in which complainants will be consulted for their perspective on the effectiveness of the Program.

Overall, both internal and external clients appear to be satisfied with the Investigative program. The evaluator made some observations that will serve to strengthen the OIC’s Investigative function and enable greater efficiencies.

With this context in mind, a work plan was created to outline how the OIC intends to capitalize on these opportunities. The Management Action Plan provides clear commitments, timelines and deliverables to address each recommendation.

Appendix A – Logic Model

Text version

Appendix A – Logic Model

Activities	Provide information to stakeholders	Conduct investigations of complaints	Conduct Special Investigations of institutions	Monitor institutions' remedial actions
Outputs	Information through various media	Investigation Reports / Follow-up Reports	Special Investigations reports	Monitoring reports
Intended Results - Immediate (Approx. 1 yr)	Institutions and individuals have become more informed the Act and OIC	The Investigations process addresses complaints efficiently	Complaints have been addressed and resolved effectively	Necessary improvements to institutions' practices have been identified
Intended Results - Intermediate (Approx. 3 to 5 yrs)	Complainants and institutions have increased their confidence in the complaint treatment and request processes	Complainants and institutions have a better understanding of the rights and obligations set out in the Act	Institutions have made sustained changes in support of the Act, for improved compliance	Proactive change management culture drives continuous improvements
Intended Results - Longer term (7+ yrs)	Institutions are fulfilling their obligations under the Act		Canadians' rights under the Act are protected effectively	Canadians have full confidence in the impartial treatment and resolution of Access to Information Complaints

Appendix B - Interviewees

Internal Interviewees

Layla Michaud, Deputy Commissioner of Investigations

Gino Grondin, Deputy Commissioner, Legal Services and Public Affairs

France Labine, Deputy Commissioner, Corporate Services, Strategic Planning and Transformation Services

Allison Knight, Senior Director, Investigations and Registry

Heather Rousson, Senior Investigator

Natasa Pushkar, Senior Investigator

Katarina Kovačević, Lead Investigator

Jeffrey Vachon, Investigator