Evaluation of the AgriRisk Initiatives Program

Abbreviations

AAFC

Agriculture and Agri-Food Canada

ACB

Administrative Capacity Building

BRM

Business Risk Management

CAP

Canadian Agricultural Partnership

GBA

Gender-based Analysis

LPIP

Livestock Price Insurance Program

PIP

Performance Information Profile

R&D

Research and Development

SCAP

Sustainable Canadian Agricultural Partnership

WLPIP

Western Livestock Price Insurance Program

Executive summary

Purpose

The Office of Audit and Evaluation of Agriculture and Agri-Food Canada (AAFC) conducted an evaluation of the AgriRisk Initiatives Program (AgriRisk) to assess its relevance, efficiency and effectiveness. The purpose of this report is to communicate the results of the evaluation.

Scope and methodology

The evaluation examined AgriRisk activities from 2018-19 to 2022-23 using multiple lines of evidence including a review of program documents, project files and literature; key informant interviews; analyses of administrative and financial data; and a case study.

Background

AgriRisk was a non-repayable contribution funding program with an annual average budget of $11 million. The program aimed to increase producers' ability to address the risks they face by facilitating the development and adoption of new risk management tools. The program was divided into 3 streams: Microgrants, R&D and ACB. Though AgriRisk was not renewed as a standalone program under the Sustainable Canadian Agricultural Partnership, an evaluation was still required by the Policy on Results and section 42.1 of the Financial Administration Act.

Findings

• AgriRisk played a unique role in improving the ability of producers in the Canadian agri-food system to address the risks they face by facilitating the development and adoption of new risk management tools.
• The program implemented promising practices to improve efficiency and partially achieved its immediate and intermediate outcomes but saw a low number of applications and service standards that were inconsistently met.
• The design of AgriRisk did not include specific objectives to facilitate or support participation by underrepresented groups, although this was not a requirement at the time for programs under the CAP. However, no barriers to participation for any group were identified.
• There are areas for improvement in the program's Performance Information Profile.

Conclusions

AgriRisk addressed a need in the agricultural sector and funded projects aligned with AAFC and federal government priorities. The program's flexible design, which included multiple streams, enabled Canada's agricultural sector to invest in the development and adoption of new risk management tools. There were processes in place to help administer the program efficiently, though at least 1 service standard was not met in 2018-19, 2020-21 and 2021-22. The program partially achieved its immediate and intermediate outcomes, in part due to a low number of applications and thus projects funded. To strengthen future programming at AAFC, there is an opportunity to better understand the degree to which underrepresented and marginalized groups are accessing and benefitting from projects supported by AAFC. This would help to ensure that there are no barriers to participation for any group. In addition, there are areas for improvement in AgriRisk performance measures that would have helped ensure that reliable and consistent performance outcome data was available.

1.0 Introduction

The Office of Audit and Evaluation of AAFC conducted an evaluation of the AgriRisk Initiatives Program (AgriRisk) as part of the 2023-24 to 2027-28 Integrated Audit and Evaluation Plan. AgriRisk was a five-year program under the CAP that had the objective of improving producers' ability to address the risks they face by facilitating the development and adoption of new risk management tools. It was not renewed as a standalone program under the Sustainable CAP (SCAP). The results of this evaluation are intended to inform current and future programming and policy decisions.

2.0 Scope and methodology

The evaluation was conducted in accordance with the Treasury Board Secretariat Policy on Results (2016) and assessed the relevance, efficiency and effectiveness of AgriRisk over a five-year period, fiscal years 2018-19 to 2022-23. The evaluation used multiple lines of evidence including a review of program documents, project files and literature; key informant interviews including AAFC staff and funding recipients; analyses of administrative and financial data; and a case study of the WLPIP. This evaluation was not an in-depth review of AgriRisk as the program is no longer delivered as a standalone program. However, evaluations of programs, even when not renewed, are required under the Policy on Results when certain conditions are met. For more detailed evaluation methodology, see Annex A.

3.0 Program profile

3.1 Overview of the AgriRisk Initiatives Program

The AgriRisk Initiatives Program (AgriRisk) was a five-year non-repayable contribution program with an annual average budget of $11 million. The program aimed to increase the ability of Canadian producers to address risk by supporting research and development as well as the implementation and administration of new risk management tools in the agriculture sector. Under the program, financial assistance was provided to non-profit and for-profit organizations to facilitate the development and adoption of risk management tools, including insurance-based products. Non-financial support was also provided to provincial and territorial governments. Funding to support AgriRisk as a standalone initiative ended on March 31, 2023, as the program was not renewed as a standalone program under SCAP.

AgriRisk was divided into 3 streams:

• R&D: funded through non-repayable contributions which provided recipients with up to $500,000 per fiscal year in funding for industry-led R&D related to risk management tools and practices. Recipients in the R&D stream were required to contribute 25% of estimated project costs as a combination of cash or in-kind.
• Microgrants: provided recipients with up to $25,000 per fiscal year in non-repayable funding to support academic research on issues relevant to business risk in agriculture.
• ACB: provided recipients with up to $5 million per fiscal year to support administration and testing of the viability of new risk management tools, such as insurance products, after R&D was completed. This contribution could include, without exceeding the maximum federal contribution of $5 million per fiscal year, a one-time federal contribution for seed funding up to $750,000. Contributions made under the ACB stream to not-for-profit organizations or provinces and territories were non-repayable, except for seed funding contributions, which were conditionally repayable. All contributions, including seed funding, made to for-profit organizations under the ACB stream were repayable. For recipients under the ACB stream, the minimum recipient contribution required for non-profit recipients was 15% of costs, while a minimum of 50% was required from for-profit recipients. For provinces and territories, the AAFC contribution was 60% and the recipient contribution was 40%.

AgriRisk's anticipated outcomes are articulated in the program Logic Model (see Annex A).

3.2 Resources

Allocation of funds

The budget allocated for AgriRisk was $55 million over 5 years (2018-19 to 2022-23) but actual spending was only $29.5 million. Figure 1 presents the federal government budgeted resources and actual expenditures from 2018-19 to 2022-23. The program underspent by about $25.5 million over the evaluation period, representing about 46% of the allocated budget. Underspending was primarily due to a lower than anticipated uptake. Unspent funds were reallocated to support other priority AAFC programs.

Figure 1: Program Budget and Actual Spending, 2018-2019 to 2022-2023 ($ millions)

(Total planned spending of $55.0 million and actual spending of $29.5 million)

Description of the above image

Figure 1 presents a vertical bar graph showing the program budget and actual program spending per fiscal year from 2018-19 to 2022-23. The table below shows the spending budget and fiscal per fiscal year over the evaluation period ($ millions).

	2018-19	2019-20	2020-21	2021-22	2022-23	Total
Program budget	12	12	12	9.6	9.6	55
Actual spending	2.9	6.3	5.8	7.4	7.1	29.5

4.0 Relevance

4.1 Alignment with AAFC and government priorities

AgriRisk's objective was to increase producers' ability to manage the risks they face by facilitating the development and adoption of new risk management tools. The program was aligned with AAFC's core responsibility of sector risk as it helped producers address the risks they face and contributed to 2 of the 6 departmental results:

• the agriculture sector is financially resilient
• the agriculture and agri-food sector is equipped with assurance systems and supporting tools

The program was also aligned with federal government priorities outlined in CAP. Under the CAP framework, one priority area was risk management. This priority area included the promotion and development of private sector risk management tools. AgriRisk contributed to this priority by providing funding to support the development of new risk management tools.

AgriRisk was aligned with other federal government priorities, including climate change. Although not an overarching priority, the program developed new risk management tools to address risks related to climate change. Program staff also mentioned that climate change has resulted in new needs for risk management tools due to changing weather patterns.

4.2 Complementary with other federal government programs

The evaluation found that AgriRisk complemented other AAFC programming as there were a few programs with related objectives. There were no non-AAFC programs identified as having complementary or overlapping objectives with AgriRisk.

AgriRisk's objectives complemented the existing core BRM suite (AgriInsurance, AgriStability, AgriRecovery and AgriInvest) as implemented under CAP. AgriRisk was the only program funding research, development and administration of new risk management tools. In addition, the program supported proactive risk management by encouraging stakeholders to take greater responsibility in developing and adopting new risk management tools and strategies.

Of all programming in place under CAP, AgriCompetitiveness and AgriAssurance had the most similar program objectives to AgriRisk. Comparable to AgriRisk, one of AgriCompetitiveness' objectives is to improve farm management, which includes risk management activities. However, AgriCompetitiveness did not cover R&D of new risk management tools under CAP. Similar to AgriRisk, AgriAssurance aims to help producers learn how to utilize and administer new risk management tools. While AgriAssurance helps not-for-profit organizations and individual farmers develop and/or adopt market and regulatory requirements to reduce risk, AgriRisk helped various agricultural sectors research, develop and/or implement tools that the sector defines as a business risk.

5.0 Program design and performance

5.1 Efficiency

AgriRisk service standards

Timeliness to access government services is a key driver of client satisfaction^{Endnote 1} and service standards help to measure the timeliness of service. During the evaluation period, 5 service standards were tracked (as summarized in Table 1). Service standards related to inquiries and acknowledgement of applications were generally met. However, the achievement of standards related to the assessment of applications and payment of claims was less consistent. During the evaluation period, the target of 80% of application approvals or rejections sent within 100 business days was met only in 2022-23. The target of sending 80% of claims payments out within 30 days was met in 3 out of 5 years.

Table 1: Service standards

Service standard	Target (%)	2018-19 (%)	2019-20 (%)	2020-21 (%)	2021-22 (%)	2022-23 (%)
Acknowledge receipt of your application within one business day	80	not tracked	100	100	n/a	n/a
Assess applications and send approval or rejection notification letter within 100 business days after receiving complete application	80	6	0	54	71	100
Respond to general inquiries by email before the end of the next business day	80	100	87	88	n/a	n/a
Respond to general inquiries by phone before the end of the next business day	80	88	100	100	93	100
Send you a payment within 30 business days of receipt of complete claim	80	74	100	81	64	86

Program management noted that changes to program delivery contributed to challenges in achieving service standards. The program transitioned from one division to another at AAFC at the end of fiscal year 2018-19. Application decisions were affected by this process in 2018-19 and 2019-20. There was also an impact on application review in 2020-21 and 2021-22 because of the COVID-19 pandemic.

Promising practices

The 2-stage application process was seen as a strength, as this model allowed for upfront screening of projects. Similarly, having open intake was seen as optimal because it supported open dialogue around complex or abstract subject matter if program officers had questions or a need for more information about projects. This approach allowed for identifying the most relevant needs and appropriate scoping of the projects submitted. Also, this method has the advantage of alleviating administrative burden on program officers as well as applicants who were not sure which stream they should be applying to.

Additionally, flexibility in the eligible activities in the program was seen as useful. This created space for adaptation, for example, allowing recipients to pivot from in-person to virtual workshops during the COVID-19 pandemic.

5.2 Effectiveness

The program had one immediate outcome to increase awareness of risk management tools and mitigation strategies, which was measured using the indicator, "number of information items shared." This indicator was tracked through performance reporting, although the original indicator in the PIP was "number of information items developed." The indicator used was revised during the evaluation period, but the PIP was not correspondingly updated. Findings from the evaluation indicate that the educational tools funded by AgriRisk were successful in raising awareness of risk management tools.

The program partly achieved its intermediate outcomes through the R&D stream by supporting the development of risk management tools. However, the program fell just short of its targets to support the development of 40 tools with 37 risk management tools being supported. This was likely primarily caused by the lower than anticipated number of new risk management tools receiving funding.

The program did succeed toward their outcome of supporting the adoption of risk management tools and strategies. The evaluation found that some projects led to insurance products and were adopted by producers. A key success was the expansion of the Western Livestock Price Insurance Program.

Decision-making tools were also created for smaller commodities, including tools developed by watershed groups to be utilized by producers when certain weather conditions arose. For example, in Quebec, a decision-making tool was developed for the sheep industry to help farmers who wanted to diversify their herd with different types of scenarios. This was beneficial to the sector because it is one where there are fewer consultants and the tool could help both new and experienced farmers address any questions they may have.

Overall, while AgriRisk was able to support insurance development and insurance uptake, there were factors beyond its control. One key driver is whether the industry is supply managed or not. In general, in a supply managed sector, provincial boards can implement mandatory insurance requirements. This can also lead to more affordable insurance options, because a higher number of subscribers will decrease premiums. This nonetheless requires that there is agreement among the various stakeholders involved with respect to considerations like reciprocals. Outside of supply managed sectors, affordability is key. If producers find the price too high, they may be less inclined to adopt the insurance product and question why government is not supporting insurance and offering it at a discounted rate.

5.3 Effectiveness — Program Information Profile

Anticipated outcomes and associated performance indicators are set out for the program in a PIP. The key components of the PIP are presented in Table 2.

Table 2: AgriRisk Program's outcomes and indicators

Result level	Result statement	Indicator
Immediate outcomes	Increased awareness of risk management tools and mitigation strategies	1. (per PIP) Number of information items developed 1. (Added in performance reporting) Number of information items shared
Intermediate outcomes	Development and examination of risk management tools and services that engage producers	2. Adoption rate (ACB Stream) 3. Number of risk management tools developed (R&D Stream)
Ultimate outcomes	Producers have adopted risk management tools and strategies	4. Number of newly developed tools implemented by the industry and available to producers to manage business risk.

The evaluation included a review of the PIP, which identified several areas for improvement. These are outlined below.

Results chain

The PIP did not adequately reflect a logical chain of results. Thus, the logic model outcomes and related PIP would require updating if AgriRisk were still being delivered as a standalone program.

To create a logical progression between results, current immediate and the intermediate outcomes (as listed in Table 2) could be either reversed or moved to the same level. In a logic sequence, tools would generally be developed before awareness of tools can be increased. Similarly, producers cannot adopt a management tool and related strategies if they are not aware of them. In the specific case of AgriRisk, the development and awareness building may also occur simultaneously, with awareness being built as tools are developed, but before they are finalized. Thus, both development of tools and awareness building could be considered immediate outcomes.

PIP — performance indicators

The evaluation found some performance indicators as defined in the PIP were not clearly linked to their associated outcome. While the program updated the indicators used in performance reporting to better align with the anticipated outcomes, they did not carry these changes through to the PIP. The PIP is intended to capture the indicators used to measure results, and to be updated throughout a program's life to reflect changes to performance reporting. Thus, the performance indicators used to measure results should have been kept up to date in the PIP.

Several areas for clarification in the PIP were identified:

• The indicator "number of information items developed," which is the only indicator included in the PIP, does not adequately capture awareness. Nevertheless, in practice the program collected information on an additional revised indicator, "number of information items shared," that helps inform the assessment of the immediate outcome.
• The indicator "adoption rate" was used to measure progress toward the anticipated outcome of "development and examination of risk management tools and services that engage producers" for the ACB stream. Whereas the R&D stream measured the development of new risk management tools. There is a clearer alignment between the adoption indicator and the ultimate outcome, suggesting that an additional outcome statement may be required to clarify program logic for the R&D stream.
• There is an ultimate outcome and associated indicator established, but there is no target to accompany the indicator. Because of this, the extent to which the outcome was achieved cannot be determined. While information was collected through final performance reports to assess whether progress was made toward the outcome, the extent to which the outcome was achieved cannot be determined without a target to use as the threshold for success. While indicators that are difficult to measure can be used appropriately in PIPs, the indicator in this case (number of newly developed tools implemented by the industry and available to producers to manage business risk) could only have a quantitative target established based on information from final performance reports.

PIP — Underrepresented groups

The evaluation examined the degree to which AgriRisk integrated GBA Plus considerations and found they were integrated into the design and delivery to a very limited extent, based on the expectations of programs at the time. Under CAP, there were limited requirements for the integration of GBA Plus into programs. Similarly, there were no parameters within the program or the applicant guide to ensure that the new risk management tools developed were targeting producers from underrepresented groups.

More specifically, there was initially no requirement in the application or the project performance report template to indicate if and how projects would help underrepresented groups, or organizations serve underrepresented groups. This was addressed by the program in a later iteration of the funding application, implemented part way through the evaluation period, where applicants were asked if their project was targeting a specific underrepresented group (Indigenous farmers and land managers; women in agriculture; young and beginning farmers; official language minority communities and farmers with disabilities). However, this information was not used to determine whether funding would be allocated and applicants could decline to respond to these questions. This resulted in a data gap with respect to the extent those groups benefitted from the program or how the program may have impacted them otherwise. Nevertheless, the evaluation did not identify barriers to participation in AgriRisk, specifically with a view to underrepresented groups or GBA Plus considerations.

6.0 Lessons learned and conclusions

AgriRisk addressed a need in the agricultural sector by funding projects aligned with AAFC and federal government priorities related to the ability of producers to manage the risks they face. The program supported AAFC's core responsibility of sector risk, contributed to 2 departmental results and complemented other AAFC's programs with similar objectives. The program was also aligned with government priorities outlined in CAP. The program implemented practices to support efficient delivery and partially achieved its immediate and intermediate outcomes, at least in part due to low intake. To further strengthen future programming at AAFC, the evaluation identified 2 areas for improvement.

First, there is an opportunity for future programming to better understand the degree to which other groups are accessing and benefitting from funded projects and making sure there are no barriers to underrepresented group participation. Integrating GBA Plus, as now required under Sustainable CAP, should help in this matter. Specifically, to measure the integration of GBA Plus and related considerations in the design and delivery of future programs and assess their impacts, mandatory data collection requirements and a clear methodology are needed to capture findings that take into account GBA Plus principles.

Second, the review of the AgriRisk PIP demonstrated the need for current and future programs to ensure that indicators and outcomes are clearly aligned with the results logic model. It would also ensure that programs have reliable performance outcome data available for monitoring and evaluation purposes in order to manage and improve programs, policies and services. Such an exercise could be part of a cyclical review of performance measurement strategies.

Annex A: Logic model

Activities

• Recipient activities: Collecting and analyzing data; establishing project management teams; Assessing self-identified risks and mitigation strategies
• AAFC: Program Management and oversight
• Recipient activities: Negotiating with financial services providers to secure delivery of new tool(s)
• Recipient activities: Knowledge transfer activities on new risk management tools and mitigation strategies

Outputs

• Data analysis and reports; Risk assessments; Business case development for new tool(s)
• AAFC: Program Management and oversight products (# of contribution agreements; % of service standards met)
• Agreement(s) with a financial services delivery agent(s)
• Launch and implementation of agricultural risk management tool(s), products; Promotion and marketing products (brochures, online content); Education seminars and products (pamphlets, etc.)

Immediate outcomes

Increased awareness of risk management tools and mitigation strategies

Intermediate outcomes

Development and examination of risk management tools and services that engage producers

Ultimate outcomes

Producers have adopted risk management tools and strategies

Source: AgriRisk Program PIP, 2018.

Annex B: Evaluation methodology

Document review

The document review assessed Program relevance, design, delivery and effectiveness. A broad range of document types were analyzed over the reference period (2018-19 to 2022-23), including, AAFC policies, mandates and commitments. Several program documents were also reviewed, including applicant guides, previous evaluation reports and Program guidelines.

Data review

A data review was conducted to support the assessment of program delivery, efficiency and effectiveness. The data review included data for all projects submitted in the evaluation period. Most of the analysis conducted examined data for all funded AgriRisk projects with signed contribution agreements from the program's database.

Project file review

A project file review was conducted to support the assessment of program relevance and effectiveness. The review provides an analysis of documents contained in all AgriRisk project files between (2018-19 to 2022-23). Analysis included project application information, budget and data from annual and final performance reports, where available.

Key informant interviews

Key informant interviews gathered insights from program stakeholders on the relevance, design, delivery and effectiveness of AgriRisk. A total of 12 interviews were conducted between October and December 2023. Recipients were preselected for interviews based on their project's stream, region and funding amount.

Case study

The evaluation included a case study on the Livestock Price Insurance (LPI) Program that received funding from AgriRisk (see Figure 2). This qualitative research method examined LPI through the collection of data from multiple sources (for example, 4 interviews and document review). This case study allowed for a more in-depth analysis of this specific project.