Evaluation of the AgriAssurance Program

On this page

Abbreviations

• AAFC: Agriculture and Agri-Food Canada;
• GBA: Plus Gender-based Analysis Plus;
• CFIA: Canadian Food Inspection Agency;
• FPT: federal-provincial-territorial;
• SME: small and medium-sized enterprise

Executive summary

Purpose

The Office of Audit and Evaluation of Agriculture and Agri-Food Canada (AAFC) conducted an evaluation of the AgriAssurance Program to provide senior management with an assessment of the relevance, design, delivery, efficiency and effectiveness of the Program.

Scope and methodology

AgriAssurance Program activities were evaluated using multiple lines of evidence: a literature review; document, data, file and consultation review; interviews with AAFC staff, recipients, partners and stakeholders; and five case studies. The evaluation focussed on AgriAssurance support to not-for-profit organizations and small and medium-sized enterprises (SMEs) between 2016–17 and 2020–21.

Background

The AgriAssurance Program supports the Canadian agriculture and agri-food industry to put in place assurance systems that provide consumers and buyers with confidence in the health, safety and quality of Canadian agricultural products. Support is provided through two program components.

• The national industry association component provides funding to not-for-profit organizations to help industry develop and adopt national systems, standards and tools to support health and safety claims about agriculture and agri-food products.
• The pilot SME component provides targeted support to companies to implement third-party assurance certifications required to access foreign markets.

Findings

• The AgriAssurance Program addresses the need for public trust in food safety and quality and is responding to emerging sector priorities, such as environmental sustainability.
• The Program is aligned with government and departmental roles and priorities, particularly the Canadian Agricultural Partnership priority of securing and supporting public trust.
• The Program is reaching national industry associations representing the majority of Canadian agricultural sectors. The pilot SME component supported agricultural processors to obtain third-party certification required to respond to export opportunities, though uptake was lower than expected.
• There are continuing challenges with the alignment of federal-only AgriAssurance investments in the development of assurance systems and FPT Cost-shared assurance investments for the implementation of these same systems.
• Performance data is limited due to a lack of ongoing collection and sharing of outcome data related to the provincial-territorial responsibility for assurance system implementation.
• The Program collects data on the diversity of groups applying for and benefiting from, program funding, but Gender-based Analysis Plus (GBA Plus) considerations are not part of funding decisions.
• The Program is meeting its service standards, but there were delays in project and funding approvals, and the Program has consistently underspent its budget allocation.
• A reliance on outputs for program and performance reporting makes it difficult to assess the Program’s effectiveness. Evaluation case studies and interviews provided evidence that some AgriAssurance projects have had a significant positive impact on their sectors and fulfilled the Program’s objectives.

Conclusion

The evaluation confirmed the relevance of AgriAssurance, as robust assurance systems are key to maintaining public confidence in the food system and the Program has adapted to respond to changing sector priorities. While it is clear from the case studies that some national industry association projects under the AgriAssurance Program have achieved their expected results, weak performance measurement meant an overall Program assessment was not possible. Refinements to the Program logic model, along with annual tracking of qualitative and quantitative outcome measures would enable AgriAssurance to better report on results.

With the exception of funding delays at the outset of the Canadian Agricultural Partnership, recipients were generally satisfied with program delivery. Under the national industry association component, AAFC expertise and its robust technical review process were seen as contributing to program outcomes. The evaluation identified continuing gaps in assurance system implementation at the provincial-territorial level in the absence of formal commitments to align federal-provincial-territorial (FPT) Cost-shared Program investments with federally funded projects to develop systems and standards. In other words, federal investments may build a system, but there are no guarantees that provincial-territorial level investments will be available to implement said system. Improved governance structures, work planning and reporting for the FPT Working Groups along with regularized meetings could help to improve FPT communication and coordination.

As identified in the previous 2017 evaluation^{Footnote 1}, the Program has again not spent its budget allocation, underspending by one-third over the evaluation period. The pilot SME component appeared to support for agricultural processors to obtain third-party certification required to respond to export opportunities, though acknowledging the deliberately limited promotion campaign, Program uptake was still less than expected. An expanded marketing or promotion campaign that directly targets exporting SMEs in the food processing sector, along with other Program participants, could position the Program to expand its reach and truly test its effectiveness in equipping the sector with assurance systems and supporting tools.

Recommendations

Recommendation 1: The Assistant Deputy Minister, Programs Branch, should improve the AgriAssurance performance measurements (including updating the project logic model and indicators) and progress reporting requirements.

Recommendation 2: The Assistant Deputy Minister, Programs Branch, should review AgriAssurance Program design elements to improve coordination with Cost-shared programming and encourage alignment of supporting provincial-territorial programming.

Recommendation 3: The Assistant Deputy Minister, Programs Branch, should develop and implement a marketing or promotion plan to broaden awareness of the Program and encourage uptake.

Management Response and Action Plan 

Management agrees with the evaluation recommendations and has developed an action plan to address them by April 2024.

1.0 Introduction

The Office of Audit and Evaluation conducted an evaluation of Agriculture and Agri-Food Canada’s (AAFC) AgriAssurance Program as part of the 2020–21 to 2024–25 Integrated Audit and Evaluation Plan. The evaluation complies with the Treasury Board of Canada’s Policy on Results and fulfills the requirements of the Financial Administration Act. The evaluation is intended to inform current and future program and policy decisions.

2.0 Scope and methodology

The evaluation assessed the relevance, design, delivery, efficiency and effectiveness of the AgriAssurance Program between 2016–17 to 2020–21. Case studies assessed longer-term performance by reviewing projects that began prior to 2016–17 and were funded under previous frameworks. The Office of Audit and Evaluation used multiple lines of evidence to evaluate the Program including a literature review; document, data, file and consultation review; AAFC staff, recipient, partner and stakeholder interviews; and five case studies. A previous evaluation of the Program was completed in 2017 ^{Footnote 2}. For the detailed methodology including definitions for qualitative descriptions such as “almost all”, “majority”, “some”, please see Annex A.

3.0 Program profile

3.1 Program description and delivery

The AgriAssurance Program supports the Canadian agriculture and agri-food industry to put in place assurance systems that provide consumers and buyers with confidence in the health, safety and quality of Canadian agricultural products. More specifically, the Program’s objective is to support industry to:

• develop, verify and integrate assurance systems to address market and regulatory requirements, such as food safety, plant and animal health, market attributes and quality standards; and
• enable industry to make credible, meaningful and verifiable claims about the health safety and quality of agricultural products.

An assurance system is a systematic process for determining whether a product or service meets specified standards or requirements. The intent of such systems is two-fold: increasing customer confidence in a product or service; and, enhancing a supplier’s credibility, and thus, its competitive advantage. In the context of the AgriAssurance Program, this includes processes and procedures that provide confidence that:

• risks along the food supply chain are addressed; and
• relevant claims can be proven – that food is safe, it was sourced from a healthy resource base and has the attributes buyers seek and for which they will pay.

Under three previous federal-provincial-territorial (FPT) agricultural policy frameworks between 2003 and 2018, the AgriAssurance Program and its predecessor (AgriMarketing Stream D: Assurance Systems) helped to develop national on-farm and post-farm assurance systems and standards. The Program continues to provide this support under the Canadian Agricultural Partnership, the 2018-23 policy framework, through the national industry association component. Within the FPT agricultural policy framework, the Program supports the development of national assurance systems, while the provinces and territories support system implementation through FPT Cost-shared programming. Under the Canadian Agricultural Partnership, the Program added a new component as a pilot providing support to small and medium-sized enterprises (SMEs).

National industry association component (NIA)

The NIA component funds industry projects to develop and adopt systems, standards and tools that support health and safety claims about Canadian agriculture and agri-food products. Priority is given to projects in the following areas:

• providing confidence in market attributes and quality standard claims;
• responding to consumer concerns, such as, animal welfare and environmental sustainability;
• advancing existing industry assurance systems in food safety and traceability, and
• creating efficiencies by integrating assurance systems.

Eligible recipients include not-for-profit organizations operating at the national level in the agriculture, agri-food and agri-based products or the fish and seafood sector. Applications from not-for-profit organizations operating in Canada at the regional level may be considered if there is no national representation, and if the applicant can demonstrate their ability to deliver a project from a national or agri-food sector-wide perspective.

The maximum AAFC contribution for a project will normally not exceed $1 million over five years. Eligible project costs are to be shared between AAFC and the applicant, with AAFC providing a maximum of 75% of project costs.

Small and medium-sized enterprises (SME) component

An SME component pilot was undertaken that provides targeted support for SMEs to obtain third-party certification for the first time that is required to respond to an export opportunity.

Eligible SMEs^{Footnote 3} must operate in the agriculture, agri-food and agri-products or the fish and seafood industries. They must be directly involved in growing, harvesting, processing or otherwise transforming or consolidating their products. The maximum AAFC project contribution will normally not exceed $50,000. AAFC provides funding for up to 50% of project costs.

3.2 Governance and Program delivery

The AgriAssurance Program is managed by AAFC’s Competitiveness Division within the Business Development and Competitiveness Directorate of the Programs Branch. Directorate staff evaluate applications to ensure they meet basic eligibility requirements before being assessed by technical experts at AAFC or the Canadian Food Inspection Agency. Applications are also sent for comment to members of the FPT Assurance Working Group. National industry association applications are then reviewed by a Committee of Directors General before ultimate approval by the Program Director, Director General, Assistant Deputy Minister or Minister depending on the value of the project. SME projects are approved by the Program Director  following a technical review.

3.3 Resources

The AgriAssurance Program was renewed in 2018–19 at a value of $74 million and continues to accept applications until September 2022 or until the Program’s budget has been fully committed. The Program was allocated $61.4 million in funding for Grants and Contributions (Vote 10) and $12.8 million for Program administration in salary and non-pay operating expenses (Vote1). At the time of Program renewal, AgriAssurance was allocated a staff of 18.1 full-time equivalents.

3.4 Intended outcomes

The AgriAssurance Program aims to achieve the following outcomes:

Immediate outcomes

• The agriculture and agri-food sector is prepared to develop assurance systems and supporting tools (for example, food safety, animal and plant health, market attributes and quality standards, and system integration).
• Industry’s awareness and knowledge of initiatives to mitigate sector risk are increased.

Intermediate outcome

• The agriculture and agri-food sector develops and implements assurance systems and supporting tools (for example, food safety, animal and plant health, market attributes and quality standards and system integration).

Ultimate outcome

• The agriculture and agri-food sector is equipped with assurance systems and supporting tools.

The AgriAssurance Program logic model in Annex B provides further detail on the Program's key activities, outputs and outcomes.

4.0 Relevance

4.1 Alignment of Program with current and emerging priorities

Public trust in the food system influences what consumers will buy. That trust can be damaged through crises, such as food recalls or food-borne illnesses^{Footnote 4}. Food assurance systems, such as certifications, origin labelling and traceability, can provide information about food safety characteristics and increase public trust^{Footnote 5 6 7  8}. Assurance systems can also be useful because they inform consumers of other important issues such as food quality or taste^{Footnote 9}, animal welfare or environmental sustainability.

In a survey of national industry associations in the agriculture sector conducted by the Program in January 2021, almost all respondents confirmed that assurance systems are important to maintaining and building public trust. An investment analysis conducted by the Program in March 2021 showed that projects funded by AgriAssurance aligned with the main priority areas identified by the survey and industry consultations, specifically: food safety, animal welfare, managing certifications, traceability, environmental sustainability and biosecurity.

The evaluation noted the potential for greater use of technology in assurance programs, particularly in food testing and in data management and analysis^{Footnote 10}, including the potential use of blockchain technology. A 2021 Organisation for Economic Co-operation and Development scoping paper suggested that governments have a role to play in addressing constraints to digitization in agriculture. The Program received two applications under the Canadian Agricultural Partnership to use blockchain technology in traceability projects but has yet to fund either of these projects. Those applications did not meet the program criteria and were ineligible to receive funding.

Interviews with AAFC staff indicated that strong relationships with national industry associations have helped the Program to meet current and emerging industry priorities, particularly those of the on-farm animal sectors that face the greatest risks. Staff confirmed that the evolution of industry applications demonstrates how the Program is adapting to the changing sector priorities and market demands in areas such as environmental sustainability. One example is the Turkey Farmers of Canada, which over the years has completed AgriAssurance projects to develop its food safety, biosecurity and animal welfare programs. With Program support under the Canadian Agricultural Partnership, the Turkey Farmers of Canada is now working on a lifecycle analysis of the environmental impacts of its products.

All interviewed recipients of AgriAssurance funding through the SME component confirmed that the Program responded to their priorities to certify the safety or quality of their products to access new markets, including Europe and Asia. Almost all of the funding recipients who had completed their final project reporting said they would apply to the Program again for assistance in obtaining food safety certification for the first time.

For both the national industry association and SME Program components, almost all funding recipients interviewed indicated their projects would either not have gone ahead or been much smaller in scale without AgriAssurance support. In one case, an industry association noted that AAFC funding enabled it to undertake innovative projects to address consumer demands for sustainably sourced meat products.

4.2 Alignment with AAFC and government role and priorities

The AgriAssurance Program aligns with the federal government's roles, responsibilities and priorities through support to public health, expanding the economy and supporting environmentally sustainable practices. At the departmental level, the Program aligns with AAFC’s priorities of increasing exports to new markets and supporting risk management practices. AgriAssurance supports government priorities by positioning industry to be more resilient and sustainable and to take advantage of areas that are opening up as a result of free trade agreements.

AgriAssurance contributes to the Canadian Agricultural Partnership priority of enhancing public trust by supporting the creation and improvement of assurance projects that enable industry to respond to consumer, buyer and market demands; gain and maintain market access; and protect against threats to plant and animal health. The evaluation confirmed that projects funded under the Canadian Agricultural Partnership aligned with the public trust priority. The evaluation also confirmed that the SME component of AgriAssurance addresses the AAFC priority of market diversification and supports the expansion of SMEs into international markets.

The majority of funded projects align with the division of federal roles and responsibilities as most assurance funding was directed at program development, while the provincial-territorial funding is mainly directed to implementation and adoption. Evidence suggests that greater awareness of funded projects among AAFC, industry and the provinces was needed to avoid any potential for overlap.

The SME component was one area where there is potential for overlap with other federal and provincial small business support programs; however, an analysis of 25 provincial and federal government programs found no duplication between AgriAssurance and similar programs supporting SMEs. While the Program’s overall objective of meeting new market demands for third-party assurance certifications did overlap with one provincial program in New Brunswick, AgriAssurance is unique in addressing international market requirements. Further, the SME component provides an opportunity for provincial and federal governments to complement each other and help to meet the high demand for funding support from SMEs.

5.0 Program design and delivery

This section presents the evaluation findings on the effectiveness of program design and delivery including implementation of Gender-based Analysis Plus requirements.

5.1 Effectiveness in reaching target population

Under the Canadian Agricultural Partnership policy framework, between 2018-19 and February 2022, the Program approved 76 projects. The national industry association component represented the vast majority of AgriAssurance funding, with $46 million in funding allocated to 33 associations for 50 projects. Funded organizations represent 11% of the approximately 300 national industry associations that the Program estimates would be eligible for funding. The evaluation found that the Program is designed to be industry-driven and funding recipients include associations representing the majority of agricultural sectors and production in Canada (for example, in beef, pork, dairy, poultry, grains and pulses). The bulk of funding for national industry associations under the Canadian Agricultural Partnership went to projects in animal health, traceability and environmental sustainability. Producer organizations in dairy, animal support activities, pork and beef accounted for more than half of AgriAssurance investments.

All industry association funding recipients interviewed were satisfied with the distribution of AgriAssurance funding and felt the application and intake process was effective. Some of the industry associations recipients noted they appreciated the thoroughness of the technical review process and that the Program application process compared favourably to other federal government programs. Pre-application discussions with AgriAssurance staff were viewed as helping applicants understand the Program requirements. Approximately two-thirds of the applications to the Program were approved for funding following review by technical experts in specific fields.

The SME pilot program was introduced during the current funding framework to support SMEs to obtain the third-party certification required to respond to an export opportunity for the first time. Twenty-six food processing businesses were approved for $685,000 in support for food safety and quality standards projects under the SME component. This was much lower than the forecast of $2.2 million in annual spending. The number of funded businesses is small when compared to the more than 1,400 exporting SMEs that would potentially be eligible to apply. Challenges for SMEs include limited circumstances covered by the Program and the requirement to obtain third-party certification to respond to an export opportunity.

The Program intentionally limited promotion of the pilot to prevent being overwhelmed by excessive demand during the initial pilot phase. Promotion efforts were restricted to sharing at the Interdepartmental Public Trust Working Group; articles in a few industry bulletins and trade association newsletters; sharing information with a mailing list; publication in AAFC’s Regional Monthly Newsletters; and presentations to regional office meetings.

The evaluation found some evidence that suggests limited uptake could be due to the 50/50 cost-share model whereby the federal government paid half of project expenses and the resulting impact on small businesses. On average, AAFC provided just over $26,000 in support to individual SMEs. Three of the four SME recipients interviewed indicated the funding only covered the cost of certification for one year, and not subsequent renewals, even though the return on investment from an applied assurance system may not be realized within one year.

5.2 Alignment with provincial-territorial programming

In the Multilateral Framework Agreements between FPT governments that have underpinned the agricultural policy frameworks, the federal role is to support the development of national assurance systems, while the role of the provinces and territories is to support system implementation through FPT Cost-shared programming. Under the Agreements, provinces and territories have the flexibility to decide how to implement federally funded systems/programs. The 2021 Program investment analysis showed there are significant gaps between federal AgriAssurance funding in system development and FPT Cost-shared investments in implementation at the provincial level. For example, in the pork sector AgriAssurance approved over $5 million in funding for the Canadian Pork Council under Growing Forward 2 and the Canadian Agricultural Partnership to develop integrated assurance programs for food safety and biosecurity, animal care, and traceability. Only two of the five major pork producing provinces then provided support for the implementation of all of those systems under the Canadian Agricultural Partnership, limiting the value of the $5 million federal investment.

The 2017 evaluation of AgriAssurance recommended closer collaboration and coordination between the Program and the provinces-territories and industry to facilitate the uptake of assurance systems. In its management response, Programs Branch committed to using the FPT Assurance Working Group as a forum for greater information sharing and consultation. Under the 2018–23 FPT agricultural policy framework, that forum has had limited effectiveness as the Working Group did not meet for 18 months between November 2019 and April 2021 and several Working Group deliverables, including a work plan, were not completed. In interviews, provincial representatives confirmed the limited effectiveness of the Working Group. The Assurance Working Group did, however, provide a forum for collecting the data for the extensive 2021 FPT investment analysis referenced above that was designed to inform the Next Policy Framework.

In addition to the lack of a formal coordination mechanism, the evaluation identified a need for improved communication between AgriAssurance and the provinces-territories. Gaps in the governance structure, defined tasks and the reporting for different FPT working groups were identified by the Program. While these working groups cannot ensure systemic FPT investment coordination, they could be used to enhance cooperation and assess whether programs are funding desired projects.

5.3 Effectiveness of performance measurement systems

In year four of the Canadian Agricultural Partnership, there was little performance information available on progress toward Program outcomes. The Program collects extensive annual reporting on the completion of project activities or outputs to monitor project implementation, but recipients are only required to submit a performance report on project outcomes or results upon completion of their multi-year projects. Typically, recipients of federal contribution programs provide annual reporting on progress toward expected outcomes.

Since the last evaluation in 2017, Programs Branch developed a new logic model and indicators to capture performance information. Given the Program’s role in systems and standards development, the short and medium-term indicators focus largely on project outputs, specifically the number of projects approved, as well as the number of assurance systems developed, advanced or implemented. Under the Canadian Agricultural Partnership, the Program has required funding applicants to submit an implementation plan with targets for uptake of the systems and standards they are developing, providing a means for benchmarking progress. The Program’s longer-term indicator is the “percentage of implementation plans reported to be functioning”. The evaluation found that little reporting was available on the progress toward meeting the targets in those project implementation plans.

For the SME component, recipients are required to submit a performance report at the end of their project which requires them to report on results achieved, challenges experienced, and lessons learned through participation in the Program. The results indicators – value of new sales, and new leads/opportunities generated as a result of the certification – reflect the outcomes one would expect from a small business support program. However, for most of the eleven recipient companies which had submitted their final performance reporting in the evaluation timeframe, it was too early for them to have generated new sales or leads.

Under the national industry association component, the lengthy time required to develop assurance systems and provincial responsibility for system implementation has discouraged attempts to measure the uptake or “use” of funded systems and their impact on industry. Since AAFC has provided funding for assurance systems and standards over the course of four agricultural frameworks, often to the same organizations, information on these longer-term results should be captured. The evaluation case studies show that such information is available from the national industry associations receiving AgriAssurance funding, but is not routinely collected by the Program.

Branch staff advised that while performance reporting may not play a key role in informing ongoing Program management, individual project results and recipient success are used to inform future funding decisions. The Program used their investment analysis to demonstrate the link between assurance systems funded by AgriAssurance and its predecessors, and whether these are being implemented on farms and in the processing of agricultural products. In the absence of other performance reporting, the investment analysis, interviews and case studies formed much of the evaluation evidence.

An examination of the Program’s performance measurement systems pointed to other areas for improvement. The ultimate outcome in the current program logic model does not capture Program results, specifically its objective of enabling industry to make credible, meaningful and verifiable claims about the health, safety and quality of Canadian agricultural products. The model lacks measures to track the specific outcomes for the SME project component. Further, performance indicators are limited to the collection of quantitative information. The collection of both quantitative and qualitative data on progress toward outcomes in annual project reporting would better enable the Program to report on results. Best practices  for collecting performance data include requiring recipients to report annually on the incremental advancement of their systems and standards, collecting data on longer term outcomes of previous projects as part of the application process for new projects, or periodic follow-up phone calls with previous funding recipients to obtain updates on system implementation.

5.4 Gender-based Analysis Plus

AgriAssurance funding applications include two questions to gather GBA Plus information. Applicants are asked if their organization’s mandate focuses on Indigenous peoples, persons with disabilities, visible minorities, women or youth, or if those groups will directly benefit from project activities. Submitting this information is voluntary and responses are not included in the assessment of project applications. Since benefits from the national industry association component supporting the development of assurance systems are broad and diffuse, the Program reported it is not possible to disaggregate the distribution of those benefits. For the SME component, AAFC’s 2022–23 Departmental Plan on GBA Plus states that AgriAssurance is supporting diversity and inclusion as many applicants are women and marginalized entrepreneurs.

The evaluation examined what measures AgriAssurance has in place to reach official language minority communities. Program staff were involved in the process of developing the department’s Official Languages Positive Measures Tool – a questionnaire designed to document the official languages impact of project funding – but the tool has yet to be used consistently in project review. From examination of a sample of projects, the evaluation found that official languages measures were included in the workplans of the majority of national industry association projects. However, only a few projects included official languages considerations in their progress or performance reports. Official languages considerations were not included in the workplans or reporting for SME projects.

6.0 Efficiency

AgriAssurance generally met or exceeded is client service standards for responding to inquiries, rendering decisions on applications and issuing payments. However, at the outset of the Canadian Agricultural Partnership in 2018–19, it did not meet its standards for responding to general inquiries, reviewing project applications and issuing payments under the national industry association component. That same year it was slow in responding to inquiries about the SME component and in 2019–20 it did not meet its standards for reviewing SME applications. Some industry association representatives interviewed were critical of the time taken for project approval and of delays in receiving contribution agreements following project approval.

The evaluation noted that, as in previous policy frameworks, delays and underspending resulted from the heavy workload in the first year of the Canadian Agricultural Partnership. The Program saw a larger volume of applications than under the previous policy framework while at the same time staff were working to close out projects from the previous framework.

AgriAssurance has underspent its allocated budget since 2013–14. Evidence from the previous evaluation showed that in the first three years of Growing Forward 2 (2013–14 to 2015–16), the Program spent only half of its grants and contributions funding. Between 2016–17 and 2020–21 the Program spent only two-thirds of its allocated budget. (See details in Table 1) Most of the underspending under the Canadian Agricultural Partnership occurred in the first year due to the additional time required to approve projects, activities being pushed to a subsequent fiscal year and put contribution agreements in place.

Table 1: AgriAssurance Program Expenditures 2016–17 to 2020–21

	Growing Forward 2		Canadian Agricultural Partnership
Year	2016-17	2017-18	2018-19	2019-20	2020-21	Total
Actual ($)	8,544,993	8,460,255	2,468,647	11,062,530	10,461,921	40,998,346
Planned ($)	13,640,693	10,483,875	12,280,000	11,761,892	12,979,361	61,145,821
Variance ($)	5,095,700	2,023,620	9,811,353	699,362	2,517,440	20,147,475

7.0 Effectiveness

7.1 Achievement of expected outcomes

As the Program only collected output data under Growing Forward and Growing Forward 2, performance reporting consists of information on the number and type of projects funded along with the number of systems and standards completed or advanced as detailed in Table 2.

Table 2: Growing Forward and Growing Forward 2 Results

On-farm	Post-farm
46 national assurance systems/standards completed	3 national assurance systems/standards completed
50 national assurance systems/standards advanced	16 national assurance systems/standards advanced

Under the Canadian Agricultural Partnership, the Program reported in November 2021 that of the 61 assurance systems under development with AgriAssurance support, 39 were more advanced than they were at the beginning of the framework. No qualitative reporting was available to provide insight or context on those numbers. To address the lack of performance information, the evaluation used interviews and case studies to gather evidence on the achievement of expected outcomes.

Program staff, external stakeholders and national industry association component funding recipients noted that AgriAssurance assisted the sector in developing assurance systems and tools they can deploy to build public trust and access new markets. All SME recipients interviewed, confirmed that support from the Program helped them to develop certification programs needed to meet safety standards. Most noted these certifications also helped to grow their export sales or is expected to grow those sales.

Five case studies of successful AgriAssurance projects under the national industry association component showed that Program funding contributed to the achievement of the Program’s intermediate outcome: the agriculture and agri-food sector develops and implements assurance systems and supporting tools, and ultimate outcome: the agriculture and agri-food sector is equipped with assurance systems and supporting tools.

In all but one of those cases, there was evidence those projects had a significant impact on industry and or consumers thus achieving the Program’s objective of enabling industry to make credible meaningful and verifiable claims about the health, safety and quality of Canadian agricultural products. Evidence from the case studies is summarized below.

Organization: Canadian Cattleman’s Association
Number of projects: Three projects (2015–2023)
Value of funding: $4.4 million

The Canadian Cattleman’s Association developed the Certified Sustainable Beef Framework using AgriAssurance Program support. The Certified Sustainable Beef Framework recognizes sustainable practices in Canadian beef production and processing through an audited process, enables sourcing of beef from CRSB Certified Sustainable Operations, and delivers science-based consumer assurances about beef sustainability. Canadian Cattleman’s Association staff indicated 1,300 producers have been certified under the Framework, which represents 17% of Canadian cattle herds. Food companies are beginning to source beef from farms and ranches that have been certified under the Framework, including McDonald’s Canada Limited for their Quarter Pounders and Gordon Food Service for the Gordon Choice boxed beef program. Loblaws Companies Limited made its first purchase of one million pounds of beef from certified farms and ranches in 2020 and has committed to purchase an additional three million pounds of sustainable beef by 2023.

Organization: Canadian Nursery Landscape Association
Number of projects: One project (2020–2023)
Value of funding: $82,000

AgriAssurance is supporting the Canadian Nursery Landscape Association to test a pest management program for the box tree moth that would comply with the phytosanitary requirements for exporting nursery plants to the United States. For example, boxwood is a common shrub used in landscaping and in May 2021, the United States closed the border to all boxwood imports from Canada due to moth infestations.

The Association has developed a certification module for the box tree moth and submitted it to the Canadian Food Inspection Agency (CFIA) for review. Growers have begun implementing the certification practices related to monitoring and surveillance which have produced valuable data. This data, when integrated with data from CFIA, confirmed that British Columbia is free of box tree moth. CFIA is in discussions with the United States Department of Agriculture to remove British Columbia from the federal order restricting imports of boxwood from Canada.

Organization: Canadian Pork Council
Number of projects: Three projects (2016–2023)
Value of funding: $5.85 million

AgriAssurance funding contributed to the development and enhancement of the Canadian Pork Excellence Program Platform that will integrate three assurance programs for food safety and biosecurity (PigSAFE), animal care (PigCARE) and traceability (PigTRACE) into one platform. Canadian Pork Council staff confirmed the PigTRACE program has been fully implemented across Canada as required under federal Health of Animals regulations and has led to improved reporting on the movement of pigs across the country. Traceability for pigs is seen as vitally important for protecting Canadian producers from the African Swine Fever pandemic that has infected pigs in Africa, Central Asia and Europe^{Footnote 11}.

The Council reports that 65% of hog producers have enrolled in the PigSAFE/PigCARE food safety, biosecurity and animal care programs which is estimated to represent 63% of the pigs in Canada. These programs are under technical review by CFIA. CFIA recognition for food safety programs is a key step in helping producers demonstrate they are part of a credible and effective assurance program that is based on current on-farm standards. One external expert indicated assurance system development has supported the access of Canadian pork to markets in Asia which have traditionally had a high focus on food safety.

Organization: Dairy Farmers of Canada
Number of projects: Five projects (2013–2023)
Value of funding: $10.1 million

Support from AgriAssurance, under Growing Forward 2 and the Canadian Agricultural Partnership, has enabled the Dairy Farmers of Canada to expand and implement its quality assurance program under the proAction Initiative. The Dairy Farmers of Canada has developed six modules related to milk quality, food safety, animal care, livestock traceability, biosecurity and environment. Five of the six modules have been fully implemented, with the remaining environment module expected to be fully implemented by September 2023. Since participation in proAction is mandatory for all milk producers in Canada, nearly all Canadian dairy farmers are registered with the Program which demonstrates that they conform with proAction standards. Farmers are required to keep detailed records and their practices are reviewed by independent validators who are trained to verify they are meeting proAction requirements. Dairy Farmers of Canada staff indicated that proAction has helped the industry to build trust in the reputation of Canadian milk and provide assurance to consumers.

Organization: Jewish Community Council of Montreal
Number of projects: Five projects (2013–2023)
Value of funding: $3.2 million

AgriAssurance funding has helped the Jewish Community Council of Montreal to develop a certification system for food safety in kosher food and expand that system to other specialty foods such as halal and lactose-free under the Global Specialty Food 22000 certification. Jewish Community Council of Montreal latest project is working to develop a global plant-based certification program.

As of fall 2021, approximately 350 SMEs were participating in Global Specialty Food 22000. The majority of these are in Europe and the United States, with 25 Canadian businesses participating. Jewish Community Council of Montreal staff noted the certification program has helped some SMEs with financial constraints to achieve the food safety standards demanded by larger buyers and retailers.

7.2 Factors contributing to or constraining outcome achievement

Evidence from case studies and interviews identified the factors impacting achievement of Program outcomes, including the development and implementation of assurance systems and supporting tools. The AgriAssurance Program logic model in Annex B provides further detail on the Program's key activities, outputs and outcomes.  

Contributing factors

• Industry and government interest and support including backing from regional industry associations and farmers, and federal and provincial governments.
• AAFC funding and expertise including favourable cost-sharing ratios for the National Industry Association component, clearly communicated program criteria, a robust technical review process and helpful Program staff.
• Mandatory implementation of assurance systems in supply managed sectors such as dairy and poultry.
• CFIA recognition which provides credibility and encourages uptake of assurance systems related to food safety.

Constraining factors

• COVID-19 slowed the progress of projects, preventing in-person meetings, farms visits and other activities such as biosecurity audits.
• Uneven implementation support for nationally developed assurance systems via the Cost-shared Program. In one case study a lack of funding for marketing was suggested to be the reason for low uptake of assurance tools.
• Heavy record keeping burden for compliance with assurance systems falls disproportionately on producers. Reducing that burden through streamlining tools could encourage the adoption of assurance systems.
• Delay receiving CFIA recognition and feedback, in part due to slowdowns with technical reviews at CFIA during the pandemic^{Footnote 12}.

8.0 Conclusions and recommendations

The evaluation confirmed the relevance of AgriAssurance as robust assurance systems are key to maintaining public confidence in the food system and the Program has adapted to respond to changing sector priorities. While it is clear that some national industry association projects under the AgriAssurance Program have achieved their expected results, weak performance measurement meant an overall assessment of the Program was not possible. There is an opportunity for the Program to review its performance reporting, including revisiting its logic model to include appropriate outcomes and a logical progression toward the Program’s objective of enabling industry to make credible, meaningful and verifiable claims about the health, safety and quality of Canadian agricultural products. Indicators should to be refined to capture Program outcomes. Collection of qualitative and quantitative data on progress toward those outcomes should be incorporated into annual project reporting. Given the long timeframe required to develop and implement assurance programs, the Program should collect data on the cumulative impact of Program funding over various agricultural frameworks.

With the exception of funding delays at the outset of the Canadian Agricultural Partnership, recipients were generally satisfied with Program delivery. Under the national industry association component, AAFC expertise and its technical review process were seen as contributing to program outcomes. The evaluation found continuing gaps in assurance system implementation at the provincial-territorial level in the absence of formal commitments to align FPT Cost-shared Program investments with federally funded projects. Improved governance structures, work planning and reporting for the FPT Working Groups along with more regular meetings could improve FPT coordination.

The pilot SME component supported agricultural processors to obtain third-party certification required to respond to export opportunities. Acknowledging the limited promotion planned for the SME pilot, uptake of the pilot was significantly below the forecast. An enhanced communication campaign would enable the Program to expand its reach not only to SMEs, but all potential recipients and truly test Program effectiveness in equipping the sector with assurance systems and supporting tools.

As identified in the previous evaluation, the Program has again not spent its budget allocation, underspending by one-third over the period of the evaluation. Consideration should to be given as to whether the Program has sufficient reach, whether demand for the program is lower than expected broadly or whether funding should be routinely reduced in the first year of the Policy Framework when new assurance projects are being reviewed and approved, taking into account the additional time needed to initially disburse funds.

Recommendations

Recommendation 1: The Assistant Deputy Minister, Programs Branch, should improve the AgriAssurance performance measurements (including updating the project logic model and indicators) and progress reporting requirements.

Recommendation 2: The Assistant Deputy Minister, Programs Branch, should review AgriAssurance Program design elements to improve coordination with Cost-shared programming and encourage alignment of supporting provincial-territorial programming.

Recommendation 3: The Assistant Deputy Minister, Programs Branch, should develop and implement a marketing or promotion plan to broaden awareness of the Program and encourage uptake.

Management response and action plan

Management from Programs Branch is supportive of the recommendations and have developed an action plan to address them by April 2024.

Annex A: Evaluation methodology

Literature review

The evaluation assessed peer-reviewed and grey literature published since 2016 to assess Program relevance and provide context for the environment in which the Program operates. This work was supported by staff from the Canadian Agricultural Library, who provided evaluators with articles on relevant topics including, agricultural and agri-food assurance systems in the areas of food safety, food quality, traceability and public trust.

Document, file and data review

A review of documents, files and data were used to collect information on all evaluation issues. This included a selection of project reports under the Canadian Agricultural Partnership and performance reports completed at the end of Growing Forward 2, key project documents such as program manuals and applicant guides, program foundational documents, the AgriAssurance Program Performance Information Profile and the previous evaluation of AgriAssurance published in 2017.

Interviews

Twenty-seven interviews were conducted with internal and external stakeholders to gather their views on Program relevance, design and delivery, efficiency and effectiveness. Those interviewed included:

• 5 program staff
• 5 funding recipients from national industry associations,
• 4 SME funding recipients,
• 11 external partners and stakeholders including provincial government representatives and other partners, and
• 2 two subject matter experts.

The following scale was used to describe the significance of the qualitative findings in terms of the relative proportion of responses: 

• “All/almost all” – findings reflect the views and opinions of 90% or more of the interviewees in the group.
• “Large majority/most” – findings reflect the views and opinions of at least 75% but less than 90% of interviewees in the group.
• “Majority” – findings reflect the views and opinions of at least 51% but less than 75% of interviewees in the group.
• “Half” – findings reflect the views and opinions of 50% of interviewees in the group.
• “Some” – findings reflect the views and opinions of at least 25% but less than 50% of interviewees in the group.
• “A few” – findings reflect the views and opinions of at least two interviewees but less than 25% of interviewees in the group.

Review of data and analysis from industry consultations by the Program

In preparation for the next agricultural policy framework, the Program conducted a survey and workshops with industry representatives. The survey was distributed to 79 associations in January 2021, including applicants under the Canadian Agricultural Partnership and previous frameworks, as well as members of the Industry-Government Advisory Committee. AAFC received 33 valid responses. The Program also conducted a series of six workshops with industry representatives in May 2021. The evaluation analysed the survey results and the notes from the six workshops to inform questions related to relevance and design and delivery.

Case studies

Five case studies were conducted of national industry associations that received Program funding under Growing Forward 2 and the Canadian Agricultural Partnership to respond to questions on relevance, design and delivery and effectiveness. Case study selection considered project budgets, duration, regions, sectors and types of recipients. The projects reviewed through the case studies represented about a third of the value of approved funding under the Canadian Agricultural Partnership.

Research for the case studies included document and file review, a review of project reports and other available documentation along with interviews with the funding recipient and responsible AAFC project officer. Case studies were conducted of projects with the following organizations.

• Canadian Cattleman’s Association
• Canadian Nursery Landscape Association
• Canadian Pork Council
• Dairy Farmers of Canada
• Jewish Community Council of Montreal

Methodological limitations

The following limitations were considered in interpreting the data:

Limitation

Long-term nature of Program outcomes. It can take many years before assurance projects generate longer-term outcomes, which could limit the evaluation’s capacity to assess program impact.

Mitigation strategy

To mitigate this challenge, the evaluation included case studies of organizations that received funding across two or more policy frameworks.

Limitation

Lack of program performance data. Given performance reporting is not required of funding recipients until project completion and the Program has relied on output indicators to track performance there was limited data available on outcomes.

Mitigation strategy

The evaluation used interviews and case studies to assess project outcomes.

Annex B: Agriassurance program logic model

Component

Activities

• Engagement and relationship building with stakeholders
• Analysis of domestic and international assurance requirements/standards
• Qualitative and quantitative data gathering and dissemination on assurance systems, strategies, plans and tools
• Receive and assess project applications and perform due diligence
• Prepare and send project approval/rejection letters
• Monitor contribution agreements, and process performance/progress reports, and financial claims

Outputs

• Industry engagement services and assurance systems benchmarked and evaluated based on domestic or international standards
• Sector development services: sector analysis, market information, industry-government meetings
• Project applications received and acknowledged
• Recommendations based on assessment of project applications
• Project approval/rejection letters sent
• Contribution agreements prepared for client review
• Recipient financial claims processed

Immediate outcomes

• The agriculture and agri-food sector is prepared to develop assurance systems and supporting tools (for example, food safety, animal and plant health, market attributes and quality standards, and system integration).
• Industry’s awareness and knowledge of initiatives to mitigate sector risk is increased.

Intermediate outcomes

• The agriculture and agri-food sector develops and implements assurance systems and supporting tools (for example, food safety, animal and plant health, market attributes and quality standards, and system integration).

Ultimate outcomes

• The agriculture and agri-food sector is equipped with assurance systems and supporting tools.

Source: AgriAssurance Performance Information Profile