Evaluation of Agricultural Climate Solutions and the Living Laboratories Initiative

Table of contents

• Abbreviations
• Executive summary
• 1.0 Introduction
• 2.0 Scope and methodology
• 3.0 Program profile
• 4.0 Relevance
• 5.0 Design approaches supporting efficient program delivery have been introduced
• 6.0 Conclusions and recommendations
• Annex A: Logic model
• Annex B: Evaluation methodology
• Annex C: Management response and action plan

Abbreviations

AAFC

Agriculture and Agri-Food Canada

ACS

Agricultural Climate Solutions

ACS–LL

Agricultural Climate Solutions — Living Labs Stream

BMP

Beneficial Management Practice

CALL-Net

Canadian Agroecosystem Living Labs Network

GBA

Gender-based analysis

GHG

Greenhouse gas

INRAE

National Research Institute for Agriculture, Food and Environment

LLI

Living Laboratories Initiative

NCSF

Natural Climate Solutions Fund

OFCAF

On-Farm Climate Action Fund

URMG

Underrepresented and marginalized groups

Executive summary

Purpose

The Office of Audit and Evaluation at AAFC conducted an evaluation of Agricultural Climate Solutions (ACS) — including both the Living Labs (ACS–LL) stream and the On-Farm Climate Action Fund (OFCAF) stream — to assess its relevance, efficiency and effectiveness. The Living Laboratories Initiative (LLI), which predated the ACS–LL, was also included in the evaluation.

Scope and methodology

The evaluation covered activities from 2018-19 to 2023-24 and used multiple lines of evidence including a review of program documents, project files and literature; key informant interviews; program logic model and process mapping workshops; and case studies. This evaluation period was chosen to include the full implementation of the LLI, through to the last fiscal year that would be completed by the end of the evaluation's conduct.

Background

The LLI was a 5–year program (2018–2023) providing $10 million in grants to support agricultural discovery science and innovation, with a focus on climate change mitigation and adaptation, soil and water quality protection and increasing biodiversity on agricultural landscapes. The ACS Program was first launched in 2021 as a $185 million, 10–year program contributing to the horizontal Natural Climate Solutions Fund (NCSF). At its launch, ACS included the ACS–LL stream. The ACS–LL stream was based on the LLI and its approach to agricultural innovation, but with an aim to primarily reduce greenhouse gases (GHGs) and sequester carbon. Later in 2021, AAFC introduced OFCAF to ACS. OFCAF was initially a  3–year, $200 million fund to support farmers in adopting beneficial management practices to store carbon and reduce GHG emissions as well as other environmental co–benefits. This was followed by an additional investment of $470 million and an extension to 2028 under the 2030 Emissions Reduction Plan.

Findings

• LLI, ACS–LL and OFCAF are well aligned with AAFC's core responsibility for science and innovation, with federal and international reporting obligations and with broader Government of Canada goals.
• The living labs innovation approach is unique. While there is uncertainty among producers resulting from the many programs focusing on BMPs (including OFCAF), federal-provincial-territorial efforts have minimized duplication among programs.
• AAFC is well regarded in the development and implementation of the agroecosystem living labs approach. This approach enables collaboration between producers, scientists and other partners to co-develop, test and evaluate innovative sustainable farming practices that are then more likely to be adopted quickly.
• The OFCAF model, utilizing external organizations to deliver funding to producers, is well suited to the program.
• The horizontal governance framework functions well to facilitate interdepartmental coordination and fulfillment of the NCSF horizontal reporting framework and results tracking.
• There is a lack of clarity for some stakeholders regarding the relationship between ACS–LL and OFCAF.
• Although not adequately measured under the Performance Information Profiles for ACS–LL and OFCAF, the development of strong relationships and trust networks emerged as a crucial outcome which may encourage increased BMP adoption.
• The LLI surpassed targets for all immediate and intermediate outcomes leading to successful results and anticipated increases in BMP adoption.
• Both ACS–LL and OFCAF are making efforts consistent with AAFC's framework to support Underrepresented and Marginalized Groups (URMG) in the sector.

Conclusion

ACS and LLI are well aligned with AAFC and broader federal priorities, with international obligations regarding reduction of GHG emissions and carbon sequestration, and with AAFC's Action Plan to Support URMG. In addition, AAFC is a leader in the concept and implementation of the agroecosystem living labs approach and collaborates with international partners in this area. Although not adequately measured under the Performance Information Profiles for ACS–LL and OFCAF, the development of strong relationships and trust networks emerged as crucial outcomes, supporting knowledge exchange, helping to maintain participant commitment and enthusiasm, and stimulating the interest of non-participants in the practices examined and supported by ACS. It is premature to assess the achievement of most other outcomes for the ACS streams. In addition, there is a need to clearly articulate the relationship between ACS–LL and OFCAF to be able to leverage potential synergies between the 2 streams. Lastly, long timeframes for approvals and claims processing have slowed progress and caused challenges, especially for smaller recipients.

Recommendations

Recommendation 1a: The Assistant Deputy Ministers of Programs Branch and Science and Technology Branch should, together, refine the performance measures of ACS–LL to better reflect and assess the relationship building, knowledge gains and behaviour changes in the sector that result from ACS–LL activities.

Recommendation 1b: The Assistant Deputy Minister of Programs Branch should refine the performance measures of OFCAF to better reflect and assess the relationship building, knowledge gains and behaviour changes in the sector that result from OFCAF activities.

Recommendation 2: The Assistant Deputy Minister of Program Branch, working with the Assistant Deputy Ministers of Science and Technology Branch and Strategic Policy Branch, should clarify the relationship between the ACS–LL and OFCAF and communicate this relationship to key stakeholders.

Recommendation 3: The Assistant Deputy Minister of Programs Branch should clarify application and assessment requirements for potential applicants; examine these requirements to determine if there are areas for improvement and, if any are found, take action to improve efficiency and timeliness with respect to application assessment and claims payment processes.

Management response and action plan

Management agrees with the evaluation recommendations and has developed an action plan to address them by December 2025. For further details see Annex C.

1.0 Introduction

The Office of Audit and Evaluation at AAFC undertook an evaluation of the ACS, including both the ACS–LL stream and the OFCAF stream. LLI, which was in place from 2018–2023, was also included in the evaluation. The evaluation was planned as part of the 2023–2024 to 2027–2028 Audit and Evaluation Plan. The evaluation complies with the requirements of the Treasury Board Policy on Results.

2.0 Scope and methodology

This evaluation assessed the relevance, efficiency and effectiveness of ACS–LL,OFCAF and the LLI. The evaluation examined activities from 2018–19 to 2023–24 and used multiple lines of evidence including reviews of program documents, project files and literature; key informant interviews; program logic model and process mapping workshops; and case studies. For the LLI, the evaluation took a summative approach to include the achievement of expected outcomes as well as best practices and lessons learned from the LLI design and delivery. A formative evaluation approach was used for ACS since it is in early stages of program implementation. This part of the evaluation focussed on elements of program design and delivery to support early intervention for program improvement. The evaluation also included Gender-Based Analysis (GBA) Plus considerations. For more detailed information on the methodology used, see Annex B.

3.0 Program profile

3.1 Program context

AAFC's Agricultural Climate Solutions Program exists within a complex context that includes provincial, federal and international policies seeking to address climate change and reduce GHG emissions. Within this context, Canada has committed to reporting on actions taken and progress made toward international goals for the reduction of GHG emissions. ACS is one of AAFC's Environment and Climate Change programs and is intended to contribute to Canada's climate commitments.^{Footnote 1} Figure 1 shows the timeline of key policy interventions and program announcements.

Figure 1: International and federal policy commitments and AAFC programs to address climate change and reduce greenhouse gas emissions

Description of the above image

Figure 1 shows a timeline of key international and federal policy commitments and AAFC programs addressing climate change and the reduction of greenhouse gas emissions since 2015. Key commitments are represented by coloured circles inside a long arrow pointing to the right hand side of the page. The blue circles represent international commitment; red circles represent federal policy commitments; and green circles indicate AAFC programs. The key commitments and programs, in order from left to right, include the following:

• 2015 Paris Agreement (international commitment)
• 2016 Pan-Canadian Framework (federal policy commitment)
• 2018 AAFC Living Labs Initiative (2018 to 2023) (AAFC program)
• December 2020 Canada's Strengthened Climate Plan (federal policy commitment)
• Budget 2021 – National Climate Solutions Fund (Horizontal with AAFC Agricultural Climate Solutions2 – Living Labs 2021 to 2031) (federal policy commitments)
• April 2021 Agricultural Climate Solutions – On-Farm Climate Action Fund announced (2021 to 2024) (AAFC program)
• June 29, 2021 Canadian Net-Zero Emissions Accountability Act (federal policy commitments)
• July 2021 - Canada’s first Nationally Determined Contribution submitted to UN
• March 2022 - 2030 Emissions Reduction Plan and Budget 2022
• July 2022 - Agricultural Climate Solutions-Living Labs–9 new LLs announced (AAFC program)
• March 2023 - Agricultural Climate Solutions-On-Farm Climate Action Fund extended to 2028 (AAFC program)

When Canada signed the Paris Agreement at the United Nations Climate Change Conference (2015), it entered a legally binding international treaty that included a commitment to submit national climate action plans to the United Nations Framework Convention on Climate Change Secretariat and to report on issues related to climate change. Since the signing of the Paris Agreement, Canada has announced several plans and programs, including the following:

• the Pan-Canadian Framework on Clean Growth and Climate Change (2016)
• the Strengthened Climate Plan for a Healthy Environment and a Healthy Economy (2020)
• the NCSF, 2021, a $4 billion 10 year (2021–2022 to 2030–2031) horizontal program which includes AAFC's ACS
• the 2030 Emissions Reduction Plan: Canada's Next Steps for Clean Air and a Strong Economy (2022), which included an additional investment of $470 million in OFCAF and an extension of OFCAF to 2028

3.2 Living Laboratories Initiative and Agricultural Climate Solutions Program

In September 2018, AAFC launched the LLI: a 5-year program (2018–2023) with total contribution funding of up to $10 million to support agricultural discovery science and innovation, with a focus on climate change mitigation and adaptation, soil and water quality protection and increasing biodiversity on agricultural landscapes.

ACS was first launched in March 2021 as a $185 million, 10-year program. It is the AAFC component of the horizontal NCSF, which also includes components implemented by Natural Resources Canada (2 Billion Trees Program) and Environment and Climate Change Canada (Nature Smart Climate Solutions Fund). ACS–LL was included in the original ACS launch. It built on the LLI and its approach to agricultural innovation. The original LLI program and projects continued to the end of their funding term (March 31, 2023). There are now 14 new projects being implemented under ACS–LL.

In August 2021, AAFC launched the OFCAF, a second stream under the ACS program. OFCAF was initially announced as a 3-year, $200 million fund to support farmers in adopting BMPs designed to support carbon storage and GHG reduction, followed in 2022 by an additional investment of $470 million and extension to 2028 under the 2030 Emissions Reduction Plan.

3.2.1 Living Laboratories Initiative (2018–2023) overview

The LLI introduced a new collaborative research approach to agricultural innovation in Canada, bringing together farmers, scientists and other collaborators to develop and test innovative practices and technologies.

The initiative aimed to develop a nationwide network of living labs focussed on developing innovative solutions to a range of agriculture-related environmental issues including climate change, soil health, water quality and biodiversity.^{Footnote 2}

4 projects were established under the LLI in Prince Edward Island, Quebec and Manitoba between 2019 and 2021. The agroecosystem living labs approach is based on 3 core principles:

• focussing on farmers' needs (farmers are key partners in the development, implementation and evaluation of proposed innovations)
• developing broad and diverse partnerships including farmers and researchers from various disciplines
• testing innovations in real-life situations

LLI resources

Total spending over the evaluation period for LLI was $25,897,068 (see Table 1). AAFC employed an average of 8.4 full-time equivalent (FTE) employees for the LLI over the reference period to support the program.

The LLI was managed by the Living Laboratories Division in AAFC's Science and Technology Branch.

Notes: Actual expenditures represent what was published in the respective year Departmental Results Report. Planned resources have not been included in this report. Planned expenditures and FTEs for LLI are part of the Funding to Advance Agricultural Discovery Science and Innovation Treasury Board submission.

However, most of this submission includes expenditures and FTEs that are not related to LLI. Therefore, the submission does not accurately represent planned resources of LLI in isolation.

Source: Corporate Management Branch (as of April 30, 2024)

3.2.2 ACS–LL stream overview

Building on the LLI, the new stream was launched in 2021. This stream now supports 14 living labs and has a 10-year budget of $185 million. Where LLI focussed on a wide range of environmental issues, ACS–LL primarily aims to reduce GHGs and to sequester carbon by accelerating the co-development, testing, adoption, dissemination and monitoring of technologies and practices. Projects may lead to other co-benefits such as

• improvements to soil health and water quality
• water conservation
• increased biodiversity
• maximization of habitat capacity

Though ACS–LL uses the same model as the LLI (the agroecosystem living labs approach to innovation), it represents a substantial scale up, with a much larger budget and more than 3 times the number of living labs spread across all provinces.

ACS–LL application intake included a 2 phased funding process in 2022. Phase 1 funding provided a grant of up to $100,000 to support organizations' ability to build capacity, develop the required networks of participants and draft comprehensive project proposals for Phase 2 contribution funding. Projects were not required to receive Phase 1 funding to be eligible for Phase 2 funding; and success in applying for Phase 1 funding did not automatically translate to a successful application in Phase 2. Phase 2 provided contribution funding to implement approved comprehensive project proposals, following the living labs approach.

Eligible applicants for contribution funding under ACS–LL are not-for-profit organizations, including producer organizations; and Indigenous groups. Successful applicants are required to develop a consortium of partners to implement their comprehensive project proposal based on the living labs approach.

Eligible activities (for Phase 2 funding) include:

• conducting outreach activities to build partnerships among potential partners for a project for implementation under contribution funding
• acquiring expertise to engage with researchers
• developing research and development, knowledge transfer and adoption plans to be implemented in a project under contribution funding
• conducting research and development
• ensuring producer engagement and participation in on-farm co-development of BMPs
• facilitating approaches and solutions by testing, piloting, demonstrating and supporting adoption of processes and technologies that can be more broadly applied
• implementing knowledge transfer plans and adoption strategies to incentivize and support BMP adoption by producers
• strategic planning to respond to environmental priorities
• data collection

For the ACS–LL logic model, see Annex A.

3 branches at AAFC are involved in the delivery of ACS–LL. Programs Branch is responsible for the overall delivery of ACS–LL including coordinating the application review process, delivering funds to successful recipients and managing annual performance reporting. The Living Laboratories Division (in Science and Technology Branch) provides overall science leadership for ACS–LL, coordinating the network of living labs and their innovation activities. The Division leads scientific integration, data management, multi-sites analysis, innovation management, socio-economic work, and knowledge exchange efforts among the national network and with international colleagues, and supports living labs with GHG and carbon sequestration reporting. The Research and Analysis Directorate within Strategic Policy Branch contributes to socio-economic research, examining the socio-economic factors affecting BMP adoption and conducts cost-benefit analysis of BMPs.

ACS–LL resources

Total spending over the evaluation period for ACS–LL was $11,453,229 (see Table 2). AAFC employed an average of 9 full-time equivalent employees for ACS–LL over the reference period to deliver the program. This includes staff that supported program delivery (annual average of 4.5) and scientific activities (annual average of 4.5).

Notes: Actual Expenditures represent what was published in the respective year Departmental Results Report.

Planned Expenditures represent what was published in the respective year Departmental Plan. Actual expenditures and FTEs for 2023-24 are not yet available.

Source: Corporate Management Branch (as of April 30, 2024)

3.2.3 ACS-OFCAF stream overview

OFCAF is managed by the Industry Development Division of Programs Branch. OFCAF provides funding to a total of 13 recipient organizations who, as delivery agents, redistribute the funding through individual application intakes. The objective of OFCAF is to support farmers in adopting already proven BMPs that are new to them and that store carbon and reduce GHG emissions, specifically in the areas of:

• nitrogen management
• cover cropping
• rotational grazing practices

Activities supported through OFCAF are expected to reduce GHG emissions by up to 2 million tonnes by 2024. These practices also provide other environmental benefits such as improved biodiversity and soil health.

Eligible activities include:

• providing extension or agronomic services to producers to support adoption of BMPs, including development of farm-specific plans
• supporting adoption of BMPs among producers, including development of farm-specific plans, purchase/rental of supplies and equipment and incentive payments
• developing and delivering training for agronomists for specific BMPs
• monitoring, assessing and reporting on project impacts
• activities related to outreach and promotion in support of BMP adoption
• establishing processes and systems in support of BMP adoption
• acquiring expertise to facilitate BMP adoption by producers
• other activities as required and approved in advance by AAFC to support the successful implementation of the program

For the OFCAF logic model see Annex A.

ACS–OFCAF resources

Total spending over the evaluation period for OFCAF was $98,401,093 (see Table 3). AAFC employed an average of 7 full-time equivalent employees for OFCAF over the reference period to support the implementation of the program. This included staff supporting program delivery (annual average of 5.5), and to provide scientific expertise (annual average of 1.5).

Notes: Actual expenditures represent what was published in the respective year Departmental Results Report.

Planned expenditures represent what was published in the respective year Departmental Plan.

OFCAF was approved during the 2021-22 fiscal year. Thus, there were no planned spending nor FTEs as part of the 2021-22 Departmental Plan. Actual expenditures and FTEs for 2023-24 are not yet available.

Source: Corporate Management Branch (as of April 30, 2024)

4.0 Relevance

4.1 LLI, ACS–LL and OFCAF are well aligned with government priorities

The LLI, ACS–LL and OFCAF are well aligned with AAFC's core responsibility for science and innovation, as this area of responsibility includes developing new knowledge and technologies as well as strengthening the sector's capacity to develop and adopt innovative practices, products and processes.

Both streams of ACS are also well aligned with federal and international reporting obligations related to carbon sequestration and GHG mitigation including both the NCSF and Canada's 2030 Emissions Reduction Plan goals of reducing GHG emissions by 40% below 2005 levels by 2030 and net zero by 2050.

Under NCSF, ACS–LL builds on LLI's objective of developing a nationwide network of living labs. Both ACS–LL and OFCAF align well with NCSF goals for additional co-benefits such as supporting biodiversity and increasing resilience of agriculture as a nature-based economic sector.

4.2 The living labs approach is unique and duplication with other BMP programs has been minimized

While AAFC supports collaborative agricultural research through, for example, the AgriScience Program, the living labs approach is characterized as innovation supported by research. The focus on producers as key partners through the entire process of co-developing, testing and evaluating innovative BMPs is a unique approach.

Several other programs in the Canadian agri-environmental space also focus on BMPs, for example, the Resilient Agricultural Landscapes Program and several provincial-territorial programs now being implemented under the Sustainable Canadian Agricultural Partnership (Sustainable-CAP), though these are generally focussed on implementation of BMPs. While some overlap between programs may be desirable given the urgency of the climate situation, federal, provincial and territorial governments have worked together, in collaboration with key industry organizations, to minimize potential duplication of federal and provincial or territorial programs supporting BMPs.

5.0 Design approaches supporting efficient program delivery have been introduced

5.1 LLI and ACS–LL represent an innovative approach, with some minor challenges

5.1.1 The living labs approach is anchored in innovation

The basis of the living labs approach is innovation supported by research. More specifically, it introduces culture change involving new processes and new ways of working together with producers and partners to undertake these innovation projects supported by strong research.

Figure 2 presents the living labs iterative innovation cycle process that includes:

1. co-developing
2. testing
3. evaluating

As the innovation cycle is repeated, learning takes place.

The innovative practices being researched are adjusted to address feedback from farmers, collaborators and scientists, as well as to address priority agri-environmental challenges. One goal for this process's co-development component is to ensure the innovations are economically and technologically feasible so that they are more likely to be adopted by producers.

Figure 1 Living Labs iterative process of improvement

Source: Adapted from Living Labs Division documents.

Description of the above image

Figure 2 depicts the innovation cycle for the Living Laboratories approach as innovative new practices and technologies are tested and evaluated. The image of the innovation cycle is represented as a circle and includes 3 steps: Step 1: Co-develop; Step 2: Test; and Step 3: Evaluate. To show that this is an iterative process, the innovation cycle figure is repeated 3 times and each time it is repeated, from the left to right hand side of the page, it moves upwards and gets larger. The far left hand side represents innovation cycles at the planning stages of the process whereas the right hand side represents cycles that are moving toward adoption stages.

AAFC is an international leader in developing and implementing the agroecosystem living labs approach

AAFC is well respected in the international community for the work undertaken in the development and implementation of the agroecosystem living labs approach. AAFC presented the approach at the meeting of G20 Agricultural Chief Scientists (G20-MACS) in May 2018. Since then, the Living Labs Division has collaborated with several international agricultural innovation and research networks including the United States Department of Agriculture's Long-Term Agroecosystem Research Network, the European Network of Living Labs, the International Society for Professional Innovation Management and the Inter-American Institute for Cooperation on Agriculture.

The Living Labs team has been a key collaborator for the European Commission in their efforts to develop agroecosystem living labs. AAFC is an international collaborator in ALL-Ready, which aims to prepare a framework for a network of living labs and research infrastructure in Europe. The department has been involved throughout the project, helping to identify the vision, scope, mission and selection criteria for the proposed network of living labs across Europe. AAFC continues to support European efforts to develop living labs through involvement with the Agroecology Partnership, which is currently reviewing proposals for research and innovation projects that support the transition of the agricultural sector to agroecology, including living labs, across Europe.

In addition, AAFC's collaborative partnership with France's National Research Institute for Agriculture, Food and Environment (INRAE) resulted in a Memorandum of Understanding, signed in May 2020, that included agroecosystem living labs as a priority area for collaboration. In October 2023, AAFC co-hosted the first International Forum on Agroecosystem Living Labs with INRAE in Quebec. The forum included 2 days of presentations and poster sessions and a field tour of a LL-QC site, co-hosted by the Union des producteurs agricoles, the lead organization for the LLI project in Quebec.

The Canadian Agroecosystem Living Labs Network (CALL-Net) supports the implementation of the Living Labs

The Living Labs Division is comprised of 4 units, which support the living labs:

• National Coordination
• Science Integration
• Innovation and Knowledge
• Socio-economics

The division undertakes activities such as coordinating science and socio-economic research activities and maintaining ongoing knowledge exchange with the international living labs community.

In 2021–2022, the CALL-Net was established to facilitate national multi-site and international scientific collaborations with “priority partners” and accelerate the growth of a national “Living Labs Network” for the development of agri-environmental innovation and increased BMP adoption.^{Footnote 3} A total of 9 CALL-Net working groups^{Footnote 4} meet regularly to:

• promote collaborative research activities and opportunities among AAFC scientists and with international collaborators
• foster agri-environmental partnerships with other government departments, industry groups and Indigenous communities
• encourage the development of standardized datasets and the facilitation of data sharing across the national network

Living Labs demonstrate adaptive management and continuous improvement

The Living Labs Division committed to an adaptive management approach to learn about the implementation of the approach and adjust as required. The International Engagement Division in the Science and Technology Branch undertook a qualitative research study in fiscal year 2019–2020 to identify early successes and challenges from the initial 2 living labs: LL-Atlantic and LL-Eastern Prairies. Based on these, the International Engagement Division worked with the Living Laboratories Division to develop a detailed action plan to address identified challenges. Examples of actions undertaken include: ongoing communications and education sessions to clarify the living labs approach and participants' roles and responsibilities, as well as efforts to streamline program processes where possible.

5.1.2 The Living Labs were able to adapt to challenges encountered

Adapting to early challenges

Given the culture change inherent in the living labs approach, there was initially a lack of clarity for some participants about their roles and responsibilities and a need for ongoing reinforcement of the vision of the Living Labs. In addition, recipients were surprised about commitments required of them, such as recruiting partners in advance of proposal submission, and AAFC scientists were uncertain of the connections between their own research and LL research. In response to this problem, a $100,000 grant was introduced under ACS–Living Labs to support proposal development. This grant gave prospective living labs lead organizations more resources to form a consortium and develop a workplan and proposal. However, the grant may also have prompted additional competition among potential applicants to recruit the same partners.

Adhoc interactions were often more productive than formal meetings

Evaluation results demonstrated that some producers found the frequency and formality of some meetings held by the living lab did not meet their needs. Meetings were considered too frequent and some producers found they did not always leave with the practical takeaways they would have liked. In contrast, when they were given the opportunity to meet with researchers in their fields, they were able to have productive discussions where their specific questions were directly addressed. In addition, scientists found the living labs approach was demanding given their time and budget constraints and their need to maintain scientific rigour to ensure data quality. Some project recipients suggested there is some tension between researchers who need trials and controls, and producers who need to be innovative.

Overall, it appears that while the culture change brought about by the implementation of the living labs approach involved inevitable challenges, partners are adapting, resulting in strengthened relationships. In turn, these relationships support innovative approaches to knowledge transfer, and innovative thinking and research.

Living Labs data sharing contributes to good governance

Given the complex nature of the Living Labs, there are some challenges related to data sharing, such as determining data ownership when there is a large network of partners; or gaining agreement from all partners or contributors in a living lab to share data with others outside their living lab.

To support information sharing and collaboration between living lab participants, specifically external partners, AAFC developed and launched an online account. Ongoing communications between living labs and CALL-Net working groups have built a sense of community and trust among participants. Furthermore, LLI developed a data management strategy and cloud solution to ensure the Initiative met its outcomes. This strategy enables standardization of measurements and variables, data interoperability between sites of the CALL-Net, replicability of living labs science and compliance with Government of Canada directives, in turn supporting the world-class science undertaken at the Living Labs.

COVID-19 created some delays for early Living Labs

In the case of LLI, the COVID-19 global pandemic significantly impacted the implementation of the Quebec and Ontario Living Labs, resulting in delays to project start dates. The pandemic caused the cancellation of the Call for Project Proposals for the British Columbia region. For the already established LL-Eastern Prairies, COVID-19 altered the scope of work for some projects in 2021. All 4 original LLI sites were impacted by COVID-19 restrictions preventing in-person co-development meetings during the lockdown period as well as AAFC scientists from accessing participating farms for data collection. Mitigation strategies were implemented to address these challenges, such as virtual and distanced in-person meetings.

Extreme weather contributed to project delays

Extreme weather events also resulted in some delays. The LL-Atlantic project on Prince Edward Island endured a drought and 2 hurricanes and the LL-Eastern Prairies experienced both drought and flooding through the program period. In both cases, this impacted the ability to carry out the project as intended, impacted data collection and made it difficult to assess the effectiveness of some BMPs.

Complex funding model

The internal funding model for the Living Labs includes 2 types of funding: Vote 1 (Operating) and Vote 10 (Grants and Contributions). While programs often include both of these components, the Operating funding took a more important role in the delivery of the Living Labs, as this type of funding is what is used to support AAFC scientists. As noted elsewhere in this report, the work undertaken by AAFC scientists was essential to the success of the Living Labs. The grant and contribution funding was to be used for the other activities undertaken and eligible expenses incurred by the Living Labs. However, there were some challenges encountered in implementing the Living Labs as a result of the inclusion of both kinds of funding in projects. For example, the distinction between what should be accounted for under each type of funding when the Living Lab is being delivered by an integrated team (of AAFC and external collaborators) is not always clear.

5.2 The OFCAF approach includes practices supporting efficient delivery

5.2.1 The OFCAF delivery model is well suited to the program

OFCAF employs a further distribution of contribution delivery model in which AAFC provides funding to initial recipients (for example, non-profit organizations and Indigenous communities) who in turn deliver the funds to the ultimate recipients (producers) who use the funds to help cover the costs associated with the adoption of BMPs. The use of this model is new for AAFC and it has been noted that both AAFC and direct recipients are working to better understand each other's needs in this new model. Despite learning while doing, this delivery model presents important advantages.

Under the current delivery model, third party delivery agents who are already familiar to local stakeholders act as a bridge between AAFC and producers. As such, the delivery agents are key to developing crucial trust networks that enable the federal government to pursue a climate change action agenda, as well as promote the discretionary on-farm adoption of BMPs. Some recipients are also delivery agents for provincial cost-shared programs, making them well-placed to help producers find the most appropriate programs, whether federal or provincial. Well-connected delivery agents are also able to foster connections between producers and experts, such as professional agrologists and agronomists.

Adaptive management supports ongoing program improvement

The Industry Development Division of Programs Branch demonstrates commitment to adaptive management practices and supports ongoing learning about what works well in OFCAF, as well as where there are opportunities for adjustment. For example, OFCAF program staff hold regular national consultation meetings with OFCAF recipients. Meetings provide the opportunity to discuss recipient concerns such as program administrative details, including project deadlines and completion of annual performance reports. The meetings also provide an opportunity to discuss approaches taken by recipients to address challenges such as reducing barriers to participation among underrepresented and marginalized producers. In addition, OFCAF administered a survey of recipients in early 2024 to supplement information received in annual reporting. Feedback through these means informs considerations for changes to the program, including as part of the OFCAF extension (2023–2028).

Some challenges may impact the delivery of OFCAF

As a new program, OFCAF faces some challenges as it evolves.^{Footnote 5}

Lack of trained professionals

In some regions, there is a lack of agronomists, agrologists and certified crop advisers trained in key BMPs. This gap in trained professionals who can support producers in adopting BMPs may impact OFCAF's ability to achieve the desired increase in adoption of BMPs. AAFC incorporated funding for training of professional agrologists, agronomists and certified crop advisers as an eligible activity under OFCAF to help address this challenge. In fiscal year 2022-23, the most recent year for which data is available, over 1,000 of these professionals were trained with the support of this funding.

A need for neutral scientific expertise

Scientific expertise at AAFC supports the identification of BMPs that are eligible for OFCAF support. Evaluation evidence suggests that having AAFC scientists certify the scientific merit of eligible BMPs lends credibility to the process because it is understood that AAFC scientists have no vested interest in developing specific products for market. The BMPs supported by OFCAF may not include all the BMPs that farmers and industry would like to see supported.

5.3 NCSF horizontal governance works well

ACS is the AAFC component of the NCSF, led by Natural Resources Canada. Under NCSF, there are 6 committees and working groups ranging from working level to Assistant Deputy Minister level. The Horizontal Reporting Working Group ensures interdepartmental coordination and fulfillment of the requirements of the NCSF horizontal reporting framework and results tracking. The Indigenous Collaboration Working Group ensures coordinated and effective approaches to working with Indigenous peoples in a way that reduces administrative burden and reflects the Government's commitment to reconciliation. Interdepartmental governance structures for NCSF appear to work well.

5.4 Complexity contributes to delivery challenges for ACS–LL and OFCAF

5.4.1 There is uncertainty over the relationship between the 2 streams of ACS

ACS–LL and OFCAF are 2 streams of the same program. However, there is a lack of clarity about how ACS–LL and OFCAF relate to each other. For example, ACS–LL is an innovation program that leverages research to develop new or improved existing BMPs while OFCAF is an adoption program supported by scientific verification of eligible BMPs. Evidence indicates that some stakeholders view LLI and ACS–LL as leading to OFCAF, with the practices being researched by the Living Labs intended to be supported for widespread adoption by OFCAF in the future. However, whether this is the intention of the program is not clearly articulated.

There is a similar lack of clarity around the roles and responsibilities for knowledge transfer under ACS–LL and OFCAF. Knowledge transfer activities are being undertaken by recipients of funding under both streams of ACS. These activities help share the on-farm benefits of BMPs and therefore help to encourage their broader adoption. However, several recipients including those who were delivery agents for both ACS–LL and OFCAF, said the program emphasized that the knowledge transfer efforts of the 2 streams can be complementary, but should not be duplicative. These recipients are uncertain about how these activities can remain separate as the producers in their area would benefit from the knowledge that is to be shared from the work undertaken by both streams. This may be especially so in geographically smaller regions where a smaller pool of producers would likely attend both ACS–LL and OFCAF knowledge transfer events. Again, the ability of recipients as delivery agents to coordinate knowledge transfer efforts was identified as key in minimizing duplication of these efforts.

5.4.2 The ACS–LL and OFCAF review processes are complex

Both ACS–LL and OFCAF have detailed application review and approval processes which include approximately 14 steps and involve staff from multiple branches, regional offices and multiple levels from program analysts through to the assistant deputy minister level. For LLI and ACS–LL, there is the additional challenge of allocating 2 different types of funding and signing collaborative research and development agreements.

The complexity of the process was considered to contribute to the time required for application reviews and subsequent signing of contribution agreements and claims.^{Footnote 6} While acknowledging that this process does take time, its rigour leads to well-informed decisions. That said, this complexity also contributes to some confusion for recipients. In addition, some report being unsure who to speak with at AAFC regarding funding decision timelines. Going forward, the program may benefit from reviewing these processes to reduce administrative burden where possible, while maintaining a high level of rigour in the review of applications, and maximize informed decision-making.

5.4.3 Administrative timeframes for applications and claims for LLI, ACS–LL and OFCAF are long

Administrative timeframes posed a significant challenge for recipients across LLI and both ACS–LL and OFCAF.

Long time frames for application and claim processes

Time frames for reviewing applications and signing contribution agreements was the most frequent problem identified by both ACS–LL and OFCAF recipients. In turn, funding timeframes led to changes in work plans. For example, timing of some steps resulted in the loss of entire growing seasons and, by extension, lost opportunities for sampling and measurement. This hindered progress toward the outcomes anticipated by both ACS–LL and OFCAF recipients.

Time frames also caused frustration and hardship for those recipients who had to self-fund a project for several months or more. While less than optimal, this was possible for some of the larger recipients, but was more challenging for smaller recipients. For LLI, funding timeframes also had potential impacts on trust building between project partners and collaborators. For OFCAF, the late launch and reduced timelines contributed to challenges in building awareness of the program among producers.

Long timeframes are echoed in the findings of the recent Office of the Auditor General report on agriculture and climate change, which reported that it took an average of 5 months to finalize contribution agreements with successful applicants, with only 1 of 22 agreements signed in time to take full advantage of the 2022 growing season. The report recommended improving the timeliness for review, approval and finalization of contribution agreements for future climate programs.

5.5 Effectiveness

5.5.1 Strong relationships and trust networks support program effectiveness

Relationship building and the development of increased trust between a diverse set of partners emerged as a crucial outcome of LLI. For example, by involving producers directly, the co-development approach permits producers and researchers to better understand each other's perspectives. This sometimes leads to revised research approaches, enhances producers' appreciation for scientific rigour and may encourage broader adoption of BMPs. In the case of LL-Atlantic, the strength of relationships fostered by living labs enhanced community resiliency in the face of a drought and 2 hurricanes. The living labs approach encourages the integration of scientific disciplines to take a broader systems approach to environment and climate change issues.

In the context of ongoing culture change, peer-to-peer learning is a crucial element of successful knowledge transfer because it leverages the understanding and trust that already exists between producers. Some recipients have introduced more informal knowledge transfer events, as well as easily accessible products that offer immediate takeaways for producers. These efforts are emerging as a key means of maintaining participant commitment and enthusiasm, and of stimulating interest as well as possible adoption among non-participants.

5.5.2 LLI, ACS–LL and OFCAF are making progress toward anticipated outcomes

LLI

As described above (see Figure 2), the Living Labs iterative cycle focuses on co-developing, testing and evaluating practices or technologies through the collaboration of producers and other stakeholders, supported by AAFC scientists, on actual farms in actual conditions. A 2021 Office of the Auditor General report found this model to be “a good example of the formal coordination of scientific activities.” Evaluation evidence indicates the process enables producers and researchers to work together to design studies that are of direct interest to both stakeholders and that yield sustainable farming practices likely to be adopted more quickly by Canadian farmers.

The immediate and intermediate outcomes for the LLI were achieved and almost all targets were met or surpassed. Notably, the capacity to adopt BMPs was increased as indicated by the fact that in all 4 original Living Labs, the number of knowledge products produced, the number of field demonstrations and peer-to-peer learning events and the number of producers participating in these events met or surpassed all targets set.

LLI projects in all of the regions led to successful results and anticipated increases in BMP adoption. For example, the use of cover crops in the Atlantic region demonstrated reduced soil erosion and nitrate leaching and improved yields, while results in Ontario indicated that double cropping can be profitable and improve soil health. Real and expected increases in BMP adoption across regions were reported based on interest and engagement from the LLI projects and the applicable nature of the LLI research activities. Evidence from some of the case studies suggest that LLI laid a foundation that will carry over to ACS–LL. For example, LL-Atlantic reported a near doubling of the number of partners as the original project concluded and the new ACS–LL funded project launched.

ACS-LL and OFCAF

Assessment of outcomes for ACS–LL and OFCAF is premature as both programs had only reported on Year 1 (2022-23) activities at the time of evaluation. Based on first year reports, baselines and targets for the remaining indicators are to be set in 2023–2024. The only known outcome at the time of evaluation is that ACS–LL is funding 14 living labs, thereby surpassing its target of 10 established living labs supported by ACS.

A majority of the 13 OFCAF recipients report being over-subscribed for the 2023–2024 intake. This indicates the popularity of the program and the program anticipates success against targets in the coming years. Finally, strengthened relationships are emerging as equally important in both ACS–LL and OFCAF as with the previous LLI. Behaviour change may be occurring beyond participating producers, with producers who did not participate in the programs adopting BMPs.

5.5.3 There are areas for improvement in the current Performance Information Profile

The ultimate outcome indicators for both ACS–LL and OFCAF are consistent with NCSF and international reporting obligations on GHG emissions. However, given the nature of ACS–LL as an innovation program supported by science, the project timelines are too short to see conclusive scientific results related to the achievement of these outcomes.

For both ACS–LL and OFCAF, the current Performance Information Profile set out expected outcomes and indicators that do not adequately measure the impact of program activities on behaviour changes which may contribute to achieving targets for reduced GHG emissions.

For example, the development of strong relationships and trust networks emerged as a key factor enabling the successful implementation of LLI, ACS–LL and OFCAF and potentially having reach beyond the programs. However, the measurement of numbers of partnerships and meetings in living lab annual reports under the ACS–LL Performance Information Profile does not capture the impact of the networks that have developed. Similarly, the OFCAF Performance Information Profile measures acres of land and hectares where BMPs have been implemented but does not capture the impact of the various field demonstrations and knowledge transfer events undertaken to support adoption.

The recent Office of the Auditor General report on agriculture and climate change also included findings related to performance measurement. Their findings focussed on issues related to targets set for indicators. Their recommendation on performance measurement includes a recommendation to finalize measurable, time-bound targets primarily linked to the reduction of GHG emissions. The focus of this evaluation's recommendation is to ensure that the indicators outlined in performance measurement documents are meaningfully capturing the programs' impact. This will likely include both qualitative and quantitative indicators.

5.5.4 Progress on Gender-based Analysis Plus indicators

The evaluation found that under ACS, both the ACS–LL and OFCAF streams are making efforts consistent with AAFC's framework to support URMG in the sector. Some ACS–LL and OFCAF recipients spoke of the importance of involving diverse producers including accommodating the specific needs of Indigenous producers, with 1 recipient describing significant efforts to include Indigenous voices in program governance.

LLI

Federal and departmental policies regarding GBA Plus were under development when the LLI was introduced in 2018. Nevertheless, LL-Eastern Prairies and LL-Quebec reported that activities undertaken as part of their projects generated benefits for Indigenous communities and supported the development of working relationships with Indigenous groups.

ACS–LL

Indigenous groups are explicitly identified as eligible applicants and partners in ACS–LL Applicant Guide. ACS–LL awarded funding to Mistawasis Nêhiyawak, its first Indigenous-led living lab on July 14, 2022.

All ACS–LL recipients are required to report on the partnerships established with URMGs in annual performance reports. In ACS–LL first year reports (2022-23), 6 of the 9 ACS–LLs introduced in 2022 reported that a small number of partnerships and core participating producers were members of underrepresented and marginalized groups. Given that only 1 year of data exists, it is not yet possible to draw conclusions regarding the extent of participation of underrepresented groups in ACS–LL partnerships or as core participating producers.

OFCAF

Similarly, the OFCAF Phase 2 application requires applicants to explain how their organization will ensure that assistance will benefit Indigenous and other underrepresented groups. To this end, recipients are required to report on the proportion of underrepresented groups among supported farmers, although actual reporting varies. As with ACS–LL, there is only one year of reporting and this reporting was inconsistent across recipients, making it difficult to draw conclusions at this time.

In addition to annual project reports, AAFC's Industry Development Division recently consulted with OFCAF recipients regarding OFCAF accessibility and reach. Results indicate that applicants from underrepresented groups are more likely to implement multiple BMPs over time and to provide leadership in the sector. However, they are also more likely to be new entrants to the sector, and to encounter a range of challenges including funding barriers, lack of awareness of programs and need for support with applications.

These results are consistent with evaluation findings suggesting that while engagement with diverse producers and related organizations has been an OFCAF focus, populations vary across regions and it is not always possible to reach underrepresented and marginalized groups in specific areas. As a result, a tension arises for the program between rewarding established players with large capacity who can be expected to implement high volumes of BMPs, versus making way for new applicants who may be smaller, have less capacity, but whose participation may encourage the wider adoption of BMPs and enhancement of sustainable agricultural practices intended by the program.

6.0 Conclusions and recommendations

6.1 Conclusions

LLI, Agricultural Climate Solutions-Living Labs (ACS–LL) stream and OFCAF stream are well aligned with AAFC and federal priorities and international obligations regarding reduction of greenhouse gas (GHG) emissions and carbon sequestration as well as with AAFC's URMG. In addition, AAFC is a leader in the implementation of the agroecosystem living labs approach and collaborates with international partners in this area.

Although not adequately measured under the Performance Information Profiles for ACS–LL and OFCAF, the development of strong relationships and trust networks emerged as a crucial outcome and early signals are positive. It is premature to assess the achievement of most other outcomes. Also, there is a lack of clarity about the relationship between of ACS–LL and OFCAF.

Lastly, long timeframes for approvals and claims processing have slowed progress and caused challenges, especially for smaller recipients.

6.2 Recommendations

Recommendation 1a: The Assistant Deputy Ministers of Programs Branch and Science and Technology Branch should, together, refine the performance measures of ACS–LL to better reflect and assess the relationship building, knowledge gains and behaviour changes in the sector that result from ACS–LL activities.

Recommendation 1b: The Assistant Deputy Minister of Programs Branch should refine the performance measures of OFCAF to better reflect and assess the relationship building, knowledge gains and behaviour changes in the sector that result from OFCAF activities.

Recommendation 2: The Assistant Deputy Minister of Program Branch, working with the Assistant Deputy Ministers of Science and Technology Branch and Strategic Policy Branch, should clarify the relationship between the ACS–LL and OFCAF and communicate this relationship to key stakeholders.

Recommendation 3: The Assistant Deputy Minister of Programs Branch should clarify application and assessment requirements for potential applicants; examine these requirements to determine if there are areas for improvement and, if any are found, take action to improve efficiency and timeliness with respect to application assessment and claims payment processes.

Annex A: Logic model

ACS Living Labs Logic Model

Activities

Program activities

• Promote the program to potential applicants
• Receive project proposals
• Review project proposals and propose projects/proposals for approval or rejection
• Issue decision letters
• Negotiate and prepare contribution agreements
• Provide funding to recipients
• Process financial claims
• Collect and analyze performance information
• Monitor projects

Recipient activities

• Conduct outreach activities to build partnerships among potential partners for a project for implementation under Contribution funding
• Acquire expertise to engage with researchers
• Develop research and development, knowledge transfer and adoption plans to be implemented in a project under Contribution funding
• Conduct research and development
• Ensure producer engagement and participation in on-farm co-development of BMPs
• Facilitate approaches and solutions by testing, piloting, demonstrating and supporting adoption of processes and technologies that can be more broadly applied
• Implement of knowledge transfer plans and adoption strategies to incent and support BMP adoption by producers
• Strategic planning to respond to environmental priorities
• Data collection

Outputs

• Recipients receive funding to establish living labs to incent, support and accelerate on-farm co-development, testing, dissemination, adoption and monitoring of technologies and BMPs that sequester carbon, mitigate climate change and provide environmental co-benefits and social benefits
• Performance reports for AAFC

Immediate outcomes

• Establishment of partnerships to encourage diversity in engagement and participation in co-development
• Producers' engagement and participation in on-farm co-development of BMPs
• Enhanced capacity building to strengthen the sector

Intermediate outcomes

• Increase in capacity of producers to adopt BMPs
• Producers' adoption of carbon sequestration BMPs is increased
• Producers' adoption of GHG emission reduction BMPs is increased
• Increase in environmental co-benefits BMPs
• Increase in scientific knowledge base in the sector
• Incorporation of targeted farm practices into National [GHG] Inventory Reporting (to be completed as part of STB work on data strategy)

Ultimate outcomes

• Sequestration of GHGs on Canadian agricultural lands is increased
• Canadian agriculture is more environmentally sustainable

Source: Environment and Climate Change Programs Performance Information Profile, Programs Branch, 2023.

ACS On-Farm Climate Action Fund Logic Model

Activities

Program activities

• Promote the program to potential applicants
• Receive project proposals
• Review project proposals and propose projects/proposals for approval or rejection
• Issue decision letters
• Negotiate and prepare contribution agreements
• Provide funding to recipients
• Process financial claims
• Collect and analyze performance information
• Monitor projects

Recipient activities

• Provide extension or agronomic services to producers to support adoption of BMPs, including development of farm-specific plans
• Support adoption of BMPs among producers, including development of farm-specific plans, purchase/rental of supplies and equipment and incentive payments
• Develop and deliver training for agronomists for specific BMPs
• Monitor, assess and report on project impacts
• Activities related to outreach and promotion in support of BMP adoption
• Establish processes and systems in support of BMP adoption
• Acquire expertise to facilitate BMP adoption by producers
• Other activities as required and approved in advance by AAFC to support successful implementation of the program

Outputs

• Recipients increase capacity to support producers in the adoption of BMPs that mitigate GHG emission, including in the areas of nitrogen management, cover cropping and rotational grazing
• Performance reports for AAFC

Immediate outcomes

• Increased participation of producers BMPs within the 3 areas (nitrogen management, cover cropping, rotational grazing) (GBA+)

Intermediate outcomes

• Increased adoption of beneficial management practices (nitrogen management, cover cropping and rotational grazing)

Ultimate outcomes

• GHG emissions are reduced in the Canadian agriculture sector

Source: Environment and Climate Change Programs Performance Information Profile, Programs Branch, 2023.

Annex B: Evaluation methodology

Document and file review

The document and file review assessed program relevance, efficiency and effectiveness. This line of evidence included a review of the program's foundational and governance documents describing the historical context for the ACS Program; departmental and federal documents outlining Government of Canada priorities; and project files from LLI, ACS–LL and OFCAF (including performance information).

Literature review

The literature review primarily focussed on assessing program relevance. Grey literature and peer-reviewed articles were reviewed using Google, Google Scholar, government (for example, federal and provincial agricultural ministries) and industry association websites (for example, living labs conference papers, presentations and documents and relevant member organizations related to LLI, ACS–LL and OFCAF delivery). Literature published between 2003 and 2023 was considered, but more recent publications published in the last 5 years were prioritized for recent developments. Theories and principles research gave priority to more widely used materials over recent decades. The scope focussed on Canadian literature using search terms specific to adoption of BMPs, adoption of agri-environmental practices and innovation support programs in Canada.

Key informant interviews

To support an assessment of program relevance, efficiency and effectiveness, interviews were conducted with key informants. A total of 45 individuals were interviewed who represented Living Labs Division and OFCAF program staff, other AAFC staff (including Senior Managers, Directors, Directors General and positions related to performance measurement, corporate reporting and intellectual property) and LLI, ACS–LL and OFCAF recipients. A total of 25 people were consulted at AAFC and 20 from recipient groups (5 from ACS–LL, 4 from OFCAF and 3 from both).

Case studies

Case studies were conducted to address relevance, efficiency and effectiveness for the LLI (2018–2023). A total of 4 case studies were completed examining the 4 Living Labs projects, including Atlantic, Ontario, Quebec and Eastern Prairies. Relevant documents reviewed included but were not limited to contribution agreements, annual and final performance reports, interview notes, news releases, newsletters and recipient association websites. Additional interviews were completed and used to support evidence obtained through other documentation.

Logic model and process map workshops

This line of evidence contributed to the assessment of program design and efficiency. 2 workshops, 1 each for LLI/ACS–LL and OFCAF, were held with program staff from both the LL Division and the Industry Development Division OFCAF team. Focus group style conversations explored the program logic models, performance information profiles and process maps developed for LLI/ACS–LL and ACS-OFCAF. Discussions identified opportunities for improvement in performance measurement and provided the evaluation team with an understanding of how funding flows from AAFC to farmers to assist in their adoption of BMPs.

Annex C: Management response and action plan