RDMC, February 8, 2019
Table of contents
- Executive Summary
- 1.0 Introduction
- 2.0 Methodology and Scope
- 3.0 Background
- 4.0 Program Relevance
- 5.0 Performance
- 6.0 Design and Delivery
- 7.0 Conclusions and Recommendations
- Annex A: Evaluation Methodology
- Annex B: Canadian Agricultural Adaptation Program 2014-2019 Logic Model
- Annex C: Achievement of Outcomes
- Agriculture and Agri-Food Canada
- Advancing Canadian Agriculture and Agri-Food Program
- Canadian Agricultural Adaptation Program
- Gender-Based Analysis Plus
The Office of Audit and Evaluation of Agriculture and Agri-Food Canada (AAFC) undertook an evaluation of the Canadian Agricultural Adaptation Program (CAAP) (2014-2019) to assess the relevance, performance, design, and delivery of the program.
Methodology and Scope
The evaluation assessed CAAP activities over a five-year period, from April 1, 2014 to March 31, 2019. Methodology included a document, file, and data review; a literature review; interviews with AAFC staff, recipients, applicants, project stakeholders, and external stakeholders and experts; and case studies.
For over 20 years, AAFC has funded adaptation programming designed to address new and emerging issues and opportunities in the agriculture, agri-food and agri-based products sector. Over the years, adaptation programming has undergone significant changes. The most recent adaptation program, CAAP, provides project-based funding to not-for-profit organizations in the agricultural sector. CAAP is intended to bridge gaps in science, adoption, and commercialization across AAFC programming, including those in the Growing Forward 2 and Canadian Agricultural Partnership policy frameworks, with flexibility to complement AAFC’s innovation programs.
The CAAP budget is $38.2-million for fiscal years 2014-2015 to 2018-2019. Projects are cost-shared with industry partners in the form of non-repayable contributions, up to 50 percent of eligible project costs.
CAAP is a valuable initiative that facilitates adaptation in the agriculture, agri-food, and agri-based products sector.
- There is a continued need for the Program. CAAP enables the agricultural sector to remain competitive in an evolving global marketplace. Flexibility is a key attribute of CAAP, making it responsive to the needs and priorities of the agricultural sector.
- The Program does not duplicate other federal government adaptation and innovation programming. CAAP aligns with other AAFC programming and is uniquely positioned to support adaptation projects that other programs cannot, increasing the sector’s ability to respond to new or emerging issues.
- The Program experienced low uptake due to a variety of factors related to overly broad objectives; changes to the program scope, eligibility, funding ratio, and delivery model; and limited promotion.
- Despite limited uptake, funded projects generated some positive results. Examples include increased collaboration and information sharing, and the development of tools, strategies, products, and processes to address critical needs, investigate solutions, and seize new opportunities.
- The actual grants and contributions expenditures represented 31 percent of planned expenditures from 2014-2015 to 2017-2018.
- For over 20 years, AAFC adaptation programming has addressed new and emerging issues in the agricultural sector enabling it to adapt and remain competitive. CAAP has made progress in facilitating sector implementation of tools and strategies to adapt to changing circumstances, and adoption of innovations developed through multiple initiatives.
- CAAP is delivered in an efficient and timely manner and is responding to identified design and delivery issues. There are opportunities to address the limitations of the current Program design and delivery model to increase uptake and better align the Program with departmental, federal government, and industry needs and priorities.
- The Program’s performance indicators and targets did not capture the full scope of the Program impacts as they rely on average or percentage levels of achievement. Results could be misleading since one or two very successful projects could significantly skew the averages. Immediate and Intermediate outcomes did not adequately consider the broad range of CAAP objectives. Longer-term outcomes could better address Program economic or other public good impacts while an absence of long-term tracking studies made it difficult to assess long-term impacts.
Recommendation 1: Assistant Deputy Minister, Programs Branch, should refine priorities, assess alternative design and delivery options, and develop a communication strategy to promote the CAAP with key national and sector stakeholders.
Recommendation 2: Assistant Deputy Minister, Programs Branch, should review its performance outcomes and indicators to ensure they align with Program objectives and priorities and develop tools or processes to better assess the long-term impacts of adaptation programming.
The Office of Audit and Evaluation of Agriculture and Agri-Food Canada (AAFC) undertook an evaluation of the Canadian Agricultural Adaptation Program (CAAP) (2014-2019) as part of the 2017-2018 to 2020-2021 Integrated Audit and Evaluation Plan. CAAP funds projects that are national or sector-wide in scope and that support the agriculture, agri-food, and agri-based products sector to adapt to new and emerging issues, opportunities, and challenges to remain competitive. The results of this evaluation are intended to inform current and future program and policy decisions.
2.0 Methodology and Scope
The evaluation took place from January 16, 2018 to November 30, 2018 and was conducted in accordance with the Treasury Board Policy on Results and Directive on Results. The evaluation assessed Program activities over a five-year period, from April 1, 2014 to March 31, 2019.
The evaluation assessed the relevance, performance, design, and delivery of the Program, including lessons learned. A comparative analysis with similar programs was used to examine possible future directions and alternate ways of achieving intended results.
Multiple lines of evidence included a document, file, and data review; literature review; interviews with AAFC staff, recipients, applicants, project stakeholders, external stakeholders, and experts; and case studies. The detailed methodology is in Annex A.
AAFC has delivered programming to support adaptation in the agriculture, and agri-food, and agri-based products sector since 1995. The Canadian Adaptation and Rural Development Program was delivered from 1995-1999 and 1999-2003 before transitioning program activities to the three subsequent programs, the Advancing Canadian Agriculture and Agri-Food (ACAAF) Program (2004-2009), CAAP (2009-2014), and CAAP (2014-2019). Despite some differences in design and delivery models, all adaptation programs have had broad, parameters intended to stimulate change, develop new approaches, or respond to urgent issues in the sector.
In 2014, the Program shifted from being largely regionally-delivered by Regional Adaptation Councils, to focusing on national and sector-wide priorities where funding was administered solely by AAFC Program Staff. The current CAAP (2014-2019) aims to fund industry-led projects that address the Program’s objectives and that are in line with national or sector-wide priorities identified by industry and/or government.
Table 1 illustrates the significant changes to the Program design and delivery from CAAP (2009-2014).
|CAAP Program||Project Scope||Eligibility||Funding Ratio||Delivery Model|
|CAAP (2009-2014)||National and regional projects||Any Canadian legal entity capable of entering into a contract, including, but not limited to, organizations and associations; cooperatives; marketing boards; Indigenous groups; for-profit companies; and individuals.||
Contributions to not-for-profit organizations and associations were non-repayable
Contributions to for-profit entities were repayable.
National and regional delivery components
|CAAP (2014-2019)||National and sector-wide, industry-led projects||Not-for-profit organizations and associations, including cooperatives, marketing boards, and Indigenous groups||
Contributions to not-for-profit organizations and associations were non-repayable
Centralized and national delivery
|Sources: Canadian Agricultural Adaptation Program (2014-2019) Applicant Guide, AAFC and Canadian Agricultural Adaptation Program (2009-2014) National Application Guide.|
CAAP (2014-2019), referred to henceforth as CAAP, is intended to bridge gaps in science, adoption, and commercialization across the suite of AAFC programming and to complement AAFC’s innovation programs. The objective, as stated on the departmental website, is to enable the agriculture, agri-food, and agri-based products sector to:
- Seize opportunities enabling industry to take advantage of a situation or circumstance to develop a new idea, product, niche, or market opportunity;
- Respond to new or emerging issues that were unknown or not a concern previously; and,
- Pathfind and/or pilot solutions to new and ongoing issues. Pathfinding is investigating new ways and/or different options of dealing with new and/or ongoing issues. Piloting is testing ideas and/or approaches to apply in the sector.
CAAP provides non-repayable contributions for not-for-profit, industry-led projects to enable the sector to adapt and remain competitive. Projects must demonstrate that the activities:
- Involve or are supported by groups or individuals that represent the targeted sector (national or sector-wide); and,
- Ultimately benefit the stakeholders of the targeted sector.
The projects and activities funded through CAAP must demonstrate that they are:
- Capturing value-added opportunities for the industry or the sector;
- Adapting to changing consumer demands within the mandate of AAFC;
- Implementing and/or developing strategies and processes to address urgent or critical issues, such as outbreaks and other threats;
- Exploiting emerging opportunities, such as optimizing processes, developing technology, and conducting feasibility studies for products, processes, and technologies;
- Building capacity and/or developing tools to anticipate and prepare for the future. This includes developing innovative approaches to take advantage of new opportunities and address issues of broad public benefit; or,
- Facilitating new approaches and solutions by testing new ideas and processes for sector-wide application.
Funded projects must align with CAAP objectives, be a priority for the sector or industry, demonstrate clear benefits to stakeholders and Canada, demonstrate capacity to achieve objectives, and generate expected outcomes. The CAAP Logic Model provides further detail on the Program’s key activities and outputs (Annex B).
3.4 Application and Reporting Process
Applications are accepted on a continuous basis, which means that they can be submitted for consideration at any point during the life of the Program until the available budget has been fully allocated. Applications are pre-screened by AAFC to determine eligibility before an application package is completed.
Project proposals are reviewed by AAFC experts, and then they are presented for funding decisions to the Directors General Innovation Committee, whose membership consists of directors general from across AAFC. Where additional expertise is required to assess project proposals, AAFC will draw upon the services of external experts. Based on delegated authorities, projects recommended for approval are signed-off by the responsible Director, Director General, and Assistant Deputy Minister. When Minister approval is required, the Deputy Minister is responsible for recommending projects to the Minister.
The maximum funding amount for each project should not exceed $1-million. An applicant can apply more than once but for each applicant, funding should not total more than $4-million over five years. Successful applicants sign a Contribution Agreement with AAFC and carry out project activities. Contribution Agreements outline the amount of funding AAFC will provide to the recipient toward eligible costs, as well as the recipient’s responsibilities and obligations.
The overall administration of the application intake and assessment, along with the fiscal management of the Program, is performed by the Commercialization and Environmental Programs Division of the Innovation Programs Directorate, Programs Branch. Eligible costs related to a project are shared 50:50 between AAFC and the applicant. For projects that are considered a high priority or that address an urgent issue, AAFC may decide to increase its percentage of contribution if the project is deemed beneficial and important to the sector and/or Canada. Program results are compiled by the Program Policy Development, Analysis and Results Division of the Innovation Programs Directorate and are reported in the Departmental Plan.
CAAP uses A-base transfer payments (a department’s authorized budget), with non-expiring terms and conditions, to support program delivery and to fund eligible costs of a project. From 2014-2015 to 2018-2019, $38.2-million was budgeted for Program activities (Table 2).Footnote 1
Due to lower than expected program uptake, adjustments were made to the CAAP grants and contributions budget, which was revised from $10.1-million yearly for 2014-2015 and 2015-2016, to $6.1-million in 2016-2017, $3.9-million in 2017-2018, and $8.1-million in 2018-2019 (Table 2). At the time of the evaluation, several CAAP projects were under review or still in progress, which may increase spending before the Program ends March 31, 2019.
|Year||Budget||Actual Expenditures||Variance (%)|
Note that 2018-2019 expenditure data is not complete since the fiscal year ends on March 31, 2019. Amounts are as of December 2018.
4.0 Program Relevance
4.1 Continued Need
Flexibility is a key attribute of CAAP, making it responsive to the needs and priorities of the agriculture, agri-food, and agri-based products sector.
There is a continued need for CAAP to help the agriculture, agri-food, and agri-based sector adapt and remain competitive. Adaptation in the agricultural sector is essential given changing local and global market conditions, consumer demands, and environmental and political concerns. Growing population and increased demand for healthy foods necessitates increased production and a heightened focus on organic products and emerging commodities (for example, hemp, pulse products, and organic foods). Competition in high-growth markets (for example, India and China) and shifting international trade conditions require the sector to remain nimble and resilient. There is also a need to increase sustainable and technologically advanced farming practices, such as soil testing and adoption of precision agriculture, to support the prioritization of environmental concerns and more efficient practices.
CAAP’s highly flexible funding mechanisms and continuous application intake process are effective in responding to industry adaptation needs as circumstances arise, which other programs may be too restrictive to fund. On average, project recipient interviewees reported there was only a 35 percent likelihood that projects submitted to CAAP would have proceeded in the absence of CAAP funding and if they had proceeded, they would have been greatly reduced in scope and impact.
CAAP’s intentionally broad Program objectives enable the Program to consider providing support for a range of adaptation projects. However, these same objectives may have prevented industry from having a clear understanding of the Program’s purpose.
4.2 Alignment with Departmental and Federal Priorities and Roles and Responsibilities
The Program objectives are well-aligned with departmental and federal priorities to support sector adaptation and innovation.
CAAP objectives align with AAFC priorities, which include a competitive and market-oriented agriculture, agri-food, and agri-based products sector that proactively manages risk; and an innovative and sustainable agriculture, agri-food, and agri-based products sector. CAAP aligns with federal roles and responsibilities to support adaptation in the sector as described in the Department of Agriculture and Agri-Food Act. The Minister of Agriculture and Agri-Food Mandate Letter highlights the Department’s overarching goal to support the agricultural sector to be a leader in innovation; develop and seize new opportunities; develop new products and expand export markets; and, address critical issues such as water management and food safety. The objectives of CAAP reflect the federal government commitment to support adaptation within the agricultural sector, as outlined in the December 2015 Speech from the Throne, and Budgets 2017 and 2018.
4.3 Comparison with Other Federal Government Programs
CAAP does not duplicate other federal government adaptation and innovation programming.
The evaluation found that there is a slight overlap between CAAP and other federal programs that support adaptation and innovation within the agriculture, agri-food, and agri-based products sector.
Several federal programs support adaptation and innovation but differ from CAAP since they have a regional or sector focus, cover different stages of the innovation continuumFootnote 2, apply different eligibility criteria, or are more focused on innovation. For example, the Atlantic Canada Opportunities Agency’s Atlantic Innovation Fund and the Business Development Program are only available to applicants within Atlantic Canada. Western Economic Diversification Canada’s Western Innovation Initiative targets for-profit entities and does not aim to support projects at the earliest stages of the innovation continuum, for example, fundamental and applied research.
Other AAFC programs supporting adaptation within the agri-sectors are fairly distinct from CAAP. For example, AgriAssurance focuses on the adoption of tools, systems, and initiatives to support the health and safety claims about agricultural and agri-food products. The AgriCompetitiveness program supports capacity building and industry awareness initiatives (Table 3).
A literature and document review examined how CAAP aligns with other federal programs by assessing eligibility criteria and program focus. This analysis, supported by interviews with AAFC staff and project stakeholders, found that CAAP aligns with other AAFC programming and is uniquely positioned to support adaptation projects that the other federal programs cannot, increasing the sector’s ability to respond to new or emerging issues.
|Delivery Organization||Program||Focus||Eligibility||Innovation Continuum|
|Atlantic Canada Opportunities Agency||Business Development Program||Strategic priorities are identified by each individual region||Small and medium-sized enterprises, not-for-profit organizations, Atlantic Canada||Fundamental/applied research, product commercialization and market development|
|Western Economic Diversification Canada||Western Innovation Initiative||Clean technology; clean resources; digital technology; advanced manufacturing; value added agriculture; or health/bio sciences||For-profit businesses (<500 FTEs) located in Western Canada||Technology development and demonstration, product commercialization and market development|
|AAFC||CAAP||National or sector-wide priorities identified by industry and/or government||Not-for-profit organizations, marketing boards, Indigenous groups||Applied research, technology development and demonstration|
|AAFC||AgriScience||Shorter-term research activities to enable industry to overcome challenges and address fiscal barriers experienced by small and emerging sectors||For-profit and not-for-profit organizations||Pre-commercial science activities and research|
|AAFC||AgriAssurance||Funds projects to help industry develop and adopt systems, standards, and tools to support health and safety claims about Canadian agricultural and agri-food products||Not-for-profit: organizations operating in the agriculture, agri-food, and agri-based products sector that operate solely for social welfare, civic improvement, pleasure or recreation, any other purpose except profit||Technology development and demonstration, product commercialization and market development, and market-entry and market volume.|
|AAFC||AgriCompetitiveness||Aims to help the agricultural sector to leverage, coordinate, and build on existing capacity, enhance safety, adapt to changing commercial and regulatory environments, seize new opportunities, share best practices, and provide mentorship opportunities||Not-for-profit organizations that operate solely for social welfare, civic improvement, pleasure or recreation, any other purpose except profit.||Technology development and demonstration, product commercialization and market development, and market-entry and market volume.|
Sources: CAAP (2014-2019) Applicant Guide; AgriScience Program, Projects Applicant Guide; Atlantic Canada Opportunities Agency Business Development Program Details; Western Economic Diversification Canada. Western Innovation Initiative Applicant Guide, 2018. AgriAssurance Program Applicant Guide. AgriCompetitiveness Program Applicant Guide.
5.1 Achievement of Output
CAAP is progressing towards achieving its targeted output of 25 projects, however the 23 approved projects represent less than a third of available funding.
CAAP collects performance data from recipients at regular intervals throughout the duration of projects, as well as through annual and final performance reports. Project reporting schedules vary depending on the projects’ level of risk (low, medium, or high), which is assessed at the time of project approval. Factors assessed include the timeframe of activities; difficulty in achieving project objectives; project materiality; recipient’s prior history, financial capability, and stability; and capacity to manage the project.
The evaluation found that from 2014-2015 to 2017-2018, only 31 percent of planned grants and contributions were expended towards CAAP projects (Table 4). This is considerably less than the 75 percent of planned grants and contributions expended during the previous CAAP. Spending levels were also lower for the current CAAP. From 2014-2015 to 2017-2018, CAAP spent an average of $2.3-milion per year compared to an average of $3.3-million per year from 2009-2010 to 2013-2014.
|Fiscal Year||Planned Grants and Contributions||Actual Grants and Contributions||Actual/
|Note: 2018-2019 was removed from the analysis since the fiscal year is not yet complete.
Source: CAAP financial data.
The evaluation found that lower uptake was due to a variety of factors related to the overly broad objectives, changes to the Program (program scope, eligibility, funding ratio, and delivery model), and the limited promotion of the changes.
Program objectives were intentionally broad, enabling a range of project proposals to meet the government’s objectives as they evolved. However, these objectives may have prevented industry from having a clear understanding of the Program’s purpose.
Industry awareness and understanding of CAAP diminished when the Program shifted from being largely regionally-delivered by Regional Adaptation Councils, to focusing on national and sector-wide priorities, creating uncertainty among prospective applicants around the renewed purpose and function of the Program. Eligibility shifted from any Canadian legal entity (including, in select regions, for-profit organizations) to not-for-profit organizations and associations. The change in industry matching requirement from 15 percent to 50 percent of eligible project costs was also a factor in the lower than expected program uptake. Interview responses indicated that not-for-profit applicants were less likely to apply since they could not cover the higher funding proportion. Program changes were not widely communicated during the outset of the Program and created a significant impediment to program uptake.
CAAP’s expected output by March 31, 2019 is 25 projects funded. During the evaluation period, 28 projects were approved for $12.8-million in funding, including two completed and two approved projects that were waiting for signed Contribution Agreements. The evaluation noted that the target for this indicator is low and does not take into consideration the materiality of projects (dollar value of the projects), since only 31 percent of planned grants and contributions were expended and the target has already been achieved.
The largest proportion of CAAP funding was approved for projects that focused on livestock (34 percent), dairy farming (23 percent), horticulture (20 percent), and poultry and eggs (7 percent) (Table 5).
|Approved Projects by Sector||Number of Projects||CAAP Funding Approved ($)||CAAP Funding Approved (%)|
|Poultry and eggs||1||844,021||7|
|Pulses and special crops||1||649,251||5|
|Other (Hemp, Organics, and Honey)||4||600,720||5|
|Grains and oilseeds||1||42,000||<1|
|Source: CAAP Project Data as of December 2018Footnote 3|
For projects associated with specific commodity groups, the amount of Program funding allocated was relatively aligned to the size of the major agriculture commodity groups, as indicated by the percentage share of total farm market receipts in 2016 (Figure 1)Footnote 4. Sectors that had higher total farm market receipts such as live animals, red meat, and other animal products, dairy, and horticulture generally received a larger proportion of funding from CAAP. The only exception was the grains and oilseeds sector which had 37 percent of farm market receipts but received less than one percent of CAAP funding. This sector was more appropriately served by other AAFC funding programs, such as the AgriScience Program Clusters and Projects, due to the sector’s capacity to manage larger scale and longer-term sector development investments.
5.2 Immediate Outcome
CAAP achieved its expected immediate outcome of establishing partnerships and collaborations, as well as disseminating project information to target groups.
The evaluation found that, on average, three partnerships and collaborations were established per project. The Program is on track to achieve the target of three partnerships per project by March 2019. However, as the indicator tracked average partnerships per project this skewed the results upwards. For example, some projects reported having no partnerships, while others reported as many as 22. There was a range of public and private sector partners involved in project activities. Partners included universities, First Nations, not-for-profit organizations, local/regional governments, research labs, industry associations, and local development corporations.
Case studies illustrated that partnerships and collaborations facilitated the success of CAAP projects. For example, the Alberta Beef Producers collaborated with a private research organization and Alberta Agriculture and Forestry to develop and test analgesic elastrator bands for use during cattle castration. This partnership was instrumental as the close working relationship provided direct access to industry through product development and field trial processes, and provided lab space, field staff (beef cattle specialists) and expert researchers in pain mitigation and animal welfare.
CAAP projects generated an average of eight information items per project, surpassing the March 2019 target of six. Examples of information items considered in project reporting include media reports, information events, and publications. A variety of information items were developed and shared across diverse stakeholder groups such as organizations, industries, and universities. Project findings and activities were commonly reported at industry meetings, presentations, conferences, and published in media reports. Detailed Program outcomes are in Annex C.
5.3 Intermediate Outcomes
CAAP achieved one of the intermediate outcome and is on track to achieve the other two of its three intermediate outcomes including the development of tools, strategies, products or processes, and has supported collaboration across stakeholders.
The Program achieved its target of developing 25 tools and strategies that facilitate adaptation to changing circumstances and/or respond to immediate sector-based concerns. Types of new products, strategies, processes, and technologies developed included datasets and samples (for example, genetics and genomics data), food processing and/or production methods, and contaminant prevention and detection technologies. The evaluation noted that the target for this indicator was low, given that the target was easily achieved even though only 31 percent of planned grants and contributions were expended from 2014-2015 to 2017-2018. Furthermore, this indicator is measuring output, number of tools and strategies rather than measuring the actual outcome of increased adaptability and responsiveness to changing circumstances.
CAAP projects are on track to achieve the second intermediate outcome of developing innovative, value-added products, processes, and practices such as new food ingredients and products, or automated production practices. Fifty percent of CAAP projects have facilitated industry development of these elements. The Program is on track to achieve its target by March 2019, this indicator was not always limited to innovation projects as defined in the indicator. For example, six projects under the current funding cycle focused solely on the development of tools and strategies for adaptation as opposed to innovation and are misrepresenting results.
In support of the third intermediate outcome, CAAP facilitates a range of ongoing collaborations between recipients and partner organizations, which contributes to establishing systematic communications among key industry stakeholders. The Program is on track to achieve its target of 75 percent of projects reporting on-going collaboration beyond project completion, with 67 percent of projects reporting planned or actual on-going collaboration. CAAP projects are likely to lead to spin-off projects or additional activities, such as new applications of technology, tools, strategies, and product varieties. For example, plans are in place to expand communications on the Alberta Beef Producers’ project which developed improved meat quality through pain management tools.
Results from a long-term tracking study conducted to examine longer-term impacts of projects funded under the Advancing Canadian Agricultural and Agri-Food (ACAAF) Program (2004-2009) showed that 74 percent of adaptation projects continued collaborations established through the Program. Although the nature of some collaborations changed (for example, became less formal or frequent), several recipients continued to work with project partners to further their initiatives.
Nearly half (43 percent) of approved projects under the current CAAP were ongoing; intermediate outcomes will be assessed in future years. In addition, case studies provide evidence of some of the beneficial products developed through CAAP funding. For example, the Charter Community of K’asho Got’ine, located in a remote region of the Northwest Territories, was successful in developing a unique type of organic fertilizer consisting of decomposed and macerated fish. This organic fertilizer effectively helped grow potatoes in modular pods and eliminated the need for chemical fertilizers and standard horse manure, which would have been costly to import from the southern provinces.
While CAAP has achieved, or is on track to achieving, its immediate and intermediate outcome targets, these targets were revised to reflect lower program uptake.
5.4 Long-Term Outcomes
For over 20 years, AAFC adaptation programming has addressed new and emerging issues in the agricultural sector enabling it to adapt and remain competitive.
CAAP has made progress in facilitating sector adoption of innovative, value-added products, processes, and practices, as well as implementation of tools and strategies developed through multiple initiatives. While AAFC adaptation programming has existed for more than 20 years, the only long-term tracking study has been for the ACAAF (2004-2009) Program. The study results indicated that 60 percent of completed projects led to the implementation of strategies to respond to emerging issues and opportunities. In the absence of other data, to further assess long-term impacts, the evaluation examined three projects from the CAAP (2009-2014) conducted by the Canadian Livestock Transport (CLT), the Canadian Centre for Swine Improvement Inc. (CCSI), and the Western Barley Growers Association (WBGA). The evaluation found that the collaborations and partnerships established through these projects are still continuing, and have effectively responded to industry needs.
Using the ACAAF long-term study as a benchmark suggests that CAAP could meet its 60 percent target. For the current CAAP, long-term program outcomes are not expected to emerge until March 2021, but there are some early results. During the evaluation period, three projects (17 percent) were identified as demonstrating long-term outcomes and had implemented tools and strategies to adapt to changing circumstances and/or respond to urgent and/or critical issues. The Program anticipates that 30 percent of projects developed through CAAP will result in sector adoption of innovation items by March 31, 2021.
Evaluation evidence found that the development of tools and strategies through CAAP projects has started to have an impact on the sector. For example, the Prince Edward Island Potato Board’s project addressed an urgent health and safety issue. The Program’s ability to rapidly respond resulted in widespread application of specialized metal detection tool equipment that detected needle contaminants in Prince Edward Island potatoes. The surveillance and security equipment put in place as a result of this CAAP project helped to preserve the integrity of the Prince Edward Island potato industry, which contributes over $1-billion annually to the province’s economy.
Case studies provide examples of achievements made on this long-term outcome:
- The Egg Farmers of Ontario moved closer to commercializing a technology (that is, hyper-spectral imaging) that can non-invasively determine the gender and fertility of eggs. Advancing this technology to a commercial level will reduce the practice of culling day-old male chicks, diverting them from the hatchery path to the table egg path or other purposes. This will enhance efficiency in egg production while generating increased revenues and addressing animal welfare concerns.
- The Canadian International Grains Institute developed four new methods for processing pulse flours to improve foods typically found to have low nutritional value (for example, bread, pasta, and breakfast cereal).
- The Canadian Beef Breeds Council increased the adoption of genomic technologies in the beef sector; anticipating multiple long-term cost savings and benefits such as enhancing feed efficiency, improving meat quality, and reducing greenhouse gas emissions.
6.0 Design and Delivery
6.1 Program Design and Delivery
Design issues, mainly related to unclear priority areas and promotion of the Program has resulted in low uptake and lapsed funds.
CAAP was designed to fund industry-led projects that address the Program’s objectives of supporting the agriculture, agri-food, and agri-based products sector. The objectives are threefold: supporting adaption to new and emerging issues, opportunities, and challenges; fostering competitiveness through broad, flexible parameters intended to stimulate change; and developing new and innovative approaches in the sector. Despite the intentionally broad objectives, the lack of defined priority areas for investment has made attracting suitable proposals difficult and low uptake has resulted in lapsed funds.
Program management is exploring alternate design and delivery options to better reflect industry needs. Shortly after the start of the 2014-2019 program funding cycle, a pre-screening stage was added to the application process. Although not widely communicated until recently, this new process enabled the Program to vet unsuitable projects at an early stage, reducing the level of effort on applications and enabling Program officers to focus more resources on projects with the potential to succeed in the Program. CAAP planned spending was revised in 2017-2018 and 2018-2019 and Program objectives are being refined to increase uptake in the next iteration.
To better align with industry and government needs and priorities, AAFC Program management proposed refocusing CAAP priority areas to environmental sustainability, adaptation to new technology, and strategic development. The evaluation found that a shift to defined priority areas is likely to have a positive impact on the Program. When asked about future program priorities evaluation interviewees identified environmental sustainability, adaptation to new technology, and strategic development as their top priorities for programming. Increased clarity regarding the purpose and objectives of the Program was identified as a key advantage of implementing targeted funding streams.
The proposed priority areas of environmental sustainability, adaptation to new technology, and strategic development are well-aligned to industry priorities and needs. In terms of environmental sustainability, supporting the sector to mitigate the impacts of climate change was a key theme that emerged from industry consultations conducted during the development of the Canadian Agricultural Partnership. Suggestions included a heightened focus on reducing greenhouse gas emissions, adopting sustainable farming practices, and developing strategies to reduce carbon emissions. There is a need to develop strategies and tools to address sustainability and productivity issues impacting the sector to capitalize on new market opportunities, such as consumer demand for sustainable production, and to manage risks such as increasing government and non-government requirements for water quality and greenhouse gas emissions.
The evaluation identified that there is low adoption of new technology and automated practices among agricultural producers and processers. According to the 2017 AAFC survey of food processors, a majority (63 percent) of the industry is just partially or minimally automated. Key barriers to adoption of new technologies include financial and space constraints. This is significant because global trends of population growth, urbanization, and climate change will necessitate the implementation of new approaches to agricultural production and processing. Feedback from the survey indicates that financial advice and assistance, as well as informational support, would be of great benefit in increasing rates of adoption of new technologies. Specifically, 76 percent of food processor survey respondents indicated that costs were a moderate to severe barrier to automation and 42 percent reported that technical advice could support them in making decisions about adopting new technologies.
In terms of strategic development, emerging sectors such as natural foods and health products, face increased consumer demand but significant labour shortages, particularly in areas such as sales, marketing, and manufacturing, which have impeded growth. Other barriers include complex domestic regulatory frameworks (for example, regulations around Canadian hemp production) and lower levels of public trust around claims made about the organic certifications of Canadian made goods. For example, 44 percent of respondents to a survey disseminated by the Canadian Organic Trade Association indicated that they trust the label, “Canada Organic certified,” compared to 55 percent and 54 percent of respondents indicating trust in “Made in Canada” and “local” claims, respectively. New and emerging Canadian commodities could benefit from increased Program support for strategies around production, product development, marketing, and building public trust for emerging Canadian agricultural and agri-food products.
Though not explicitly stated as current Program priorities, many CAAP projects are already addressing a number of proposed priority or similar thematic areas (Figure 2). Evaluation evidence supports improving clarity around the Program’s purpose and objectives, to support Program uptake.
CAAP funding is available to not-for-profit applicant organizations or associations, marketing boards, and Indigenous groups. In the previous iteration of the Program, for-profit applicants in select regional councils were funded. There is an opportunity for CAAP to attract new for-profit applicants and more projects that fill a public good, national, or sector-wide need by amending some of the funding terms and eligibility rules. This could also improve uptake. Evaluation interviews support that for-profit entities are integral to sector-based adaptation as they are often more willing or able to undertake higher-risk projects and are able to share some of the benefits with others in the sector.
While CAAP requires that industry matches 50 percent of funding, other regionally focused programs have a lower matching requirement. For example, CanNor’s Strategic Investments in Northern Economic Development Program and the Atlantic Canada Opportunities Agency’s Atlantic Innovation Fund require a minimum contribution of only 20 percent for not-for-profit applicants. Stakeholder interviews indicated that not-for-profit led projects with higher needs or urgency are less likely to be able to meet CAAP’s 50 percent industry matching requirement.
AAFC staff and CAAP stakeholders suggested that any changes to Program parameters should consider maintaining sufficient adaptability, enabling it to address emerging issues while remaining focussed on identified priorities.
While one of the roles of evaluation is to explore alternate design and delivery options, Senior Program management had already been in discussions to revise the Program and how it is promoted to increase uptake in the next iteration. The evaluation assessed two alternate models of program design or delivery against CAAP to present a consideration of different delivery options. The programs examined were the Western Diversification Program and the Impact Canada Challenge Platform. These programs were chosen as examples of Treasury Board models of experimentation. It was found that the Western Diversification Program experimented with outcomes-based achievement payments, where the recipient receives varying levels of payment for different kinds of achievement. This approach was found to be administratively effective and heightened the focus on outcomes, which is a gap in the current CAAP design. The federal government is increasingly using challenge functions, team recognition, and prize rewards, as funding mechanisms, which motivates stakeholders and teams to solve public policy problems. Under this model, the government selects priority areas to be addressed such as clean technology, and participating organizations are encouraged to submit their innovative solutions. The Impact Canada Challenge Platform encourages applicants under specific program streams to generate innovative solutions to some of the biggest concerns that face Canadians and their communities, that is, program objectives are clearly defined.
While these Programs provide good examples of experimental design or delivery options, CAAP’s adaptation focus makes it somewhat unique and difficult to align with these delivery methods. The evaluation found that the Program could explore experimentation options to improve delivery and encourage increased uptake in the Program.
6.2 Operational Efficiency
CAAP has generally exceeded its current operational efficiency service standards for timely processing of applications and claims payments.
CAAP programming has prescribed processes and service standards to ensure value-added support to the agriculture, agri-food, and agri-based products sector. This includes notifying applicants in writing of final approval or rejection within 100 business days of the application being considered complete and sending the recipient a payment within 30 business days of receipt of a completed and documented claim. The Program aims to meet its service standards a minimum of 80 percent of the time.
CAAP has generally exceeded its operational efficiency targets with respect to timely processing of applications and claims payments. In 2015-2016, only 59 percent of rejection letters were sent out within the timeframe and only 71 percent of payments were issued in the timeline. For the last quarter of 2016-2017 both service standards were only met 50 percent of the time. AAFC program management reported that these specific delays were the result of unexpected staff resource issues, which impeded the timely processing of applications and payments at some points (Figure 3).
The evaluation found that CAAP had straight-forward application, reporting, and claims processes, which enabled more resources to be allocated to project delivery rather than administrative project management. However, interviewees suggested that the Program could be streamlined, for example the time required to process applications and claims payments. The funding mechanisms appear to be fair and comparable to similar programs and most recipients interviewed were relatively satisfied with the reporting process. Interviewees noted that efficient communications were a result of strong working relationships between recipients and Program officers, and the officers’ in-depth knowledge of projects.
6.3 Performance Measurement
Many performance indicators and targets did not capture the full scope of the Program impacts, which impedes the ability to fully assess Program results.
There are multiple areas where performance outcomes and indicators could be improved to better assess Program impacts. For example, current indicators and targets do not take into account the planned program grants and contribution spending, which impedes the ability to fully assess Program results. The immediate outcome indicators track average collaborations and information items per project. However, these indicators are misleading since one or two very successful projects could skew the averages. For example, among the projects reviewed as part of this evaluation one project had 22 partners, while several projects had none. Instead, the indicator could track the percentage of projects that had at least one new partner.
The broad nature of the outcomes makes it difficult to understand the overall impacts of the Program. The long-term outcomes focus on adoption of innovations and tools but do not further explain the expected economic impacts of these interventions (for example, increased revenues, operational cost-savings, or improved gross margin of project stakeholders). Such impacts could be included in long-term tracking studies of the projects to better understand the significance to the sector of adopting these technologies. There may also be opportunities to consider new outcomes and indicators, depending on the future direction and priorities for the Program, such as those related to environmental and social impacts (for example, estimated reductions in greenhouse gas emissions based on the adoption of a new technology by project stakeholders).
6.4 Government of Canada Policy Considerations
Diversity and inclusion factors have not yet been formally implemented as part of CAAP, but Program management intends to more explicitly consider these in future program iterations.
As per the Treasury Board Directive on Results, the evaluation examined the extent that Gender-Based Analysis Plus (GBA+) was considered. GBA+ is an analytical tool, process, or product used to assess the potential impacts of the policies, programs, services, and other initiatives on diverse groups taking into account identity factors beyond gender including the examination of a range of other intersecting identity factors (sex, age, sexual orientation, disability, race, ethnicity, religion, education, language, geography, culture, and income).
The evaluation found that GBA+ analysis is in the preliminary stages of implementation in CAAP as the policy was introduced mid-way through the funding cycle in 2016-2017. Moving forward, Program management expressed that diversity and inclusion factors will be more systematically considered by the Program in the application process and advertising, as well as to assess the impacts of projects and the Program on diverse groups. Project recipients did not explicitly consider diversity and inclusion in project implementation citing demographic limitations of the sector and their higher prioritization of outcomes and skillsets, regardless of demographic background.
7.0 Conclusions and Recommendations
There is a continued need for CAAP to enable the agriculture, agri-food, and agri-based sector to adapt and remain competitive in the evolving global marketplace, as the Program facilitates responsiveness to industry priorities and needs. CAAP objectives are well-aligned with departmental strategic outcomes, as well as federal government priorities and responsibilities to support sector adaptation. The Program does not duplicate other government programs.
CAAP experienced lower than expected uptake due to changes to the Program (program scope, eligibility, funding ratio, and delivery model), its overly broad objectives, and limited program promotion. Promotion of the Program’s purpose and priorities, and the building of relationships with key national and sector stakeholders, could increase awareness and uptake. Despite limited Program utilization, funded projects generated some positive results related to increased collaboration; information sharing; and the development of tools, strategies, products, and processes to address critical needs. Projects also investigated new options for dealing with new and/or ongoing issues and seized new opportunities. The major factors that contributed to the success of CAAP projects included industry support for CAAP initiatives, effective external collaborations, and efficient project communications. Supportive Program staff and effective AAFC processes further contributed to success.
There are multiple areas where performance outcomes and indicators could be improved to better assess impacts. Many performance indicators and targets did not capture the full scope of the Program impacts since they rely on average or percentage levels of achievement instead of aggregate impacts. Long-term outcomes did not consider the economic or other public good impacts of the Program. New outcomes and indicators should be considered, depending on the future direction and priorities for the Program, such as those related to environmental and social impacts. Long-term tracking studies are critical to better enable the Program to measure and report on adaptation program impacts.
7.3 Design and Delivery
CAAP generally exceeded its claims processing services standards. Evaluation evidence noted some delivery issues identified by recipients. Program management is exploring alternate design and delivery options, including refined Program priorities, to improve uptake. Alternate models of program delivery, such as, but not limited to, outcomes-based achievement payments, challenge functions, or prize rewards, may contain elements that could be adapted to CAAP design and delivery needs. Proposed Program priorities of environmental sustainability, adaptation to new technology, and strategic development are directly linked to sector needs and federal government priorities relating to environmental sustainability and adoption of clean technologies.
Recommendation 1: Assistant Deputy Minister, Programs Branch, should refine priorities, assess alterative design and delivery options, and develop a communication strategy to promote the CAAP with key national and sector stakeholders.
Recommendation 2: Assistant Deputy Minister, Programs Branch, should review its performance outcomes and indicators to ensure they align with Program objectives and priorities and develop tools or processes to better assess the long-term impacts of adaptation programming.
Management Response: Management has agreed to refine program priorities; assess alternative design and delivery options; develop a communication strategy targeted to relevant industry stakeholders; review performance outcomes; and investigate appropriate methodology to assess the impacts of adaptation programming. All actions will be completed by December 2022.
Annex A: Evaluation Methodology
The evaluation was conducted in accordance with the Treasury Board Policy on Results and Directive on Results. It assessed the relevance, performance, and design and delivery of the Program. Specific focus was given to assessing the Program design and delivery. The evaluation took place from January 16, 2018 to November 30, 2018 and assessed Program activities over a five-year period, from April 1, 2014 to March 31, 2019.
Various methodologies were used to undertake the evaluation and to obtain the data required to address each of the evaluation questions and issues:
- Document, File, and Data Review: The document, file, and data review provided a comprehensive perspective of Program outputs, activities and performance. Relevant documents included departmental performance reports, documentation of Program activities, contribution agreements, and performance data collected through annual and final performance reports. The operational data review included data collected through the performance measurement system (that is, ACCESS database), the Programs Branch database (for example, SPED Service Standards Database) and the financial expenditures database.
Literature Review: A review of relevant literature was conducted to address evaluation questions related to relevance, performance, and design and delivery. Industry priorities and needs were identified by reviewing documents from academic literature and research reports. A gap and needs analysis was conducted to identify sector issues and needs.
Alternative program design and delivery options were identified through an extensive review of similar programs implemented by AAFC and the Government of Canada. The review included an examination of program websites, descriptions, documents and files. It included gathering examples of best practices and lessons learned from program implementation. The review relied upon publications, including AAFC and other federal organizations, and telephone interviews with representatives.
- Interviews: Fifty-five interviews were conducted between June and August 2018 with AAFC staff (11), recipients (12), applicants (12), project stakeholders (12), and external stakeholders and experts (8). An interview guide was developed for each target group and interviewees were provided with interview guides in advance. The interviews focused on relevance, performance, and design and delivery. The evaluation included GBA+ analysis and incorporated interview questions exploring identity dimensions, where relevant.
- Case Studies: Eight case studies of funded projects provided insights into the outcomes and activities implemented through CAAP. Projects were selected based on project type, sector and region. The case studies included interviews with recipients and partners and a content analysis of project documents and materials. The selected case studies are outlined in the following table.
|1||Transforming the Canadian cattle industry through the widespread adoption of genomic technologies – Canadian Beef Breeds Council|
|2||Advancing Pulse Flour Processing and Applications – Canadian International Grains Institute|
|3||Technique de production, diètes et transformation d’insectes comestibles pour l’alimentation humaine et animale – Centre de recherche agroalimentaire de Mirabel|
|4||Analgesic Elastrator Bands Development – Alberta Beef Producers|
|5||Soil Fertility and Cold Climate Root Vegetable Growth Trials – Charter Community of K'asho Got'ine|
|6||Hyper Eye - Innovative hyper-spectral imaging technology to advance the poultry sector – Egg Farmers of Ontario|
|7||Foreign Material Mitigation Measures – Prince Edward Island Potato Board|
|8||Growing Canadian sweet potato industry: from slips to market – Vineland Research and Innovation Centre|
The following methodological limitations were considered in interpreting the data:
|Limitation||Mitigation Strategy||Impact on Evaluation|
|Small sample sizes of interviewees.||Triangulated interview feedback with other lines of evidence.||May not have fully represented stakeholder views.|
|Program cycle is not yet complete.||Used a variety of information sources to assess Program performance.||Do not have full results on Program performance.|
|Too early to assess long-term outcomes of funded projects.||Used interview data and secondary data from a long-term tracking study conducted by AAFC on the results of a previous adaptation program, Advancing Canadian Agricultural Adaptation Program (ACAAF) to anticipate long-term outcomes.||Do not have full results on Program outcomes and impacts.|
Canadian Agricultural Library
The Office of Audit and Evaluation would like to thank the Canadian Agricultural Library for its contribution to the evaluation work. Library staff conducted a literature search providing relevant material to assist the evaluation team in its assessment of the Program’s relevance and performance.
Annex B: Canadian Agricultural Adaptation Program 2014-2019 Logic Model
The CAAP logic model describes the key activities and outputs of the Program and the sequence of short-term, medium-term, and long-term outcomes.
|Program Management of AAFC
|Program Management of AAFC
|Source: Performance Information Profile (PIP) for Canadian Agricultural Adaptation Program, 2017.|
Annex C: Achievement of Outcomes
The Program’s achievement of outcomes indicates that all outcomes are on track or achieved as of the evaluation period.
July 19, 2018
|Collaboration and partnership established/ Information shared with target groups/project stakeholders||Number of partners involved in projects||3 / project
(March 31, 2019)
|3 / project||Achieved|
|Number of information items (brochures, guides, etc.) shared with target groups/project stakeholders||6 / project
(March 31, 2019)
|7.8 / project||Exceeded|
|Program recipients develop tools and strategies to adapt to changing circumstances and/or respond to urgent and/or critical issues||Number of tools and strategies (training materials, methodologies, etc.) are developed to adapt to changing circumstances and/or respond to urgent and/or critical issues||25
(March 31, 2019)
|Program recipients develop innovative, value-added products, processes, or practices||Percentage of projects producing innovative, value-added products, processes or practices are developed||60%
(March 31, 2019)
|Systematic, ongoing collaboration/ information sharing among sector groups||Percentage of projects reporting on-going collaboration beyond project completion||75%
(March 31, 2021)
|Sector implements tools and strategies to adapt to changing circumstances and/or respond to urgent and/or critical issues||Percentage of projects where tools and strategies are implemented to adapt to changing circumstances and/or respond to urgent and/or critical issue||60%
(March 31, 2021)
|Sector adopts innovative, value–added products, processes or practices||Percentage of projects where innovative, value-added products, processes or practices are adopted||30%
(March 31, 2021)
|Not applicable||Not applicable|
|[a] Note: This data was collected through interviews with recipients as performance data was not yet available for this indicator.
Sources: Performance Information Profile (PIP) for Canadian Agricultural Adaptation Program (2014-2019) (internal document), CAAP Annual and Final Performance Data.