Results and Delivery Management Committee,
December 12, 2019
Abbreviations
- AAFC
- Agriculture and Agri-Food Canada
- AGGP
- Agricultural Greenhouse Gases Program
- Gs & Cs
- Grants and Contributions
- GHG
- Greenhouse Gas
- OAE
- Office of Audit and Evaluation
Executive summary
Purpose
The Office of Audit and Evaluation of Agriculture and Agri-Food Canada (AAFC) undertook an evaluation of the Agricultural Greenhouse Gases Program (AGGP) (2016-21) to assess the relevance, performance, and design and delivery of the program. This evaluation fulfils a requirement of the Financial Administration Act and the Treasury Board’s Policy on Results. The results are intended to inform current and future program and policy decisions.
Scope and methodology
The evaluation assessed the program covering the period of April 1, 2016 to July 24, 2019. The evaluation also assessed long-term impacts of the program through examining outcomes from the previous iteration of the AGGP (2010-11 to 2015-16) and from the current program. Multiple lines of evidence were included: a review of previous evaluations; a document, file, and data review; a literature review; a bibliometric review; interviews with AAFC staff, recipients, and project stakeholders; and long-term case studies.
Background
There are two iterations of Agricultural Greenhouse Gases Program (AGGP) funding. The first AGGP began on September 1, 2010 and ended on March 31, 2016. The program was initiated as part of Canada’s contribution to the Global Research Alliance for the mitigation of agricultural greenhouse gases (GHG). The program was subsequently renewed, and the current AGGP began on April 1, 2016, and will end on March 31, 2021. Both the original and the current program support non-AAFC led research projects through contribution agreements. The research projects are intended to create technologies, practices, and processes that can be adopted by farmers to mitigate GHG emissions. These projects are also intended to help farmers increase their understanding of GHG emissions.
Findings
- The AGGP provides needed, and unique, funding for research on agricultural GHG mitigation strategies, technologies, and beneficial management practices.
- The AGGP aligns with AAFC and Government priorities, supporting the mission and results directives of the Department.
- The AGGP aligns with federal roles and responsibilities and responds to the federal government’s commitments to various national and international agreements related to climate change and GHG mitigation.
- While the AGGP claims processes are aligned with the Department’s standard procedures, changes to claims policies and processes since the first iteration of the program, along with communication gaps at the time of the launch of the current program resulted in increased administrative burden and financial impacts for recipients. However, some of these concerns may be related to the inherent incompatibilities with research funding expectations and contribution program requirements.
- The program’s logic model and performance indicators do not accurately reflect all of the activities and impacts of funded projects.
- Missed opportunities to facilitate information sharing with other researchers and to leverage existing knowledge and capacity, diminished program effectiveness.
- The program has signed contribution agreements for twenty research projects, meeting the program target.
- Projects funded by the AGGP are on track to meet the immediate and intermediate outcomes, and have already met the long-term program outcome indicator.
Conclusions
The evaluation found that the AGGP meets an existing need for research on agricultural GHG emissions and farming technologies and practices. The program is aligned with Federal and Departmental roles, responsibilities, and priorities, and is on track to meet its performance outputs and outcomes. However, changes in the design and delivery of claims processes along with gaps in communication, created confusion, and in some cases significantly increased the administrative and financial burden on recipients. This may be due to the inherent incompatibilities with research funding expectations and contribution program requirements. Current program performance indicators do not capture some of the impacts these projects are having in the sector, and the evaluation identified missed opportunities for the program to enable the sharing of knowledge and results of the program.
Recommendations
Recommendation 1
The Assistant Deputy Minister, Programs Branch, should review program design and delivery aspects to ensure administrative efficiency and effectiveness for both AAFC and recipients.
Recommendation 2
The Assistant Deputy Minister, Programs Branch, should review and revise the AGGP logic model and performance measurements to better capture the activities and impacts of funded projects.
Recommendation 3
The Assistant Deputy Minister, Programs Branch, should facilitate opportunities to enable the sharing of knowledge and results of the program.
1.0 Introduction
The Office of Audit and Evaluation of Agriculture and Agri-Food Canada (AAFC) undertook an evaluation of the Agricultural Greenhouse Gases Program (AGGP) (2016-21) as part of the 2018-19 to 2021-22 Integrated Audit and Evaluation Plan. This evaluation fulfils a requirement of the Financial Administration Act and the Treasury Board’s Policy on Results. The results of this evaluation are intended to inform current and future program and policy decisions.
2.0 Scope and methodology
The evaluation assessed the relevance and performance of program activities covering the period of April 1, 2016 to July 24, 2019. The evaluation examined the Agricultural Greenhouse Gases Program (AGGP) using a variety of methods including: a document, file, data, and literature review; bibliometric analysis review; interviews with Agriculture and Agri-Food Canada (AAFC) staff, recipients, and project stakeholders; and long-term case studies. The evaluation assessed long-term impacts through examining outcomes from projects funded through the first iteration of the program, and from the current program.
For the detailed evaluation methodology, see Annex A.
3.0 Program profile
3.1 Context
The Agricultural Greenhouse Gases Program (AGGP) was initiated to represent Canada’s contribution to the Global Research Alliance for the mitigation of agricultural greenhouse gases (GHG). Canada was one of the founding members of the Global Research Alliance, which was formally launched at the United Nations Climate Change Conference, in Copenhagen, in December 2009. The Global Research Alliance is an international network of more than 30 member-countries devoted to collaboration in agricultural research in GHG mitigation, and beneficial management practices. The goal of the Global Research Alliance is to increase international cooperation and investment in research activities to help reduce the emission intensity of agricultural production and increase the potential for soil carbon sequestration or storage (Global Research Alliance Charter, 2011).
Canada’s official GHG inventory (2017) indicates that 8% of Canada's GHG emissions are from crop and livestock production, excluding emissions from the use of fossil fuels or from fertilizer production. The main gases emitted by agricultural activities are:
- Carbon dioxide, which is released during soil cultivation;
- Methane, which is associated with cattle and livestock manure; and
- Nitrous oxide, which comes from using fertilizer and manure.
With beneficial management practices, agriculture can help minimize factors contributing to climate change. Agricultural soils have the potential to store (sequester) carbon and capture carbon dioxide in perennial vegetation, offsetting carbon emissions in other sectors.
The first AGGP was from September 1, 2010 to March 31, 2016. The agricultural sector and the Government of Canada have continued to recognize that there is a need to develop and adopt beneficial management practices to minimize the sector’s environmental impacts. On January 19, 2016, the Minister of Agriculture and Agri-food Canada (AAFC) approved the renewal of the program for five years (April 1, 2016 to March 31, 2021). In the second program iteration, there were no major changes in the program objectives and requirements. Both the first and the current program support non-AAFC led research projects through contribution agreements (federally provided funding which must be spent according to agreed upon conditions). The research projects are intended to create technologies, practices, and processes that can be adopted by farmers to mitigate GHG emissions.
3.2 Program objectives
The objective of the AGGP is to enhance the understanding and accessibility of agricultural technologies and beneficial management practices that can be adopted by farmers to mitigate GHG emissions in Canada.
Research under this program is intended to support an increased understanding of the chemical, physical, and biological processes that lead to GHG emissions from agricultural systems into surface water, ground water, and the atmosphere across four key priority areas: Livestock Systems, Cropping Systems, Agricultural Water Use Efficiency, and Agroforestry. The research projects are intended to examine:
- Livestock Systems to determine activities that reduce GHG emissions which may include manure management, as well as grazing and feeding strategies;
- Cropping Systems for developing knowledge and technology to reduce GHG emissions and maximizing carbon sequestration from forage, pasture and rangeland, as well as wetland and zero tillage management;
- Agricultural Water Use Efficiency to quantify GHG emissions and enhance carbon sequestration through improved irrigation and drainage practice; and
- Agroforestry to understand, develop and promote practices across Canada, focusing on carbon sequestration on agricultural land and the adoption and efficacy of agroforestry practices.
The AGGP research are intended to result in the development of beneficial management practices and diagnostic tools which can help to increase knowledge of agricultural practices that can mitigate and/or reduce the impacts of GHG emissions and nutrient losses.
The program should ultimately contribute to the mitigation of agricultural GHG emissions, which, in turn, can contribute to the Government of Canada’s commitments to the environment and to climate change.
3.3 Program governance and delivery
The AGGP is managed by the Commercialization and Environmental Programs Division in the Programs Branch. The program material was developed in consultation with the Science and Technology Branch, as Science and Technology Branch administered the program prior to April 2012, and have the scientific expertise and capacity to assess program objectives and activities. Representatives from the Science and Technology Branch were involved in the assessment of project applications, and provide a review function on project performance reports. All administrative and financial processes, and communications are administered by the Programs Branch.
The maximum funding available for an individual project is $2-million. Eligible costs related to a project are to be shared between AAFC and the successful applicant (and others where relevant, for example other industry supporters and other governments). The maximum level of total government funding (federal, provincial/territorial and municipal funding) for the AGGP will generally not exceed 85% of the project’s budget.
Eligible activities are:
- research;
- development of knowledge, tools, and information products;
- pilot projects, technology assessment, and testing;
- technical assistance; and
- promotion of GHG mitigation, beneficial management practices, and technologies through extension and awareness activities, enhanced partnerships, networking, and information sharing.
Applicants eligible for this contribution program include:
- not-for-profit organizations, including co-operatives (if applicable);
- academic institutions;
- other levels of government (provincial/territorial/municipal); and
- aboriginal groups.
During the lifetime of a project, recipients are required to submit annual and final performance reports by May 31 of each year. All performance reports are reviewed and approved by the Programs Branch, with the Science and Technology Branch providing scientific informational assistance. Performance results are collected, analyzed and stored in an Access database.
The AGGP Logic Model in Annex B provides further detail on the Program's key activities, outputs, and outcomes.
3.4 Resources
The AGGP is an application-based $27-million, five-year (April 1, 2016 to March 31, 2021) program. Table 1 represents the budget and actuals of this A-base funded program, which is permanent funding provided to the Department at the onset of each budget period, and is distributed to recipients as Vote 10, which is Grants and Contributions (Gs & Cs) funding.
| 2016-17 Fiscal year |
2017-18 Fiscal year |
2018-19 Fiscal year |
2019-20 Fiscal year |
2020-21 Fiscal year |
Total | |
|---|---|---|---|---|---|---|
| Gs & Cs budget ($) | 2,360,500 | 5,835,245 | 7,320,342 | 5,382,000 | 5,382,000 | 26,280,087 |
| Gs & Cs actual ($) | 1,857,891 | 5,771,446 | 7,138,259 | - | - | 14,767,596 |
| Percent of budget spent | 79 | 99 | 98 | - | - | - |
|
Source: Programs Branch, 2019. Note: As recipient claims are submitted and reimbursed each quarter, actuals for the fiscal years 2019-20 and 2020-21 will not be available until the completion of those fiscal years. |
||||||
During the evaluation period, financial information was available up to the end of the fiscal year 2018-19. Table 1 shows that budgeted funds in the first year of the program were under utilized; reflecting a slow start to program initiation. The contribution agreements were signed near the end of the fiscal year, providing only three months for recipients to spend their funding for the first fiscal year. Since no carry-over of funds is permitted, this reduced the time for recipients to conduct activities, and resulted in budget revisions for the following fiscal years.
4.0 Relevance
This section summarizes the evaluation findings related to the relevance of the Agricultural Greenhouse Gases Program (AGGP). It does this by exploring the demonstrable need for the program, its alignment with government priorities, and its consistency with the roles and responsibilities of the federal government.
4.1 Continued need for the program
The AGGP provides needed, and unique, funding for research on agricultural greenhouse gas (GHG) mitigation strategies, technologies, and beneficial management practices.
The evaluation assessed whether the program continues to address a need, and is responsive to the needs of Canadians. The evaluation’s environmental scan found that within Canada, the AGGP is the only program with the purpose of funding research that specifically targets the development of beneficial management practices to address agricultural GHG emissions. Other national research funding programs only have environmentally beneficial practices as a secondary outcome of innovation and production driven projects. There is a continuing need for research on agricultural practices and GHG emissions, since GHGs have been identified, by the Global Research Alliance, as driving global climate change, affecting local weather, and agricultural production.
The 2018 Government of Canada National Inventory Report notes that agricultural GHGs account for 8% of Canada’s total GHG emissions, and are responsible for 28% of national methane emissions and 71% of nitrous oxide emissions. Improvements in management practices of cropland and pastures have the potential to reduce emission intensity, increase productivity and efficiency, and mitigate the impacts of climate change on production.
The International Panel on Climate Change reported in 2014 that climate change has significant impacts on agricultural production, with an increased intensity and frequency of droughts, violent storms, and other extreme weather events posing significant challenges to the sector. The unpredictability of these events emphasizes the need for adaptation and mitigation research and strategies. The evaluation found that the AGGP provides funding targeted specifically at this research.
There is a need within the sector for sustainable agricultural production alternatives to provide farmers with options on how to reduce environmental impacts while still increasing their production. Evaluation evidence determined that the beneficial management practices developed through AGGP research projects have the potential to generate the environmental and economic data farmers can use to assess the trade-offs and costs of implementing beneficial management practices. The evaluation literature review found that many factors influence a producer’s decision to adopt beneficial management practices including: economic and environmental outcomes; existing policy and programs; and the cost of adoption.
The evaluation determined that without the program funding, the agricultural GHG research projects would not have proceeded. If projects had been able to proceed, they would have taken place at a much smaller scale. There are no other national funding sources for agricultural GHG research, and only one provincial funding program (Saskatchewan). While there are other funding programs available for agriculture GHG research, they typically only provide funding for short-term projects, and are not designed specifically for developing beneficial management practices for agriculture in Canada.
The need for funding to conduct this research is demonstrated by the number of academic institutions and not-for-profit organizations that applied under AGGP. This iteration of the program received 45 applications and funded 20 projects, of which 7 were continuations of projects from the original Program. Even though the current program iteration had a longer application intake period (April – July) it received 23 fewer applications than the first iteration of the Program, possibly due to the atypical intake timing, as fall is the typical timing for research applications from other funding bodies.
4.2 Alignment with AAFC and government priorities
The AGGP aligns with Agriculture and Agri-food Canada (AAFC) and Government priorities, supporting the mission and results directives of the Department.
The evaluation found that the AGGP objectives align with departmental priorities. Specifically, the mission of Agriculture and Agri-Food Canada (2019-20) is “To provide leadership in the growth and development of a competitive, innovative, and sustainable Canadian agriculture and agri-food sector.” The evaluation evidence found that the AGGP supports this mission three ways:
- contributes to a sustainable agriculture sector through the development of environmental beneficial management practices which can be adopted by farmers;
- funds research for innovative practices and technologies which reduce GHG emissions; and
- enables Canadian producers to be more competitive in international markets by marketing their products as having environmental attributes or being grown in a sustainable manner (Buller and Morris, 2004).
The outcomes of the program discussed under performance (section 5.0), also support the following departmental results:
- The Canadian agriculture and agri-food sector knowledge base is increased through AAFC supported scientific and innovative research.
- The Canadian agriculture and agri-food sector is effective in transforming ideas into new products processes or practices.
4.3 Alignment with federal roles and responsibilities
The AGGP aligns with federal roles and responsibilities and responds to the federal government’s commitments to various national and international agreements related to climate change and GHG mitigation.
The AGGP was originally developed to contribute to the Government of Canada’s commitment to the Global Research Alliance, an international alliance aimed at increasing collaboration in agricultural research to mitigate agricultural GHGs worldwide. The Government of Canada has continued to commit to reduce GHG emissions through various national and international agreements. In 2015, the Government of Canada signed the Paris Agreement, with the goal to reduce GHG emissions to 30% below 2005 levels by 2030. In the same year, Canada adopted the 2030 Agenda for Sustainable Development at the United Nations General Assembly, which includes a specific goal of “Zero Hunger, Responsible Consumption and Production, and Climate Action”.
There have also been several national level frameworks and commitments to reduce GHG emissions and support sustainable development. The 2016 Pan-Canadian Framework on Clean Growth and Climate Change, and the Federal Sustainable Development Strategy (2019-22) both include agriculture-related actions and goals as a means of addressing climate change and sustainable development in Canada.
The evaluation found that the AGGP supports initiatives that have the potential to lead the way to broader adoption or application of beneficial management practices for Canadian farms. This should help the Canadian agricultural sector reach its goals related to agriculture and climate change in international and national agreements. The program is strategically placed to be responsive to current international concerns, such as the 2019 request of the United Nations’ Intergovernmental Panel on Climate Change for countries to address climate change and land management issues. AGGP funded research projects have the potential to contribute to the mitigation of GHG emissions, and increase the environmental sustainability of the Canadian agriculture sector. The program can also support the Government of Canada’s commitments to the environment, sustainable development, and to climate change.
5.0 Program design and delivery
This section summarizes the evaluation findings related to the design and delivery of the Agricultural Greenhouse Gases Program (AGGP). It includes an examination of the claims processes, operational efficiency, and the selection of program performance measures.
5.1 Context
Similar to the approach taken for the first AGGP, the second AGGP launched with a one-time open call for application proposals from eligible applicants. Applicants were required to submit a full application to be considered for funding, consisting of a completed Agriculture and Agri-food Canada (AAFC) Project Application Form (including a Project Work Plan and Budget), the AGGP supplemental questionnaire, and additional supporting documentation required by the program to fully assess the project proposal.
The AGGP intake period was from early April, 2016 to early July, 2016. Applications were assessed by an AAFC Technical Committee, which included members from the Programs Branch and the Science and Technology Branch, who formed teams to rate the proposals. All proposals were ranked against program criteria and compared to each other to ensure diversity in priority areas; across the Research, Development and Technology Transfer continuum; and geographically. The Technical Committee made recommendations to the AAFC Director General Innovation Committee, which made final recommendations on funding proposals to the Minister. After the selection process, successful applicants’ contribution agreements were signed in January 2017, and the research projects were then able to begin submitting claims for eligible expenses.
5.2 Claims processes
While the AGGP claims processes are aligned with the Department’s standard procedures, changes to claims policies and processes since the first iteration of the Program, along with communication gaps at the time of the launch of the current program resulted in increased administrative burden and financial impacts for recipients.
The AGGP claims process was impacted by two factors; the first being the difference between the AGGP and other Federally funded research programs, the second being the changes in the claims process of the Program. Federally funded grant programs are common funding sources for research activities; however, the AGGP is a transfer payment contribution program. The inherent differences between the two systems presented challenges for AGGP recipients in the understanding of eligible and ineligible expenses. These differences, and recipients limited level of understanding of the differences, impacted their financial planning for their projects, and the administrative time required to reconcile the budgets and eligible expenditures.
The claims process of the AGGP has evolved over time including moving the claims processes from Program Officers in the second year of the program (2017-18) to the Claims and Repayment Division in Innovation Programs Directorate; switching to a “pay now, sample after” approach; and adopting per diem rates for travel to be consistent with other programs and best practices. The changes were designed to reduce the burden of program recipients, and ensure that recipients have sufficient training and support to meet program reporting requirements. Although training sessions were organized, evidence suggests the timing, content and participation at the start of the program, impacted its effectiveness. Interviewed recipients and financial officers at the academic institutions consistently mentioned the helpfulness of AAFC claims officers in navigating the confusing policies around eligible expenditures.
The changes in the claims process were reported by interviewed recipients (six of ten) as creating confusion in reporting claims. Recipients who had participated in the previous program found the changes particularly challenging since they noted rules appeared to have changed over time, when the rules were in fact being more consistently enforced for this iteration. Recipients who had participated in both the original and current AGGP reported a 10 to 20% increase in administrative time for the current Program, above the allowed overhead. Three financial officers from the academic institutions who had worked on the first iteration of AGGP noted changes in the level of detail required, the pre-approval of certain travel and hospitality expenditures, and determining eligibility of expenses with the current iteration, as challenges of the current AGGP.
Three separate documents outlined eligible and ineligible claims; the Applicant Guide, the contribution agreement, and the Recipient Guide. Commonalities were found between the three documents; however, discrepancies were found in the level of detail provided for items, as well as terms not being defined (for example, treatment of standard research expenses such as interpretation of in-kind contributions, maintenance fees on lab testing, and fees from another university department), which contributed to user confusion. There was also no mention of a recipient guide in the contribution agreement, which states “This Agreement and the Program Applicant Guide constitute the entire Agreement between the Parties”.
Four of the six university financial staff interviewed reported challenges in understanding the eligibility requirements of the Program, and cited a lack of clear guidance documents available at the start of the Program. These communication gaps caused issues with budget calculations used in the contribution agreement and resulted in additional time-consuming clarification being sought from program and claims staff. The number of recipients with issues impacted the administrative time spent on AGGP projects by both recipients and AAFC claims staff.
The evaluation found that the Recipient Guide and Annex (February 2017), which included a more detailed list of ineligible expenditures, was shared with recipients after their contribution agreements, including their associated budgets, had been drafted and approved (Figure 1). Recipients believed that their project budgets were approved by AAFC upon signing of the contribution agreements, and subsequently learned that some of their budgetary items were considered ineligible by per the Recipient Guide Annex. Without an in depth understanding of transfer payment policy, recipients were unable to accurately plan their project budget.
Description of above image
| Date | Event |
|---|---|
| June 2016 | Applicant Guide posted online |
| July 2017 | Application intake closed |
| December 2016 | Recipients received acceptance letters |
| January 2017 | Contribution Agreements (including project budget) signed |
| February 2017 | Recipient Guide sent to recipients |
The “sampling based on risk” approach (pay now, sample after) instituted in 2012 reduced the documentation burden of recipients, but increased the risk of confusion around eligible costs. The selective sampling method required program participants to provide receipts for a selected 10% of claims each quarter, post payment . In the remaining 90% of claims not sampled, items that may not have fit the Programs definition of eligibility have already been reimbursed, giving the recipients the impression they were eligible. If those same expenses are subsequently sampled and not approved, claim eligibility appears to be inconsistent, which created additional confusion. In addition, claims review for this iteration of AGGP also sampled recipient funded expenditures, which must also meet program eligibility guidelines.
While recipients experienced challenges with claims eligibility, the Claims and Repayment Division at AAFC has made efforts to improve the understandability of the claims processes and reduce the administrative burden on recipients. In 2018-19 a new Financial Reporting Template was rolled out to simplify the process. Program recipients were provided training on how to complete and submit the template. The new template standardized the transaction listings to ensure only necessary information was reported.
5.3 Operational efficiency
The Programs Branch goal is to achieve service standards a minimum of 80% of the time, under normal circumstances. The evaluation observed that the Branch achieved three out of four of the service standards (Table 2). Only one service standard was not fully met which was to assess applications and send approval or rejection letters within 100 business days after the applicant had received a fully complete application package. This service standard was met for 56% of project applicants, however the remainder of applicants received their letters only one day late.
| Service standards | Fiscal year | Total occurrences | Frequency standard was met (%) | Standard target (80%) achieved |
|---|---|---|---|---|
| Respond to general inquiries (AAFC phone and email) before the end of the next business day | 2016-171 | 84 | 94 | Yes |
| Acknowledgement of receipt of an application (within one business day) | 2016-171 | 45 | 100 | Yes |
| Application assessment and approval or rejection letters (within 100 business days) | 2016-171 | 45 | 56 | No |
| Payment of a claim (within 30 business days of receipt of a duly completed and documented claim) | 2016-17 | 3 | 100 | Yes |
| Payment of a claim (within 30 business days of receipt of a duly completed and documented claim) | 2017-18 | 35 | 86 | Yes |
| Payment of a claim (within 30 business days of receipt of a duly completed and documented claim) | 2018-19 | 33 | 82 | Yes |
|
Source: Programs Branch, 2019 |
||||
It was found that there was a reduced efficiency in claims processing since claims officers were assigned a different program and/or different project recipient each time they reviewed claims samples. This meant that claims officers needed time to familiarize themselves with the different Programs, and projects, each time they received a claim file to review. While this way of processing claims did not to impact the Program’s ability to meet service standards, there is an opportunity for improved program efficiency.
5.4 Performance measurement
The program's logic model and performance indicators do not accurately reflect all of the activities and impacts of funded projects.
The evaluation found that the program logic model could be improved, in particular, the outputs are very similar to the intermediate outcomes. The project activities were not fully captured by the performance indicators, with some indicators being too narrow, or too broad. The projects funded in this program are very diverse; representing various agricultural sectors, four different priority areas, and regions across Canada. As well, the research projects fall at varying points along the Research, Development and Technology Transfer continuum. The pre-defined performance measures hamper the ability to capture and measure the diversity of projects.
| Recipient Activities |
|
|---|---|
| Recipient outputs |
|
| Immediate outcomes |
|
| Intermediate outcomes |
|
| Long-term outcome | 1. Farmers' knowledge in the areas of GHG mitigation, beneficial management practices and technologies is increased |
The evaluation determined that the indicator chosen for the long-term outcome of transferring knowledge to farmers did not sufficiently capture the extent of the Program’s influence. The indicator of number of farmers in attendance at knowledge sharing events does not guarantee knowledge transfer, and does not measure the quality of the interaction. AGGP projects undertake a variety of knowledge sharing activities that vary in their ability to effectively communicate information to farmers; grouping attendance to all of these activities does not capture the extent of influence. As this is the sole indicator for the long-term outcome, the long-term impacts of the program are not measured in the current logic model.
Performance indicators could highlight and recognize some of the unanticipated benefits of the program which includes the training and development of highly qualified personnel; input for policy development; and present opportunities to facilitate agricultural products being marketed as sustainable.
Training and development of highly qualified personnel
The evaluation observed that the research project funding enabled the development and training of highly qualified personnel to grow the community of greenhouse gas (GHG) researchers. AGGP project funding enables the hiring of undergraduate students, graduate students, research associates, and post-doctoral fellows. The funding provides the opportunity to train individuals in agricultural GHG related sciences, while also providing the human resources needed to complete the projects. Case study evidence, examining seven projects, found that at least 146 highly qualified personnel were trained from the original AGGP to the first two years of the second AGGP. Qualitative evidence indicated that the program enabled these individuals to use their knowledge and training to find other employment opportunities in the agri-environmental field, where they were in demand.
Policy development
Research projects were generating and contributing to policy discussions and development at the provincial level. Evaluation evidence reported interactions or involvement with provincial government policy representatives around the development of programs and GHG regulations. These discussions were found to greatly enhance the adoption of initiatives, after the end of the Program. Policy discussions and implementation are overlooked in performance indicators and should be recognized as playing an important role in the path from research to new programming and to the adoption of practices.
Products marketed as sustainable
The evaluation noted that program recipients’ research on beneficial management practices was enabling some farmers to begin producing agricultural goods in a more environmentally sustainable or green manner. Implementing these beneficial management practices contributes to farmers’ ability to market their goods to consumers as environmentally friendly and sustainable, potentially opening up new markets and enhancing sales. While this economic by-product of the research was qualitative and difficult to measure, it still represents a positive outcome of the Program.
5.5 Collaboration opportunities
Missed opportunities to facilitate information sharing with other researchers and to leverage existing knowledge and capacity, diminished program effectiveness.
The evaluation found that the program missed the opportunity to facilitate involvement and information sharing among the AGGP project researchers, and between the AGGP project researchers and AAFC scientists involved with GHG mitigation activities. Collaboration among researchers facilitates the sharing of knowledge, skills, and techniques related to agriculture GHG research and mitigation, which is mutually beneficial to researchers and the Program. Enhanced collaboration improves the chances for success and supports solving of research challenges.
Evaluation evidence from the long-term case studies noted that the first iteration of the AGGP included informal collaboration between AAFC scientific researchers from the Science and Technology Branch and AGGP funded researchers; this collaboration was absent in the second iteration. The evaluation observed that while the program complements some initiatives occurring at AAFC, such as the Living Labs initiative, scientific researchers at AAFC were uncertain how to obtain the results of AGGP projects, since there was no information or database that they could access. This may prevent the inclusion of developed information and techniques into current AAFC research projects and represents a missed opportunity to leverage existing capacity and knowledge, and to improve the impact of outcomes.
Increased collaboration between the program and researchers may also support the selection and development of performance measures that capture the timing and activities of research projects. Performance measures typically represent events that occur in sequence, from program outputs to immediate outcomes, to intermediate outcomes, and finally to long-term outcomes. The achievement of program long-term outcomes prior to program Outputs, immediate outcomes, or intermediate outcomes suggests that the long-term outcomes of the program may not be fully captured by that measure.
Formal collaboration among AGGP projects is not supported by the Program. In the first iteration of the AGGP, a workshop, facilitated by AAFC, and hosted by a university, brought together project recipients providing an opportunity for sharing best practices and knowledge transfer between projects. Facilitation of collaboration between AGGP projects would enable recipients to share lessons learned, and best practices, for facing program and research challenges.
As one of the founding members of the Global Research Alliance, Canada provided the AGGP as one of the first domestic responses to agricultural GHGs. The evaluation observed that there were very limited connections between the Global Research Alliance and the AGGP, with only five out of twenty projects reporting having Global Research Alliance related activities. As a result, the AGGP, and Canada, are missing the chance to showcase its research and accomplishments, and project researchers are not provided an international forum to gain and advance knowledge.
6.0 Performance
This section summarizes the evaluation findings related to the achievement of the Agricultural Greenhouse Gases Program's (AGGP) expected outputs and outcomes, and the impacts of the Program.
6.1 Outputs
The program has signed contribution agreements for twenty research projects, meeting the program target.
The evaluation found that the program has successfully funded its target number of research projects, which cover the four priority areas, with some projects representing multiple priority areas. Table 3 summarizes the number of research projects that address greenhouse gas (GHG) emissions within the priority areas. The number of research projects funded is a function of the quality and scope of research proposals received by the Program.
| Priority area | Number of research projects |
|---|---|
| Single priority area projects - livestock systems | 2 |
| Single priority area projects - cropping systems | 6 |
| Single priority area projects - agricultural water use efficiency | 3 |
| Single priority area projects - agroforestry | 5 |
| Multi-priority area projects - livestock systems and cropping systems | 2 |
| Multi-priority area projects - agricultural water use efficiency and cropping systems | 1 |
| Multi-priority area projects - all priority areas | 1 |
| Total | 20 |
| Source: Programs Branch, 2019 | |
Recipients included 17 academic institutions, and 3 not-for-profit organizations. The recipient research projects were found to be broadly distributed across the country (Table 4). Research projects were concentrated in provinces with the highest agricultural GHG emissions and the highest GHG emission intensity in Canada (Alberta, Ontario, Quebec, and Saskatchewan) (Statistics Canada, 2017). The distribution of AGGP projects succeeded in covering the agricultural areas with greatest need for research on GHG emissions.
| Province | Number of research projects |
|---|---|
| British Columbia | 2 |
| Alberta | 4 |
| Saskatchewan | 3 |
| Manitoba | 1 |
| Ontario | 4 |
| Quebec | 3 |
| New Brunswick | 1 |
| Nova Scotia | 1 |
| Prince Edward Island | 1 |
| Total | 20 |
| Source: Programs Branch, 2019 | |
The evaluation found that the projects were beginning to achieve their outputs of developing collaborative agreements. program recipients were found to have developed nine collaborative agreements and partnerships for the development, and sharing, of applied research on GHG mitigation. Many are still underway, with 18 agreements and partnerships anticipated to be developed by the end of the projects. Recipients noted that there were some initial challenges in developing collaboration agreements, such as finding suitable research personnel and research sites, as well as unforeseen technical issues (delays from suppliers and adverse weather conditions). There was no program performance target for achieving this measure, but as the program is in its second iteration, there should be a target set for this indicator.
6.2 Impact
Projects funded by the AGGP are on track to meet the immediate and intermediate outcomes, and have already met the long-term program outcome indicator.
During the evaluation period, the availability of performance information was limited to one fiscal year and a limited percentage of reports from the second year. As these are five-year research projects, the achievements in the first two years are expected to be lower as many of the performance measures are only achieved at the end of a research project. Based on the accomplishments on projects to date, the trajectory suggests that all program targets will be met by the end of the program cycle. However, the targets that are being met, or close to being met, are the ones that would be expected to occur at the end of the research projects, suggesting the targets and their indicators should be refined to better reflect the timing and achievements of research projects.
The evaluation found that the program is on track to achieve its immediate outcomes of developing, verifying, and validating new GHG mitigation technologies and beneficial management practices for the sector, and making recommendations for future areas of research (Table 5). At the time of the evaluation, 10 beneficial management practices had been developed of the 65 that are planned. The indicator for recommendations for future research is intended to only capture recommendations if the proposed beneficial management practice is unable to be validated, in which case the researcher would make a recommendation for future studies in that area. Therefore, the program did not set a target for recommendations, and would only expect to receive this information in the final year of the Program.
| Target (number) | Achieved as of July 2019 (number) |
Percent of target | |
|---|---|---|---|
| New technologies and practices developed | 60 | 10 | 17 |
| Information products developed | 150 | 37 | 25 |
| Newly developed technologies and practices demonstrated | 50 | 20 | 40 |
| Workshops and field days organized | 80 | 28 | 35 |
| Peer reviewed publications | 140 | 42 | 30 |
| Professional presentations | 140 | 136 | 97 |
| Farmers that participated in workshops or field days | 3,500 | 4,068 | 116 |
Intermediate outcomes include making GHG mitigation technologies and beneficial management practices available to farmers through knowledge transfer, and knowledge sharing in academic settings. At the time of the evaluation, recipients had developed information products to demonstrate new technologies and beneficial management practices to farmers, in the form of fact sheets, brochures, and videos, and had organized workshops and field days. Recipients had also published peer-reviewed publications and conducted presentations at professional forums.
The evaluation observed that the AGGP is providing an effective opportunity for farmers to receive information on new technologies and beneficial management practices. The majority of AGGP research projects directly involve farmers in the research, by having research plots on their land, providing farmers with an opportunity to create their own opinions on the benefits of implementing these new practices and processes. The Farm Financial Survey (2017) found that the three most important activities for a farmer to consider trying something new were seeking information from independent consultants, relying on their own experience, and participating in producer associations. Even though the farmers did not rank attending workshops or field days as important activities when deciding to try something new with their operations, these activities provide researchers an opportunity to connect with farmers and producer associations.
The evaluation found that AGGP projects are succeeding in achieving the chosen long-term program outcome of increasing farmer knowledge in the areas of GHG mitigation, beneficial management practices, and technologies. The information collected through AGGP research projects provides farmers with the necessary knowledge about the benefits, costs and the trade-offs of adopting beneficial management practices. The factors affecting farmers’ decisions to try something new are: improved efficiency of their operations safety; time to implement the change; and increase in value of product (Farm Financial Survey, 2017).
Performance information indicated that farmers who participated in field days or workshops that demonstrated GHG mitigation technologies and beneficial management practices had already surpassed the program target of 3,500. Nonetheless, there was no direct evaluation evidence that all of the farmers who were exposed to AGGP project information were understanding or responding to the information they were receiving. This indicator does not capture the quality of the knowledge transfer, or usefulness of the knowledge to the attending farmer. The ability of recipients to meet the long-term outcome indicator so early suggests that the target was too low, or that a different measure may be needed.
Through the examination of long-term case studies, the evaluation found that recipients who participated in both the first and second iterations of the AGGP were beginning to be successful in reaching out to farmers to enhance their understanding of the projects. There was limited evidence of adoption of beneficial management practices from AGGP among farms. Other long-term case study recipients mentioned anecdotal cases of farmers who had become early adopters of these beneficial management practices, or who had expressed interest in learning how to adopt the developed practices, as well as interest in the research from industry associations. The interest and involvement of producer and industry groups in these projects may encourage adoption of the developed beneficial management practices.
The adoption of the beneficial management practices developed by AGGP projects may take a significant time, ranging from six to twenty-two years after being introduced (Kuehne et al., Predicting farmer uptake of new agricultural practices: A tool for research, extension and policy, 2017). As a result, the adoption of the beneficial management practices developed through the AGGP by farmers is expected to occur well after the projects conclusion.
See Annex C for a detailed performance information profile.
7.0 GBA+
As per the Treasury Board Directive on Results, the evaluation examined the extent that Gender-Based Analysis Plus was considered. Gender-Based Analysis Plus is an analytical tool, process, or product used to assess the potential impacts of policies, programs, services, and other initiatives on diverse groups taking into account identity factors including sex, age, sexual orientation, disability, race, ethnicity, religion, education, language, geography, culture, and income. The evaluation noted that diversity and inclusion factors have not yet been formally implemented as part of Agricultural Greenhouse Gases Program (AGGP).
8.0 Conclusions and recommendations
Relevance
The evaluation findings confirm that there is a continued need for the federal government to support the Agricultural Greenhouse Gases Program (AGGP) research projects to develop beneficial management practices to reduce Greenhouse Gas (GHG) emissions in agriculture. The results of these projects can provide Canadian farmers with the information they need to make knowledgeable decisions about adoption of beneficial management practices.
The objectives of the program are consistent with government priorities and Agriculture and Agri-food Canada (AAFC) departmental strategic outcomes, and align with federal government roles and responsibilities. The research conducted by the AGGP recipients supports the mission and results directives of the Department. The research responds to the federal government’s commitments to various national and international agreements related to climate change and GHG mitigation.
Design and delivery
Evaluation evidence found that the changes in policies and processes since the first iteration of AGGP and within the current iteration, coupled with the gaps in communication, contributed to confusion and challenges for program recipients. The evaluation found that administrative time for principal investigators to navigate challenges in claims resulted in fewer hours spent on research related project activities. These budgetary and expenditure issues negatively affected recipients’ interest in participating in this kind of program again. These challenges coupled with the incompatibility between university funding systems has been highlighted by the claims administrative challenges of both recipients and AAFC staff.
The harmonization of program claims with departmental standard practices has increased internal economy and efficiency. However, the evaluation found that multiple recipients (60% of respondents) encountered challenges with the new claims system. The changes in policies, lack of communication, and discrepancies in guiding documents contributed to the financial burden and confusion of recipients. This resulted in claims being denied, inconsistencies in claims eligibility, and increased financial and administrative burden to recipients. These issues, along with a lack of flexibility, compared to other science funding programs, resulted in a loss of researcher time spent on science activities. Typically, research funding is secured through grants programs.
The evaluation identified outcomes that are being achieved that are not captured by performance measures, and collaboration opportunities for the program to leverage knowledge and capacity. The program has an opportunity to enhance performance information gathering to better capture some of the unintended outcomes and benefits of the program and to facilitate greater information sharing with other researchers, leveraging existing knowledge, skills, and techniques. Facilitation of collaboration among recipients could support sharing of best practices and methods for overcoming challenges. Increased external collaboration could also assist in the selection of performance outcomes, indicators, and targets, to better reflect achievements and timing of research. Some mechanisms could include information sharing, such as holding an AGGP recipient/producer organization conference; providing funding for AAFC scientist participation in projects; and updating the AAFC website to show AGGP project outcomes.
Performance
While the 20 funded AGGP projects are only in the second year of their five year funding, they have already met, or are on track to meet, their intended outputs and outcomes, suggesting these are not the appropriate measures and/or targets. Over 4,000 farmers have participated in field days and workshops hosted by the program recipients, surpassing the long-term indicator target of 3,500. This long-term indicator has already been met, prior to many of the immediate and intermediate indicators. Suggesting that the long-term indicator is not an appropriate target, or measure, for the long-term program outcome.
While the long-term outcome was reached, the quality of the interactions and the increase in farmers understanding of the beneficial management practice remains largely unknown. Case studies provided some evidence that farmers were increasing their knowledge in the areas of GHG mitigation, beneficial management practices and technologies by participating in AGGP field days or workshops.
Recommendations
Recommendation 1
The Assistant Deputy Minister, Programs Branch, should review program design and delivery aspects to ensure administrative efficiency and effectiveness for both AAFC and recipients.
Recommendation 2
The Assistant Deputy Minister, Programs Branch, should review and revise the AGGP logic model and performance measurements to better capture the activities and impacts of funded projects.
Recommendation 3
The Assistant Deputy Minister, Programs Branch, should facilitate opportunities to enable the sharing of knowledge and results of the Program.
9.0 Management Response and Action Plan
| Recommendation | Management Response and Action Plan (MRAP) | Target Date1 | Responsible Leads |
|---|---|---|---|
| 1. The Assistant Deputy Minister, Programs Branch, should review Program design and delivery aspects to ensure administrative efficiency and effectiveness for both Agriculture and Agri-food Canada (AAFC) and recipients. | Agreed. Innovation Programs Directorate staff will examine and review the recipient claims issues identified through the evaluation and determine whether program design or delivery revisions could reduce similar issues going forward. In addition, they will examine and document how changes made to the claims process towards the end of the evaluation period have already contributed to consistencies in the process and improved compliance with the Transfer Payments Policy. | June 2020 | Director General, Innovation Programs Directorate |
| Additionally, should a similar program be launched following Agricultureal Greenhouse Gases Program (AGGP) 2, Innovation Programs Directorate will determine whether the current program recipient type is appropriate given the incompatibilities between research funding expectations and government grant and contribution program requirements. During the program development stage, consideration will be given to changing the eligibility requirements. | April 2021 | Director General, Innovation Programs Directorate | |
| Prior to the launch of a new program in 2021, program staff will ensure that the applicant guide and recipient guide are consistent and available prior to the signing of contribution agreements. | April 2021 | Director General, Innovation Programs Directorate | |
| 2. The Assistant Deputy Minister, Programs Branch, should review and revise the AGGP logic model and performance measurements to better capture the activities and impacts of funded projects. | Agreed. Innovation Program’s Directorate will work collaboratively with representatives from Corporate Management Branch and the Office of Audit and Evaluation to review the logic model and determine how to better capture the activities and impacts of funded projects, ensuring that the level of resources required to collect data to report against the selected indicators is commensurate with project spending levels. | April 2021 | Director General, Innovation Programs Directorate |
| Several changes have already been made to the final performance report template for AGGP, in order to capture the impacts of this iteration of the program’s projects. These include the addition of questions to capture policy discussion impacts and adoption of beneficial management practices and technologies. The program’s Performance Information Profile will be revised to capture these changes. | April 2021 | Director General, Innovation Programs Directorate | |
| Innovation Programs Directorate is also committed to continue to explore the challenge of measuring impacts of research and development programs, as recognized across federal science departments, and determining best practices in impact measurement for AGGP and the rest of its suite of programs. | April 2021 | Director General, Innovation Programs Directorate | |
| 3. The Assistant Deputy Minister, Programs Branch, should facilitate opportunities to enable the sharing of knowledge and results of the Program. | Agreed. Innovation Programs Directorate will consult with recipients and within the Department, including Public Affairs Branch, to determine the most appropriate means of sharing the results of AGGP projects. | April 2021 | Director General, Innovation Programs Directorate |
| The Directorate is committed to compiling a list of beneficial management practices and technologies developed under AGGP for communication internally and externally using available platforms. | April 2021 | Director General, Innovation Programs Directorate | |
| At the same time, the Directorate recognizes that one of the greatest challenges of this program is in ensuring that knowledge of the beneficial management practices developed under AGGP is disseminated to the actual producers who would be likely to adopt these practices. The Directorate will continue to examine ways to better communicate the results of AGGP to their potential adopters. | April 2021 | Director General, Innovation Programs Directorate | |
| 1. When long-term target dates are required for action items, the management response should explain why the target date is in the long-term. | |||
Annex A: Evaluation methodology
Literature and document review
To assess relevance a review of policy documents and literature on agricultural greenhouse gas (GHG) emissions and beneficial management practices were completed. Provincial and federal programs providing funding for agricultural initiatives that support climate change mitigation, and the development or adoption of beneficial management practices were examined. Private sector surveys and governmental surveys were reviewed to determine indicators which may track adoption of beneficial management practices.
Review of program files and databases
To assess program impacts and outputs a review of program files, specifically performance reports from recipients, were completed. Program data on recipients and applicants were also used in the evaluation.
Key informant interviews
To support an assessment of program relevance, efficiency and effectiveness, interviews were conducted to assess the continued need for the Program, its progress towards expected outcomes, internal and external factors that have facilitated or hindered progress, and program efficiencies. The key informant interviews were used to support evidence obtained through other lines of evidence.
Interviews were conducted with 15 members of Agriculture and Agri-food Canada (AAFC) staff (program officials, senior management, and Science and Technology Branch representatives) and 16 project recipients; including researchers and financial officials.
Case studies
To assess performance and effectiveness seven case studies were conducted on previous and current Agriculture Greenhouse Gases Program (AGGP) projects. Four case studies assessed the achievement of long-term outcomes through projects funded under the first AGGP that received continued funding under the second AGGP. Three additional case studies were selected from current AGGP projects. The case studies consisted of an in-depth analysis of program files, interviews with project recipients and stakeholders, and a bibliometric analysis (a quantitative analysis of academic literature based on metrics, such as citation rates) of professional journals and citations. Network mapping was used to demonstrate connections between the researchers, government, and other stakeholders in the agri-environmental research community.
Methodological limitations
The following methodological limitations were considered in interpreting the data:
| Limitation | Mitigation strategy | Impact on evaluation |
|---|---|---|
| Long-term nature of program It can take many years before the projects funded under the AGGP generate the expected intermediate and long-term outcomes, which could limit the evaluation’s capacity to assess the program impacts. | Six case studies were conducted of projects under AGGP1, and continued under AGGP2. | Some of the longitudinal impacts of AGGP were assessed. |
| Changes in performance indicators between program iterations. Changes in indicators may limit the evaluation’s ability to assess improvements achieved in the second iteration of AGGP. | Interviews were conducted with program staff and program recipients. | Anecdotal evidence was collected of efficacies and efficiencies achieved, and some of the impacts of program changes were assessed. |
Canadian Agricultural Library
The Office of Audit and Evaluation would like to thank the Canadian Agricultural Library for its contribution to the evaluation work. Library staff conducted a literature search providing relevant material to assist the evaluation team in its assessment of the Program’s relevance and performance.
Annex B: Logic model
The Agricultural Greenhouse Gases Program (AGGP) logic model describes the key activities and outputs of the program and the sequence of short-term, medium-term, and long-term outcomes.
| Long-term Outcome | 1. Farmers’ knowledge in the areas of greenhouse gas (GHG) mitigation, beneficial management practices and technologies is increased |
|---|---|
| Intermediate Outcomes |
|
| Immediate Outcomes |
|
| Outputs | Recipient
|
| Program Management of Agriculture and Agri-food Canada (AAFC) 1. Approved and fulfilled contribution agreements |
|
| Activities | Recipient
|
Program Management of AAFC
|
|
| Inputs | 1. $26,910,000 over five years (April 1, 2016 to March 31, 2021) |
| Source: AGGP Administrative Program Data, Programs Branch, 2019 | |
Annex C: Performance in information profile
| Output/outcome | Indicator | Program target | Planned | 2017-18 achieved | 2018-19 achieved | Total achieved | Percent achieved |
|---|---|---|---|---|---|---|---|
| Collaborative agreements and partnerships for developing and sharing of applied research on greenhouse gas (GHG) mitigation | MOUs and/or collaborative agreements signed with agriculture industry and provincial governments and agencies | No target | 18 | 9 | 4 | 13 | - |
| Approval and fulfillment of contribution agreements | Projects approved by program | 20 | 20 | 20 | - | 20 | 100 |
| Output/outcome | Indicator | Program target | Planned | 2017-18 achieved | 2018-19 achieved | Total achieved | Percent achieved |
|---|---|---|---|---|---|---|---|
| New GHG mitigation technologies and beneficial management practices for the sector are developed, verified and validated by program recipients | New technologies and beneficial management practices available for adoption are developed, verified and validated | 60 | 65 | 9 | 1 | 10 | 17 |
| Areas for further research with respect to GHG mitigation technologies and beneficial management practices are identified | Recommendations made for future research | No target | 4 | 0 | 0 | 0 | - |
| Output/outcome | Indicator | Program target | Planned | 2017-18 achieved | 2018-19 achieved | Total achieved | Percent achieved |
|---|---|---|---|---|---|---|---|
| GHG mitigation technologies and beneficial management practices are available to farmers (knowledge transfer) | Information products such as factsheet and brochures are developed | 150 | 154 | 25 | 12 | 37 | 25 |
| GHG mitigation technologies and beneficial management practices are available to farmers (knowledge transfer) | Newly developed technologies and beneficial management practices demonstrated through field days and workshops | 50 | 67 | 20 | 0 | 20 | 40 |
| GHG mitigation technologies and beneficial management practices are available to farmers (knowledge transfer) | Workshops and field days organized by project to demonstrate new technologies and beneficial management practices | 80 | 76 | 21 | 7 | 28 | 35 |
| Knowledge is shared in academic settings | Peer reviewed publications published | 140 | 139 | 31 | 11 | 42 | 30 |
| Knowledge is shared in academic settings | Presentations at professional forums on GHG delivered | 140 | 141 | 86 | 50 | 136 | 97 |
| Output/outcome | Indicator | Program target | Planned | 2017-18 achieved | 2018-19 achieved | Total achieved | Percent achieved |
|---|---|---|---|---|---|---|---|
| Farmers' knowledge in the areas of GHG mitigation, beneficial management practices and technologies is increased | Farmers who participated in field days or workshops that demonstrate GHG mitigation beneficial management practices and technologies | 3,500 | 5,058 | 3,558 | 510 | 4,068 | 116 |
| Source: Programs Branch, July 2019. | |||||||