Evaluation of the AgriInnovation Program – Stream A: Research Accelerating Innovation

March 30, 2017

List of abbreviations

AAFC

Agriculture and Agri-Food Canada

AIP

AgriInnovation Program

CLF

Crops Livestock and Food

GF2

Growing Forward 2

GHG

Greenhouse Gas

IT

Information Technology

PMS

Performance Measurement Strategy

R&D

Research and Development

RDT

Research, Development, and Transfer

SMS

Science Management Solution

SSTA

Sustainable Science and Technology Advancement

STB

Science and Technology Branch

Executive summary

Background

The AIP is a five-year program (2013-14 to 2017-18) designed to accelerate the pace of innovation by supporting research and development activities in agri-innovation and facilitating the demonstration, commercialization and/or adoption of innovative products, technologies, processes, practices and services. The program funds activities under three streams: Stream A: Research Accelerating Innovation; Stream B: Research, Development, and Knowledge Transfer; and Stream C: Enabling Commercialization and Adoption.

This evaluation focuses on Stream A under which AAFC conducts research where industry does not have a natural lead but where there is both a strong sector link and a broader Government of Canada role. Stream A is composed of two components:

• Crops Livestock and Food (CLF) includes far-from adoption research activities related to: threats to Canadian crops from invasive plants, insects, and crop pathogens; the nutritive components of agricultural products and their role in disease prevention; how contaminants enter into agricultural products and associated mitigation measures.
• Sustainable Science and Technology Advancement (SSTA) includes research related to: the necessary scientific knowledge and tools to: improving the sustainable and productive use resources; increasing sector productivity; improving environmental performance; and assisting the sector in accessing new markets and in meeting sustainability requirements.

Stream A is managed by the Science and Technology Branch (STB) and funding for research projects is distributed through an annual project application process. Only internal AAFC scientists are eligible to receive project funding under Stream A, but other organizations such as universities, and individuals may participate in activities as collaborators. A total of $150 million was budgeted for Stream A activities from 2013-14 to 2017-18.

Key findings and conclusions

The key findings and conclusions resulting from the evaluation are as follows:

Relevance

The evaluation found that there is a continued need for Stream A scientific research that helps the agriculture and agri-food sector to optimize productivity, identify and mitigate various production risks, capture market opportunities and keep pace with sustainability considerations. Scientific research is a key driver of product and process innovation required to address the competitiveness and profitability needs of the agriculture and agri-food sector. Studies suggest that long-term research has the potential to yield some of the highest returns available from public investments and producer groups recognize that investment in research and development is key to the success of the sector.

Research is critical to understanding the causal mechanisms underlying threats to Canada's food production and distribution systems, permitting the development of strategies and practices to detect, monitor and control hazards through the food chain as well as understanding the nutritive and health attributes of agricultural products. It can take many years (10 years or more) to develop new innovations, such as disease-resistant crop varieties or management practices that optimize soil productivity and minimize negative impacts on the environment. The results of public good and foundational research provide a critical base for more sustainable agricultural practices and processes, as well as more targeted adoption and commercialization projects.

The objectives of Stream A are well aligned with the priorities of the federal government and the strategic outcomes of AAFC, particularly with respect to supporting agricultural science and innovation and recognizing the importance of the environment in supporting a strong Canadian economy. Stream A activities are particularly aligned with AAFC's Strategic Outcome 2: An innovative and sustainable agriculture, agri-food and agri-based products sector. The 2015 Mandate Letter to the Minister of Agriculture and Agri-Food highlighted the Government's priority to "invest in agricultural research to support discovery science and innovation in the sector."^{Footnote 1} Budget 2016 also emphasized the importance of investing in public agricultural science and research, as well as modernizing agricultural and environmental science infrastructure.^{Footnote 2}

While there are many stakeholders involved in agricultural research activities, AAFC has an important role in delivering the type of research and development funded through Stream A because it has a public good focus (for example, improving sustainability of agricultural practices) or is too far from commercialization for private sector to realize a return on investment in the near term, yet it provides a critical base for more targeted adoption and commercialization projects. Other research organizations, such as some university scientists tend to focus on theoretical discoveries without a clear application in the markets. Private sector organizations are inclined to focus on applied research that can generate faster returns.

Though provincial government research programs often have a similar focus to AAFC research, they have varying capacity for agricultural research as some provinces devote a significant proportion of their resources to extension services, knowledge transfer activities and commercialization. On the whole, provinces have less capacity than the federal government to fund agricultural research. The federal government share of total government agricultural research expenditures in Canada has been historically larger than that of provincial governments, averaging 69.4% and 30.6%, respectively, between 1985-86 and 2015-16. Furthermore, AAFC has the unique advantage of being able to conduct research in different ecological zones due to its national network of research centres.

Stream A complements other agricultural research and innovation activities conducted or funded by AAFC and other government and non-government, organizations. Policies guiding Stream A project selection are intended to prevent overlap or duplicate other sources of research and innovation support. The program complements other activities by advancing scientific findings to a point where the risks for industry are lower, which provides them with the incentive to fund more applied projects. Stream A also facilitates collaboration and leveraging of expertise, facilities, and resources with other federal, provincial, industry, and university research partners.

Performance (effectiveness)

The evaluation examined projects underway as of March 31, 2016. The evaluation found that, three years into the five-year program, Stream A has made considerable progress towards the achievement of its intended outputs and outcomes. As of March 31, 2016, Stream A has committed $59.3 million in funding to 159 research and innovation projects. Most of the first round projects are complete (82%), with the majority starting in 2013-14 and ending in 2015-16. These research projects span a wide variety of sector portfolios, with the largest proportion of committed to the Crops Livestock and Food^{Footnote 3} sub-program and the Agro-Environment Resiliency^{Footnote 4} sector portfolio.

Stream A has made considerable progress with respect to knowledge generation and the dissemination of research results. As of March 31, 2016, the program has already exceeded its Performance Measurement Strategy (PMS) targets as follows:

• 278 peer reviewed publications developed (PMS target of 143 by March 31, 2018) (see Table 4); and
• 419 presentations at conferences events and annual meetings (PMS target of 150 by March 31, 2018) (see Table 9).

Progress has also been made in developing new products, processes, and technologies to enhance agricultural productivity and market potential as well as innovations related to environmental sustainability. The results achieved as of March 31, 2016 in relation to the PMS targets to be achieved by March 31, 2018 are as follows:

• 37 new products, processes, and technologies relevant to the minimization of catastrophic threats to crops, optimization of livestock efficiencies, and food health and safety (PMS target of 51 by March 31, 2018) (see Table 5); and
• 15 innovations related to environmental sustainability such as new sustainability metrics tools, new or adapted beneficial management practices, and new or improved decision-support tools (PMS target of 65 by March 31, 2018) (see Table 7).

The evaluation identified several examples of significant innovations and new products, processes, and technologies resulting from Stream A projects. One example of these innovations is a national project resulted in the development of a new wheat variety that is higher yielding and resistant to Ug99 (an emerging fungus with devastating consequences) and other diseases. The project could result in economic benefits to the sector of up to $700 million annually from reduced losses and reduced fungicide use, if Ug99 were to arrive in Canada.

Several projects have also resulted in additional opportunities for further collaboration and research. According to project reports, 30 projects resulted in 71 follow-on research grants, awards, honours or prizes at the time of the evaluation. Most of the projects reviewed as part of case studies resulted in (or will likely result in) follow-on projects or investments such as investments by AAFC and key industry stakeholders, and new areas of investigation or collaboration. For example, a project to develop strategies to reduce nitrogen losses has generated significant interest and led to the development of an AIP Stream B agri-science project with funding from Fertilizer Canada.

Performance (efficiency and economy)

The evaluation found that many aspects of the Stream A design have facilitated effective delivery, while a few factors have constrained delivery. The major factors contributing to the effective delivery of Stream A projects include:

• Stream A builds on other Science and Technology Branch investments, thereby taking advantage of complementary science activities and existing infrastructure, such as highly qualified scientists and technicians, research centres across multiple eco-zones, and leading edge scientific technology and equipment. For example, in many cases the salaries of scientists working on AIP Stream A projects are paid under the AAFC Science sub-program. Furthermore, it builds AIP Stream B by focussing on science requirements that are necessary to support industry-led research and development projects, but that are not close enough to commercialization to warrant industry investment.
• The project selection process promotes science excellence, accountability and directs resources to areas where there is the greatest need and projects that are most likely to have significant impacts.
• Collaborative research teams with multiple areas of expertise have leveraged internal and external scientific expertise and capacity, and facilitated knowledge dissemination and transfer to the scientific community. On average, Stream A projects involved nine AAFC scientist participants and three AAFC technical staff. Over one third of Stream A projects involved external collaborators, most frequently from Canadian universities. A 2016 study of scientific output and collaboration patterns of 13 federal government entities showed that AAFC scientists frequently engage in collaboration and that the department more than doubled its international collaboration rate from 2000 to 2014^{Footnote 5}.
• The program supports activities, such as regular outreach with industry stakeholders, hosting regional meetings with industry and research stakeholders, and distributing publications that highlight major research achievements, that have facilitated the transfer and adoption of innovations.

The evaluation found that the design and delivery of Stream A projects is sometimes constrained by the broader government context in which it is required to operate. These constraints include:

1. The current three year timeframe for funding projects. A longer funding timeframe for projects (for example 5 years) would allow more time to focus on longer-term research goals, demonstrate program results, and assess the incremental value of continuing the activities in subsequent rounds of funding.
2. Data storage and computation capacity constraints. The evaluation identified some constraints with respect to available data storage and computation capacity. As well as challenges in procurement timelines associated with acquisition processes. This is particularly affecting areas of research related to genetics which is producing more and more large and complex datasets.

The evaluation found that STB has developed performance measurement and data collection mechanisms such as the Science Management Solutions (SMS) to track the implementation and outcomes of Stream A projects. These tools have improved over time and are useful for communicating project results. Despite the progress to date, some issues were identified with the performance measurement data such as limited and missing information on some outputs and outcomes, duplicate entries across multiple projects, and different interpretations of output and outcome categories. Furthermore, it is difficult to measure the intermediate and end outcomes based on the existing performance indicators.

Recommendation and management response and action plan

The key issue identified and recommendation resulting from the evaluation is:

Issue #1:

The evaluation found that performance measurement requires more details about the results of projects and the importance of the impacts in terms of the discoveries, innovations and developments. Training guidelines and quality assurance protocols are not sufficient to assist scientists in reporting project impacts.

Recommendation #1:

In line with the new Policy on Results and the Government of Canada results and delivery agenda, AAFC should continue to refine the program's approaches to performance measurement to ensure that the indicators used and the performance data collected can accurately and consistently report on the impacts of scientific research activities.

Management response and action plan

Agreed

1. STB is revising all its sector strategies to include logic models to better align with the results and delivery agenda.

   Target Date: June 30, 2017

   Responsible Position: Director General, Partnerships and Planning Directorate, Science and Technology Branch.

2. STB will create Program Information Profiles (PIPS) for all programming as per the new Results and Delivery Agenda for the Government of Canada.

   Target Date: November 1, 2017

   Responsible Position: Director General, Partnerships and Planning Directorate, Science and Technology Branch.

1.0 Introduction

1.1 Purpose of the report

This report represents the findings of the evaluation of the Agri-Innovation Program (AIP) Stream A: Research Accelerating Innovation. Stream A provides non-repayable funding to support Agriculture and Agri-Food Canada (AAFC)-led cross-cutting, long-term research (about 10 years from commercialization) where industry does not have a natural lead, but where there is both a strong sector link and a broader Government of Canada role.

The evaluation was conducted as part of AAFC's five-year Departmental Evaluation Plan (2014-15 to 2018-19). The evaluation was undertaken by the department's Office of Audit and Evaluation.

2.0 Program profile

2.1 Program context

In April 2013, the Government of Canada launched the Growing Forward 2 (GF2) policy framework for Canada's agriculture and agri-food sector. GF2 provides the foundation for government agricultural programs and services over the five year period of 2013 to 2018. By focusing on innovation, competitiveness and market development, GF2 provides Canadian producers and processors with tools and resources that help them to innovate and capitalize on emerging market opportunities.

Within GF2, the AIP is a five-year program (2013-14 to 2017-18) with a budget of up to $698 million, of which $468 million is available for funding projects based on applications from industry. The remaining funds go towards AAFC-led research, development and knowledge transfer activities that complement industry-directed initiatives. The program is designed to accelerate the pace of innovation by supporting research and development activities in agri-innovation and facilitating the demonstration, commercialization and/or adoption of innovative products, technologies, processes, practices and services. The aim is to enhance economic growth, productivity, competitiveness, adaptability and sustainability of the Canadian agriculture, agri-food and agri-based products sector and assist in capturing opportunities for the sector in domestic and international markets.

The AIP resides under AAFC's Program Alignment Architecture 2.1 Science, Innovation, Adoption and Sustainability and is delivered through three streams:

• Stream A: Research Accelerating Innovation (2.1.2);
• Stream B: Research Development and Knowledge Transfer (2.1.3); and
• Stream C: Enabling Commercialization and Adoption (2.1.4).

AAFC also conducts long-term research and development activities through Sub-Program 2.1.1: Science Supporting an Innovative and Sustainable Sector, which is complementary to the AIP, as well as the Collaborative Framework and other sources of funding.

2.2 Overview of Stream A: Research Accelerating Innovation

Stream A: Research Accelerating Innovation is designed to address emerging science-based requirements of the Canadian agriculture, agri-food and agri-based products sector by generating and providing access to scientific knowledge that helps the industry identify and mitigate risks to agriculture and agri-food production, keep pace with sustainability considerations, improve productivity and capture market opportunities. The specific objectives of Stream A are to:

• Conduct innovative research to address key challenges and opportunities facing the sector;
• Transform scientific knowledge into agricultural products, processes and practices that improve competitiveness;
• Conduct research to address the key environmental sustainability challenges facing Canadian farmers;
• Encourage the transformation of scientific knowledge into agricultural practices that improve the environmental sustainability and profitability of farming operations;
• Support scientific measurement and analysis of the environmental sustainability performance of the sector that will facilitate competitiveness; and
• Work with industry to target AAFC efforts towards key business risks and opportunities.

Stream A is managed by AAFC's STB. Stream A is composed of two components:

1. Crops, Livestock and Food (CLF) supports early research activities on: threats to Canadian crops from invasive plants and insects, and crop pathogens; the nutritive components of agricultural products and their role in disease prevention, so as to help meet demands for more nutritious products; the manner in which contaminants enter into the production of agricultural products as well as prevention and remedial strategies.
2. Sustainable Science and Technology Advancement (SSTA) plays a role in equipping the sector with the knowledge and tools to make more sustainable and productive use of the available resource base, address sector productivity goals and meet pressures to be good environmental neighbours; and assisting the sector in accessing new markets and retaining current markets in the face of increasing government and non-government sustainability requirements.

The Sustainability Metrics project which resides with SSTA fulfills AAFC commitments to provide information required for reporting to the Government of Canada on the Federal Sustainable Development Strategy, to the Organization for Economic Co-operation and Development, and to the United Nations Framework Convention on Climate Change. The project builds on past AAFC accomplishments under National Agri-Environmental Health Analysis and Reporting System and National Carbon and Greenhouse-gas Accounting and Verification System. The project supports improvements to improve accuracy and applicability of a suite of priority measures in areas of soil quality, water quality and quantity, air quality, and biodiversity that are commonly required by various national and international schemes for assessing the environmental sustainability of agriculture.

Only internal AAFC scientists are eligible to be funded under Stream A activities. Other organizations, universities, researchers and individuals may participate in project activities as collaborators. Stream A program funding is distributed through an annual project application process, with the exception of Sustainability Metrics. A call out to AAFC scientists is made for Stream A, the Knowledge Technology Transfer element of Stream B, and A-base funding at the same time. The application process has three stages: priorities are identified in the summer, a call for letters of intent is typically issued in the fall, detailed proposals are solicited in the winter, and final funding decisions are made in March. Only selected applicants are invited to prepare a detailed proposal. Applications are evaluated by various internal and external reviewers (for example, Research Development and Technology Transfer (RDT) Directors, Marketing and Industry Services Branch, Strategic Policy Branch, Programs Branch, Canadian Food Inspection Agency, and external scientific reviewers).

Letters of intent and detailed proposals are assessed based on various criteria, such as the extent to which the project activities are: an appropriate role for the AAFC, aligned with departmental priorities; build on synergies; likely to result in substantial impacts for the sector; and will open new lines of research and lead to new technologies. Projects are normally supported for one to three years in duration. In exceptional cases projects are funded for more than three years.

The STB developed the SMS information system to support the management of science projects and to improve transparency, accountability and the visibility of science. Launched in 2013-14, this system captures project information from the initial proposal, project approval decisions, and project results on an annual basis. It also captures data on how each project supports the delivery of the strategic objectives associated with the sector science strategy. It also captures over 75 types of outputs from the research projects, such as scientific publications, technologies, licenses, collaborators and beneficial management practices. The information is used to develop annual Program and Sector Reports that track progress made toward achieving goals and set priorities for new calls for proposals.

2.3 Program resources

The following table indicates that the total budget for Stream A activities is approximately $150 million over a five year period, with about one third of the budget dedicated to salaries and two thirds dedicated to non-pay operating funds. The Sustainability Metrics Project is a $4.4 million ($880,000/year) project within SSTA.

Table 1: Stream A – Research Accelerating Innovation Planned Spending (million)

Budget	2013-2014	2014-2015	2015-2016	2016-2017	2017-2018	Total
Salary	$9.98	$9.98	$9.98	$9.98	$9.98	$49.90
Non-pay operating	$20.82	$20.64	$20.74	$20.25	$17.73	$100.19
Total	$30.80	$30.62	$30.72	$30.23	$27.71	$150.08
Source: AAFC STB: Science Planning and Reporting, Partnerships and Planning Directorate; excludes internal services, EBP and accommodation.

3.0 Evaluation methodology

3.1 Evaluation scope

The evaluation included a comprehensive assessment of the relevance and performance of Stream A, in accordance with the Treasury Board of Canada's Directive on the Evaluation Function (2009). The evaluation covers the period from April 1, 2013 to March 31, 2016.

The following lines of evidence were used to address the evaluation questions:

1. Document, File and Operational Data Review
   An extensive review of internal and external documents was undertaken to collect information on program relevance, design and delivery, and activities, outputs and outcomes. The methodology included a review of internal data and documents including administrative data from the SMS databases, program files such as guidelines for letters of intent and proposals, and departmental documents such as the Science and Technology Branch Plan and Science Sector Strategies. External documents reviewed included federal government Speeches from the Throne and the Minister of Agriculture and Agri-Food Mandate Letter.
2. Benchmarking Analysis
   A benchmarking analysis was undertaken of similar programs in Canada (for example, other federal and provincial government programs) and other jurisdictions including the United States, Europe, United Kingdom, France, Australia, Brazil and China. The analysis aimed to examine how efficiently and economically Stream A is being delivered and to identify alternative delivery mechanisms, promising practices, and lessons learned.
3. Interviews
   A total of 48 interviews were completed between March and June, 2016 to obtain input on the relevance and performance of Stream A program activities. Respondents included the following stakeholders: project collaborators and participants (federal government agencies, universities, and private companies) (6); Stream A project proponents (AAFC research scientists and RDT Leads) (37); and AAFC senior managers (directors and other senior staff) (5).
4. Case Studies
   Case studies of ten Stream A projects were conducted to gain details on the impact and need for the project, key factors that contributed to or constrained the project's success, and perspectives on program design and delivery. The projects were selected to represent a variety of sub-programs, science sectors and primary project locations, among other factors. Data collection for the case studies took place between March and May, 2016 and involved interviews with project leads and participants (by telephone or in person during site visits) and a review of relevant project databases and documents.
5. Site Visits
   Site visits were made to six AAFC research and development centres including centres in Summerland, Agassiz, Harrow, London, Saint-Hyacinthe, and Lethbridge between March and May, 2016. The centres visited were selected to represent a mix of regions, sectors and areas of scientific expertise, among other criteria. The purpose of the site visits was to obtain information concerning the research activities being undertaken at each research centre, with a particular focus on research activities related to Stream A. Each site visit included discussions with senior management staff followed by group or individual interviews with scientists and a tour of the facilities. Interviews were conducted with a total of 44 representatives during the site visits, including 30 AAFC scientists leading Stream A projects, eight AAFC senior managers, and six students, postdoctoral researchers and technicians.

3.2 Evaluation limitations

The major limitation of this evaluation is nature of the program and timing of the evaluation. It requires many years for far-from-market research projects to generate expected impacts. The Stream A science activities under review only considered activities in the past three years, which cannot be expected to have resulted in significant impacts during this time period. To mitigate this challenge, the evaluation focused on immediate impacts of the program and also considered how AAFC scientific activities have evolved over a broader timeframe.

4.0 Evaluation findings

4.1 Relevance

4.1.1 Continued need for program

The evaluation found that there is a continued need for foundational research that helps the agriculture and agri-food sector to keep pace with sustainability considerations, optimize productivity, identify and mitigate various production risks, and capture market opportunities.

Long-term scientific research is a key driver of product and process innovation required to address the competitiveness and profitability needs of the agriculture and agri-food sector. Studies suggest that long-term research has the potential to yield some of the highest returns available from public investments^{Footnote 6} and producer groups in GF2 consultations recognized that investment in research and development is key to the success of the sector.^{Footnote 7} Long-term, far-from market research is critical to understanding the causal mechanisms underlying threats to Canada's food production and distribution systems, permitting the development of strategies and practices to detect, monitor and control hazards through the food chain and understanding the nutritive and health attributes of agricultural products. It can take many years (10 years or more) to develop new innovations, such as disease-resistant crop varieties or management practices that optimize soil productivity and minimize negative impacts on the environment. The progress made in conducting foundational research provides a critical base for more targeted adoption and commercialization projects.

There is a need for scientific research to address sustainability and productivity issues impacting the sector in order to capitalize on new market opportunities, such as consumer demand for sustainable production, and manage risks such as increasing government and non-government requirements with respect to water quality^{Footnote 8} and greenhouse gas (GHG) emissions.^{Footnote 9} The agriculture sector is emissions-intensive relative to other sectors; in 2013, the sector was responsible for 10% of national GHG emissions.^{Footnote 10} Research into areas that reduce environmental waste also tend to enhance productivity by optimizing the use of inputs such as fertilizers for crops and feed for livestock. Such productivity improvements are required for producers to remain competitive internationally, since about 58% of the value of primary agriculture production in Canada is exported and producers are facing substantial increases in input prices, with production costs increasing by over 47% over the 2004-2014 period.^{Footnote 11}

Three years into the five-year program, Stream A has spent $59.3 million in funding 159 research and innovation projects^{Footnote 12} that aim to develop solutions to emerging threats and strategies for improving the sustainability of production systems, among other areas. Most of the projects reviewed are complete (82%), with the majority starting in 2013-14 and ending in 2015-16. 120 new projects were approved in March 2016 and are now in their first year of funding and activity.

As demonstrated in the following table, 55% of all project funding to date has been dedicated to projects targeting Crops Livestock and Food^{Footnote 13} while the balance of funding (45%) has been committed to Sustainable Science and Technology Advancement. The largest proportion of Stream A funding has been committed to projects in the Agro Environment Resiliency portfolio.

Table 2a: Stream A: Research Accelerating Innovation Projects and Committed Funding as of March 31, 2016

Stream A Program Component	Number of Projects (#)	Number of Projects (%)	Budget ($)	Budget (%)
Crops Livestock and Food	94	59%	$32,572,408	55%
Sustainable Science and Technology Advancement	65	41%	$26,766,679	45%
Total	159	100%	$59,339,087	100%
Source: STB Project Planned Cost Information, Extracted April 18, 2016. Includes one Stream A project which received an additional $320,000 in funding from 2011-12 to 2012-13.

Table 2b: Stream A: Research Accelerating Innovation Projects and Committed Funding as of March 31, 2016

Primary Portfolio	Number of Projects (#)	Number of Projects (%)	Budget ($)	Budget (%)
Agro Environment Resiliency	63	40%	$29,013,222	49%
Horticulture	17	11%	$6,181,605	10%
Cereals and Pulses	17	11%	$6,135,088	10%
Agri-Food	26	16%	$5,358,307	9%
Dairy, Pork, Poultry and other Livestock	11	7%	$4,811,061	8%
Oilseeds	9	6%	$3,272,150	6%
Beef and Forage	9	6%	$2,932,704	5%
Bioproducts	3	2%	$852,700	1%
Biodiversity and Bioresources	4	3%	$782,250	1%
Total	159	100%	$59,339,087	100%
Source: STB Project Planned Cost Information, Extracted April 18, 2016. Includes one Stream A project which received an additional $320,000 in funding from 2011-12 to 2012-13.

Most collaborators, project proponents and senior managers interviewed stated that Stream A projects are addressing the major sector needs with respect to research and innovation.^{Footnote 14} Interviewees explained that the most critical needs facing the sector with respect to research and innovation include (in order of frequency mentioned): long-term scientific research, research related to addressing environmental and climate change issues, and research that addresses threats to agriculture and agri-food production.

4.1.2 Alignment with federal government priorities and Agriculture and Agri-Food Canada strategic outcomes

The objectives of Stream A are well aligned with the priorities of the federal government and the strategic outcomes of AAFC, particularly with respect to supporting agricultural science and innovation and recognizing the importance of the environment in supporting a strong Canadian economy. The 2015 Mandate Letter to the Minister of Agriculture and Agri-Food Canada identifies the investment in agricultural research to support discovery science and innovation in the sector as a priority for the government."^{Footnote 15},^{Footnote 16} The federal government Budget 2016 also identifies to over $70 million in planned investments in agricultural science, including research and infrastructure.^{Footnote 17} The objectives and activities of Stream A are also well aligned with AAFC's Strategic Outcome 2: An innovative and sustainable agriculture, agri-food and agri-based products sector, particularly in terms of "generating new knowledge, fostering innovation and increasing adoption and commercialization of agricultural, agri-food and agri-based products, processes or practices."^{Footnote 18}

4.1.3 Alignment with federal government roles and responsibilities

Mandated by federal legislation under the Experimental Farm Stations Act (1985),^{Footnote 19} STB plays an important federal role in conducting long-term research with an applied focus, which is unlikely to be undertaken or funded by other organizations. The federal government has a roll in undertaking public good research and the capacity to address foundational agricultural research and development such as environmental sustainability research. AAFC, in particular, has the unique advantage of being able to conduct analysis in different ecological zones due to its national network of research centres.

Stream A is led by STB and appears to complement other agricultural research and innovation activities conducted or funded by AAFC and other federal and provincial government, academic, private and not-for-profit organizations. The project selection processes for Stream A includes an assessment to avoid overlap or duplicate other sources of research and innovation support. Senior management conducts an investment framework analysis that examines ongoing AAFC research as well as funding support from other sources, such as Natural Sciences and Engineering Research Council of Canada funding to university scientists. The program considers criteria such as science excellence and whether the project is an appropriate role for AAFC or whether industry should contribute. Most project proponents interviewed perceived that it is very unlikely that their Stream A project would have proceeded without AAFC assistance.

Non-government organizations tend to either focus on fundamental research that contributes to new theory or research that can be commercialized. For example, some university scientists tend to focus on theoretical discoveries without a clear application in the markets. Private sector organizations are inclined to focus on applied research that can generate faster returns.

Though provincial government research programs often have a focus similar to AAFC, they have varying capacity for agricultural research as some provinces devote a significant proportion of their resources to extension and commercialization. In total, provinces have less capacity than the federal government to fund agricultural research and even less capacity to conduct it since most of the funding tends to go to universities. The federal government share of total government agricultural research expenditures in Canada has been historically larger than that of provincial governments, averaging 69.4% (federal) and 30.6% (provincial), between 1985-86 and 2015-16.^{Footnote 20} Furthermore, AAFC has 20 research and development centres, sub-stations, and other offices across Canada and a significant amount of expertise developed and maintain under separate programs, such as AAFC Science and Collaborative Framework. In summary, while it is clear there are many stakeholders involved in agricultural research activities; AAFC has an important role in delivering the type of research funded through Stream A.

4.2 Performance – effectiveness

The following section examines the performance of Stream A in terms of the extent to which Stream A has achieved its intended outputs and outcomes.

4.2.1 Intended outputs

The evaluation found that, as of March 31, 2016, Stream A is producing outputs as intended. The program has already produced an estimated 151 peer reviewed publications (submitted and/or published) under SSTA and 127 under CLF.^{Footnote 21} This exceeds the target of 54 publications by March 31, 2018 for SSTA and 89 publications by March 31, 2018 for CLF (Table 3).

Table 3: Achievement of Stream A Outputs

Program Output Topic	Indicator	Target	Date to Achieve Target	Outputs Achieved as of March 31, 2016
Addressing environmental sustainability challenges (SSTA)	# of peer reviewed publications submitted and/or published	54	March 31, 2018	151
New products, processes, technologies relevant to improving the productivity and/or competitiveness of the sector, minimization of catastrophic threats to crops, optimization of livestock efficiencies, and food health and safety (CLF)	# of peer reviewed publications submitted and/or published	89	March 31, 2018	127
Source: Stream A Program Performance and Risk Management Strategy and AAFC Databases, Extracted April 18, 2016.

Table 4 summarizes the different types of scientific publications resulting from Stream A projects. In addition to peer reviewed articles and papers in scientific journals, the other types of scientific publications generated most frequently are conference proceedings, abstracts, and posters as well as presentation materials and book chapters.

Table 4: Scientific Publications Resulting from Stream A Projects^{Footnote 22}

Results from Stream A Projects	SSTA	CLF	Total
Article/Paper	151	127	278
Conference - Proceedings	75	68	143
Conference - Abstract	22	21	43
Conference - Poster	22	23	45
Presentation Materials	17	22	39
Chapter in Book	16	10	26
Report	5	5	10
Total	308	276	584
Source: AAFC Databases, Extracted April 18, 2016.

4.2.2 Intermediate outcomes

Innovations resulting from crops livestock and food projects

As of March 31, 2016, Stream A projects have resulted in an estimated 37 innovations^{Footnote 23} related to the number of new products, processes, and technologies developed relevant to the minimization of catastrophic threats to crops, optimization of livestock efficiencies, and food health and safety. This is almost three quarters of the target of 51 innovations by March 31, 2018.

Table 5: Achievement of Immediate Outcomes Related to Crops Livestock and Food

Program Outcome	Indicator	Target	Date to Achieve Target	Outputs Achieved as of March 31, 2016
New products, processes, technologies, relevant to improving the productivity and/or competitiveness of the sector, minimization of catastrophic threats to crops, optimization of livestock efficiencies, and food health and safety are available for transfer.	Number of new products, processes, technologies.	51	March 31, 2018	37
Source: Stream A Program Performance and Risk Management Strategy and AAFC Databases, Extracted April 18, 2016.

Examples of innovations resulting from Crops Livestock and Food projects are provided in Table 6. Over one third of the innovations (16/37) are related to genetic sequencing and varieties.

Table 6: Reported Innovations Resulting from Crops Livestock and Food Projects

Innovation Type	#	Examples
Genetic Material	16	600 novel nucleotide sequences of agriculturally important symbiotic bacteria added to public databases 2,000 new haploid lines based on six targeted winter triticale crosses 470 sequences from wheat and wheatgrass pertaining to expressed genes associated with flowering in perennial grasses New wheat variety resistant to stem rust Two Fusarium avenaceum genome sequences deposited at the National Center for Biotechnology Iinformation GenBank in the Whole Genome Shotgun database
Process, Systems or Methodologies	11	A novel strategy for detecting Apple Maggot was developed using geographic information systems and ecological knowledge of the target insect; transferred to the Canadian Food Inspection Agency An ergot nursery was planted at Lacombe, the first nursery of this kind for triticale in Canada
Designs	3	Adapting the genotyping-by-sequencing approach to assist with addressing Johne's disease, a chronic debilitating disease affecting ruminant animals
Patent	2	Patent obtained as part of a project that aimed to reduce food safety risks of unavoidable natural mycotoxins significant to Canada through the development of feed additives in livestock production
Software Tools	1	Panneton B. Matlab Software suite for modeling particulate transport in the surface layer of the atmosphere
New/adapted Beneficial Management Practices (BMPs)	1	In vitro demonstration (liver tissue culture) that enterolactone can protect liver cells against oxidative stress, in dairy cows.
Other	3	New Juice-based Lure for Monitoring Female Apple Clearwing Moths Encapsulation system for protecting thermal sensitive probiotic bacteria
Total	37	n/a
Source: AAFC Databases, Extracted April 18, 2016. n/a = not applicable

Most collaborators, project proponents, and senior managers interviewed felt that Stream A projects have been successful in developing new products, processes, and technologies to minimize catastrophic threats to crops, optimize livestock efficiencies, and increase food health and safety. However, several representatives explained that more time is required to commercialize the innovations and apply the research findings to benefit the agriculture and agri-food sector.

The case studies and site visits provided evidence that CLF projects have contributed to the development of new products, processes, and technologies to minimize catastrophic threats to crops, optimize livestock efficiencies, and increase food health and safety. A number of significant examples are illustrated in the following text box.

Text Box 1. Case Study Examples of Impacts of Crops Livestock and Food Projects

• Improving productivity and/or competitiveness of the sector - Major discoveries in the area of crop genetics that could lead to significant input cost savings and improved environmental performance from agricultural production
  • Developing perennial traits in wheat
    • Rationale: Perennial crops, such as wheatgrass, possess economic benefits such as savings on input costs (because it only needs to be seeded every three years), and environmental benefits (for example, full year moisture, carbon sequestration, improved nutrient use, weed control, etc.). However, most cereal crops are annual and breeders have failed to develop effective perennial cereal varieties after decades of efforts.
    • Achievement: The project resulted in several discoveries such as how similar wheat and wheatgrass are in terms of gene sequencing (90-99% the same) which have led to increased collaborations with U.S. partners.
  • Engineering nitrogen fixation in cereal crops
    • Rationale: Commercial agricultural production is very dependent on the application of chemical fertilizers, which are costly to farmers and pose significant risks to the environment. Some plants, such as legumes, are able to access nutrients that are naturally occurring in the atmosphere through a process known as nitrogen fixation, but most agricultural crops do not possess this ability.
    • Achievement: A project was conducted to better understand the genetic properties of nitrogen fixing plants to enhance crop nutrient use with a long-term goal of being able to engineer this process in cereal crops. The research has resulted in major discoveries such as a better understanding of the genes and bacteria driving the nitrogen fixation processes.

• Minimizing catastrophic threats to crops
  • Developing a new wheat variety resistant to an emerging fungus with potentially devastating economic consequences for Canadian farmers
    • Rationale: Ug99 is a recently detected strain of wheat stem rust that has spread quickly throughout eastern Africa and as far as Iran. Though the strain is not currently present in Canada, it can be transported by wind, About 80% of Canadian wheat is susceptible to the strain.
    • Achievement: A national project resulted in the development of a new variety that is higher yielding and resistant to Ug99 and other diseases. The project stands to result in economic benefits to the sector of up to $700 million annually from reduced losses and reduced fungicide use, if Ug99 were to arrive in Canada.
  • Identifying a strawberry virus that was never detected before in the world
    • Rationale: Strawberry nurseries in Nova Scotia are major suppliers of plants for commercial growers in Canada and the United States. Growers were recently hit by an emerging disease which resulted in the destruction of hundreds of acres of strawberry plants in the region and in parts of the United States.
    • Achievement: A national project was undertaken to investigate the causes and effects of the disease. The project resulted in the identification of a new strawberry virus and findings that will likely contribute to improved monitoring and mitigation protocols.

• Validating health benefits of foods
  • Improving the functionality of an in vitro digestion platform that helps agri-food processors and scientists to validate nutrient and health benefits of foods
    • Rationale: Agri-food producers and scientists require mechanisms to validate food products in terms of their health and functional attributes (for example, to see how proteins or probiotics are digested).
    • Achievement: A project improved the functionality of an in vitro digestion platform that has assisted six companies to test and validate healthy attributes of their food products and AAFC scientists in 30 other projects , for example, related to encapsulation.

• Optimizing livestock efficiencies and increasing food safety - Identifying strategies which minimize hazardous contaminants in food production systems
  • Mitigating human health risks by developing antibiotic-free solutions in livestock production
    • Rationale: Livestock production has relied greatly on growth enhancing antibiotics, which have contributed to an increase in antibiotic-resistant diseases (such as E. coli) that can impact human health.
    • Achievement: A project was undertaken to investigate bio-based alternatives to antibiotics. The project made several discoveries one example of which is that antibiotic growth promoters used in livestock production work by impacting the animals' immune system.
  • Providing regulators and farmers with strategies to minimize human health risks and reduce the prevalence of antibiotics, pharmaceuticals, and microbial agents in food systems
    • Rationale: Contaminants such as pharmaceuticals, pathogens, and antibiotic resistant bacteria in agricultural production pose serious threats to human health through water- or food-borne disease.
    • Achievement: A national project that examined these issues resulted in various impacts such as validating the safety of Ontario provincial sewage sludge management and identifying BMPs related to controlled drainage, which have been adopted by Ontario conservation authorities. The research team received a significant honour from the Water Environment Association of Ontario for Exemplary Biosolids Management in the category of Significant Research.^{Footnote 24}.

Innovations from Sustainable Science and Technology Advancement Projects

As of March 31, 2016, Stream A projects have resulted in 15 innovations or about 25% of the targeted number of 65 innovations by March 31, 2018 regarding new sustainability metric tools and beneficial management practices and tools related to the Sustainable Science and Technology Advancement component of Stream A (Table 7).^{Footnote 25}

Table 7: Achievement of Immediate Outcomes Related to Sustainable Science and Technology Advancement Projects

Program Outcome	Indicator	Target	Date to Achieve Target	Outputs Achieved as of March 31, 2016
New sustainability metrics tools made available to the sector	Number of new sustainability metrics tools (for example, methods, models, algorithms, coefficients)	30	March 31, 2018	5
New beneficial management practices and decision-support tools relevant to environmental sustainability are available for transfer to the sector.	Number of new/adapted beneficial management practices	15	March 31, 2018	5
New beneficial management practices and decision-support tools relevant to environmental sustainability are available for transfer to the sector.	Number of new or improved decision-support tools	20	March 31, 2018	4
New beneficial management practices and decision-support tools relevant to environmental sustainability are available for transfer to the sector.	Other	n/a	March 31, 2018	1
Source: Stream A Program Performance and Risk Management Strategy and AAFC Databases, Extracted April 18, 2016. n/a = not applicable

Similar to CLF projects, the number and categorization of SSTA innovations reported in AAFC's program database are dictated by the manner in which project proponents interpret and report innovations resulting from their projects. The evaluation found that there are variations in how project proponents categorize their innovations. Also, some indicators, such as number of beneficial management practices or decision-support tools, were only recently added to reporting systems. The number of innovations shown for each indicator in Table 7 has been updated based on a review of the project outcome descriptions, to ensure they align with the targeted outcomes.

As indicated in Table 8, the types of innovations resulting from SSTA projects include new sustainability metrics tools such as methodologies for measuring GHG emissions from livestock production, new or adapted beneficial management practices that optimize nutrient use in crops, and tools to support sustainable decision making.

Table 8: Reported Innovations Resulting from Sustainable Science and Technology Advancement Projects

Innovation Type	#	Examples
New Sustainability Metrics Tools	5	Methodology for the estimate of NH3 emissions from domestic animals applicable to GHG and air pollutant inventories Farm environmental management survey fertilizer and tillage tables Various farm type, soil, and systems databases (for example, national high-accuracy, high resolution land use maps for 1990, 2000 and 2010)
New or Adapted BMPs Relevant to Environmental Sustainability	5	Cluster thinning as a tool to hasten ripening of wine grapes Interaction of irrigation and soil management on sweet cherry productivity and fruit quality at different crop loads Identification of honeycrisp apple rootstocks with superior and inferior abilities to translocate key nutrients to leaves Suitable orchard floor management strategies in organic apple orchards that augment soil organic matter and maintain tree performance Method to augment berry yeast-assimilable nitrogen in wine grapes from late-season foliar applications
Decision-support Tools Relevant to Environmental Sustainability	4	Farms to Regions Envision Decision Support Tool for Eastern Ontario. Development of the fuzzy inference systems and forecast rainfall-based SCAN (Soil, Crop, Atmosphere for Nitrogen management) platform for trial Panneton B. Matlab code for image segmentation using the Triangle method Carleton University Envision Ontario data and information portal.
Other	1	Genetic material: small subunit ribosomal RNA sequences (impact of land use on arbuscular mycorrhizal fungal communities)
Total	15	n/a
Source: AAFC Databases, Extracted April 18, 2016. n/a = not applicable

Most collaborators, project proponents, and senior managers interviewed perceived that Stream A projects have been successful in developing new beneficial management practices relevant to environmental sustainability. Several representatives explained that there has been considerable progress in this area, particularly in terms of identifying beneficial management practices and decision-support tools which promote both environmental sustainability and improved agricultural productivity.

The case studies and site visits provided additional evidence that SSTA projects have contributed to the development of new sustainability metrics tools and beneficial management practices and decision-support tools relevant to environmental sustainability. A number of significant examples are illustrated in the following text box.

Text Box 2. Case Study Examples of Impacts of Sustainable Science and Technology Projects

• New/improved sustainability metrics tools
  • More accurate measures of environmental impacts from agriculture production to help farmers and regulators identify sustainable practices and reduce harmful impacts
    • Rationale: Regulators and producers require rigorous indicators to measure nutrient losses from agricultural production to identify practices that optimize input use and minimize waste to the environment.
    • Achievement: Two national projects aimed to address this issue. One aimed to enhance, calibrate and validate tools to assess the risk of phosphorus losses and another aimed to develop a comprehensive indicator of reactive nitrogen release from Canadian agriculture. The projects resulted in enhanced environmental impact indicators from agricultural production for farmers and regulators to identify sustainable management practices.

• New/adapted beneficial management practices relevant to environmental sustainability
  • Identifying beneficial management practices that result in substantial economic gains and reductions in environmental contamination
    • Rationale: Nitrogen is an essential plant nutrient input for all intensive cropping systems in Canada. However, losses from ammonia volatilization and nitrous oxide emissions represent a major economic loss and contribution to environmental contamination.
    • Achievement: A national project was undertaken to develop ways to measure the losses and test the impact of various management strategies on nitrogen losses and productivity. The project resulted in new technologies for measuring nitrogen losses and identified beneficial management practices (BMPs) that will improve yields and environmental performance. Preliminary findings from the first two years estimated the BMPs to result in economic gains of $900 to $5,400 per 100 acres compared to the control treatment, 57% to 97% reduction in ammonia losses, and 21% reduction in nitrous oxide emissions.
  • Soil/manure application practices that maximize productivity and minimize environmental impacts
    • Rationale: Preventing nutrient losses from agricultural production is becoming an increasingly important issue for regulators.
    • Achievement: A national project was undertaken to assess phosphorus loss vulnerability in various soil and manure samples collected across Canada. The research resulted in key findings with respect to the soil/manure applications that maximize productivity and minimize environmental impacts in different parts of Canada. Findings regarding effective BMPs that optimize nutrient use in fresh and composted manures were disseminated at over 50 different events in North America and internationally and through various industry publications.
  • Helping farmers to understand the long-term economic and environmental costs and benefits of existing management practices
    • Rationale: Research was conducted to examine how current farm practices, such as crop rotation, fertility management, residue removal, and tillage impact sustainability and productivity (for example, soil quality, productivity, economic costs and benefits, plant disease, insects, and GHGs).
    • Achievement: The project resulted in key findings that will provide farmers with evidence of financial and environmental impacts to assist them in determining which farm practices to adopt.

Dissemination of Knowledge to Key Stakeholders

Stream A has well exceeded its targets with respect to knowledge dissemination. As of March 31, 2016, project participants have disseminated knowledge through 419 presentations at conferences or other events, which is considerably greater than the target of attendance at 150 conferences, events and annual meetings by March 31, 2017. Similarly, as of March 31, 2016, there were a total of 771 scientific and technology transfer publications which is greater than the target of 100 by March 31, 2018.^{Footnote 26}

Table 9: Achievement of Immediate Outcomes Related to Knowledge Dissemination

Program Outcome	Indicator	Target	Date to Achieve Target	Outputs Achieved as of March 31, 2016
Program knowledge is disseminated to stakeholders	Number of conferences, events, annual meetings attended	150	March 31, 2018	419 presentations at conferences or other events
Program knowledge is disseminated to stakeholders	Number of transfers to STB outreach staff, the science community or other stakeholders.	100	March 31, 2018	53 technology transfer publicationsFootnote 27 and 278 peer reviewed scientific publications
Source: Stream A Program Performance and Risk Management Strategy and AAFC Databases, Extracted April 18, 2016.

Information and expertise related to Stream A projects was disseminated through a variety of knowledge and expertise contribution activities, most frequently consisting of presenting at conferences, events and annual meetings, as demonstrated in Figure 1.

Figure 1: Types of Knowledge and Expertise Contributions from Stream A Projects

Source: AAFC Databases, Extracted April 18, 2016.

Most collaborators, project proponents, and senior managers perceived that Stream A projects were successful in disseminating information and knowledge developed or obtained through the project to sector representatives or other stakeholders as well as advancing knowledge in the science community. Some projects received significant recognition. For example, a project which investigated soybean resistance to root rot at the genetic level was featured on the cover page of a major scientific publication: The Plant Journal. A few senior AAFC managers indicated that more could be done to disseminate information to a wider audience by using web-based platforms.

4.2.3 Intermediate and end outcomes

As indicated in Table 10, the performance measurement indicator of an increase in agriculture net value added has been used for the intermediate outcome of Stream A which is an "agriculture and agri-food sector that utilizes science to improve or transform commodities into new value-added products". Performance data indicates that Stream A seems to be achieving the intermediate outcome because the net value-added^{Footnote 28} in agriculture was $14.9 billion in 2014, $2.6 billion above the target of $12.3 billion by March 31, 2018. However, this change is not likely attributable of Stream A research because at the time the data was collected, Stream A research had only been underway for one year. There are also many other factors that contribute to an increase in agriculture net value added in addition to Stream A projects such as other activities conducted by AAFC, provincial governments, not-for profit organizations and the private sector, as well as events outside the program's control (for example, commodity prices and input costs). It is difficult to determine the extent to which an increase in agriculture net-value added resulted from Stream A projects.

Table 10: Achievement of Stream A Intermediate and End Outcomes

	Program Outcome	Indicator	Target	Date to Achieve Target	2014 Statistics Canada Data
Intermediate Outcome	Agriculture and agri-food sector that utilizes science to improve or transform commodities into new value-added products	Increase in agriculture net value added	$12.3 billion	March 31, 2018	$14.9 billion (2014)
End Outcome	An innovative agriculture, agri-food and agri-based products sector	Total R&D expenditures by business enterprises in food manufacturing	$172.7 million	March 31, 2018	$125.9 million (2014)
Source: Stream A Program Performance and Risk Management Strategy and Statistics Canada.

As indicated in Table 10, the performance measurement indicator of total "R&D expenditures by business enterprises in food manufacturing" has been used for the end outcome of Stream A, which is an innovative agriculture, agri-food and agri-based products sector. Although total R&D expenditures by business enterprises in food manufacturing was $125.9 billion in 2014 as compared to the target of $172.7 billion by March 31, 2018, it is difficult to attribute this changes to the program. Similar to the performance measurement indicator for the intermediate outcome, there are limitations to this indicator because there are many other factors that contribute to an increase in total R&D expenditures by business enterprises in food manufacturing. These factors include other activities conducted by AAFC, and separate organizations as well as events outside the program's control (for example, global markets, profitability of food manufacturers, and the investment climate).

4.2.4 Additional benefits and opportunities

Stream A projects have resulted in a number of positive additional benefits, such as subsequent spin-off research projects and enhanced scientific capacity among students and new scientists.

Several projects have resulted in additional opportunities for further collaboration and research. According to project reports, 30 projects resulted in 71 follow-on research grants, awards, honours or prizes at the time of the evaluation. Most of the projects reviewed as part of case studies resulted in (or will likely result in) follow-on projects or investments such as investments by AAFC and key industry stakeholders, and new areas of investigation or collaboration. For example, a project to develop strategies to reduce nitrogen losses has generated significant interest and led to the development of an AIP Stream B agri-science project with funding from Fertilizer Canada.

Stream A projects have also helped to develop existing and future scientific capacity and highly qualified people through research projects and salary dollars. Stream A projects enable the hiring and support of Post Doctorate Fellows and student placements at AAFC research centres. This support and development enables AAFC to retain employees and also build capacity for recruitment for future employees. The area of focus of these students can range from disciplines such as environmental science, geography, and engineering. Post Doctorate Fellows and students learn skills, such as the process to undertake scientific research (for example, how to design a project and the quality control, patience, and creativity required) and gain experience working in a professional laboratory setting.

4.3 Performance – efficiency and economy

4.3.1 Effectiveness of Stream A design and delivery

The evaluation found that many aspects of the Stream A design have facilitated effective delivery, while some factors have constrained delivery.

Contributing factors to effective program design and delivery

Multiple lines of evidence show that the major factors contributing to the effective delivery of Stream A projects include the complementary science activities, the project selection process, collaborative research teams with multiple areas of expertise, infrastructure, equipment and technical assistance supported through other STB programming, and activities facilitating the transfer and adoption of innovations.

Complementary science activities and infrastructure

Stream A projects have benefited from complementary science activities and infrastructure supported through other STB investments. For example, several Stream A projects built on findings from previous projects. The projects benefited from AAFC scientific staff and infrastructure, including approximately 400 scientists with expertise in 19 agricultural science disciplines^{Footnote 29} and 20 research centres across multiple ecological zones of Canada. Through this infrastructure, scientists have access to superior facilities and equipment such as a Level 3 plant pathology lab, cutting edge gene sequencing equipment, and long-term plots with data on different treatments and nutrient impacts over time. The availability of experienced and highly qualified AAFC technicians has contributed to the success of Stream A projects and is a unique advantage compared to the level of support available to university scientists.

Project selection process

The design and delivery of Stream A programming promotes accountability and directs resources to areas where there is the greatest need and to projects that are most likely to have significant impacts. For example, the project application process assesses if the projects respond to the priorities identified in the call for proposals and the potential impact of projects during three stages of the review process (that is, management review of letters of interest and detailed proposals and external peer review of proposals). Science excellence is also assessed through the comprehensive peer review process, which ensures that only projects with scientific merit are funded. The evaluation found that the process has been reviewed and improved in terms of its timeliness and efficiency since the inception of the program. It was suggested by some key informants that the process could be further refined by facilitating increased opportunities for communication with respect to priorities and the project selection process between scientists, senior managers and external stakeholders (for example, with provincial government representatives with respect to sustainability priorities).

The Stream A project selection process is also aligned with how other similar programs allocate research funding. For example, the Department of Fisheries and Oceans Canada's Aquaculture Collaborative Research and Development Program undertakes project selection on an annual competitive basis. Applications are screened against all eligibility requirements and undergo a two-part peer-review process: a technical review by internal and (or) when appropriate, external scientists, followed by a comprehensive review by the Technical Review Committee, comprised of representatives from the Department of Fisheries and Oceans, provinces, industry and others. Agricultural research organizations in other jurisdictions such as the French National Institute for Agricultural Research and the Brazilian Agricultural Research Corporation (Embrapa) also allocate funding through a competitive call for proposals process only open to their internal scientific staff. Both the French National Institute for Agricultural Research and Embrapa have also organized their research and development activities around strategic programs with each having their own selection process depending on the activities to be funded, usually requiring an initial expression of interest or inquiry letter, followed by a detailed proposal.

Collaborative research teams

AAFC and external science expertise is leveraged through the collaborative research teams formed to undertake Stream A projects. For example, Stream A projects benefited from AAFC scientists with different areas of expertise (for example, agronomic modelling, plant pathology, and soil science), production systems and geographic regions. On average, Stream A projects involved nine AAFC scientist participants and three AAFC technical or other support staff. Stream A also provides the opportunity for AAFC scientists to work with scientists at universities and other stakeholder organizations, further leveraging expertise and facilitating knowledge dissemination and exchange. Of the 159 Stream A projects reviewed, 55 projects involved external collaborators. Each of these projects had an average of four individual collaborators. The types of collaborators most commonly included representatives from Canadian universities or research institutions (particularly the Universities of Guelph, Manitoba, and Alberta), provincial governments (particularly Alberta and Ontario), and other federal government departments or agencies (particularly Environment Canada), as demonstrated in the following figure. International collaborators were most frequently located in the United States, France, and China.

Figure 2: Characteristics of External Collaborators in Stream A Projects

Source: AAFC Databases, Extracted April 18, 2016.

According to a review of literature related to networking in science, collaboration is an important factor in disseminating knowledge to key stakeholders. Studies have shown that the size of research teams and extent to which scientists participate in conferences contributes to the quantity^{Footnote 30} and quality^{Footnote 31} of publications a scientist produces. The literature suggests that being in an open network is a key predictor for career success^{Footnote 32} and that interpersonal networks are important in determining patterns of knowledge diffusion.^{Footnote 33} A recent study of scientific output and collaboration patterns of 13 federal government entities showed that AAFC scientists frequently engage in collaboration. This study stated that over 75% of AAFC's scientific publications were co-authored by researchers from another organization and the department more than doubled its international collaboration rate from 2000 to 2014.^{Footnote 34}

The benchmarking analysis found that similar programs in other jurisdictions also promote collaboration and partnership to enhance the quality and scope of their research projects. In addition to regional and national collaborations, the French National Institute for Agricultural Research, Embrapa and the Chinese Academy of Agricultural Sciences, as national agricultural research institutions, invest in the formal development of international collaborations and have implemented programs designed specifically for building these types of partnerships.

Additional mechanisms could be investigated to encourage further collaboration between AAFC scientists and with external organizations. Forums (for example, quarterly webinars) could be developed to increase communication between the International Engagement Division and scientists with respect to international collaboration opportunities. Policies governing travel and research collaboration could be reviewed to further facilitate effective partnership both within AAFC and with external organizations and stakeholders.

Activities facilitating knowledge transfer and adoption

Outreach activities are important mechanisms for disseminating the results of Stream A projects. Some examples of outreach activities include AAFC scientists sending bi-weekly emails with key research data to industry stakeholders; publishing data on industry websites; involving industry stakeholders in science activities (for example, using producer plots to conduct research); undertaking open houses and regional research user meetings, which help to inform stakeholders of recent scientific activities, identify research gaps, and coordinate activities with key collaborators; hosting events at the research centres; and developing handouts describing major research achievements.

Design and Delivery Constraints

The evaluation found that the design and delivery of Stream A projects is sometimes constrained by the broader government context in which it is required to operate. These include human resource challenges, limited project and program timeframes, and administrative and data storage constraints.

Further facilitating multi-disciplinary science

Some scientists interviewed suggested implementing alternative science delivery models to address capacity constraints. For instance, they indicated that AAFC could reduce duplication of efforts by developing a more integrated focus on production systems rather than commodities, or develop more specialized centres of excellence (for example, in microbiology) within particular research centres. More recently, STB has begun hosting Transformative Workshops that bring together groups of scientists that are multi-disciplinary, multi-generational, and from multiple stakeholder groups to generate ideas to address key agricultural challenges and support policy and investment decisions.

Current funding timeframes

The evaluation found that some Stream A projects may be constrained by the current timeframes of funding. Alternative models that may further optimize the likelihood that projects will result in innovations could be considered. While many key informants interviewed recognized the incremental nature of Stream A projects and that it is valuable to reassess project results at different stages, they indicated that the risk of shorter timeframes is that scientists may limit the scope of their research goals and creativity required for innovation. Although there is no timeframe for a research project, the program only supports projects for up to three years before the project is required to submit a new project proposal. This process allows the program to ensure that project continue to be relevant and funded at the right level. Interviewees suggested that individual projects could be funded over longer timeframes (up to five years), which would encourage research teams to investigate more fundamental science issues and maintain longer-term experiments and technical staff expertise.

The programs reviewed as part of the benchmarking analysis demonstrated a variety of funding models, some of which had longer timeframes for science activities (up to 5 years). For example, the Australian Research Council manages the National Competitive Grants Program, which provides funding for up to five years.^{Footnote 35} The United States Department of Agriculture's Agricultural Research Service also funds research through a five-year national program cycle. Agricultural Research Service scientists receive funding through a non-competitive, peer-review process. The long-term funding commitment helps their scientists to conduct long term research without any funding interruptions or changes in national research priorities.^{Footnote 36}

Data storage and other IT constraints

The evaluation identified some constraints with respect to available data storage and computation capacity, as well as challenges in procurement timelines associated with acquisition processes. This is particularly affecting areas of research related to genetics which is producing more and more complex datasets.

4.3.3 Performance measurement and reporting systems

The evaluation found that STB has developed comprehensive mechanisms to track the progress and impacts associated with Stream A projects.

Scientists are required to prepare annual progress reports for each of the Stream A projects in which they are involved. The SMS reporting system compiles comprehensive information obtained from project plans and reports such as project objectives, costs, collaborators, progress against milestones, lessons learned, and project outputs.

The evaluation found some challenges with respect to the using the SMS data as part of this evaluation. For example, it was challenging to categorize innovations according to the programs intended outcomes (for example, whether an innovation was a new process, system, methodology or beneficial management practice) which made it difficult to report on the achievement of targeted types of innovations described in the PMS. Limited information was also provided in the SMS database with respect to the significance of the impacts or discoveries. Though the mechanisms are somewhat new and still in the testing phase, the systems are useful tools for communicating project results.

5.0 Conclusions, recommendation and management response and action plan

Conclusions

Relevance

The evaluation found that there is a continued need for Stream A scientific research that helps the agriculture and agri-food sector to optimize productivity, identify and mitigate various production risks, capture market opportunities and keep pace with sustainability considerations. Scientific research is a key driver of product and process innovation required to address the competitiveness and profitability needs of the agriculture and agri-food sector. Research is also critical to understanding the causal mechanisms underlying threats to Canada's food production and distribution systems, permitting the development of strategies and practices to detect, monitor and control hazards through the food chain and understanding the nutritive as well as health attributes of agricultural products.

The objectives of Stream A are well aligned with the priorities of the federal government and the strategic outcomes of AAFC, particularly with respect to supporting agricultural science and innovation and recognizing the importance of the environment in supporting a strong Canadian economy.

Furthermore, AAFC is mandated under the Experimental Farm Stations Act (1985)^{Footnote 37} to conduct agricultural research and plays an important role in conducting long-term research with an applied focus, which is unlikely to be undertaken or funded by other organizations. More specifically the type of research funded through Stream A because it has a public good focus (for example, improving sustainability of agricultural practices) or is too far from commercialization for private sector to realize a return on investment in the near term, yet it provides a critical base for more targeted adoption and commercialization projects. Other research organizations, such as some university scientists tend to focus on theoretical discoveries without a clear application in the markets. Private sector organizations are inclined to focus on applied research that can generate faster returns.

Stream A complements other agricultural research and innovation activities conducted or funded by AAFC and other federal and provincial government, academic, private and not-for-profit organizations. Policies guiding Stream A project selection are intended to prevent overlap or duplicate with other sources of research and innovation support. The program complements other activities by advancing scientific findings to a point where the risks for industry are lower, which provides them with the incentive to fund more applied projects. Stream A also facilitates collaboration and leveraging of expertise, facilities, and resources with other federal, provincial, industry, and university research partners.

Performance (effectiveness)

The evaluation found that, three years into the five-year program, Stream A has made considerable progress towards the achievement of its intended outputs and outcomes. In particular, Stream A has met or already exceeded its performance targets with respect to knowledge generation and the dissemination of research results. Progress has been made in developing new products, processes, and technologies to enhance agricultural productivity and market potential as well as innovations related to environmental sustainability. The evaluation identified several examples of significant innovations and new products, processes, and technologies resulting from Stream A projects.

Performance (efficiency and economy)

The evaluation found that many aspects of the Stream A design have facilitated effective delivery, while a few factors have constrained delivery. The major factors contributing to the effective delivery of Stream A projects include:

• The project selection process promotes accountability and directs resources to areas where there is the greatest need and to projects which are most likely to have significant impacts. It also ensures science excellence through an external peer review process.
• Collaborative research teams with multiple areas of expertise have leveraged internal and external scientific expertise and capacity and facilitate knowledge dissemination and transfer to the scientific community.
• Stream A builds on other Science and Technology Branch investments, thereby taking advantage of complementary science activities and existing infrastructure, such as highly qualified scientists and technicians, research centres across multiple eco-zones, and leading edge scientific technology and equipment.
• The program engages in activities, facilitating the transfer and adoption of innovations, such as undertaking regular outreach with industry stakeholders, hosting regional meetings with industry and research stakeholders, and distributing publications that highlight major research achievements that have facilitated the transfer and adoption of innovations.

The evaluation found that the design and delivery of Stream A projects is sometimes constrained by the broader government context in which it is required to operate. These type of constraints include:

• The current three year timeframe for funding projects. A longer funding timeframe for projects (for example 5 years) would allow more time to focus on longer-term research goals, demonstrate program results, and assess the incremental value of continuing the activities in subsequent rounds of funding.
• Data storage and computation capacity constraints. The evaluation identified some constraints with respect to available data storage and computation capacity. As well as challenges in procurement timelines associated with acquisition processes. This is particularly affecting areas of research related to genetics which is producing more and more large and complex datasets.

The evaluation found that STB has developed performance measurement and data collection mechanisms such as the Science Management Solutions (SMS) to track the implementation and outcomes of Stream A projects. These tools have improved over time and are useful for communicating project results. Despite the progress to date, some issues were identified with the performance measurement data such as limited and missing information on some outputs and outcomes, and different interpretations of output and outcome categories. Furthermore, it is difficult to measure the intermediate and end outcomes based on the existing performance indicators.

5.2 Recommendation and management response and action plan

The key issue identified and recommendation resulting from the evaluation is:

Issue #1:

The evaluation found that performance measurement requires more details about the results of projects and the importance of the impacts in terms of the discoveries, innovations and developments. Training guidelines and quality assurance protocols are not sufficient to assist scientists in reporting project impacts.

Recommendation #1:

In line with the new Policy on Results and the Government of Canada results and delivery agenda, AAFC should continue to refine the program's approach to performance measurement to ensure that the indicators used and the performance data collected can accurately and consistently report on the impacts of scientific research activities.

Management response and action plan

Agreed

1. STB is revising all its sector strategies to include logic models to better align with the results & delivery agenda.

   Target Date: June 30, 2017

   Responsible Position: Director General, Partnerships and Planning Directorate, Science and Technology Branch.

2. STB will create Program Information Profiles (PIPS) for all programming as per the new Results and Delivery Agenda for the Government of Canada.

   Target Date: November 1, 2017

   Responsible Position: Director General, Partnerships and Planning Directorate, Science and Technology Branch.

Appendix A: Program logic model

The following logic model describes the linkages between the Stream A: Research Accelerating Innovation Program activities, outputs and intended outcomes. It also demonstrates a partial contribution to the departmental Program Activity and Strategic Outcome in the Program Alignment Architecture. It should be noted that the expected results of the program end at the immediate outcome level.

Figure 3: Logic Model for Stream A: Research Accelerating Innovation

Activities	Outputs	Immediate Outcomes	Intermediate Outcomes	End Outcomes	Program Activities	Strategic Outcomes
Research activities related to technologies, processes, products, and services for the production of sustainable, safe agri-food & agri-based products.	Reports related to: addressing environmental sustainability challenges new products, processes, technologies relevant to: improving the productivity and/or competitiveness of the sector, minimization of catastrophic threats to crops, optimization of livestock efficiencies, and food health and safety are available for transfer	New products, processes, technologies, relevant to the minimization of catastrophic threats to crops, optimization of livestock efficiencies, and food health and safety are available for transfer New beneficial management practices relevant to environmental sustainability are available for transfer. Program knowledge is disseminated to stakeholders	Agriculture and agri-food sector that utilizes science to improve or transform commodities into new value-added products.	An innovative and sustainable agriculture, agri-food and agri-based products sector.	2013-14 3.1 Science, Innovation and Adoption 2014-18 2.1 Science, Innovation, Adoption and Sustainability	An innovative and sustainable agriculture, agri-food and agri-based products sector
Source: Stream A Program Performance and Risk Management Strategy.

Activities and outputs

Research activities in this initiative are related to the development of technologies, processes and products including research aimed at understanding agriculture's interaction with the environment. Other research activities include research relating to the mitigation of threats to Canadian agricultural production and addressing long-term challenges or opportunities facing the sector.

There are five distinct output areas. These areas are expected to generate project level reports on results which will generate peer reviewed publications related to the following:

• addressing environmental sustainability challenges and improving the productivity and/or competitiveness of the sector;
• minimizing catastrophic threats to Canadian crops and optimizing livestock efficiencies;
• addressing threats to the food value chain and developing potential mitigation strategies;
• the health attributes of various food components.

Expected Results

The knowledge generated by these outputs is intended to lead to the creation of technologies, processes, and products that are available for transfer to stakeholders. There are two immediate outcomes related to this transfer:

• New products, processes, technologies, relevant to the minimization of catastrophic threats to crops, and food health and safety are available for transfer to the sector;
• New beneficial management practices relevant to environmental sustainability and/or improved productivity are available for transfer to the sector.

The initiative is at the beginning stage of the science continuum in terms of measuring the horizontal movement of information from primary research scientists to outreach experts in AAFC and to other members of the science community. There is an immediate outcome related to measuring this horizontal movement:

• Outcome related to the transition of information from research scientists to outreach experts in AAFC and to other members of the science community.

Higher Order Outcomes

Research accelerating innovation is aligned precisely to AAFC's Program Alignment Architecture at the intermediate and end outcome levels. The expected results for Research Accelerating Innovation are a part of a collection of expected results that contribute to Program Activity 3.1 Science Innovation and Adoption and Strategic Outcome #3: An innovative agriculture, agri-food and agri-based products sector.

Appendix B: Detailed evaluation methodology

The evaluation included a comprehensive assessment of the Agri-Innovation Program Stream A: Research Accelerating Innovation, in accordance with the Treasury Board of Canada's Directive on the Evaluation Function (2009). The evaluation covers the period from April 1, 2013 to March 31, 2016. The evaluation was designed to address the following two evaluation issues: relevance and performance. Relevance examined whether there is a continued need for the program, its alignment with government priorities and federal roles and responsibilities. Performance assessed the extent to which the program has resulted in the achievement of expected outcomes and how efficiently and economically the program was delivered. Within each issue specific evaluation questions were explored, which are presented in the following table.

Evaluation Issues and Questions

Categories	Evaluation Issues and Questions
Relevance: Continued Need for Program	Does the programming implemented through Stream A: Research Accelerating Innovation address the needs of the agriculture and agri-food sector with respect to research and innovation? 1.1 What are the needs of the sector related to research and innovation? 1.2 In what manner and to what extent does the programming implemented through Stream A programs respond to the need? In what manner and to what extent does Stream A complement, duplicate or overlap other AAFC initiatives, and other programming designed to promote innovation in the agriculture and agri-food sector?
Relevance: Alignment with Government Priorities	How do the program's objectives align with the priorities of the federal government with respect to innovation and the strategic outcomes of AAFC?
Relevance: Alignment with Federal Roles and Responsibilities	To what extent is the program aligned with federal roles and responsibilities?
Performance: Achievement of Expected Outcomes	In what manner and to what extent has Stream A contributed to targeted immediate outcomes: Developed new products, processes, technologies to minimize catastrophic threats to crops, optimize livestock efficiencies, and increase food health and safety? Developed new beneficial management practices relevant to environmental sustainability? Disseminated information and knowledge developed/obtained through the program to sector representatives and other stakeholders? In what manner and to what extent has Stream A contributed to targeted intermediate outcomes: Supported agriculture and agri-food sector that utilizes science to improve or transform commodities into new value-added products? Advanced the knowledge in the science community in the areas of food safety and security, bioresources and efficient integrated production systems Contributed to sector capacity to produce, adopt, and commercialize innovative products, processes, practices, services and technologies In what manner and to what extent has Stream A contributed to targeted ultimate outcomes: Contributed to an innovative and sustainable agriculture, agri-food and agri-based products sector? Increased safety of the food systems? Are there any unintended outcomes, positive or negative, that can be attributed to the activities implemented under Stream A? What factors, internal and external to the program, contributed to and constrained achievement of the expected outcomes?
Performance: Efficiency and Economy	How efficiently and economically is the program being delivered? What are the opportunities to improve the program? Are there alternative delivery mechanisms that may produce the intended outcomes more efficiently or economically? To what extent are the performance measurement and reporting structures effective in reporting on the achievement of outcomes? How and to what extent is the performance information used?

The data collected for the evaluation was extracted and triangulated to address each of these questions. Data was analyzed to develop a summary response to each evaluation question. As part of this step, the strengths and limitations of each line of inquiry were taken into account to develop valid, reliable, and credible conclusions. A variety of data analysis techniques were employed to rule out alternative explanations of the results and enhance the generalizability of the specific causal linkages between the program components. Various statistical tests and methods were used to analyze the data such as frequency tables, cross tabulations, and correlation analysis.

The evaluation employed multiple lines of evidence drawing from primary data sources including interviews of collaborators, project proponents (AAFC scientists), and senior managers, in-depth case studies of particular projects, and site visits to AAFC research and development centres, and secondary data sources including a document, file and operational data review, and a benchmarking analysis.

Research and data collected from the various lines of evidence were used to develop supporting technical reports to contribute towards the evaluation of AAFC's AgriInnovation Program Stream A. Each data source is described in more detail in the following paragraphs.

1. Document, file and operational data review

An extensive review of internal and external documents was undertaken to collect information on the program relevance, design and delivery, and activities, outputs and outcomes. The methodology included an review of internal data and documents including administrative data from the SMS databases, program files such as the Performance Measurement and Risk Management Strategy, guidelines for letters of intent and proposals, departmental documents such as the Science and Technology Branch Plan and Science Sector Strategy, departmental performance reports and reports on plans and priorities. External documents reviewed included federal government Speeches from the Throne, the Minister of Agriculture and Agri-Food Mandate Letter and research on the relevance and need for innovation programming, among other documents.

2. Benchmarking analysis

A benchmarking analysis was undertaken of similar programs in Canada (for example, other federal and provincial programs) and other jurisdictions including the United States, Europe, the United Kingdom, France, Australia, Brazil and China. The analysis aimed to provide evidence to examine how efficiently and economically the program is being delivered as well as identify alternative delivery mechanisms, promising practices, and lessons learned. Similar programs were examined that reflect the key focus areas of each AgriInnovation Program Stream: Stream A: Research Accelerating Innovation; Stream B: Research, Development, and Knowledge Transfer; and Stream C: Enabling Commercialization and Adoption.

A literature and internet review was conducted to identify relevant programs and organizations and obtain the available information. Where available, information was gathered on program and administration costs, dollars leveraged from external sources,TEs employed to deliver the program, outputs, outcomes, and other details on the design and delivery mechanism of the programs. In addition, a number of alternative delivery mechanisms and promising practices were identified. Programs were categorized by their focus area and compared to the relevant AIP Stream.

3. Interviews

A total of 48 interviews were conducted with key informants between March and June, 2016 to obtain input regarding the relevance and performance of the program activities. The number of interviews completed exceeded the targeted 40 interviews. Respondents included Stream A project proponents (AAFC research scientists and Research and Development Technology Leads), project collaborators, and AAFC senior management. Interviews were conducted with representatives in provinces across Canada. The following table provides the number of completed interviews per respondent group and a brief description of the respondents.

Description of Key Informants Interviewed

Target Group	Number Completed	Description of the Target Group
Stream A Project Proponents	37	The project proponent target group is comprised of AAFC research scientists. The roles of representatives interviewed was that of project lead or principal investigator of a research project. The scientists interviewed covered a range of expertise and diverse project subject matter. Most projects were complete. Projects represented a mix of Stream A sub-programs and sectors.
Stream A Project Collaborators	6	The collaborator target group is comprised of individuals involved or impacted by AAFC AgriInnovation research projects. The roles of these representatives interviewed include scientists in federal government agencies, a university professor, and a CEO of a private company.
AAFC Senior Management	5	STB directors and other senior AAFC staff.
Total	48	n/a
n/a = not applicable

The following steps were taken to complete the key informant interviews:

• Developed interview guides for each target group in consultation with the AAFC Evaluation Team.
• Developed a list of Stream A project proponents and senior managers. Individuals that had not been recently interviewed as part of other AAFC science evaluations (for example, Science 2.1.1) were given a priority. In addition, we asked participating project proponents to recommend key partners or collaborators.
• Sent all the collaborators and project proponents a letter of introduction from Ference and Company Consulting introducing the evaluation, and explaining the purpose and timing of the interviews. A relevant interview guide was attached to each email.
• Sent follow-up emails or contacted the representatives by telephone to schedule the interview at a time of their convenience in the official language of their choice or to encourage their completion of the questionnaire. Reminders were sent to all those who did not respond to the initial invitation and follow-up call or email.
• Conducted telephone interviews or obtained completed questionnaires via email.
• Compiled and analyzed the responses using Excel and synthesized the results for each evaluation question and indicator.

4. Case studies

Case studies of ten Stream A projects were conducted to gain details on the impact and need for the project, key factors which contributed to or constrained the project's success, and perspectives on program design and delivery. The projects were selected based on the following criteria:

• Mix of Stream A sub-programs;
• Mix of science sectors;
• Mix of primary project locations;
• Projects that have started implementation and are near completion or completed;
• Projects that have resulted in outputs to date which have been reported in the SMS project reporting database (for example, publications, media articles, and inventions or patents, etc.); and
• Projects identified during site visits that demonstrated unique or innovative components, promising practices, and major achievements.

To conduct the case studies, a case study interview guide and template were developed. Data collection for the case studies took place between March and May 2016 and involved interviews with project leads and participants (by telephone or in person during site visits) and a review of relevant project databases and documents (for example, media releases).

Of the 10 projects reviewed, 6 projects were part of the Crops Livestock and Food sub-program (3 of which focused on food safety) and 4 projects were part of the Sustainable Science and Technology Advancement sub-program. The case study projects represented a mix of science sectors as well including 5 Agro-Envrionment Resiliency projects, 2 agri-food projects, 1 cereals and pulses project, 1 horticulture project, and 1 oilseeds project. Seven projects were completed and three were ongoing at the time of the case study review. Projects budgets represented approximately $6.1 million in funding and an average budget of $613,980. Projects had an average of 11 AAFC scientists participating in the implementation of the project activities as well as several technicians, students, postdoctoral researchers, and external collaborators. Primary project locations included Guelph (3), Brandon (2), London (2), Harrow (1), Kentville (1), and Ottawa (1); however 8 of the 10 projects also involved scientific activities at other AAFC research centres across Canada. In total, interviews were conducted with 13 representatives including 10 project leads and 3 collaborators and project participants.

Site visits

Site visits were made to six AAFC research and development centres including centres in Summerland, Agassiz, Harrow, London, Saint-Hyacinthe, and Lethbridge from March to May 2016. Centres were selected to represent a mix of eco-zones, sectors, and areas of scientific expertise. Centres which featured a higher number and value of Stream A projects were prioritized and centres which had already been visited recently as part of a recent evaluation of other STB programming were excluded. Some consideration was given to the proximity of some of the centres to each other (for example, London and Harrow). The purpose of the site visits was to obtain information concerning the research activities being undertaken at each research centre, with a particular focus on research activities related to Stream A: Research Accelerating Innovation, the factors contributing to and constraining success, and the major impacts and intended outcomes resulting from the activities.

Each site visit included discussions with senior management staff followed by group or individual interviews with scientists and a tour of the facilities. Information was assembled based on observations, documentation, and interviews with scientists and their teams. Associate Directors RDT facilitated meetings with the scientists and centre tours. Project leads for Stream A projects were invited to participate in interviews as part of the site visits and were provided with a discussion guide explaining the purpose of the evaluation and topics that would be covered. Students, postdoctoral researchers, and technicians were also interviewed during the tours. Interviews were conducted with a total of 44 representatives during the site visits, including 8 AAFC management and other staff (for example, Directors RDT, Associate Directors RDT, and Knowledge Technology Transfer staff), 30 AAFC scientists specializing in a variety of sectors and areas of scientific expertise, and 6 students, postdoctoral researchers and technicians. Interviews and data collected from six scientists during the site visits contributed to information for in-depth case studies of specific Stream A projects.