Evaluation of Science 2.1.1: Science Supporting an Innovative and Sustainable Sector

March 30, 2017

List of Abbreviations

AAFC

Agriculture and Agri-Food Canada

ADM

Assistant Deputy Minister

AIP

AgriInnovation Program

CAGR

Canadian Animal Genetic Resource

CCFC

Canadian Collection of Fungal Cultures

CNC

Canadian National Collection of Insects, Arachnids and Nematodes

DOA

National Collection of Vascular Plants

DOAM

Canadian National Mycological Herbarium

DNA

Deoxyribonucleic acid

IT

Information Technology

KTT

Knowledge Technology Transfer

NIS

National Identification Service

NPO

Non-pay Operating

OIPC

Office of Intellectual Property and Commercialization

PGRC

Plant Gene Resources of Canada

R&D

Research and Development

RDT

Research, Development, and Transfer

SMS

Science Management Solution

STB

Science and Technology Branch

U.S.

United States

Executive summary

Purpose of the Evaluation

The purpose of the evaluation is to assess the relevance and performance of Agriculture and Agri-Food Canada Science 2.1.1: Science Supporting an Innovative and Sustainable Sector (AAFC Science) sub-program in the Program Alignment Architecture^{Footnote 1}. The following areas were also assessed where they support AAFC Science activities: Collections, International Engagement Division and the Office of Intellectual Property and Commercialization (OIPC). The evaluation covers the period from April 1, 2009 to March 31, 2016.

Background

AAFC Science is an A-base program consisting of research, development, and technology and knowledge transfer projects led by AAFC science professionals. AAFC Science is administered by the Science and Technology Branch (STB) and is the largest Program Alignment Architecture sub-program within the Department encompassing much of the department's expenditures under the Strategic Outcome: an innovative and sustainable agriculture, agri-food agri-based products sector. AAFC Science supports research and development work to enhance the agriculture and agri-food sector's ability to increase agricultural productivity, improve environmental performance, improve the attributes of agricultural products for food and non-food uses, and address threats to the agriculture and agri-food value chain. The major activities supported by AAFC Science include:

• Maintaining an agricultural science capacity, including a network of 20 research and development centres across the country with scientific expertise, technical staff and specialized equipment and facilities to conduct agricultural science research;
• Maintaining several large national biological reference collections and gene banks which play a key role in agricultural science and research;
• Conducting scientific research, development and knowledge and technology transfer;
• Facilitating collaborative partnerships and international engagements; and
• Facilitating the transfer and commercialization of research results.

A total of $1.2 billion in funding was budgeted for AAFC Science activities from 2009-10 to 2014-15. The intended outcomes of the program are:

• Enhanced scientific knowledge with benefits for the agriculture and agri-food sector
• Use/value of activities related to collections and collaborative research agreements
• Dissemination and transfer of enhanced scientific knowledge to the scientific community
• Utilization of enhanced scientific knowledge, technologies and practices by the scientific community that ultimately contributes to the sector's ability to:
  • Increase agricultural productivity
  • Address threats to the agriculture and agri-food value chain
  • Improve environmental performance
  • Improve attributes for food and non-food uses

The scientific capacity provided by AAFC Science also contributes to impacts under other STB program areas, such as the AgriInnovation Program Stream A and B, through its support for scientist and other indeterminate staff salaries, infrastructure and equipment used in projects funded by these programs.

Key Findings

The key findings resulting from the evaluation are as follows:

Relevance

Research and development is essential for driving innovation and ultimately improving sector productivity, enhancing competitiveness, and capitalizing on opportunities such as changing demands for agri-food and agri-based products. AAFC Science supports innovation by conducting foundational research that lays the groundwork for more applied research activities undertaken in the AgriInnovation Program Stream A, as well industry-led research that is supported under AgriInnovation Program Stream B. AAFC Science activities are part of an integrated innovation continuum that is intended to focus foundational research on areas that ultimately support adoption and/or commercialization of new products and services. In order to identify specific sector needs, STB conducted internal and external consultations to develop nine sector strategies that guide funding decisions under AAFC Science and other related research and development programs. Three of the nine strategies^{Footnote 2} capture cross cutting challenges associated with the need to preserve biodiversity and improve environmental performance. The strategies are as follows:

1. Forage and Beef
2. Cereals and Pulses
3. Oilseeds
4. Horticulture
5. Dairy, Pork, Poultry and Other Livestock
6. Bioproducts (discontinued in 2016-17)
7. Agri-Food
8. Agro-Environment Resiliency(cross-cutting)
9. Biodiversity and Bioresources (cross-cutting)
10. Clean Technology (cross-cutting) (new in 2016-17)

AAFC Science objectives and activities are well aligned with federal government priorities and AAFC departmental strategic outcomes, particularly with respect to maintaining a strong science infrastructure, fulfilling international obligations to conserve genetic materials, and investing in agricultural science research, development and knowledge and technology transfer. The Mandate Letter to the Minister of Agriculture and Agri-Food (November 2015) emphasizes the government's plan to "invest in agricultural research to support discovery science and innovation in the sector"^{Footnote 3}. AAFC Science is aligned with AAFC's Strategic Outcome 2: An innovative and sustainable agriculture, agri-food and agri-based products sector, particularly in terms of generating new knowledge, fostering innovation, and improving the competitiveness and adaptability of the agriculture, agri-food and agri-based products sector. AAFC Science also aligns well with current federal government priorities. The federal government Budget 2016 specifically refers to planned investments in agricultural science infrastructure.

Delivering AAFC Science is an appropriate role for the federal government since the activities align with roles and responsibilities established in the Experimental Farm Stations Act (1985). AAFC Science plays an important role in maintaining the capacity to undertake agricultural science research, particularly foundational research with longer-term Canadian public and industry benefits. No other entity in Canada (for example, provincial government, industry, or universities) matches AAFC in terms of its facilities, equipment and breadth of expertise to support research in agricultural science. Furthermore, there are very limited incentives for industry to invest in foundational research due to the lack of near-term commercial returns, and little capacity for universities to engage in public good non-academic related scientific activities, such as Collections, that underpin much of the activities under AAFC Science.

Achievement of Intended Outcomes

AAFC Science has made a significant contribution to its intended immediate, intermediate and end outcomes:

1. AAFC Science has made substantial contributions to its immediate outcome of enhanced scientific knowledge with benefits for the agriculture and agri-food sector.

   The enhanced knowledge has resulted in numerous contributions to scientific publications and innovations with benefits for the agriculture and agri-food sector. Between 2013-14 and 2015-16, projects fully funded by AAFC Science resulted in 88 innovations and 1,508 scientific publications, which included peer reviewed scientific articles and papers, conference proceedings and book chapters.

2. The Collections and Collaborative Research Agreements have significant value.

   These activities support scientists and other stakeholders in a variety of ways such as classifying and preserving specimens for research purposes, identifying pests, invasives and biological threats to agricultural production systems, facilitating collaborations between AAFC researchers and external stakeholders, and assisting AAFC to protect the intellectual property generated from its research, and helping generate over $36 million in royalties during the period from 2009-10 to 2015-16. The royalties generated are a strong indicator that the sector is utilizing the research from AAFC Science, particularly related to cereals, pulses and oilseeds (as the majority of royalties are generated in these commodities).

3. AAFC Science has made substantial contributions to its intended intermediate outcome which is the dissemination and transfer of enhanced scientific knowledge to the scientific community and other stakeholders.

   Between 2013-14 and 2015-16, projects fully funded by AAFC Science resulted in the development of 49 technology transfer publications and 608 knowledge and expertise contributions. The most frequent types of technology transfer publications consisted of bulletins, newsletters and trade journal publications while the knowledge and expertise contributions included AAFC scientists speaking or presenting their findings, training personnel, students or other experts, hosting demonstration/field days, acting as a scientific or technical expert and conducting media interviews.

4. It is difficult to determine the full scope of impacts of the AAFC Science activities since science and innovation activities are collaborative in nature, require many years to achieve their intended long-term outcomes and the scope of the evaluation only considered AAFC Science activities in the past six years. However, the evaluation identified several examples of how AAFC Science is working towards these outcomes. The case studies of science projects provided examples of contributions towards:
   • increased agricultural productivity,
   • methods to address biological threats to the agriculture and agri-food chain,
   • improved environmental performance, and
   • Improved attributes of agricultural products for food and non-food uses.

Program Design and Delivery

The evaluation found that the following factors are contributing to the success of AAFC Science:

1. Long-term investments in scientific capacity

   Long-term investments in infrastructure and scientific expertise have been integral in driving research results and contributing to the reputation of AAFC as a world leader in agricultural science knowledge and expertise. Investments in equipment, infrastructure and technical support staff have facilitated AAFC research activities on a larger scale and over longer periods of time than most other agricultural science organizations in Canada including universities, provincial governments and the private sector. AAFC scientists have access to equipment and facilities, such as extensive laboratory, greenhouse, and field facilities that can be used for long-term field studies. They also have access to support staff including AAFC technicians that are highly skilled in specialized areas.

2. Multidisciplinary, collaborative approach to research

   AAFC Science's collaboration with a variety of partners both internally (with AAFC research facilities and experts across Canada) and externally (with universities, provincial and federal government departments, industry, and international partners) generates synergies and enhances the likelihood of significant discoveries. The extent to which AAFC scientists collaborate has increased in the past 15 years. According to a 2016 study that examined the scientific output and collaboration patterns of 13 federal government entities from 2000-2014, over 75% of AAFC's scientific publications were co-authored by researchers from another organization and AAFC more than doubled its international collaboration rate in the last 15 years^{Footnote 4}. This reflects an increase in research projects that examine issues involving different fields of research and large teams with complementary expertise from a variety of organizations.

3. Coherent management of science investments

   The centralization of AAFC Science management has helped to prioritize and direct resources more strategically. The creation of the STB in 2012, which integrated the former Research Branch and Agri-Environmental Services Branch, consolidated all AAFC's scientific expertise and has allowed for a more coherent, national perspective on science needs and priorities.

   AAFC Science activities are directed strategically to promote science excellence and align with priorities and gaps. Extensive consultations were conducted with internal and external stakeholders to develop nine sector science strategies, each with clearly identified priority areas. The alignment of AAFC Science activities with longer-term industry strategies helps to direct resources to priority areas and leverage industry investment in more applied industry research projects and clusters.

The evaluation found that there are some aspects of the AAFC Science design and delivery model which constrain the success of program activities. These include:

1. Information Technology (IT) and equipment

   The evaluation identified some constraints with respect to available data storage and computation capacity, as well as challenges in procurement timelines associated with acquisition processes. This is mainly affecting areas of research related to microbiology and genomics, which are producing larger and more complex datasets.

2. Performance measurement

   AAFC Science is working towards improved performance measurement. Processes have been established to support more centralized monitoring of science activities. Work plans have been developed for each sector science strategy with key indicators to further facilitate monitoring, evaluation, and planning. STB has a project information system (such as, Science Management Solution) to monitor project investments and outputs, and developed mechanisms to report on national progress against the sector strategies and work plans. Although this system is enabling project tracking, its ability to report on program-level outcomes is limited.

   The program requires a performance measurement strategy with output and outcome indicators and associated performance targets. Without this strategy, it is difficult to monitor annual progress and overall achievements against intended program objectives. Further, a clear articulation of how Science 2.1.1, AgiInnovation Streams A, B and C, and the Collaborative Framework work in concert with each other to support the innovation continuum in agriculture sector (such as, a program theory) would be very useful for telling the performance story.

Recommendation and Management Response and Action Plan

The key issue identified and recommendation resulting from the evaluation is:

Issue #1:

The evaluation found that although AAFC Science is monitoring research activities through SMS, it does not have a consistent approach to performance measurement which is limiting the ability to monitor scientific outputs and measure the impact of AAFC Science activities.

Recommendation #1:

In line with the new Policy on Results and the Government of Canada results and delivery agenda, AAFC should develop performance measurement approaches to ensure that STB can accurately and consistently report on the outputs and outcomes of AAFC Science.

Management Response and Action Plan

Agreed

1. STB is revising all its sector strategies to include logic models to better align with the results & delivery agenda.
   • Target Date: June 30, 2017
   • Responsible Position: Director General, Partnerships and Planning Directorate, Science and Technology Branch.
2. STB will create Program Information Profiles (PIPS) for all programming as per the new Results and Delivery Agenda for the Government of Canada.
   • Target Date: November 1, 2017
   • Responsible Position: Director General, Partnerships and Planning Directorate, Science and Technology Branch.

1.0 Introduction

The purpose of the evaluation is to assess the relevance and performance of the AAFC Science 2.1.1: Science Supporting an Innovative and Sustainable Sector sub-program in the Program Alignment Architecture. AAFC Science is a very large and complex sub-program that forms the underpinning of much of the scientific research conducted by AAFC. This includes supporting salaries and science equipment that are used to deliver other research and development programs such as AIP Stream A, B and the Collaborative Framework. As such, it is difficult to disentangle AAFC Science expenditures from other program areas. Therefore, the evaluation assessed the relevance of AAFC Science as whole, in terms of the department's capacity to conduct foundational research, and assessed performance based only on the results of projects that were fully funded by AAFC Science (such as, the performance analysis does not include projects where salaries were funded by AAFC Science and non-pay operating costs were funded under a separate program).

The following areas were also assessed where they support ongoing AAFC Science activities: Collections, International Engagement Division and the Office of Intellectual Property and Commercialization (OIPC). The evaluation does not include the Genomics Research and Development Initiative since this was evaluated through a horizontal evaluation led by the National Research Council Canada.

The evaluation was conducted as part of AAFC's five-year Departmental Evaluation Plan (2014-15 to 2018-19). The evaluation was undertaken by the Office of Audit and Evaluation, with support from Ference and Company.

2.0 Program Profile

2.1 Overview of Program

AAFC Science (program area: 2.1.1: Science Supporting an Innovative and Sustainable Sector, residing under section 2.1: Science, Innovation, Adoption and Sustainability) is an A-base program administered by STB, the Department's largest branch. STB employees work in 20 research and development centres, sub-stations, and other offices throughout Canada (Figure 4). STB is divided-up among three regions and one directorate:

• Coastal Region
• Prairie Region
• Ontario-Québec Region
• Partnerships and Planning Directorate

Description of above image

Figure 1: Science and Technology Branch Research and Development Centres by Region.

Region	Province	City
Coastal	British Columbia	Summerland
Coastal	British Columbia	Agassiz
Coastal	Newfoundland	St. John's
Coastal	Prince Edward Island	Charlottetown
Coastal	Nova Scotia	Charlottetown
Coastal	Nova Scotia	Kentville
Prairie	Alberta	Lacombe
Prairie	Alberta	Lethbridge
Prairie	Saskatchewan	Saskatoon
Prairie	Saskatchewan	Swift Current
Prairie	Manitoba	Brandon
Prairie	Manitoba	Morden
Ontario-Quebec	Ontario	Harrow
Ontario-Quebec	Ontario	London
Ontario-Quebec	Ontario	Guelph
Ontario-Quebec	Ontario	Ottawa
Ontario-Quebec	Quebec	St-Jean-sur-Richelieu
Ontario-Quebec	Quebec	Sherbrooke
Ontario-Quebec	Quebec	Saint-Hyacinthe
Ontario-Quebec	Quebec	Quebec

AAFC Science is the largest sub-program within the Program Alignment Architecture encompassing much of the department's expenditures under the Strategic Outcome: an innovative and sustainable agriculture, agri-food agri-based products sector. AAFC Science supports foundational research and development work to advance the agriculture and agri-food sector's resiliency and enhance understanding of the resource base upon which agriculture depends, threats to Canadian agriculture production, mechanisms to protect and conserve Canadian bio-resources and genetic diversity, and new opportunities for the sector. It also supports salaries and science equipment used by other research and development programs such as AIP Stream A and B and the Collaborative Framework.

2.1.1 Sector Strategies and Strategic Objectives

AAFC Science activities are guided by nine sector strategies^{Footnote 5} and four strategic objectives. The strategies outline STB's objectives and focus areas, provide a framework for scientists to propose areas of work, and describe the role STB will play in relation to, and in collaboration with, other organizations^{Footnote 6}. Seven of the sector strategies are commodity-focused, encompassing science activities with respect to:

• 1. Forage and Beef
• 2. Cereals and Pulses
• 3. Oilseeds
• 4. Horticulture
• 5. Dairy, Pork, Poultry and Other Livestock
• 6. Bioproducts (discontinued in 2016-17)
• 7. Agri-Food

Three other strategies capture cross-cutting agricultural challenges:

• 8. Agro-Environment Resiliency
• 9. Biodiversity and Bioresources
• 10. Clean Technology (new in 2016-17)

Each science strategy is framed around the following four objectives^{Footnote 7}:

• Increasing agricultural productivity, which includes research that increases the yield potential of crops, improves feed efficiency for livestock, and decreases the yield gap (such as, the gap between potential yield and realized yield) by addressing biotic stresses (for example, weeds, insects and diseases), abiotic stresses (for example, nutrients, water, cold, salinity, soil structure, heat), and system-level productivity (for example, crop rotation, pasture systems). This area of research also includes finding suitable uses for marginal lands and using agricultural biodiversity for greater economic sustainability.
• Improving environmental performance, which includes research focused on ways to improve the efficiency of nutrient utilization and recycling, enhance integrated pest management practices, improve management of water resources and the energy sustainability of agricultural practices, mitigate greenhouse gas production, improve soil quality, and more generally reduce the environmental footprint of agriculture and agri-food production and processing.
• Improving attributes for food and non-food uses, which includes research in the food and feed area that examines nutrition and health promotion, feed quality for animal production, food quality for marketable traits and characteristics, shelf life and post-harvest storage, edible vaccines, food processing and packaging. Research in the area of non-food, or industrial applications encompasses biofuels and bioenergy, feedstocks for pharmaceuticals, biochemicals, and fibre, as well as biopesticides and biofumigants.
• Address threats to the value chain, which includes research that addresses significant production impacts related to biotic and abiotic stresses, and ensures food safety in agricultural production systems as well as processed and pre-packaged foods^{Footnote 8}.

The sector strategies are implemented through sector work plans and reports and guided by Sector Leads and Champions. Sector Leads (Director level) are responsible for planning, delivering and reporting on the sector strategies. Sector Champions (Director General level) provide high level input into the plans and strategies, and ensure they take a national perspective. Sector work plans explain objectives, activities and required resources to deliver the strategy.

2.1.2 Activities

AAFC Science activities include maintaining agricultural science capacity across the country, maintaining collections of preserved biological specimen and genetic resources, undertaking scientific research and development, and facilitating collaborative partnerships, international engagements, and the transfer and commercialization of research results.

Maintaining Agricultural Science Capacity

AAFC maintains a network of 20 research and development centres across Canada. The research and development centres are located in three regions, each with a unique ecological environment: Coastal Region, Prairie Region, and Ontario-Quebec Region. Each centre possesses scientific expertise, technical staff, and specialized equipment and facilities to conduct agricultural science research. This network helps STB to approach challenges facing the agriculture and agri-food sector in a holistic manner and provides STB with a coordinated interface with industry and other stakeholders in different regions across Canada^{Footnote 9}.

The infrastructure, facilities, and equipment vary at each centre. Research and development centres possess laboratories with specialized scientific equipment, research farms, livestock facilities, greenhouses, as well as other specialized facilities. Laboratories are assigned to a specific research scientist and his/her team, which may be composed of technicians, postdoctoral researchers and students working on short-term placements. Many research and development centres possess specialized labs (for example, analytical chemistry) and unique facilities that are used by AAFC scientists in multiple regions.

Maintaining Collections

AAFC currently has seven collections of over 19 million physical holdings of insects, plants, fungi, bacteria, nematodes as well as plant and animal reproductive material. The collections are managed by the Biodiversity and Collections Division that supports research and regulatory activities, by acquiring, documenting, and distributing germplasm and specimens nationally and internationally and providing information digitally. The National Biological Collections include the Canadian National Collection of Insects, Arachnids and Nematodes; National Mycological Herbarium; Canadian Collection of Fungal Cultures; National Collection of Vascular Plants; Plant Gene Resources of Canada; Canadian Animal Genetic Resources; and the Canadian Plant Virus Collection. The material and information contained in these collections enable public and private research that benefits the economy and trade, food and agriculture, public health and food safety, monitoring of invasive species and food security. These collections are also the foundation for essential research and development activities that help the agricultural sector adapt to changes in climate, pests and diseases. This continued effort also provides fundamental knowledge and materials to support the mandate of organizations such as the Canadian Food Inspection Agency, the Canadian Forestry Service and the Canada Border Services Agency through the National Identification Service (NIS), and the availability of digitized data and information related to the specimens^{Footnote 10}.

Scientific Research Projects

AAFC Science plays a prominent role in foundational research where public investment is needed to support further innovation in the sector. This type of research often involves activities that provide a better understanding of the challenges facing the resource base and the interactions across the production system. Examples of the focus of AAFC Science's fundamental research include soil, water and other inputs to agricultural production; inputs to animal nutrition, meat and dairy production; and the transformation of agricultural production to food.

Most AAFC Science project funding is distributed through an application process. AAFC requests letters of intent and detailed proposals from AAFC scientists for targeted research on an annual basis. A single call process is used for all AAFC-funded research, development, and technology and knowledge transfer projects led by AAFC scientists, including AAFC Science, other targeted activities funded through Growing Forward 2 such as AgriInnovation Program Streams A and B, and through STB's Collaborative Framework. Applications are evaluated by internal and external reviewers at different stages, based on specific criteria and funding priorities. In some cases, projects are directed by management, for example, if they address an urgent sector or stakeholder need. Typically, priorities are identified in the summer, calls for letters of intent are issued in the fall, detailed proposals are solicited in the winter, and final funding decisions are made in March for projects to start April 1st. Projects are normally supported for one to three years in duration. In cases where projects are longer than 3 years, a new project proposal is required. This provides a gating process to ensure continued relevance and appropriate funding levels.

Facilitating Collaborative Partnerships and International Engagements

The International Engagement Division facilitates collaboration between AAFC Scientists and international partners that include industry, academia, other governments and non-government organizations^{Footnote 11}.

Facilitating Collaborative Research and Commercialization of Results

The OIPC supports the AAFC scientific community by providing advice on collaborations, publication, technology transfer and the management of intellectual property. The OIPC negotiates and prepares various agreements to facilitate research collaboration, such as collaborative research agreements, confidentiality agreements, material transfer agreements, research support agreements, and license agreements for AAFC technologies and plant varieties. The OIPC also evaluates invention disclosures; develops intellectual property protection strategies; and manages the AAFC portfolio of patents, trademarks and official marks, and royalty payments on AAFC licenses^{Footnote 12}.

Facilitating Knowledge Transfer

Following the creation of the Science and Technology Branch in 2012, Knowledge and Technology Transfer (KTT) officers were identified in each region to give the agriculture and agri-food sector easier access to knowledge acquired and technologies developed by STB researchers. KTT staff undertake various activities such as attending industry meetings, organizing knowledge-sharing events, developing databases of knowledge and technologies, and coordinating with KTT offices across the country^{Footnote 13}.

2.1.3 Target Groups and Stakeholders

Only internal AAFC employees are eligible to participate in AAFC Science (A-base) activities as funding recipients. Funding is available for research in other organizations, such as universities and industry partners, through other programs such as AgriInnovation Stream B. AAFC Science funding is intended to support foundational and public good research that other organizations are not doing and that is not funded through other programs. Researchers in other organizations may participate in project activities as collaborators. External stakeholders include Canadian producers, industry associations related to agricultural and agri-food products, processes and services, representatives of universities and academia, and other federal and provincial government representatives involved in agriculture and agri-food-related research or regulation.

2.1.4 Governance Model

STB is led by an Assistant Deputy Minister (ADM) and an Associate ADM. Each of the three geographic regions is led by a Director General Research, Development and Transfer (RDT) (such as, Coastal Region, Prairie Region, and Ontario-Quebec Region). Directors RDT are responsible for managing science activities in these regions. The Partnerships and Planning Directorate is led by a Director General who is responsible for providing strategic direction, investment planning, IT planning, HR planning, overseeing the project proposal process, and science policy.

2.2 Program Resources

The overall expenditures for AAFC Science during the six-year period from fiscal 2009-10 to 2014-15 is approximately $1.2 billion. Annual program expenditures have declined by 35% from $235 million in 2009-10 to $152 million in 2014-15. As indicated in Table 1, the major factors contributing to the decline in program expenditures are the merger of the Research Branch and the Agri-Environmental Services Branch to form the Science and Technology Branch, and the phasing out of some programs.

Fiscal Year	Salaries	Non Pay Operating*	Total Expenditures	Notes
2009-10	$172,581,037	$61,919,538	$234,500,576	n/a
2010-11	$172,381,118	$47,292,222	$219,673,340	Reduced Non-pay operating (NPO) due to termination of Agri-Bioproducts Innovation Program and phasing out of Matching Investment Initiative
2011-12	$166,727,199	$42,233,128	$208,960,328	Reduced NPO and salary due to sunsetting of Action Plan funding
2012-13	$172,764,634	$45,943,793	$218,708,426	Transition funding to support merger of AgriEnvironmental Science Branch and Research Branch into STB
2013-14	$134,006,075	$46,550,138	$180,556,214	Reduced salary due to government measures to increase efficiency (271 fewer FTEs)
2014-15	$113,544,034	$38,576,628	$152,120,662	Reduced NPO and salary due to structural transformation (175 fewer FTEs)
Total	$932,004,098	$282,515,448	$1,214,519,546	n/a
Source: AAFC Corporate Finance, SAP n/a = not applicable * NPO includes materials and supplies, professional services, scientific and technical equipment, travel, repairs and maintenance, training and other expenses. Note: Vote 1 expenditures have been restated to reflect the current Science Supporting an Innovative and Sustainable Sector PAA using the crosswalks available and historic information. This information is not comparable to the DPR information that includes other Branches

In 2015-16 AFFC Science funded 1,575 Full-time equivalent employees, which includes AAFC indeterminate employees, visiting fellows (roughly 150 in STB as of December 31, 2015) and (an average of 160 researchers in STB under the AAFC's Foreign Research Participant Program). STB is also one of the largest employers of students in the federal public service, with roughly 800 students hired per year, many of which are graduate students.

3.0 Evaluation Methodology

3.1 Evaluation Scope

The evaluation included a comprehensive review of the relevance, performance and design and delivery of AAFC Science, in accordance with the Treasury Board of Canada's Policy on Results. The evaluation covers the period from April 1, 2009 to March 31, 2016.

3.2 Evaluation Methodologies

The evaluation employed the following lines of evidence to address the evaluation questions:

• A document, file and database review was conducted to collect information on AAFC Science inputs, activities, outputs and outcomes, including a review of internal data and documents such as administrative data from the Science Management Solution (SMS) databases, departmental documents such as Departmental Performance Reports and Reports on Plans and Priorities, and financial and full-time equivalent (FTE) data. External documents reviewed included evaluations of similar science programs, studies examining the impact of networking and collaboration in science, federal government documents and studies on the relevance and need for government-funded science programming.
• A literature review was conducted to compare the design and delivery of AAFC Science with similar programs in Canada and other countries. Profiles were developed on how countries such as the United States, the United Kingdom and Australia are organized to undertake agricultural research. A comparative analysis was conducted including an analysis of strengths, weaknesses, opportunities, and threats impacting the agricultural research and innovation system in each region.
• Stakeholder interviews were completed with 68 representatives including 12 AAFC senior managers and program staff, 37 AAFC scientists, four technicians and 15 external collaborators (for example, university, provincial government, and industry partners). The purpose of the interviews was to obtain input regarding the relevance, performance and design and delivery of AAFC Science. Contact lists were developed to ensure a mix of early, mid and late career AAFC scientists, areas of specialization and geographic regions throughout Canada. Interviews were conducted by telephone between May 2015 and April 2016.
• Case studies were conducted of AAFC research and development centres in Guelph, Sherbrooke, Fredericton, Charlottetown, Swift Current and Saskatoon. The purpose of the case studies was to collect detailed information on the activities and impacts of AAFC Science to address evaluation questions related to program performance, design and delivery. The case studies were selected to ensure a mix of strategic objectives, sector science strategies, and geographic locations. Each case study focused on the major science and innovation achievements arising from AAFC Science activities within AAFC research and development centres with respect to specific sector science strategies and strategic objectives.
• Site visits and focus groups were conducted at AAFC research and development centres in Guelph, Sherbrooke, Fredericton, Charlottetown, Swift Current, Saskatoon, Ottawa, and Vineland^{Footnote 14} between June 2015 and March 2016. The site visits were selected to represent a mix of focus areas and geographic locations. The purpose of the site visits and focus groups was to gather evidence for the case studies and to obtain evidence with respect to the relevance, design and delivery of AAFC Science activities. Each visit consisted of interviews with research centre management personnel followed by focus groups or individual interviews with scientists and a tour of the facilities. Interviews were conducted with a total of 60 representatives, including 11 AAFC research management staff; 37 AAFC scientists; seven students, post docs and technicians; and five collaborators.

3.3 Evaluation Limitations

The most significant limitation for this evaluation is the long term nature of scientific impacts. Given the 6 years scope of the evaluation is difficult to assess the extent to which AAFC is achieving its outcomes. Furthermore, it is difficult to isolate the contribution of AAFC Science to outcomes as a result of the integrated nature of research and development programs at AAFC (such as, AgriInnovation, AAFC Science and projects funded through STB's Collaborative Framework contribute to similar outcomes that are difficult to separate for reporting purposes). To mitigate these limitations the evaluation focussed on projects that were fully funded by AAFC Science and conducted case studies on projects that have shown early impacts and serve to demonstrate the ability of science projects to achieve long term outcomes.

4.0 Evaluation Findings

4.1 Relevance

4.1.1 Continued Need for Program

Research and development is essential for driving innovation and ultimately improving sector productivity, enhancing competitiveness, and capitalizing on opportunities such as changing demands for agri-food and agri-based products. AAFC Science supports innovation by conducting foundational research that lays the groundwork for more applied research and development activities undertaken in AgriInnovation Program Stream A, as well as industry-led research and development that is supported under AgriInnovation Program Stream B and through STB's Collaborative Framework. AAFC Science activities are part of an integrated innovation continuum that is intended to focus foundational research on areas that ultimately support adoption and/or commercialization of new techniques, practices, processes, products and services.

The evaluation found that AAFC Science is generally addressing the needs and priorities of the agriculture and agri-food sector. Extensive consultations were conducted with internal and external stakeholders to develop the nine sector strategies and each strategy has clearly identified priority areas. Funding envelopes are established to address these key priorities and gap areas. AAFC Science funding is directed to areas that are less likely to be funded by industry in order to fill the gaps. The following sub-sections provide a discussion of some of the broader challenges the sector in facing.

Sector Productivity and Competitiveness Challenges

The Canadian agriculture and agri-food sector faces a variety of challenges impacting its productivity and competitiveness. Variable input and commodity prices, influenced by world market and exchange rate shifts, threaten the profitability of the agriculture and agri-food sector in Canada. Sector input costs increased by about 47% from 2004 to 2014.^{Footnote 15} Canada's ability to supply agricultural products is also impacted by environmental threats such as climate change, which has increased the severity and frequency of extreme weather, particularly in semi-arid regions, and the nature of crop pests and diseases. Extreme events, like the 2001 and 2002 droughts and floods of 2010 and 2011, can have a devastating impact on crop yields, reducing yields by as much as 50%.^{Footnote 16} Demand for agricultural exports is impacted by steep global competition, particularly from emerging exporters such as Brazil, which recently outpaced Canada as the largest agricultural power behind the U.S. and European Union.^{Footnote 17} Demand is also affected by factors in the global economy such as global financial crises, political instability and sanctions in export destination countries.

Vulnerable Biodiversity

The preservation of biodiversity is a major concern for agricultural production. There is a need to preserve and maintain the genetic diversity of Canadian agriculture through collections of seeds and genetic materials to repopulate particular varieties or breeds if there is a major disease outbreak, reintroduce particular plant or animal material which align with consumer preferences, characterize beneficial attributes of different plant varieties and breeds, and fulfill international obligations. Canada is a signatory to the Food and Agriculture Organization of the United Nations International Treaty on Plant Genetic Resources for Food and Agriculture. One obligation under this Treaty is to preserve the genetic diversity of crop plants, their wild relatives and plants present and unique in the Canadian biodiversity in the Plant Genetic Resources of Canada genebank and make it accessible to the appropriate users.

AAFC Science helps to conserve biodiversity by maintaining a variety of collections of preserved specimens such as the Canadian National Collection of Insects, Arachnids and Nematodes; the National Mycological Herbarium; the National Collection of Vascular Plants; as well as live genetic resources such as the Canadian Collection of Fungal Cultures; the Plant Gene Resources of Canada; the Canadian Animal Genetic Resources; and the Canadian Plant Virus Collection. The material and information contained in these collections enable public and private research that supports improved sector productivity, public health and safety, and the monitoring of invasive species as well as enabling the Government of Canada to meet its international obligations.

Opportunities for New Markets and Products

The Canadian agriculture and agri-food sector faces several opportunities that could benefit from agricultural science activities. For example, global demand for agricultural products is expanding due to growing populations, higher incomes, urbanization and changing diets^{Footnote 18}. Consumption of agricultural commodities will increase most rapidly in emerging markets. As an illustration, China's imports of oilseeds are expected to rise by 40% over the next ten years, accounting for 59% of global trade^{Footnote 19}. Canada's export potential to these expanding markets is increasing due to an increase in the number of bi-lateral trade agreements, which currently cover 38 countries and 44% of the world's agriculture and agri-food markets^{Footnote 20}. There is also an increasing demand for value-added agricultural products, foods with specific attributes (for example, related to food safety, nutrition, environmental stewardship, animal welfare, and fair trade) and non-food products such as biodiesel. Rising global temperatures present an opportunity to expand agricultural production in northern regions of Canada. In addition, evolving technologies such as bioengineering, precision agriculture, remote sensing, and decision modeling present opportunities to research new ways to improve productivity and develop new products.

4.1.2 Alignment with Federal Government Priorities and Agriculture and Agri-Food Canada Strategic Outcomes

AAFC Science objectives and activities are well aligned with federal government priorities and AAFC departmental strategic outcomes, particularly with respect to maintaining a strong science infrastructure, fulfilling international obligations to conserve genetic materials, and investing in agricultural science research and development.

The recent Mandate Letter to the Minister of Agriculture and Agri-Food emphasizes the government's plan to "invest in agricultural research to support discovery science and innovation in the sector."^{Footnote 21} The federal government Budget 2016 also specifically refers to planned investments in agricultural science infrastructure.^{Footnote 22} A total of $37.6 million in new investments has already been committed to support infrastructure projects at 10 AAFC research and development centres, including the installation of a bioreactor to help investigate processes for treating farm waste, the renovation of a research and pilot plant facility to improve food safety and food processing, the installation of new efficient growth chambers in several centres, as well as several roof replacements, building expansions and modernizations.^{Footnote 23} Although these investments are not within the evaluation scope, they indicate alignment between AAFC Science and current government priorities.

AAFC Science is also aligned with AAFC's Strategic Outcome 2: An innovative and sustainable agriculture, agri-food and agri-based products sector, particularly in terms of generating new knowledge, fostering innovation, and improving the competitiveness and adaptability of the agriculture, agri-food and agri-based products sector.^{Footnote 24} The program activities also assists AAFC to meet its commitment to provide expertise and other support for Canada's participation in the Commission on Genetic Resources for Food and Agriculture and International Treaty on Plant Genetic Resources for Food and Agriculture, as indicated in the AAFC Departmental Sustainable Development Strategy 2014-15. Treaty objectives include conserving plant genetic resources essential for food security, providing access to them and using them in a sustainable manner.^{Footnote 25}

4.1.3 Alignment with Federal Government Roles and Responsibilities

Delivery of AAFC Science is an appropriate role for the federal government since activities align with the roles and responsibilities in the Experimental Farm Stations Act (1985), which states that the duties of research and development centre staff are to conduct "experiments and research bearing on the agricultural industry of Canada."^{Footnote 26}

No other entity in Canada (for example, provincial government, industry, or university) matches AAFC in terms of its facilities, equipment and breadth of expertise to support research and development in agricultural science. Furthermore, there are very limited incentives for industry to invest in foundational and public good research and development due to the lack of near-term commercial returns, and little capacity for universities to engage in public good non-academic related scientific activities, such as Collections, that underpin much of the activities under AAFC Science.

4.2 Performance

This section provides the evaluation findings regarding the extent to which AAFC Science has achieved its intended immediate, intermediate and end outcomes.

4.2.1 Immediate Outcomes

Outcome #1: Enhanced Scientific Knowledge with Benefits for the Agriculture and Agri-Food Sector

AAFC Science has made substantial contributions to its immediate outcome of enhanced scientific knowledge with benefits for the agriculture and agri-food sector. This enhanced knowledge has resulted in numerous contributions to scientific publications and innovations with benefits for the agriculture and agri-food sector.

As indicated in Table 2, between 2013-14 and 2015-16, AAFC Science projects (projects funded exclusively with A-Base Science Supporting an Innovative and Sustainable Sector funding) resulted in 88 innovations.^{Footnote 27} The most frequent types of innovations are new plant varieties; innovative processes, systems or methodologies; new genetic materials; and software tools. The sector portfolios that generated the highest number of innovations are biodiversity and bio-resources followed by horticulture.

Type	Number
Plant Variety	27
Process, Systems or Methodologies	17
Genetic Material	12
Software Tools	10
Designs	7
Dataset/Database	5
Other	10
Total	88
Source: AAFC Databases, Extracted April 18 2016. Includes outputs for project fully funded by Science Supporting an Innovative and Sustainable Sector (A-Base) between 2011-12 and 2015-16.

Portfolio	Number
Biodiversity and Bioresources	26
Horticulture	24
Dairy, Pork, Poultry and other Livestock	12
Other (N/A or corporate initiatives)	10
Agro-Environment Resiliency	8
Cereals and Pulses	5
Beef and Forage	2
Agri-Food	1
Total	88
Source: AAFC Databases, Extracted April 18 2016. Includes outputs for project fully funded by Science Supporting an Innovative and Sustainable Sector (A-Base) between 2011-12 and 2015-16.

As indicated in Table 3, between 2013-14 and 2015-16, 1,508 scientific publications have been associated with AAFC Science in SMS. This number of publications is an under-representation, given that associating publications with projects in SMS is optional and that STB has generated significantly more publications than are associated with specific projects in SMS. The scientific publications consisted largely of peer reviewed scientific articles or papers, conference proceedings, and book chapters. Note that there is a lag between research and publication and therefore the number of-publications may be under-represented in this report. Further, many projects are only partially supported by AAFC Science (for example for many projects salaries are funded through AAFC Science and Non-Pay Operating costs are funded through other programs), and their products are not included in these results. The sector portfolios that generated the highest proportion of publications included Biodiversity and Bioresources, followed by Horticulture and Agro-Environment Resiliency.

Type	Number
Article/Paper	763
Conference Proceedings	271
Chapter in Book	111
Conference Poster	97
Conference Abstract	110
Presentation Materials	89
Report	29
Book	13
Journal Abstract	15
Other	10
Total Scientific Publications	1,508
Source: AAFC Databases, Extracted April 18 2016. Includes outputs for project fully funded by Science Supporting an Innovative and Sustainable Sector (A-Base) between 2013-14 and 2015-16.

Portfolio	Number
Biodiversity and Bioresources	541
Horticulture	325
Agro-Environment Resiliency	278
Beef and Forage	74
Cereals and Pulses	91
Bioproducts	32
Dairy, Pork, Poultry and other Livestock	43
Agri-Food	9
Oilseeds	13
Total Scientific Publications	1,508
Source: AAFC Databases, Extracted April 18 2016. Includes outputs for project fully funded by Science Supporting an Innovative and Sustainable Sector (A-Base) between 2013-14 and 2015-16.

According to several of the collaborators interviewed, AAFC's level of expertise is very high and compares very favourably with the research being conducted in other countries. In some cases, AAFC scientists are considered leaders in their area of research and are well-respected in the international scientific community.

Outcome #2: Use/Value of the Collections and Collaborative Research Agreements

The activities related to collections and collaborative research agreements have been of significant value. These activities have supported scientists and other stakeholders in a variety of ways such as identifying, classifying and preserving specimens for research purposes, facilitating collaborations between AAFC researchers and external stakeholders, and assisting AAFC to protect the intellectual property generated from its research, and generating over $36 million in royalties from 2009-10 to 2015-16.

Collections

AAFC's non-living collections provide biosystematics information (identification, classification, genetic relatedness) on insects, arachnids, nematodes, weeds, poison plants, alien/invasive species, native biodiversity, fungi and bacteria of importance to Canadian agriculture. The plant and animal genebanks help Canada meet international commitment to preserve genetic resources and helps meet Canada's obligation under the International Treaty for Plant Genetic Resources for Food and Agriculture and the Commission of Genetic Resources for Food and Agriculture, both under the Food and Agriculture Organization. Major research activities have included characterizing agriculture-related macro and micro-organisms, developing new preservation methodologies, identifying biological strategies to control invasive species and contributing to the development of models to mitigate threats to agricultural production systems. As indicated in Table 4, most of the collections are the largest of their kind in Canada. Several collections have a long history of building their inventory and accumulating collections of specimens from Canada and around the world that would be very costly to rebuild to the same degree today.

Collection	Description
Plant Gene Resources of Canada (PGRC)	World base collections of barley (39,000 accessions) & oats (28,000) World leader in genetic resources conservation related to flax and oats* 110,444 accessions of crops from Canada and around the world Established in 1970
Canadian Animal Genetic Resource (CAGR)	Only public livestock and poultry germplasm collection of its kind in Canada available for industry breeding and research
National Vascular Plants Collection	One of the largest vascular plant collections in Canada and serves plant systematics worldwide 1.6 million+ specimens of vascular plants
National Mycology Herbarium	Largest fungarium of non-lichenized fungi in Canada 350,000+ fungal and fungal plant disease specimens Documentation of plant diseases and Canadian fungi spanning 100 years
Canadian National Collection of Insects, Arachnids and Nematodes	Largest collection of its kind in Canada 17 million+ specimens of insects, arachnids and nematodes Established in 1886
Canadian Collection of Fungal Cultures	Major national collection (repository and distributor) for fungal genetic resources 17,000+ living strains of fungal cultures
Note: More information on AAFC Science Collections can be found on the AAFC's research centres website. * AAFC. 2015. Prairie Region: Overview and Centre Profiles – Saskatoon Research and Development Centre (December 2015).

AAFC Science collections serve various purposes such as responding to biosecurity, environmental, and food safety risks; protecting genetic diversity in Canadian livestock and crops (for example, reintroducing particular species if there were a disease outbreak or responding to changes in consumer preferences); improving the competitiveness of the sector through research (for example, with respect to crop disease resistance and productivity); and meeting international obligations with respect to the preservation, maintenance, and exchange of various agricultural collections, materials and resources. Some examples of the contributions of AAFC biological collections are as follows:

• Identifying, classifying and preserving specimens. The National Identification Service (NIS) is the portal through which specimens of fungi, vascular plants, insects, arachnids, nematodes and their relatives can be submitted to the taxonomists at the Canadian Collection of Fungal Cultures (CCFC), Canadian National Mycological Herbarium (DAOM), National Collection of Vascular Plants (DAO) or Canadian National Collection of Insects, Arachnids and Nematodes (CNC) to be identified. As such, the NIS is one of Canada's first lines of defense in protecting its environment and biological resources, as the identification and classification of organisms is key to understanding our native biodiversity and threats that may destabilize it.
• Acquiring and sharing specimens. PGRC regenerates and characterizes an average of 3,000 crops and crop wild varieties annually. CAGR cryopreserves semen, ova, embryos (germplasm), and DNA of domestic livestock and poultry. PGRC ships an average of 4,200 samples within Canada and to over 60 partner countries, and acquires an average of 870 samples annually. Supplying information and loans worldwide as part of an international network of preserved specimen collections (of which DAO, DAOM and the CNC belong) enables AAFC scientists to reciprocally borrow and exchange material, and obtain information for research relevant to Canadian agriculture.
• Assessing disease resistance, genetic characteristics, molecular diversity, and archeological origins of different crops. For example, PGRC facilitated research into the genome size in 99 accessions from 26 species of genus Avena, which generated new insights in size of A, B, C, and D genomes. PGRC has also undertaken extensive disease resistance screening since 2013 of wheat, oat and flax varieties providing useful information for breeders.
• Developing new methods for discovery, collection, growth and preservation of species and germplasm, such as the development of a research agri-environmental bacterial collection to support AAFC research, regulation and biosecurity, and support research on quarantine, assessment and release. CAGR has contributed to improved methodologies for the preservation of chicken, bison, and swine germplasm, which has generated industry interest.
• Conducting biosystematics and environmental metagenomics research. Biosystematics research can include research on organisms critical to Canadian agriculture such as crop plants, weeds, pests, beneficial and biocontrol insects, and plant-growth-promoting bacteria. Environmental metagenomics exploits new advances in next generation sequencing and bioinformatics analysis for the complete profiling of microorganisms in agricultural ecosystems.
• Responding to emerging threats. The collections have facilitated the delivery of numerous authoritative identifications of quarantine and invasive plant species and weeds, invasive insect pests, nematodes, and fungal and bacterial diseases to Canadian Food Inspection Agency to prevent their spread or entry into Canada as well as reducing and preventing economic losses to industry from reduced exports. For example, the National Plant Collections helped to identify invasive weeds (such as Kudzu vine and European common reed), helping industry to save costs through early control measures.
• Establishing regulations. The presence of these collections associated with science expertise is essential for the establishment of fruitful collaborations with other federal government departments and agencies to address a variety of regulatory issues (for example, mycotoxins, allergens, plant genetically modified organisms, plant pests, quarantine organisms, importation of insects et cetera). Authoritative identifications provided through the National Identification Services provide support to regulations (for example, Plant Protection Act) and help to identify biological control agents and mitigation strategies.
• Maintaining and enhancing databases, for example, integrating morphological and molecular biosystematics (for example, DNA-barcoding, multigene phylogenies and comparative genomics through whole genome sequencing) and contributing the data to public resources such as GenBank and the Barcode of Life Database. Activities also involve modernizing facilities for biological collections and strategically developing biosystematics libraries to support research on economically important organisms and provide information to clients.

About half of the scientists interviewed made use of the AAFC Science collections.

Intellectual Property Activities

AAFC Science generated approximately $36.2 million in royalties from the innovations developed by AAFC scientists from 2009-10 to 2015-16. Canadian companies accounted for approximately 94% of AAFC Science royalty revenues while other sources included academic institutions, provincial governments, international governments and international companies. In terms of sectors, the largest sources of royalty revenues are Cereals and Pulses and Oilseeds.

About half of the scientists interviewed indicated they had worked with the Office of Intellectual Property and Commercialization. Some of these scientists reported they had worked with OIPC in the development of Material Transfer Agreements or Collaborative Research and Development Agreements. About a quarter of the scientists interviewed reported that their research had led to patents.

4.2.2 Intermediate Outcomes

Outcome: Dissemination and transfer of enhanced scientific knowledge to the scientific community

AAFC Science has made substantial contributions in the dissemination and transfer of enhanced scientific knowledge to the scientific community and other stakeholders through a variety of mechanisms such as publications, conference presentations, regional meetings and through various internal and external stakeholders. The knowledge generated by AAFC Science is also a critical input to other AAFC programs, such as AIP, and contributes to further innovation in the agriculture and agri-food sector.

AAFC Science projects (projects funded exclusively with A-Base Science Supporting an Innovative and Sustainable Sector funding) that were conducted between 2011-12 and 2015-16 resulted in a variety of knowledge dissemination and technology transfer activities in addition to the scientific publications described earlier. As demonstrated in Table 5, AAFC Science projects resulted in the development of 82 technology transfer publications. The most frequent types of technology transfer publications consisted of bulletins, newsletters and trade journal publications.

Type of Technology Transfer Publication	Number
Bulletin/Newsletter/Trade Journal Publication	36
Business Opportunity Documents	15
Market assessments for commercial potential of new technologies	10
Technologies posted on Flintbox, available for transfer	8
Factsheet/Flyer/Brochure	1
Web page	1
Other	11
Total Technology Transfer Publications	82
Source: STB SMS Databases, Extracted April 18 2016. Includes outputs for project fully funded by Science Supporting an Innovative and Sustainable Sector (A-Base) between 2013-14 and 2015-16.

During the period from 2013-14 and 2015-16, AAFC Science projects resulted in 608 knowledge and expertise contributions. As demonstrated in Table 6, the most frequent type of knowledge and expertise contributions are AAFC scientists speaking or presenting their findings; training personnel, students or other experts; acting as a scientific or technical expert; and conducting media interviews. Also note that the department has hosted a number of demonstration days to disseminating knowledge to broad audiences that are captured under the generic activity types in the table below. The three sector portfolios that resulted in the most knowledge dissemination activities are Biodiversity and Bioresources, Agro-Environment Resiliency, and Horticulture.

Type	Number
Speaker / Presenter	260
Training of Personnel, Students or Experts	83
Acting as a Scientific or Technical Authority / Expert	35
Media Interviews	31
Event Organizer	25
Manuscript Reviewer / Referee	23
Editor - Journal, Book, Conference Proceedings, Newsletter, et cetera	22
Executive Position - Professional Body, Scientific Society, et cetera	19
Member - Professional Body, Scientific Society, et cetera	18
Adjunct Professor	17
Grant Proposal Evaluation	12
University Teaching Involvement	11
Conference Technical Chair / Session Chair / Panel Expert	10
Acting as Representative / Delegate	4
Other	38
Total Knowledge and Expertise Contributions	608
Source: STB SMS Databases, Extracted April 18 2016. Includes outputs for project fully funded by Science Supporting an Innovative and Sustainable Sector (A-Base) between 2013-14 and 2015-16.

Portfolio	Number
Biodiversity and Bioresources	287
Agro-Environment Resiliency	123
Horticulture	79
Beef and Forage	62
Cereals and Pulses	21
Dairy, Pork, Poultry and other Livestock	18
Bioproducts	8
Agri-Food	4
Oilseeds	4
Corporate Initiatives	2
Total Knowledge and Expertise Contributions	608
Source: STB SMS Databases, Extracted April 18 2016. Includes outputs for project fully funded by Science Supporting an Innovative and Sustainable Sector (A-Base) between 2013-14 and 2015-16.

Another method employed by AAFC Science to disseminate scientific knowledge is the writing and distribution of success stories that document the achievements of scientific research. During the period from 2010 to 2015, over 400 success stories documenting the achievements of AAFC Science projects were written and published on the AAFC website. These success stories help to disseminate scientific knowledge, the majority of which were in the areas of Horticulture (81 success stories), Agro-Environment Resiliency (81), and Cereals and Pulses (59).

The evaluation identified a variety of other mechanisms through which scientific findings are disseminated. For example, information is disseminated at regional meetings and events such as Regional Research User Meetings, where representatives from AAFC, universities, industry, and provincial government technology transfer offices identify research needs that could be addressed by AAFC Science. Some AAFC research and development centres host open houses or stakeholder days, with targeted invitations to representatives of different stakeholder groups (for example, industry associations, businesses, investors, colleges, universities, provincial government ministries, et cetera). In one research and development centre, the stakeholder day led directly to the development of an industry-led AIP project.

4.2.3 End Outcomes

Outcome: Utilization of enhanced scientific knowledge, technologies and practices by the scientific community that ultimately contributes to the sector's ability to increase agricultural productivity, address threats to the agriculture and agri-food value chain, improve environmental performance and improve attributes for food and non-food uses.

It is difficult to determine the full scope of impacts of the AAFC Science activities since science and innovation activities are collaborative in nature (presenting challenges with making attribution of results to one program) and require many years to achieve their intended end outcomes. However, the evaluation found some evidence through case studies of projects that the scientific knowledge, technologies and practices generated by AAFC Science have contributed to the sector's ability to increase agricultural productivity, improve environmental performance, improve attributes for food and non-food uses, and address threats to the agriculture and agri-food value chain.

The following text boxes provide examples of how AAFC Science has contributed to end outcomes directly and indirectly and over a broader time period. The capacity and funding provided by AAFC Science has also enhanced the department's ability to generate impacts by leveraging by resources through other STB programming.

4.2.4 Other Impacts

AAFC Science activities have resulted in other impacts such as developing scientific capacity through student and postdoctoral researcher placements and new areas of scientific investigation or collaboration.

AAFC Science activities have helped to attract and train new scientists and develop scientific capacity through student and postdoctoral placements. Some placements have attracted researchers from foreign universities (for example, Brazil, China, Ghana), many of which indicated they were attracted to the position because of the international reputation of the scientist and AAFC. Most students and postdoctoral researchers reported that they have gained knowledge and skills during the placement (for example, in preparing scientific publications, designing a scientific project, and using scientific and technical equipment). Some AAFC scientists started as postdoctoral researchers or as technicians that obtained their Doctor of Philosophy (PhD) and transitioned into indeterminate scientist positions with AAFC.

Activities supported by AAFC Science research have led to work in new areas of research and recognition for scientific achievements. For example, research projects have led to the development of new assessment techniques, additional areas of research and collaborative projects. Between 2013-14 and 2015-16, 14 AAFC Science projects resulted in 31 instances of recognition including 17 research grants, six awards and eight other achievements.

4.3 Program Design and Delivery

4.3.1 Effectiveness of the Agriculture and Agri-Food Canada Science Design and Delivery

The following section summarizes some of the main factors contributing to the success and the constraints of the design and delivery of AAFC Science.

Contributing Factors to the Success of the Design and Delivery

The evaluation found that the major factors which have contributed to the success of AAFC Science include long-term investments in scientific capacity, collaborative approach to research, and centralized management of science investments.

Long-term Investments in Scientific Capacity

Canada has a long history of federal government involvement in agricultural research. These long-term investments in infrastructure and scientific expertise have been integral in driving research results and contributing to the reputation of AAFC as world a leader in agricultural science knowledge and expertise. Similar long-term investments in infrastructure and scientific expertise were identified in the other jurisdictions reviewed, such as the U.S., United Kingdom, and Australia. The level of support for AAFC Science is proportionate to the United States Department of Agriculture's support to its in-house science activities relative to the size of the agricultural sector in each country.

Investments in equipment, infrastructure and other technical support staff have allowed AAFC scientists to conduct research activities on a larger scale and over longer periods of time than most other agricultural scientists in Canada including those affiliated with universities, provincial government and the private sector. According to interviewees, AAFC scientists have access to state-of-the-art equipment and facilities, such as extensive laboratory, greenhouse, and field facilities which can be used for long-term field studies. They also have access to support staff including AAFC technicians that are highly skilled in specialized areas. In addition, they have access to postdoctoral fellows and graduate students who are able to assist with day-to-day research and data collection activities.

Multidisciplinary, Collaborative Approach to Research

AAFC Science's ability to collaborate with a variety of partners both internally (with AAFC research facilities and experts across Canada) and externally (with universities, provincial and federal governments departments, industry, and international partners) generates synergies and enhances the likelihood of significant discoveries. The AAFC Science delivery model promotes cross-sectoral and scientific synergies in conducting agricultural research. Sector strategies are integrated and draw from various areas of scientific expertise. AAFC scientists also leverage capacity through external collaborations. For example, a scientist may co-advise a PhD student with a university professor. AAFC scientists often find it beneficial to take positions as adjunct professors at universities to facilitate these placements and leverage research capacity and facilities.

The extent to which AAFC scientists collaborate has increased in the past 15 years. According to a 2016 study which examined the scientific output and collaboration patterns of 13 federal government entities from 2000 to 2014, over 75% of AAFC's scientific publications were co-authored by researchers from another organization and AAFC more than doubled its international collaboration rate in the last 15 years.^{Footnote 28} This reflects an increase in research projects that look at issues involving different fields of research and large, multidisciplinary teams with complementary expertise from a variety of organizations. As demonstrated in Figure 2, the types of collaborators vary by STB funding stream with foundational research projects such as AAFC Science and AIP Stream A involving a broader range of Canadian and international collaborators, while downstream projects such as AIP Stream B clusters and agri-science projects involve a higher proportion of industry collaborators (see Annex C).

Description of above image

The following Table illustrates Figure 2 as follows:

Program	Type	Number
AAFC Science (A-base)	Canadian universities	87
AAFC Science (A-base)	Industry associations and farming organizations	16
AAFC Science (A-base)	Provincial governments	52
AAFC Science (A-base)	Canadian companies	48
AAFC Science (A-base)	Foreign universities, governments and organizations	60
AAFC Science (A-base)	Other federal departments and agencies	34
AAFC Science (A-base)	Other Canadian experts/organizations	16
AAFC Science (A-base)	Canadian research institutions	9
AAFC Science (A-base)	Local governments, aboriginal and non-profit organizations	12
AAFC Science (A-base)	International organizations	8
AIP Stream A (CLF, SSTA, FSI)	Canadian universities	80
AIP Stream A (CLF, SSTA, FSI)	Industry associations and farming organizations	9
AIP Stream A (CLF, SSTA, FSI)	Provincial governments	30
AIP Stream A (CLF, SSTA, FSI)	Canadian companies	9
AIP Stream A (CLF, SSTA, FSI)	Foreign universities, governments and organizations	39
AIP Stream A (CLF, SSTA, FSI)	Other federal departments and agencies	26
AIP Stream A (CLF, SSTA, FSI)	Other Canadian experts/organizations	3
AIP Stream A (CLF, SSTA, FSI)	Local governments, aboriginal and non-profit organizations	8
AIP Stream A (CLF, SSTA, FSI)	International organizations	1
AIP Stream B (Clusters, Agri-Science Projects, KT)	Canadian universities	109
AIP Stream B (Clusters, Agri-Science Projects, KT)	Industry associations and farming organizations	170
AIP Stream B (Clusters, Agri-Science Projects, KT)	Provincial governments	59
AIP Stream B (Clusters, Agri-Science Projects, KT)	Canadian companies	30
AIP Stream B (Clusters, Agri-Science Projects, KT)	Foreign universities, governments and organizations	33
AIP Stream B (Clusters, Agri-Science Projects, KT)	Other federal departments and agencies	14
AIP Stream B (Clusters, Agri-Science Projects, KT)	Other Canadian experts/organizations	2
AIP Stream B (Clusters, Agri-Science Projects, KT)	Canadian research institutions	11
AIP Stream B (Clusters, Agri-Science Projects, KT)	Local governments, aboriginal and non-profit organizations	8

Source: AAFC Databases, Extracted April 18, 2016. Includes projects project fully funded by AAFC Science that started between 2011-12 and 2015-16.

According to a review of literature related to networking in science, the size and composition of a scientist's network have a significant impact on their scientific performance^{Footnote 29}, particularly among newer scientists^{Footnote 30}. Evidence also suggests that networks are important in determining patterns of knowledge diffusion.^{Footnote 31} Less formal collaboration and networks, such as participation in social media and other open access networks, are reshaping how scientific research is conducted and communicated.^{Footnote 32}

The evaluation identified a few opportunities to further strengthen collaboration. These opportunities include developing additional mechanisms to structure collaborative research agreements with universities, other federal government departments and international partners. STB has also initiated a series of Transformative Workshops that are intended to bring together groups of domestic and international scientists from various disciplines to help foster collaboration and generate ideas that address key agricultural challenges and support policy and investment decisions. To facilitate collaborations, it is also necessary to continue striving for efficient approval of projects and distribution funding.

Coherent Management of Science Investments

The centralization of AAFC Science management has helped to prioritize and direct resources more strategically. The creation of the STB in 2012, which integrated the former Research Branch and Agri-Environmental Services Branch, consolidated all AAFC's scientific expertise and has allowed for a more coherent, national perspective on science needs and priorities.

AAFC Science activities are directed strategically to promote science excellence and ensure they align with priorities and gaps. Extensive consultations were conducted with internal and external stakeholders to develop nine sector science strategies, each with clearly identified priority areas. The priority areas inform science funding decisions in the annual calls for Letters of Intent. The current governance model of AAFC Science ensures that some resources are directed to priority areas that are less likely to receive industry support, such as Agro-Environment Resiliency. The alignment of AAFC Science activities with longer-term industry priorities has also helped to leverage industry investment in more applied industry research projects and clusters.

Processes have been established to support more centralized monitoring of science activities. Work plans have been developed for each sector science strategy with key indicators to further facilitate monitoring, evaluation, and planning. AAFC Science has introduced project information systems (for example, Science Management Solution databases) to monitor project investments and outputs and developed mechanisms to report on national progress against the sector science strategies and work plans.

Design and Delivery Constraints

The evaluation found the following constraints with respect to the design and delivery of AAFC Science:

• IT and Equipment Constraints

  The evaluation identified some constraints with respect to available data storage and computation capacity, as well as challenges in procurement timelines associated with acquisition processes. This is mainly affecting areas of research related to microbiology and genomics, which are producing larger and more complex datasets.

  STB originally formed Tiger Teams to determine STB's various IT needs, and since have partnered with Information Systems Branch to create a DG Steering Committee to better determine and integrate all STB and Information Systems Branch processes for Science computing requirements.

  The evaluation found some issues with respect to aging AAFC Science equipment and infrastructure. Although key informants felt that overall scientists have access to state-of-the-art equipment and facilitates, there are some cases where AAFC Science is not keeping pace with acquiring the latest equipment, particularly in quickly changing fields such as microbiology and genomics. Some AAFC research and development centres require renovation or replacement, since they do not meet the needs of modern science and technology.

• Need for an Enhanced Performance Measurement Strategy

  AAFC Science is working towards improved performance measurement. Processes have been established to support more centralized monitoring of science activities. Work plans have been developed for each sector science strategy with key indicators to further facilitate monitoring, evaluation, and planning. AAFC Science has introduced project information systems (for example, Science Management Solution databases) to monitor project investments and outputs and developed mechanisms to report on national progress against the sector strategies and work plans.

  The program requires a performance measurement strategy with output and outcome indicators and associated performance targets. Without an enhanced strategy, it is difficult to monitor annual progress and overall achievements against intended program objectives. Further, a clear articulation of how Science 2.1.1, Agi-Innovation Streams A, B and C, and the Collaborative Framework work in concert with each other to support the full continuum of R&D in agriculture (such as, a program theory) would be very useful for telling their performance stories.

5.0 Evaluation conclusions

Relevance

Research and development is essential for driving innovation and ultimately improving sector productivity, enhancing competitiveness, and capitalizing on opportunities such as changing demands for agri-food and agri-based products. AAFC Science supports innovation by conducting foundational and public good research and development that lays the groundwork for more applied research activities undertaken in the AgriInnovation Program Stream A, as well industry-led research that is supported under AgriInnovation Program Stream B and through the Collaborative Framework. AAFC Science activities are part of an integrated innovation continuum that is intended to focus foundational research on areas that ultimately support adoption and/or commercialization of new products and services. In order to identify specific sector needs, the program conducted internal and external consultations to develop nine sector strategies that guide funding decisions. Three of the nine strategies capture cross cutting challenges associated with the need to preserve biodiversity and improve environmental performance.

AAFC Science objectives and activities are well aligned with federal government priorities and AAFC departmental strategic outcomes, particularly with respect to maintaining a strong science infrastructure, fulfilling international obligations to conserve genetic materials, and investing in agricultural science research. The recent Mandate Letter to the Minister of Agriculture and Agri-Food emphasizes the government's plan to "invest in agricultural research to support discovery science and innovation in the sector."^{Footnote 33} AAFC Science is aligned with AAFC's Strategic Outcome 2: An innovative and sustainable agriculture, agri-food and agri-based products sector, particularly in terms of generating new knowledge, fostering innovation, and improving the competitiveness and adaptability of the agriculture, agri-food and agri-based products sector. AAFC Science also aligns well with current federal government priorities. The federal government Budget 2016 specifically refers to planned investments in agricultural science infrastructure.

Delivering AAFC Science is an appropriate role for the federal government since the activities align with roles and responsibilities established in the Experimental Farm Stations Act (1985). AAFC Science plays an important role in maintaining the capacity to undertake agricultural science research and development, particularly foundational and public good research with longer-term Canadian public and industry benefits. No other entities in Canada (provincial government, industry, or universities) match AAFC in terms of its facilities, equipment and breadth of expertise to support research in agricultural science. Furthermore, there are very limited incentives for industry to invest in foundational research due to the lack of near-term commercial returns, and little capacity for universities to engage in public good non-academic related scientific activities such as collection that underpin much of the research, development and knowledge and technology transfer activities under AAFC Science.

Achievement of Intended Outcomes

As discussed below, AAFC Science has made a significant contribution to achieving its intended immediate, intermediate and end outcomes:

1. AAFC Science has made substantial contributions to its immediate outcome of enhanced scientific knowledge with benefits for the agriculture and agri-food sector. Between 2013-14 and 2015-16, AAFC Science projects resulted in 88 innovations and 1,508 scientific publications which included peer reviewed scientific articles or papers, conference proceedings, and book chapters. The knowledge generated by AAFC Science is also a critical input to other AAFC programs such as the AIP.
2. The collections and collaborative research agreements are of significant value. These activities have supported scientists and other stakeholders in a variety of ways such as identifying, classifying and preserving specimens for research purposes, assisting AAFC to protect the intellectual property generated from its research, and generating over $36 million in royalties during the period from 2009-10 to 2015-16.
3. AAFC Science made substantial contributions to its intended intermediate outcome which is in the dissemination and transfer of enhanced scientific knowledge to the scientific community and other stakeholders. Between 2013-14 and 2015-16, AAFC Science projects resulted in the development of 49 technology transfer publications and 608 knowledge and expertise contributions.
4. The evaluation found evidence that the enhanced scientific knowledge, technologies and practices generated by AAFC Science has contributed to its end outcome which is to enhance the agriculture and agri-food sector's ability to increase agricultural productivity, improve environmental performance, improve attributes of agricultural products for food and non-food uses, and address threats to the agriculture and agri-food value chain. The evaluation identified several examples of how AAFC Science is working towards this outcome, both directly (through A-base funding) and indirectly by providing scientific capacity to undertake other STB-supported programming.

Program Design and Delivery

The major aspects of the program design contributing to the success of AAFC Science include: long-term investments in infrastructure and scientific expertise (capacity); multidisciplinary, collaborative approaches to research; and centralized management of science investments. The evaluation found that there are some other aspects of the AAFC Science design and delivery model which constrain the success of the activities. These include: challenges with Information technology and equipment (for example available data storage and computation capacity to support rapidly evolving fields such as microbiology and genomics) and performance measurement, particularly with respect to measuring end outcomes.

6.0 Recommendation and Management Response and Action Plan

The key issue and recommendation resulting from the evaluation are:

Issue #1:

The evaluation found that although AAFC Science is monitoring research activities through SMS, it does not have a consistent approach to performance measurement, which is limiting the ability to monitor scientific outputs and measure the impact of Science activities.

Recommendation #1:

AAFC should:

In line with the new Policy on Results and the Government of Canada results and delivery agenda, AAFC should develop performance measurement approaches to ensure that STB can accurately and consistently report on the outputs and outcomes of AAFC Science.

Management Response and Action Plan

Agreed

1. STB is revising all its sector strategies to include logic models to better align with the results & delivery agenda.

   Target Date: June 30, 2017

   Responsible Position: Director General, Partnerships and Planning Directorate, Science and Technology Branch.

2. STB will create Program Information Profiles (PIPS) for all programming as per the new Results and Delivery Agenda for the Government of Canada.

   Target Date: November 1, 2017

   Responsible Position: Director General, Partnerships and Planning Directorate, Science and Technology Branch.

Appendix A: Program Logic Model

The program logic model presents the activities, outputs and outcomes of the AAFC Science.

Activities	Outputs	Immediate Outcomes	Intermediate Outcomes	End Outcomes	Strategic Outcomes
Maintain science capacity (for example, expertise, assets infrastructure) through a network of research and development centres and offices Undertake research through call for proposal process Participate in collaborative partnerships with industry, academia, other governments and non-governmental organizations Non-hypothesis activities (for example, maintenance of collections and intellectual property records)	Research project reports Collections and intellectual property records	Enhanced scientific knowledge with potential benefits for the agriculture and agri-food sector Use/value of the collections, intellectual property records and other non-hypothesis activities	Dissemination and transfer of enhanced scientific knowledge to the scientific community (This knowledge is also input to other program areas and contributes to further innovation in the agriculture and agri-food sector)	Utilization of enhanced scientific knowledge, technologies and practices by the scientific community that ultimately contributes to the sector's ability to: Increase agricultural productivity Address threats to the agriculture and agri-food value chain Improve environmental performance Improve attributes for food and non-food uses	An innovative and sustainable agriculture, agri-food and agri-based products sector

Appendix B: Detailed Evaluation Methodology

The evaluation included a comprehensive assessment of AAFC Science, in accordance with the Treasury Board of Canada's Policy on Results. The evaluation covers the period from April 1, 2010 to March 31, 2016. The evaluation was designed to address the following evaluation issues: relevance, performance, and design and delivery. Relevance examined how the program activities and focus areas address sector needs and program alignment with federal government and departmental priorities and roles. Performance assessed the extent to which the program has resulted in the achievement of expected outcomes with a particular focus on immediate and intermediate outcomes. Design and delivery examined how efficiently and economically the program is delivered. Within each issue, specific evaluation questions were explored, which are presented in the following table.

Categories	Evaluation Issues and Questions
Relevance: Continued Need for Program	How do the science strategic objectives and areas of focus within AAFC Science 2.1.1 support the Canadian agriculture and agri-food sector in its continuing need to be more competitive, innovative and sustainable? Are there any gaps that could be addressed?
Relevance: Alignment with Government Priorities	How do the science strategic objectives and areas of focus of AAFC Science 2.1.1 align with the current direction of federal government priorities and AAFC departmental strategic outcomes?
Relevance: Alignment with Federal Roles and Responsibilities	Does the federal government have a role and responsibility for delivering AAFC Science 2.1.1 with the objective of knowledge generation and dissemination and can this be improved to better ensure the productivity and sustainability of the agriculture and agri-food sector?
Performance: Achievement of Expected Outcomes	To what extent has AAFC Science 2.1.1 contributed to scientific knowledge in the agriculture and agri-food sector? To what extent has AAFC Science 2.1.1 resulted in the dissemination and transfer of enhanced scientific knowledge to the scientific community? How does AAFC Science 2.1.1 contribute to: increasing agricultural productivity, improving environmental performance, improving attributes for food and non-food uses, and addressing threats to the agriculture and agri-food value chain? What factors have contributed to or constrained the achievement of expected outcomes? What other impacts have resulted from activities undertaken as part of AAFC Science 2.1.1?
Design and Delivery: Efficiency and Economy of the Design and Delivery Model	How has the design and delivery of AAFC Science 2.1.1 evolved in terms of organizational structure? Are these changes enabling resources to be directed to the most beneficial research areas to support the agriculture and agri-food sector? How does the scientific capacity provided in AAFC Science 2.1.1 contribute to achieving the objectives of Program Alignment Architecture sub-program elements 2.1.2 and 2.1.3 and what are the linkages? What examples of return on investment does AAFC Science 2.1.1 provide? Are there efficiencies and economies that could be improved in the design and delivery of AAFC Science 2.1.1? Are there any best practices, lessons learned in terms of design and delivery that could be applied to AAFC Science 2.1.1?

The data collected for the evaluation was extracted and triangulated to address each of these questions. Data was analyzed to develop a summary response to each evaluation question. As part of this step, the strengths and limitations of each line of inquiry were taken into account to develop valid, reliable, and credible conclusions. A variety of data analysis techniques were employed to rule out alternative explanations of the results and enhance the generalizability of the specific causal linkages between the program components. Various statistical tests and methods were used to analyze the data such as frequency tables, cross tabulations, and correlation analysis.

The evaluation employed multiple lines of evidence drawing from primary data sources including interviews of AAFC science professionals, AAFC senior managers, and external collaborators, case studies of particular AAFC Science activities, and site visits to AAFC research and development centres, and secondary data sources including a document and file review, and a literature review. For the ease of reporting, the following quantifiers were used in reporting interview results:

• "A few" = less than 25 per cent of respondents
• "Some" = 25 to 40 per cent of respondents
• "Several" = 41 to 45 per cent of respondents
• "Approximately half" = 46 to 55 per cent of respondents
• "Most" = 56 to 75 per cent of respondents
• "Large majority" = over 75 per cent of respondents

Research and data collected from the various lines of evidence were used to develop supporting technical reports to contribute towards the evaluation. Each data source is described in more detail in the following paragraphs.

1. Document and File Review

An extensive review of internal and external documents was undertaken to collect information on AAFC Science inputs, activities, outputs and outcomes. The methodology included a review of internal data and documents including administrative data from the SMS project reporting databases and Science Publications Database (SPD), departmental documents on the history and context of science and research programming, financial, salary, and FTE data, the design and delivery of science programming (for example, IT systems and databases, project application processes, travel and hiring policies, et cetera), and departmental performance reports and reports on plans and priorities. External documents reviewed included evaluations of other similar science programs, studies examining the impact of networking and collaboration in science, federal government Speeches from the Throne, the Mandate Letter to the Minister of Agriculture and Agri-Food and studies on the relevance and need for government-funded science programming, among other documents.

2. Literature Review

A literature review was conducted to compare the design and delivery of AAFC Science with similar programs in Canada and other countries. Information was compiled through a detailed review of literature (for example, existing studies and reports from organizations such as the Organisation for Economic Co-operation and Development) and online resources (for example, websites, annual reports, business plans and strategies, et cetera). Profiles were developed of how leading developed countries such as Canada, the United States, the United Kingdom and Australia are organized to undertake scientific research with respect to the agriculture and agri-food sector. The profiles described key characteristics of the jurisdictions including background and contextual information (for example, country and sector statistics, history of agricultural research, examples of major discoveries, key stakeholders in the agriculture and agri-innovation system, and trends in levels and sources of support for agricultural research) and major mechanisms of federal government support (for example, strategic objectives, activities, levels of funding, staff and facilities, priority-setting mechanisms, evaluation and monitoring, and reported impacts). A comparative analysis was conducted including an analysis of the different strengths, weaknesses, opportunities, and threats (such as, SWOT analysis) impacting the agricultural research and innovation system in each region. A list of the literature reviewed is provided in Annex C.

3. Stakeholder Interviews

A total of 68 stakeholder interviews were conducted between May 2015 and April 2016 to obtain input regarding the relevance, performance, and design and delivery of AAFC Science. The number of interviews conducted exceeded the targeted 55 interviews. Respondents included AAFC senior management and other staff, AAFC research scientists and other science professionals, and external collaborators. Interviews were conducted with representatives across Canada. The following table provides the number of completed interviews per respondent group and a brief description of the respondents.

Target Group	Number Completed	Description of the Target Group
AAFC Senior Management and Other Staff	12	Director Generals, Directors, RDT Directors and Associate Directors, among other staff. On average, senior management staff reported they had been in their position for 2.7 years.
AAFC Research Scientists	37	Scientists possessed a broad range of specializations such as livestock phenomics, plant molecular biology, soil microbiology, soil biochemistry and nutrient cycling, ruminant nutrition, and biofuels. Scientists reported having an average of 22 years of experience with AAFC and most frequently reported their position as a RES-3 or RES-4; some also included RES-1, RES-2, and RES-5 scientists.
Other AAFC Science Professionals	4	Biologists, post-doctoral researchers, and technicians, specializing in molecular plant biology, pest management biology, beef microbiology, and food chemistry. On average, scientific personnel reported having an average of 22 years of experience at AAFC. While the post-doctoral researcher reported having under two years of experience at AAFC.
External Collaborators	15	University, industry, non-profit, provincial government, and other federal government representatives. Representatives included senior managers and experts specialized in animal and poultry science, food and nutritional science, plant science, life and environmental science, biotechnology, conservation, and other areas related to research and innovation and the agriculture and agri-food sector.
Total	68	n/a
n/a = not applicable

The following steps were taken to complete the stakeholder interviews:

• Developed interview guides for each target group in consultation with the AAFC Evaluation Team.
• Developed a list of AAFC research scientists and senior managers based on referrals from the Evaluation Team and senior managers within STB. The lists also ensured a mix of representatives were contacted in terms of their years of experience (for example, late, mid and early career), area of specialization, and region. Some contacts were identified through snowball sampling from interviewee referrals (for example, collaborators).
• Sent an email to targeted representatives introducing the evaluation, and explaining the purpose and timing of the interviews. A relevant interview guide was attached to each email.
• Sent follow-up emails or contacted the representatives by telephone to schedule the interview at a time of their convenience in the official language of their choice to encourage their completion of the questionnaire. Reminders were sent to all those who did not respond to the initial invitation and follow-up call or email.
• Conducted telephone interviews or obtained completed questionnaires via email.
• Compiled and analyzed the responses using Excel and synthesized the results for each evaluation question and indicator.

4. Case Studies

Six case studies were conducted of AAFC research and development centres in Guelph, Sherbrooke, Fredericton, Charlottetown, Swift Current and Saskatoon. The purpose of the case studies was to collect detailed information on activities and impacts of AAFC Science to address evaluation questions related to program performance and design and delivery. The case studies were selected to ensure a mix of strategic objectives, sector science strategies, and geographic locations of the project (East, Central and West). Each case study focused on major science and innovation achievements arising from the AAFC Science activities within the research and development centres, with respect to specific sector science strategies and strategic objectives, as described in the table below.

AAFC Research and Development Centre	Sector Science Strategy	Strategic Objective
Guelph Research and Development Centre	Agri-Food and Bioproducts	Improve attributes for food and non-food uses
Sherbrooke Research and Development Centre	Dairy, Pork, Poultry, and Other Livestock	Enhance environmental performance
Fredericton Research and Development Centre	Horticulture	Address threats to the agriculture and agri-food value chain
Charlottetown Research and Development Centre	Agro-Environment Resilience	Increase agricultural productivity
Swift Current Research and Development Centre	Forages and Beef	Enhance environmental performance
Saskatoon Research and Development Centre	Biodiversity and Bioresources	Address threats to the agriculture and agri-food value chain

Data collection for the case studies involved a detailed document and file review as well as focus groups, interviews, other photo evidence and documents collected during the site visits to the research and development centres (see the description below).

5. Site Visits and Focus Groups

Site visits were conducted of AAFC research and development centres in Guelph, Sherbrooke, Fredericton, Charlottetown, Swift Current, Saskatoon, Ottawa, and Vineland between June 2015 and March 2016. The site visits were selected using similar criteria as the case studies, ensuring a mix of sector focus areas and geographic locations (East, Central and West). The purpose of the site visits was to gather evidence for the case studies and to gather evidence with respect to the relevance and design and delivery of the AAFC Science activities. Each visit consisted of one-on-one interviews with management staff followed by focus groups or individual interviews with scientists and a tour of the facilities. Associate Directors RDT facilitated meetings with the scientists and centre tours. Students, postdoctoral researchers, and technicians were interviewed during the tours. In some cases, industry and university collaborators were also interviewed as part of the visits. Interviews were conducted with a total of 60 representatives during the site visits, including 11 AAFC management and other staff (for example, Directors RDT, Associate Directors RDT, and KTT staff), 37 AAFC research scientists specializing in a variety of sectors and areas of scientific expertise, seven students, postdoctoral researchers and technicians, and 5 collaborators.

Site visit participants were provided with a discussion guide explaining the purpose of the evaluation and topics that would be covered. Focus group and interview discussion topics varied depending on the involvement of the participants. Examples of topics included how research priorities and activities are established and the extent to which the research is addressing industry needs, major impacts generated by the research, factors that have facilitated or constrained the research (for example, capacity), AAFC's role in undertaking scientific research, and suggestions for improvement. Students and post-doctoral researchers were asked to speak more generally about their experience and career plans to better understand how AAFC supports capacity building among new scientists.

As part of an evaluation of the AIP Stream A, site visits were also made to AAFC research and development centres in Summerland, Agassiz, Harrow, London, Saint-Hyacinthe, and Lethbridge from March to May, 2016. These site visits provided additional evidence regarding the design and delivery of AAFC Science.