Evaluation of Science 2.1.1: Science Supporting an Innovative and Sustainable Sector – Summary Report

March 30, 2017

Purpose

To provide a summary of the Science 2.1.1 evaluation report and management response and action plan.

Evaluation scope and methodology

The Evaluation of Science 2.1.1 was included in Agriculture and Agri-Food Canada (AAFC)'s Five-Year Evaluation Plan (2014-15 to 2018-19) and fulfils evaluation requirements under the Treasury Board Policy on Evaluation (2009) and subsequent Policy on Results (2016).

AAFC Science is an A-base program consisting of internal research, development, and technology and knowledge transfer projects. The average annual expenditures during the evaluation period were approximately $200 million. It is led by the Science and Technology Branch (STB) and is the largest Program Alignment Architecture sub-program within the Department. The major activities supported by AAFC Science include: maintaining an agricultural science capacity and biological reference collections, conducting scientific research, and facilitating collaborative partnerships and the transfer of research results.

Methodology:

• The evaluation was undertaken by the Office of Audit and Evaluation, with support from management consulting firm Ference and Company.
• The evaluation used multiple lines of evidence (including document, file and database review; literature review; interviews with internal/external representatives; site visits and focus groups at 14 centres; and case studies on six AAFC research and development centres) to assess program activities undertaken between April 1, 2009 to March 31, 2016.

Evaluation findings

Relevance:

• AAFC Science supports innovation by conducting foundational research that lays the groundwork for more applied research activities undertaken in the AgriInnovation Program Stream A, as well industry-led research that is supported under AgriInnovation Program Stream B. AAFC Science activities are part of an integrated innovation continuum that is intended to focus foundational research on areas that ultimately support adoption and/or commercialization of new products and services. AAFC Science, and other related research and development program, funding decisions are guided by nine sector strategies that identify specific sector needs.
• The objectives of AAFC Science are well aligned with the priorities of the federal government and the strategic outcomes of AAFC, particularly with respect to the priority to invest in agricultural research^{Footnote 1} and AAFC's Strategic Outcome 2: An innovative and sustainable agriculture, agri-food and agri-based products sector.
• Delivering AAFC Science is an appropriate role for the federal government since the activities align with roles and responsibilities established in the Experimental Farm Stations Act (1985). AAFC Science plays an important role in maintaining the capacity to undertake agricultural science research, particularly foundational research with longer-term Canadian public and industry benefits. No other entities in Canada (e.g., provincial government, industry, or universities) match AAFC in terms of its facilities, equipment and breadth of expertise to support research in agricultural science.

Effectiveness:

• AAFC Science has contributed to enhanced scientific knowledge with benefits for the agriculture and agri-food sector. Between 2013-14 and 2015-16, projects fully funded by AAFC Science resulted in 88 innovations and 1,508 scientific publications, which included peer reviewed scientific articles and papers, conference proceedings and book chapters.
• The Collections and Collaborative Research Agreements activities have supported scientists and other stakeholders in a variety of ways such as classifying and preserving specimens for research purposes, identifying pests, invasive species and biological threats to agricultural production systems, facilitating collaborations between AAFC researchers and external stakeholders, assisting AAFC to protect the intellectual property generated from its research, and helping generate over $36 million in royalties during the period from 2009-10 to 2015-16.
• AAFC Science has made substantial contributions to the dissemination and transfer of enhanced scientific knowledge to the scientific community and other stakeholders. Between 2013-14 and 2015-16, AAFC Science projects resulted in the development of 49 technology transfer publications (such as bulletins, newsletters and trade journal publications) and 608 knowledge and expertise contributions (such as AAFC scientists speaking or presenting their findings, training personnel, students or other experts, acting as a scientific or technical expert and conducting media interviews).
• It is difficult to determine the full scope of impacts of the AAFC Science activities since science and innovation activities are collaborative in nature, require many years to achieve their intended long-term outcomes and the scope of the evaluation only considered AAFC Science activities in the past six years. However, the evaluation identified several examples of how AAFC Science is contributing to:
  • increased agricultural productivity,
  • methods to address biological threats to the agriculture and agri-food chain,
  • improved environmental performance, and
  • Improved attributes of agricultural products for food and non-food uses.

Program design and delivery

• The evaluation found that the following factors are contributing to the success of AAFC Science:
  • Long-term investments in facilities, equipment and scientific expertise have been integral in driving research results and contributing to the reputation of AAFC as a world leader in agricultural science knowledge and expertise.
  • AAFC Science's collaboration with a variety of partners both internally and externally generates synergies and enhances the likelihood of significant discoveries. The extent to which AAFC scientists collaborate has increased in the past 15 years, which reflects an increase in research projects that examine issues involving different fields of research and large teams with complementary expertise from a variety of organizations.
  • The centralization of AAFC Science management has helped to direct resources more strategically and allowed for a more coherent, national perspective on science needs and priorities. The project approval process promotes science excellence, and aligns resources to priority areas while leveraging industry investment in more applied industry research projects and clusters.
• The evaluation identified some constraints with respect to available data storage and computation capacity, as well as challenges in procurement timelines associated with acquisition processes. This is mainly affecting areas of research related to microbiology and genomics, which are producing larger and more complex datasets.
• STB's performance measurement and data collection tools are useful for communicating project results, but some issues were identified with their ability to report on program-level outcomes.

Recommendation and Management Response and Action Plan

Recommendation

In line with the new Policy on Results and the Government of Canada results and delivery agenda, AAFC should develop performance measurement approaches to ensure that STB can accurately and consistently report on the outputs and outcomes of AAFC Science.

Management Response and Action Plan

Agreed

1. STB is revising all its sector strategies to include logic models to better align with the results & delivery agenda. Target Date: June 30, 2017. Responsible Position: Director General, Partnerships and Planning Directorate, Science and Technology Branch.
2. STB will create Program Information Profiles (PIPS) for all programming as per the new Results and Delivery Agenda for the Government of Canada. Target Date: November 1, 2017. Responsible Position: Director General, Partnerships and Planning Directorate, Science and Technology Branch.