Evaluation of the AgriInnovation Program – Stream B: Research, Development and Knowledge Transfer

May 1, 2017

List of abbreviations

AAFC: Agriculture and Agri-Food Canada
ADM: Assistant Deputy Minister
AIP: AgriInnovation Program
R&D: Research and Development

Executive summary

Purpose of the Evaluation

This Evaluation examined the relevance and performance of Stream B: Research, Development and Knowledge Transfer of the AgriInnovation Program (AIP). The evaluation covers April 1, 2013 to March 31, 2016, and was conducted as part of Agriculture and Agri-Food Canada's (AAFC) 2014-15 to 2018-19 Departmental Evaluation Plan, in accordance with the Treasury Board of Canada's Policy on Results.

Background

The AIP is a five-year Growing Forward 2^{Footnote 1} program designed to accelerate the pace of innovation by supporting research and development (R&D) activities in agri-innovation and facilitating the demonstration, commercialization and adoption of innovative products, technologies, processes, practices and services. The program funds activities under three streams: Stream A: Research Accelerating Innovation (usually ten years or more from commercialization); Stream B: Research, Development, and Knowledge Transfer (usually two to ten years from commercialization); and Stream C: Enabling Commercialization and Adoption (usually two years or less from commercialization). Similar programming was implemented from 2008-09 to 2012-13 under the first Growing Forward framework.

Stream B includes two components:

Industry-led R&D supports pre-commercialization research, development and knowledge transfer leading to innovative agriculture, agri-food and agri-based practices, processes and products. Eligible recipients include non-profit organizations and private sector companies. Recipients may undertake R&D activities directly or in collaboration with AAFC, universities or other organizations. The industry-led R&D component provides non-repayable contributions to organizations and/or collaborative assistance by departmental research scientists and experts. This support is provided through Agri-science clusters, which are national in scope and Agri-science projects, which are less comprehensive. Both clusters and projects are composed of multiple research activities.
AAFC-led Knowledge Transfer supports activities that facilitate the demonstration and transfer of new or improved products, processes or technologies developed by AAFC scientists. Activities include the development of written and electronic products and active industry outreach. Only AAFC science and technology and knowledge transfer professionals are eligible as funding recipients.

As of March 31, 2016, $268.1 million in AAFC funding was budgeted for Stream B activities from 2013-14 to 2017-18.

Methodology

The evaluation used the following quantitative and qualitative lines of evidence:

File and operational data review
Document review on the state of agricultural research capacity in Canada
Interviews with 17 key informants including Agri-science cluster leads and AAFC Programs Branch representatives involved in the delivery of Stream B
Benchmarking analysis of similar programs in other jurisdictions, and an
Economic impact assessment of program investments

Key findings and conclusions

Relevance

There is a continued need for Stream B programming to develop innovations to address threats to the profitability and competitiveness of the agriculture and agri-food sector. The sector faces numerous threats such as variable input and output prices, extreme weather, and new pests and diseases. There is also a need to capitalize on new opportunities for value-added products due to increasing global demand for food with specific attributes and non-food products. Investments in agricultural R&D contribute to improved productivity and generate a substantial return on investment in the long-term.

Limited funding, weak linkages between agricultural R&D performers, and capacity challenges constrain the sector's ability to meet its innovation needs. The federal government continues to play a lead role in funding and maintaining the capacity to undertake agricultural R&D and knowledge transfer to complement and leverage limited investment, foster innovation in small and emerging agricultural industries and encourage industry to invest in research with long-term public good benefits (for example, soil, water, and air quality; biodiversity). There is an ongoing need for collaboration among federal and provincial governments, universities, industry and other organizations in conducting agricultural R&D in Canada. R&D collaboration has many benefits such as leveraging investment, enhancing access to highly qualified personnel, infrastructure and equipment, and generating research excellence. AAFC collaboration rates are increasing at a higher rate than other federal government departments involved in scientific research. This increase coincides with the introduction of the cluster funding model under Growing Forward and its continued support under Growing Forward 2 through Stream B, which was designed to enhance collaboration.

Program-supported R&D activities can also provide opportunities for sector highly qualified personnel to gain research experience, which in turn supports the sector's future scientific and innovation capacity. This addresses several challenges that may constrain the ability to meet future highly qualified personnel needs, such as the high number of expected scientist retirements across the sector, limited networks and mentorship opportunities for newer scientists, declining enrolment rates in agricultural programs, and competition from other sectors.

Performance (effectiveness)

The evaluation found that three years into the five-year program, Stream B has made significant progress towards the achievement of its intended outputs and outcomes. As of March 31, 2016, Stream B has committed $220.5 million over the five-year period of 2013-14 to 2017-18 to 14 Agri-science clusters, 86 Agri-science projects and 20 AAFC-led Knowledge Transfer projects. In total, 60 Collaborative Research and Development Agreements have been signed, allowing for support in the form of collaborative assistance from AAFC research scientists and experts in Industry-led R&D activities. Program activities span a variety of sectors and portfolios, with the largest proportion of Industry-led R&D funding committed to the oilseeds sector portfolio and the largest proportion of AAFC-led Knowledge Transfer funding committed to agro-ecosystem productivity and health.

Private sector investments in Stream B activities have more than doubled compared to private sector investments in similar programs under Growing Forward, due to increased AAFC investments, improved cost-share ratios, and increased industry awareness and interest in the program. The level of R&D collaboration has increased. New research networks have been established and existing networks have been strengthened and expanded across provinces and different types of organizations. Though most Agri-science cluster and project activities were not complete at the time of the evaluation, considerable progress had been made in developing and identifying new products, processes, and technologies for industry commercialization or adoption. Several examples of innovations resulting from Stream B activities were identified, some of which arose from activities that were initiated under previous programming and tracked over time. All technology, knowledge and information transfer targets have been exceeded. The program has enhanced R&D capacity in the agriculture and agri-food sector by providing research experience for individuals completing an MSc or PhD.

The program has made some progress towards its intended end outcome that the sector produces, adopts, and commercializes innovative products, processes, practices, services and technologies. It is too early to assess the full impact of Stream B activities since the program is not yet complete and it can take several years to further develop, adopt and commercialize products and practices resulting from Stream B supported R&D. A cost-benefit analysis conducted by Programs Branch in 2015-16 of two successful innovations identified high rates of return from long-term innovation investments by AAFC, provincial governments and industry in agricultural R&D activities across policy frameworks^{Footnote 2}.

Program activities have resulted in significant economic impacts. Industry-led R&D expenditures of $120.2 million from April 1, 2013 to March 31, 2016 resulted in an estimated $289.9 million in spending across the Canadian economy during the same time period: $2.4 in total facilitative economic impacts for every dollar of AAFC funding. According to interviews, approximately 80%, or $232.0 million, of the total facilitative economic impact is estimated to be directly attributable to the Industry-led R&D component of Stream B (that is, it would not have occurred in the absence of AAFC assistance).

Description of this image follows. — **Total Facilitative Economic Impact from Stream B Industry-led R&D Expenditures, 2013-14 to 2015-16 (Estimated)**

AAFC funding	$120.2 million
Leveraged funding	$74.5 million
Indirect economic impact	$95.2 million
Total facilitative economic impact over 3 years	$289.9 million

Performance (efficiency and economy)

Stream B has achieved program outcomes more efficiently than similar programming implemented under Growing Forward in terms of higher levels of private sector investment leveraged per dollar of AAFC investment and lower AAFC costs per innovation developed.

The major factors contributing to the effective delivery of Stream B activities include collaboration across R&D funders and performers, cost-shared funding mechanisms, and continuous improvement of program delivery. Industry-led R&D funding mechanisms facilitate collaboration across a variety of R&D funders and performers, which contributes to improved capacity and research excellence and facilitates more coordinated and efficient R&D funding. The evaluation found some evidence that opportunities or mechanisms could be explored to facilitate increased collaboration between AAFC and other federal and provincial government organizations. AAFC's matching funding encourages industry stakeholders to fund larger scale and longer-term R&D activities that have the potential to yield substantial returns. The ongoing efforts to refine program delivery have improved the efficiency of delivery processes over time.

The main challenges that constrained the effective delivery of stream B activities include complex and time consuming application review, approval, and claims reporting processes due to the various collaborators involved and types of agreements required and limited ability to respond to emerging R&D needs and sectors. Some interviewees felt the clusters model does not allow sufficient flexibility to respond to emerging R&D needs, because it is difficult to reallocate funds between activities or towards new activities in a timely manner. Smaller, emerging sectors struggle to access Stream B funding due to capacity constraints.

Program spending as a percentage of the available budget authority increased from 49% in 2013-14 to 93% in 2015-16. The primary factors contributing to under-spending in 2013-14 were a large backlog in applications and delays in processing these applications at the outset of Growing Forward 2.

Mechanisms in place to track the performance of Stream B program activities have been successful in collecting a significant amount of performance data. However, there are limitations with respect to its consistency and accuracy. For example, the clarity of definitions and categories could be improved. There are also opportunities to improve the utility of performance data to industry and AAFC stakeholders in order to increase its use.

Recommendations

The key recommendations resulting from the evaluation are:

Recommendation #1:

Continue to implement programming which supports industry-led R&D. Facilitate a smooth transition between policy frameworks by applying lessons learned under Growing Forward 2 with respect to program delivery. Explore opportunities to enhance federal, provincial, and territorial government R&D collaboration by addressing constraints related to government funding rules and improving joint information sharing.
Recommendation #2:

Examine options to enhance the flexibility of the program to respond to emerging needs and opportunities. These could include clearly articulating government priorities and examples of activities that may be supported and streamlining processes to facilitate more timely approval of funds for new opportunities or needs. Consider introducing a new program element that targets smaller, emerging sectors with less capacity to undertake large scale R&D activities and higher risk demonstration projects that have a higher potential for impact as a result.
Recommendation #3:

Examine options to enhance performance measurement processes and indicators to ensure that the information collected is accurate, consistent, meaningful, and useful for AAFC and program stakeholders in reporting on program results, and that it demonstrates the long-term impacts of investments in innovation.
Recommendation #4:

Continue to streamline the application review, approval, and claims processes for Industry-led R&D activities. Some potential opportunities to increase efficiencies that could be investigated include reducing duplication in the technical peer review process. Claims processes could be further streamlined by revising requirements to reflect an appropriate level of financial reporting based on risk and assisting funding recipients in developing standardized approaches.

1.0 Introduction

1.1 Purpose of the report

This report represents the findings of the evaluation of the AgriInnovation Program (AIP) Stream B: Research, Development and Knowledge Transfer. Stream B provides non-repayable funding and collaborative assistance to support industry-led research and development (usually about two to ten years from commercialization), and knowledge transfer led by Agriculture and Agri-Food Canada (AAFC). The evaluation was conducted as part of AAFC's 2014-15 to 2018-19 Departmental Evaluation Plan by the department's Office of Audit and Evaluation.

2.0 Program profile

2.1 Program context

In April 2013, the Government of Canada launched the five-year, multilateral Growing Forward 2 policy framework for Canada's agriculture and agri-food sector. Growing Forward 2 is a $3 billion dollar investment by federal, provincial and territorial governments and the foundation for government agricultural programs and services.^{Footnote 3} By focusing on innovation, competitiveness and market development, Growing Forward 2 provides Canadian producers and processors with tools and resources that help them to innovate and capitalize on emerging market opportunities. The AIP is a five-year program (2013-14 to 2017-18) under Growing Forward 2 that resides within AAFC's Program Alignment Architecture Program Activity 2.1 Science, Innovation, Adoption and Sustainability and is delivered through three streams (Table 1).

**Table 1: Description of AgriInnovation Program Streams**
AIP Stream ( Program Alignment Architecture Sub-program)	Description	Budget
A: Research Accelerating Innovation (2.1.2)	Discovery research stage (usually ten years or more from commercialization)	$150.1 million
B: Research, Development and Knowledge Transfer (2.1.3)	Pre-commercialization/adoption and knowledge transfer stage (usually two to ten years from commercialization)	$268.1 million
C: Enabling Commercialization and Adoption (2.1.4)	Commercialization/adoption stage (usually two years or less from commercialization)	$137.3 million
Total		$555.5 million

The AIP is designed to accelerate the pace of agri-innovation at each stage of the innovation continuum by supporting research, development, and knowledge transfer activities and facilitating the demonstration, commercialization and/or adoption of innovative products, technologies, processes, practices and services. The aim is to enhance the economic growth, productivity, competitiveness, adaptability and sustainability of the Canadian agriculture, agri-food and agri-based products sector and assist in capturing opportunities for the sector in domestic and international markets.

The AIP Stream B: Research Development and Knowledge Transfer (Stream B) is a continuation of similar programming implemented under the first Growing Forward framework from 2008-09 to 2012-13, consisting of Agri-science clusters and the Developing Innovative Agri-products Program, supported activities at the pre-commercialization stage of the innovation continuum.

2.2 Overview of Stream B: Research, development and knowledge transfer

This section provides a description of the Stream B objectives and activities, program delivery and resources. A logic model, details on program governance, and description of stakeholders for Stream B are in Annex A.

Program objectives and activities

The objective of Stream B is to accelerate the pace of innovation in the sector. The program includes two components: industry-led research and development (R&D) and AAFC-led knowledge transfer related to innovative products, practices, processes and systems in the agriculture, agri-food and agri-based products sector. The ultimate goal of the program is to increase market opportunities, strategic research planning and development, and foster innovation, industry investment and leadership.

Industry-led research and development

The objective of industry-led R&D is to encourage private sector investments and facilitate greater academia-industry-government and international collaborations to implement applied science and technology development projects that lead to new market opportunities for innovative agricultural products, practices, processes and/or services. This component also aims to accelerate the pre-commercialization development of new products, practices and processes by supporting the required applied science, technology development and knowledge transfer activities. Industry-led R&D is delivered through:

Agri-science clusters^{Footnote 4} are national in scope, focused on a particular sector or cross-cutting area, and address components of the sector's applied science plan. Each Agri-science cluster addresses several themes that are priorities to the industry and support several research activities across the country. Agri-science cluster leads generally select research projects that address these themes through calls for proposals to potential collaborators, though other mechanisms are also used (for example, annual general meetings and planning sessions). Agri-science clusters, on average, run the length of the five-year framework, and many are continuations of prior clusters, with activities they are interested in continuing under future policy frameworks. Total departmental funding cannot exceed $20 million per Agri-science cluster over the duration of the program under Growing Forward 2.
Agri-science projects are less comprehensive than Agri-science clusters and last, on average, from two to five years. AAFC funding per Agri-science project cannot exceed $5 million over the duration of the program.

Examples of the types of activities funded under this component include:

Targeted applied science, research or development activities that address sector priorities, increase market opportunities, and foster innovation
Research support related to product, practice, process or technology development
Projects that pilot, test solutions or explore/adapt technologies to maintain the sector's ability to adapt and remain competitive, and
Knowledge transfer and the development of tailored approaches, working to bring science and technology to the field and improve farm profitability and sustainability

Eligible recipients for Agri-science clusters and projects include non-profit organizations and private sector companies. Non-profit organizations are eligible to receive up to 75% of total eligible costs and for-profit organizations are eligible to receive up to 50% of total eligible costs. Eligible cost categories under the Industry‐led R&D component include administrative expenditures, salaries/benefits, contracted services, travel, capital/asset costs, and other expenditures directly associated with Agri-science cluster and project implementation. In-kind contributions can be counted toward recipient contributions, but the value of such contributions cannot normally exceed 10% of the total eligible costs of the Agri-science cluster or project. The Industry-led R&D component clusters' and projects' activities can be undertaken by the recipient and/or contracted to a research partner, or undertaken by AAFC researchers.

For work undertaken by the recipient or contracted to a research partner external to AAFC, AAFC provides non-repayable contributions funded through Vote 10 (grants and contributions) funds. Approved requests for contribution funding are formalized through a Contribution Agreement between AAFC and the recipient that details how contribution funding will be expended by the recipient. Contribution Agreements are managed through AAFC's Programs Branch. For research, development and knowledge transfer activities undertaken by AAFC researchers, AAFC Vote 1 funding (that is, internal operating funding) is used to support salaries and operating costs associated with AAFC scientists collaborating on R&D activities. Approved requests for AAFC research collaboration are formalized through a Collaborative Research and Development Agreement, an agreement managed by the Science and Technology Branch outlining research to be conducted by AAFC researchers.

AAFC-led knowledge transfer

The objective of this component is to facilitate the federally-led transfer of innovative ideas, tools, and practices resulting from R&D conducted by AAFC science professionals. Regionally relevant and commodity specific knowledge transfer approaches are informed by advice from industry users, such as farms and businesses, to ensure knowledge will be transferred according to local circumstances and needs, thereby enhancing sector competitiveness, profitability, sustainability and adaptability.

This component supports activities that facilitate the demonstration and transfer of new or improved products, processes or technologies developed by AAFC scientists. Only AAFC science and technology and knowledge transfer professionals are eligible as funding recipients (through Vote 1 funding). Projects are normally supported for one to three years in duration. Examples of Knowledge Transfer activities conducted by AAFC's Science and Technology Branch include, but are not limited to:

Development of written and electronic products (for example, technical manuals and on-line decision support tools), and
Active outreach, such as demonstration sites, field days, organized tours and participation in industry events

2.3 Purpose of the report

The total budget for Stream B activities from 2013-14 to 2017-18 is $268.1 million (Table 2). Of this total budget, $260.5 million is available for the Industry-led R&D component, mostly in the form of Vote 10 contribution funding (66%). The budget for the AAFC-led Knowledge Transfer component of Stream B is $7.6 million over the five-year period.

**Table 2: Stream B – Research, development and knowledge transfer budget ($ millions)**
	Vote/total	2013-14	2014-15	2015-16	2016-17	2017-18	Total
Industry-led R&D	Vote 1	19.1	16.7	16.6	17.1	19.6	89
	Vote 10	26	31.1	40.9	36.8	36.8	171.5
	Total	45.1	47.7	57.5	53.8	56.3	260.5
AAFC-led knowledge transfer	Vote 1	1.5	1.5	1.5	1.5	1.5	7.6
AAFC-led knowledge transfer	Total	1.5	1.5	1.5	1.5	1.5	7.6
Total Stream B	Vote 1	20.6	18.2	18.1	18.6	21.1	96.6
	Vote 10	26	31.1	40.9	36.8	36.8	171.5
	Total Stream B	46.6	49.3	59	55.3	57.9	268.1

3.0 Evaluation methodology

3.1 Evaluation scope

The evaluation of Stream B was conducted in accordance with the Treasury Board of Canada's Policy on Results. The evaluation covers the period from April 1, 2013 to March 31, 2016, though the program is ongoing until March 31, 2018. The evaluation was conducted midway through the program in order to inform the development of innovation programming under the next agricultural policy framework. The evaluation was designed to address the issues of relevance (continued need) and performance (the extent to which the program's activities resulted in the achievement of expected outcomes, and how efficiently and economically the program was delivered).^{Footnote 5}

The following lines of evidence were used to address the evaluation questions:

File and operational data review
Document review on the state of agricultural research capacity in Canada
Interviews with 17 key informants including Agri-science cluster leads and AAFC Programs Branch representatives involved in the delivery of Stream B
Benchmarking analysis of similar programs in other jurisdictions, and
Economic impact assessment of program investment

Annex B provides more detailed descriptions of each of the specific lines of evidence and evaluation questions.

3.2 Evaluation limitations

The most significant limitations of the evaluation as well as strategies that were used to address those limitations include:

Timing of the evaluation. It requires many years for far-from-market scientific research, development, and transfer activities to generate expected impacts. The Stream B activities reviewed included only those undertaken in the period covered by the evaluation, which means that sufficient time had not lapsed to measure the progress made against intended long-term outcomes. To mitigate this limitation, the evaluation focused on immediate and intermediate impacts of the program. A review of success stories from similar activities funded under previous policy frameworks was conducted to examine the potential impact of the activities in producing expected outcomes in the long-term.
Respondent biases. Given that the Agri-science cluster leads and AAFC senior managers interviewed are involved in the program activities and have a vested interest in the program, there is a potential for respondent biases. Several measures were taken to reduce the effect of respondent biases and validate interview results including: i) ensuring that representatives selected were knowledgeable of the program and comprised a representative sample of the target groups (for example, Agri-science clusters represented a mix of agricultural commodities and products); ii) clearly communicating the purpose of the evaluation, methodology, and strict confidentiality of responses to respondents; iii) cross-checking answers from each sample of respondents with the other groups for consistency and validation; and iv) validating the findings of the interviews with the results of the file, operational data, and document reviews.
Benchmarking limitations. The review of similar programs in other jurisdictions including the United States and Australia demonstrated that the countries have adopted a variety of different approaches and mechanisms, each of which is unique to the country's specific circumstances and needs (for example, historical, geographic, economic). As a result, it is impossible to draw exact comparisons between countries and devise an agricultural innovation delivery model that could be directly transferred to the Canadian context. To mitigate these challenges, the analysis focused on the identification of similar trends that are impacting agricultural innovation in each jurisdiction to identify potential weaknesses, issues or risks in the Canadian agricultural innovation system.
Economic impact assessment limitations. The input-output model is a useful model to capture the relationship amongst industries in the economy. However, one should be mindful of the model's limitations when interpreting results. Some of the major limitations include that the relationships of input-output models are simple proportionalities that imply that marginal changes are equal to average changes (that is, economies of scale cannot be represented); input-output models are static models where time is not explicitly represented; and although input-output analysis incorporates economic structures and linkages, it often ignores any economic displacement that may occur in existing industries as new projects are completed. The estimates used to conduct the analysis rely on various assumptions regarding planned spending and expected likelihood that activities would have proceeded as planned based on the interview results. Another limitation of this methodology is that it does not take into account the quality of the R&D results and longer-term impacts from the innovations developed as a result of the activities. To address these limitations, the economic impact assessment results are presented as estimates and the scope of the impact is clearly defined.

4.0 Evaluation findings

4.1 Relevance – Continued need for the Program

The agriculture and agri-food sector faces ongoing competitiveness and productivity challenges, low levels of investment in agricultural R&D, research capacity limitations and collaboration challenges. These issues are limiting the extent that resources can be leveraged to produce relevant, high-impact research.

Competitiveness and Productivity Challenges

According to studies conducted by AAFC and the Organization for Economic Co-operation and Development, there is a continued need for investment in agricultural R&D to address ongoing competitiveness challenges in the Canadian agriculture and agri-food sector. Threats such as variable input and output prices, influenced by world market and exchange rate shifts, threaten the profitability and global competitiveness of Canadian agriculture and agri-food industries. Climate change is increasing the severity and frequency of extreme weather and bringing new pests and diseases to Canadian crops and livestock.^{Footnote 6} Global demand for agricultural products is expanding due to growing populations, higher incomes, urbanization and changing diets. There is also an increasing demand for food with specific attributes (for example, food safety, nutrition, environmental stewardship, and animal welfare) and non-food products such as biodiesel.^{Footnote 7} To remain competitive, Canadian producers need to compete on costs and value-added attributes.^{Footnote 8}

Agri-science cluster leads interviewed for this evaluation recognize the importance of innovation in addressing these competitiveness challenges, both in terms of addressing threats and seizing new, value-added opportunities. The primary objectives of the Stream B Agri-science cluster activities are to increase sector productivity (for example, through improved agronomic and feed practices, genetic traits, and technology) and expand market demand (for example, by developing new uses for agricultural products or developing premium products that consider animal welfare, sustainability, food safety and quality, and health benefits).

Evidence from the literature suggests that investments in agricultural R&D contribute to improved productivity and generate a substantial return on investment in the long-term. A recent study that examined agricultural Total Factor Productivity in Canada, Australia, and the United States found that investment in R&D capacity is a key contributor to long-term productivity growth.^{Footnote 9} The study assessed Total Factor Productivity impacts using a dynamic panel regression analysis. After controlling for climate conditions and services from public infrastructure, the study found that coefficients associated with R&D knowledge stocks are positive and significant at the one percent level, with consistent results using different scenarios. The results imply that a one per cent increase in the R&D knowledge stock tends to raise the agricultural Total Factor Productivity level by more than 0.3 per cent.

Investment in agricultural research and development

The federal government continues to play a lead role in funding agricultural R&D and Knowledge Transfer to complement and leverage limited investment, foster innovation in small and emerging agricultural industries and encourage industry to invest in research with long-term public good benefits.

There are limited public and private sector resources to invest in agricultural R&D in Canada. Total public agricultural R&D expenditures have been trending downwards in real terms since 2010-11 with $650 million in public sector investments in 2015-16, about 70% of which is from the federal government.^{Footnote 10} Similar trends have been reported in other jurisdictions, such as the United States and United Kingdom. Canada is likely to be impacted more by this trend, since federal and provincial governments generally represent the largest source of funding for agricultural R&D in Canada.^{Footnote 11} Canada continues to rank low compared to other countries with respect to private sector investment in R&D. According to a recent Organization for Economic Co-operation and Development study on Canadian agricultural innovation policies, Canada does not perform well in terms of business expenditures on R&D (Business Enterprise R&D Spending) and patenting compared to other Organization for Economic Co-operation and Development countries. ^{Footnote 12} Though private sector investments in agricultural R&D have grown steadily since the 1980s (particularly in crop varieties and food processing) reaching approximately $226 million in 2011, several factors constrain private sector R&D investment such as narrow profit margins and lower intellectual property protection for plant breeding compared to other Organization for Economic Co-operation and Development countries.^{Footnote 13}

Private sector investment in agricultural R&D varies by sector size, profitability, province, and mechanisms in place to raise industry investment. Commodity-based sectors, such as beef, canola and pulses, often raise industry funding for marketing and research from mandatory levies or check-off systems. Investments are managed by non-profit industry associations, which tend to have more capacity to fund agricultural R&D since structured systems are in place. Other smaller and emerging sectors, such as bioproducts, tend to raise funds from individual industry partners, with specific goals to commercialize technologies. For Agri-science clusters in these industries, it is more difficult to raise private investment due to the higher number of partners and agreements required.

The level of industry investment in agricultural R&D also varies in terms of the focus of the activities and timing of the expected return on investment. Industry stakeholder interviewees reported that in the absence of AAFC financial assistance they would likely continue to fund some productivity-related research expected to generate short-term returns, but less likely to fund R&D related to market expansion, as it tends to be more costly and take a longer time to yield returns. Market expansion activities also tend to be associated with more public good benefits such as agricultural and agri-food products that consider animal welfare, sustainability, food safety and quality, and health benefits.

Agricultural research and development capacity

There is an ongoing need to leverage agricultural R&D capacity, including highly qualified personnel, in Canada. The Canadian agricultural research and innovation system is complex and involves a range of federal and provincial government, university, industry, and other stakeholders. A recent study conducted by AAFC on Canadian agricultural innovation capacity estimates that agricultural R&D is performed at over 300 research facilities across Canada, each with slightly different roles and areas of focus:^{Footnote 14}

Federal government facilities. For over 125 years, AAFC has played an important role in maintaining the capacity to undertake agricultural science research in Canada, particularly early-stage research with longer-term benefits to the Canadian public and to industry. According to the 2016-17 Science and Technology Branch Plan, the Branch employed 1,986 individuals as of December 31, 2015, making up approximately 45% of AAFC's workforce. The Branch is the most regionally distributed in the Department, with an extensive network of Research and Development Centres, farms, labs, and offices distributed across the country. Some other federal organizations also maintain facilities that are used to conduct agricultural R&D such as the National Research Council of Canada's Canadian Wheat Improvement program and the Canadian Institute of Health Research probiotics research. Federal government facilities focus on a range of sub-sectors and areas such as biodiversity and bioresources, crops, and agri-food.
Provincial government facilities. Research at the provincial level is often performed through universities or by co-funded research institutes.^{Footnote 15} Only four provincial governments (Quebec, Alberta, Manitoba, and Ontario) fund and perform research and their capacity varies. Provincial government facilities tend to focus on agri-food and processing. Compared to the federal government, provincial governments allocate fewer financial resources to maintain R&D facilities, spending less than one third of the amount spent by the federal government on R&D activities in 2015-16.^{Footnote 16}
University facilities. There are over 90 academic research facilities at universities in Canada whose primary focus is the agriculture sector. There are five veterinary colleges and nine dedicated agricultural universities. The primary role of academic institutions is to develop new highly qualified personnel and research tends to focus on sustainable practices, plant variety development, and plant and animal genetics.
Private sector facilities. There are at least 80 noteworthy corporate players in the agriculture and agri-food sector operating R&D facilities. Industry stakeholders can be involved as funders, performers, marketers, adopters and implementers of agricultural research. Companies are most frequently active in the area of crops, nutrition and processed food. Private sector companies operating in Canada have limited capacity to maintain R&D facilities compared to, for example, United States counterparts, due to less integrated supply chains and lower product volumes resulting in narrower profit margins and limited funding for R&D.^{Footnote 17}
Independent facilities. Approximately 130 independent R&D facilities conduct agricultural research and tend to focus on crops and technology transfer.

An estimated 3,580 highly qualified personnel^{Footnote 18} conduct agricultural R&D at these facilities, of which 380 are AAFC scientists.^{Footnote 19} Highly qualified personnel are recruited through various mechanisms such as Canadian universities (for example, 28,332 students were enrolled in agricultural and natural resources programs in Canadian colleges and universities in 2012) and international recruitment, exchanges and partnerships.^{Footnote 20}

Several challenges may constrain the ability to meet future highly qualified personnel needs to maintain the sector's R&D capacity. It is estimated that more than 60% of agricultural sciences highly qualified personnel in Canada will retire over the next five to twenty years.^{Footnote 21} Similar trends have been reported in the United States and Australia. The disciplines facing the greatest potential shortages include both traditional areas such as plant, soil and animal science, agronomy, biosystematics/taxonomy, and entomology as well as emerging areas such as genetics and bioinformatics. The potential consequence of a lack of highly qualified personnel in agricultural sciences in the future is that the sector will not have the capacity to meet its R&D needs, which could impact its competitiveness. There are also challenges to integrating newer scientists due to limited opportunities to train with more experienced scientists and lack of industry awareness of newer scientists. The supply of new highly qualified personnel is further constrained by competition from other sectors, a lack of awareness of agricultural science careers on the part of students, and declining enrolment rates in agricultural programs (for example, 1.42% of all students enrolled in Canadian colleges and universities were enrolled in agricultural and natural resources programs in 2012, compared to 1.71% in 2002).^{Footnote 22}

The Industry-led R&D component addresses the issue of limited sector R&D capacity by providing the opportunity for firms that are interested in innovation but lack the highly qualified personnel and the facilities to undertake research to work in collaboration with AAFC scientists and other scientists from universities, the private sector or other R&D organizations. Stream B-funded R&D activities can also provide opportunities for university graduate students to gain research experience.

Collaboration and knowledge transfer

There is a need to facilitate agricultural R&D collaboration since it leverages R&D investment and capacity and generates research excellence. Collaboration optimizes the capacity available for agricultural R&D by leveraging resources across Canada to support national goals and priorities. According to the Organization for Economic Co-operation and Development, collaboration is needed due to increasing demands for more specialized or capital-intensive areas of science and due to the fact that few organizations have sufficient funds or highly qualified personnel capacity to address all of these needs without working with external collaborators.^{Footnote 23}

R&D collaboration between different organizations contributes to improved research excellence and tends to result in increased knowledge and technology transfer. A study of collaboration among thirteen federal government departments found that compared to overall publications, publications with co-authors tended to result in more highly cited papers, a measure of a paper's contribution to the field of research. AAFC's highly cited paper rates were higher among co-publications involving external organizations compared to overall publications (1.36 versus 1.24), particularly among co-publications with international organizations (1.65), independent R&D facilities (1.61), private sector organizations (1.47), and academic institutions (1.41).^{Footnote 24}

The collaboration study also found that AAFC collaboration rates are increasing at a higher rate than other federal government departments involved in scientific research. AAFC showed steady year-over-year increases, from a low of 63.8% publications involving external collaborators in 2000 to a high of 87.4% in 2014 (Figure 1). This growth was driven by increased collaboration with partners outside the federal government. The trend coincides with the introduction of the cluster funding model under Growing Forward and its continued support under Growing Forward 2 through Stream B, which is designed to support and strengthen collaboration. International collaboration rates have shown steady growth since 2000, and are approaching the 50% mark in recent years (for the thirteen federal departments and agencies, overall). AAFC and Environment Canada are leading the way in increases to international collaboration.^{Footnote 25}

Federal organization	Number of co-publications	Collaboration rate	Minimum	Maximum
Total – 13 Federal Canadian federal departments	49,554	79.6%	69.9%	87.9%
PHAC – Public Health Agency of Canada	2,601	94.2%	79.0%	100%
TC – Transport Canada	52	86.7%	33.3%	100%
ECCC – Environment and Climate change Canada	7,975	86.6%	76.1%	91.0%
DFO –Fisheries and Oceans Canada	5,108	86.6%	72.3%	92.7%
AECL – Atomic Energy Canada Ltd.	1,036	81.9%	66.7%	90.8%
CSA – Canadian Space Agency	342	79.9%	62.5%	94.7%
NRCan – Natural Resources Canada	7,049	79.9%	73.2%	87.4%
NRC Canada – National Research Council	13,190	79.2%	69.6%	91.2%
DRDC – Defence Research and Development Canada	1,601	78.5%	68.2%	86.3%
HC – Health Canada	2,248	76.4%	70.5%	82.9%
AAFC – Agriculture and Agri-Food Canada	10,221	75.5%	63.8%	87.4%
CFIA – Canadian Food Inspection Agency	1,004	74.6%	58.6%	83.3%
CRC – Communications Research Centre Canada	445	69.2%	50.0%	90.7%

Figure 2 shows the external R&D collaboration patterns between Canadian federal departments and other organizations (academic, private, and other). The thickness of the lines represents the frequency in which the departments collaborate with particular types of external organizations. AAFC collaborates most frequently with academic institutions. Among the thirteen federal government departments reviewed, AAFC, National Research Council Canada, and Environment Canada are the most central actors in the collaboration network (represented by the size of their representative circles).

Department or agency	Number of collaborations with academic partners	Number of collaborations with private sector	Number of collaborations with other sector	Total collaborations by department or agency
Agriculture and Agri-Food Canada - AAFC	6,530	812	2,553	9,895
Atomic Energy of Canada Limited - AECL	302	92	136	530
Canadian Food Inspection Agency - CFIA	391	63	276	730
Communications Research Centre of Canada - CRC	283	53	25	361
Canadian Space Agency - CSA	224	57	91	372
Department of Fisheries and Oceans - DFO	2,989	374	1,813	5,176
Defence Research and Development Canada - DRDC	968	236	184	1,388
Environment and Climate Change Canada - EC	5,048	578	2,234	7,860
Health Canada - HC	929	57	254	1,240
National Research Council - NRC	9,402	1,177	2,908	13,487
Natural Resources Canada - NRCan	4,473	762	2,085	7,320
Public Health Agency of Canada - PHAC	1,609	135	880	2,624
Transport Canada - TC	27	10	5	42
Total collaborations by type	33,175	4,406	13,444	51,025

Despite increasing collaboration rates, the Organization for Economic Co-operation and Development study on Canadian agricultural innovation policies identified major collaboration and knowledge transfer gaps: "No other country among a group of high R&D spenders, displays such a large divergence between human resources and research infrastructure, and firm R&D and patenting activity. The imbalance between world class academic research and lacklustre business R&D has led policy makers to re-examine the linkages between academia and business."^{Footnote 26} Another study found that Canadian academic institutions involved in agricultural R&D have limited interest in collaborating with industry partners, while industry partners see these institutions as critical to meeting their R&D needs.^{Footnote 27} Stream B addresses these needs by facilitating collaboration and knowledge transfer to maximize the use of limited resources and facilitate uptake of research results. The Industry-led R&D Agri-science clusters model addresses these constraints by encouraging connections between industry, academia and AAFC to further sector goals. The AAFC-led Knowledge Transfer component complements these activities by facilitating the transfer of AAFC-led R&D results to sector stakeholders.

4.2 Performance – Effectiveness

The following section examines the performance of Stream B in terms of the extent to which it has achieved its intended outputs and outcomes.

4.2.1 Activities

As of March 31, 2016, $220.5 million in AAFC funding has been committed to Stream B activities taking place from 2013-14 to 2017-18. A total of $217.0 million has been committed to Industry-led R&D activities, the largest share of which is dedicated to the oilseeds, cereals and pulses, and horticulture sub-sectors (Figure 3). A total of $3.5 million has been committed to AAFC-led Knowledge Transfer projects taking place from 2013-14 to 2017-18, primarily in the area of agro-ecosystem productivity and health (Figure 4).

Program component	Funding committed in millions
Oilseeds	42.4
Cereals and pulses	37.5
Horticulture	31.3
Dairy, pork, poultry, other livestock	27.4
Beef and forages	13.2
Bioproducts	12.4
Agri-food	9.5
Agro-ecosystem productivity and health	6.4
Organic	6.0
Greenhouse	3.4
Other	27.6

Program component	Funding committed in millions
Agro-ecosystem productivity and health	2.0
Biodiversity and bioresources	0.6
Horticulture	0.2
Beef and forages	0.2
Cereals and pulses	0.2
Bioproducts	0.1
Agri-food	0.1
Other	0.1

4.2.2 Outputs

Stream B has achieved the targets of 14 Agri-science clusters formed in priority areas for the sector and 20 Knowledge Transfer project proposals approved (Table 3). More than two-thirds of the remaining targets have been achieved: 60 of the targeted 70 Collaborative Research and Development Agreements have been signed, 86 of the targeted 120 Agri-science projects have been approved, and $217.0 million of the budgeted $260.5 million in AAFC investment has been committed.

**Table 3: Achievement of Stream B Outputs**
	Program outputs	Indicator	Growing Forward results	Target for March 31, 2018	Achieved as of March 31, 2016
Industry-led R&D	Collaborative Research and Development Agreements	# of Collaborative Research and development Agreements	62	70	60
	Agri-science Clusters	# of clusters formed in priority areas for the sector	10	14	14
	Agri-science Projects to develop new products, process or technologies	# of projects to develop new products, process or technologies	81	120	86
	Funding provided by AAFC	AAFC Investment	$171.0 million	$260.5 million	$217.0 million (Committed)
AAFC-led knowledge transfer	Project Proposals	# of Knowledge Transfer proposals approved	14	20	20
Source: Stream B Program Performance and Risk Management Strategy and AAFC Databases

Since Agri-science projects are accepted through a continuous intake process, it is expected that these numbers will increase over the remaining two years of the program. Stream B has already surpassed Growing Forward program results with respect to most outputs. Four additional Agri-science clusters have been formed, more industry-led Agri-science and AAFC-led Knowledge Transfer projects have been approved, and AAFC investment in Industry-led R&D activities has increased by 27%.

4.2.3 Outputs

Outcome: The sector has access to government and private R&D capacity and funding to support research, development and the application of knowledge and technologies.

Stream B has facilitated increased private sector investment in Industry-led R&D. As of June 2015, Stream B has resulted in $97.2 million in private sector investment (Table 4), more than double the $44.7 million in private investment for similar activities under Growing Forward and 90% of the level of private investment in Stream B activities targeted for March 31, 2018. The increase in private investment is attributable to increased AAFC investments, modified cost-shared ratios, and increased industry awareness and interest in the program. Approximately $4.0 million in other government investment has also been raised toward Stream B activities, exceeding Growing Forward results but below the amount of $9.0 million targeted for March 31, 2018.

**Table 4: Achievement of Stream B Outcomes – R&D Funding Leveraged - Industry-led R&D**
Program outcome	Indicator	Growing Forward results	Target for March 31, 2018	Achieved as of June 2015
The sector has access to government and private R&D capacity and funding to support research, development and the application of knowledge and technologies	Dollar value of private sector investments and other government investments	Private investment: $44.7 million Other Government investment: $3.7 million	Private investment: $108 million Other Government investment: $9.0 million	Private investment: $97.2 million Other Government investment: $4.0 million
Source: Stream B Program Performance and Risk Management Strategy and AAFC Investment Analysis Data, June 2015

Various factors influence the level of private sector investment in Industry-led R&D activities. Stream B guidelines require that activities led by private companies meet a minimum 50% industry matching requirement, whereas activities led by not-for-profit organizations require 25% matching. According to an investment analysis prepared by AAFC's Innovation and Growth Policy Division (Figure 5), a higher proportion of industry investment was leveraged by Agri-science projects (36% of budgeted project costs) compared to Agri-science clusters (28%). Figure 6 shows that those focused on developing new machinery and computer-related equipment leveraged a higher percentage of industry investment compared to other sectors and cross-cutting areas. This is expected since Agri-science projects and machinery and computer-related equipment R&D activities are more likely to involve private sector companies. It was also reported by AAFC staff that during the fall of 2013, the policy was changed to give preference to applicants that raised 50% in matching funding due to the high level of demand for the program. This impacted the proportion of private sector funding from Agri-science projects more than Agri-science clusters since most clusters were approved during the first intake.

Project type	Source	Percentage
Clusters	Vote 10	49
Industry	28
Vote 1	23
Projects	Vote 10	44
Industry	36
Vote 1	17
Other	3

Sector and cross-cutting area	Source	Percentage
Machinery and computer related	Vote 1	9
Vote 10	41
Other government department	2
Industry	48
Bioproducts	Vote 1	7
Vote 10	51
Other government department	4
Industry	38
Agro-ecosystem productivity and health	Vote 1	15
Vote 10	44
Other government department	4
Industry	37
Oilseeds	Vote 1	20
Vote 10	47
Industry	32
Agri-food	Vote 1	24
Vote 10	43
Other government department	1
Industry	32
Horticulture	Vote 1	21
Vote 10	45
Other government department	2
Industry	32
Dairy, pork, poultry, other livestock	Vote 1	9
Vote 10	59
Other government department	1
Industry	31
Cereal & pulses	Vote 1	38
Vote 10	31
Other government department	2
Industry	28
Organic	Vote 1	25
Vote 10	50
Industry	25
Forages & beef	Vote 1	34
Vote 10	44
Industry	22

Stream B has helped to improve the efficiency of private sector investments in agricultural R&D. Many of the Agri-science cluster leads interviewed reported that the clusters have facilitated a more efficient use of research investments by pooling provincial industry resources to support sector research priorities on a national scale. For example, the Swine Agri-science Cluster successfully introduced a national research check-off^{Footnote 29} of 2.5 cents per head in 2013 through a memorandum of understanding signed with seven provinces and attracted 40 industry partners in the call for proposals. The proportion of the Canadian beef cattle producer national check-off dedicated to research increased from 6% in 2010-11^{Footnote 30} to 16% in 2015-16.^{Footnote 31} According to Agri-science cluster lead interviewees, the increase in industry investment in research is due, in part, to the cluster's ability to articulate national research priorities and demonstrate impacts as a result of the cluster. Some Agri-science clusters are seeking to leverage further private sector investment from new sources such as seed companies (Field Crops Agri-science Cluster) and the food industry (Pulse Agri-science Cluster).

Outcome: The sector increases its collaboration and partnership for research and development

Collaboration between AAFC scientists and other scientists appears to be increasing. As of October 2016, 42% of Agri-science projects and clusters involved both Vote 1 and Vote 10 activities, exceeding the Growing Forward results of 40% but less than the 50% target expected by March 31, 2018 (Table 5). AAFC's rate of scientific collaboration with external organizations (that is, co-publication as a percentage of total publications) has been increasing over time.^{Footnote 32} Interviews with AAFC representatives suggest that this is due to Agri-science cluster programming.

**Table 5: Achievement of Stream B Outcomes – Research Collaborations - Industry-led R&D**
Program outcome	Indicator	Growing Forward results	Target for March 31, 2018	Achieved as of October 2016
The sector increases its collaboration and partnership for research and development	% of projects involving both Vote 1 and Vote 10 activities	40%	50%	42%
Source: Stream B Program Performance and Risk Management Strategy and AAFC Databases

Agri-science cluster lead interviewees suggest that Stream B activities have strengthened research networks and the capacity to leverage investment in R&D. New research networks have been established, and networks that had been developed during Growing Forward were strengthened and expanded across provinces and different types of organizations. Agricultural scientists are connecting with researchers they had not engaged with before. For example, Beef Agri-science Cluster activities led to the development of an integrated network of specialists focusing on antimicrobial resistance in beef production including veterinarians, feedlot researchers and other federal government researchers from the Public Health Agency of Canada.

An estimated 88% of all Industry-led R&D activities involved one or more collaborator type in addition to the recipient. About one fifth of activities involved two or more collaborator types, with 27 activities involving larger research teams with four or more collaborator types. The most frequent collaborators on Agri-science cluster or project activities were AAFC research centres and post-secondary institutions (Figure 7).

Collaborator types also varied by sector and cross-cutting area. For example, there was more frequent involvement of AAFC Research Centres in cereals and pulses activities and more frequent involvement of post-secondary institutions in dairy, pork, poultry, and other livestock activities (Figure 8).

Collaborator type	Number of activities
AAFC research centres	366
Post-secondary institutions	342
Other for-profit corporations	65
Supply chain organizations	54
Other research centres	50
Provincial government departments	43
Producers	31
Other not for-profit corporations	29
Provincial research centres	28
Processors	13
Other federal government departments	9

Sector and cross-cutting area	Collaborator type	Percentage
Cereals and pulses	AAFC research centres	62
Post-secondary institutions	27
Other provincial or federal governments	3
For-profit companies	8
Other research centres or not-for-profit	0
Machinery and computer related	AAFC research centres	47
Post-secondary institutions	22
Other provincial or federal governments	3
For-profit companies	3
Other research centres or not-for-profit	25
Organic	AAFC research centres	40
Post-secondary institutions	45
Other provincial or federal governments	4
Oilseeds	AAFC research centres	38
Post-secondary institutions	33
Other provincial or federal governments	3
For-profit companies	22
Other research centres or not-for-profit	6
Horticulture	AAFC research centres	30
Post-secondary institutions	28
Other provincial or federal governments	12
For-profit companies	16
Other research centres or not-for-profit	14
Agro-ecosystem productivity and health	AAFC research centres	27
Post-secondary institutions	28
Other provincial or federal governments	21
For-profit companies	18
Other research centres or not-for-profit	7
Agri-food	AAFC research centres	25
Post-secondary institutions	36
Other provincial or federal governments	5
For-profit companies	23
Other research centres or not-for-profit	11
Bioproducts	AAFC research centres	24
Post-secondary institutions	24
Other provincial or federal governments	3
For-profit companies	47
Other research centres or not-for-profit	3
Dairy, pork, poultry, other livestock	AAFC research centres	21
Post-secondary institutions	58
Other provincial or federal governments	2
For-profit companies	8
Other research centres or not-for-profit	11

The capacity built through Stream B activities has leveraged further follow-on projects, leading to additional R&D impacts. For instance, a Field Crops Agri-science Cluster-funded soybean breeding systems research collaboration between a Quebec not-for-profit, two universities and an AAFC research centre has led to a follow-on project with $8 million in funding from Genome Canada and various other provincial, industry and private sector organizations.^{Footnote 34} Agri-science cluster lead interviewees note that the increased collaboration in research has also resulted in better value for money since a broader range of expertise and capacity is leveraged per dollar of investment. Collaboration with industry has helped to better link research to commercial priorities, which makes the research more relevant for industry applications.

Outcome: AAFC-led Innovations Developed and Identified for Transfer: Technology, knowledge and information covering the full range of AAFC research, development and technology is developed for stakeholders, target groups and the sector

AAFC-led Knowledge Transfer projects resulted in 8 new/improved products identified and made available for transfer to the sector, 71 scientific publications (for example, articles or papers, conference proceedings and abstracts), and 4 technology transfer publications (Table 6). Examples of AAFC-led Knowledge Transfer activities are provided in Text Box 1.

**Table 6: Achievement of Stream B outcomes – AAFC-led innovations developed and identified for transfer - AAFC-led Knowledge Transfer**
Program outcome	Indicator	Growing Forward results	Target for March 31, 2018	Achieved as of March 31, 2016
Technology, knowledge and information covering the full range of AAFC research, development and technology is developed for stakeholders, target groups and the sector	# of existing technologies identified and made available for transfer to the sector	N/A	N/A	8 new/ improved products
Knowledge Transfer Instruments	# of knowledge transfer instruments developed (for example, brochures, flyers and trade journals, etc.)	N/A	N/A	71 scientific publications 4 technology transfer publications
N/A: Not available Source: Stream B Program Performance and Risk Management Strategy and AAFC Databases

Text Box 1. Examples of AAFC-led Knowledge Transfer Activities

Helping First Nations to better understand ancestral varieties and growing methods related to the Three Sisters: maize, squash, and beans for potential food applications (for example, maize flour). This project resulted in the development of flyers, translated in English, French and First Nations languages. Three First Nations signed memorandums of understanding with AAFC agreeing to participate in the project.
Enhancing knowledge of beneficial management practices related to wastewater from confined livestock sites among producers and regulators to minimize health and environment risks. The project resulted in the development of a factsheet and presentations to six different producer organizations and groups.
Improving access to weather and climate decision support tools, which can help agricultural producers to better manage climate-related risks and optimize pesticide, nitrogen and irrigation water use. The project resulted in publications such as an AgWeather Atlantic factsheet, guide, and YouTube video.
Transferring research on water and nutrient management to the horticultural sector, helping growers to better respond to regulatory requirements. This project aimed to utilize AAFC research to quantify and communicate the environmental risks for different horticultural production systems in Ontario and resulted in presentations to thirteen grower and regulatory organizations.

4.2.4 Intermediate outcomes

Outcome: Development of Innovations: The sector develops or advances knowledge and technologies for industry uptake for commercialization or adoption

Stream B activities have resulted in many new discoveries, products and technologies. As of March 31, 2016, Industry-led R&D activities have exceeded both Growing Forward results and Stream B targets with respect to the number of discoveries and technologies generated. In total, 870 new discoveries and technologies including new Intellectual Property items, new or improved products, processes, practices, varieties and knowledge have been developed, greater than the 540 targeted by March 31, 2018 (Table 7). A total of 844 peer reviewed publications have been published, achieving 84% of the 1,000 targeted by program close. As R&D progresses in each year of the program, the results are higher year over year. Therefore, it is expected that these targets will be met. The Achievement database indicates that 35 products developed have commercial potential.

**Table 7: Achievement of Stream B Outcomes – Development of Innovations - Industry-led R&D**
Program outcome	Indicator	Growing Forward results	Target for March 31, 2018	Achieved as of March 31, 2016
The sector develops or advances knowledge and technologies for industry uptake for commercialization or adoption	# of discoveries, technologies generated (for example, new Intellectual Property items, new or improved products, processes, practices, varieties and knowledge)	540	540	870
	# of peer reviewed scientific publications	1,000	1,000	844
	# of products that have commercial potential	N/A	N/A	35
N/A: Not available

There are some challenges with respect to how innovations are categorized and counted. For instance, some scientists may define a step towards developing an innovation as a type of innovation, while others may use a more stringent definition. It was found that performance measures could be refined under a new policy framework to reflect only final results (for example, new varieties) and not those representing a step on the way (for example, gene sequences). The clarity of definitions and categories could also be improved.

Several examples of innovations resulting from Stream B activities were identified (Text Box 2), some of which were continuations of activities initiated under Growing Forward or other programming predating Growing Forward 2. Innovations are often a result of many years of research, development and scale-up work. For most Stream B activities, it is too early to have seen an impact in the sector due to the long-term nature of the R&D work undertaken under Stream B.

Text Box 2. Examples of Innovations and Discoveries Resulting from Stream B Activities

New Uses for Agricultural Products

Alternatives to conventional fibreglass vehicle parts using flax and hemp fibers. Recent investments are focused on efforts to develop a process to test natural fibres for viability in bio-composite products. These applications will open up a new revenue stream for farmers and provide more environmentally sustainable product options in the long-term.
Process to convert mushroom waste into nutrient-rich fertilizers, providing a new revenue stream for the 2.5 billion kilograms of waste discarded by mushroom growers in North America each year.
Camelina-based eco-friendly alternative to castor oil. Camelina is a drought tolerant oilseed crop that is being developed as an eco-friendly alternative to castor oil. Samples have been sent to a large multinational company for possible use in polyurethane coating applications.

Improved product attributes

Defining the environmental 'hoofprint' of beef production. Beef is often portrayed as environmentally unfriendly due to concerns about Greenhouse Gases and other waste. This project identified that producing 1 kg of beef today creates 15% less GHGs than in 1981 due to improved production practices.
Environmentally friendly water extraction system in nutraceutical processing. Growing Forward and Growing Forward 2 investments support the development of a more environmentally friendly water extraction system, which consumes less energy and whose by-products are non-toxic. Four new extract products were developed through this new system for use in nutraceuticals.
Tools which will improve meat quality and animal welfare in swine production. Research is being conducted into ways of identifying hormones associated with an unpleasant smell when meat is cooked, in order to remove affected pigs from the breeding stock and improve animal welfare practices.

Technologies and products that improve productivity

Drone-based technology to quickly assess soil yield in corn fields. Four new products have been developed including a quick, inexpensive and reliable test that identifies high and low yielding soil areas in corn fields by employing drones equipped with multispectral sensors.
Wireless sensors to assess crop disease prediction and prevention. Growing Forward and Growing Forward 2 investments support the development and testing of new wireless sensors that virtually collect data needed for crop disease prediction and prevention for producers in real-time, saving in labour costs, crop losses, and excess use of pesticides.
Bio-based honeybee crop control system. A honeybee dispenser system has been developed to enable crop control through pollination without the adverse consequences of spraying chemicals. The project has demonstrated that the system functions effectively on large field scale canola crops.

Outcomes: Technology, Knowledge and Information Transferred:

The sector transfers research findings and methods to target groups/project stakeholders;
Technology, knowledge and information is transferred to intended users, farms and firms, thereby enhancing sector competitiveness and profitability

All technology, knowledge and information transfer targets have been exceeded and have surpassed Growing Forward results (Table 8). As of March 31, 2016, Stream B Industry-led R&D activities have resulted in 2,208 knowledge transfer instruments (for example, brochures, flyers, trade journals, etc.) and 2,645 knowledge transfer events (for example, presentations at industry tradeshows and conferences), tripling the 700 instruments and 930 events targeted. There were a reported 1,065,437 participants in these knowledge transfer events, exceeding the target of 50,000, among which 25,535 indicated that they intend to adopt the knowledge and technologies presented at these events, surpassing Growing Forward results and the Stream B target of 1,000. In light of the higher than expected outcomes, performance measures related to knowledge transfer could be refined under a new policy framework to ensure they accurately capture activities where knowledge is transferred to a target audience that could adopt the innovations.

**Table 8: Achievement of Stream B Outcomes – Technology, knowledge and information transferred**
	Program output/outcome	Indicator	Growing Forward results	Target for March 31, 2018	Achieved as of March 31, 2016
Industry-led R&D	Knowledge Transfer Instruments	# of knowledge transfer instruments developed (for example, brochures, flyers and trade journals, etc.)	700	700	2,208
	Knowledge Transfer Events	# of knowledge transfer events organized through the sector	930	930	2,645
	Knowledge Transfer Events	# of individuals participating in knowledge transfer events	50,000	50,000	1,065,437
	The sector transfers research findings and methods to target groups/project stakeholders	# of individuals participating in knowledge transfer events intending to adopt new knowledge and technologies	1,000	1,000	25,535
		# of media reports	480	480	623
AAFC-led knowledge transfer	Technology, knowledge and information is transferred to intended users, farms and firms, thereby enhancing sector competitiveness and profitability	# of knowledge transfer events organized by AAFC	N/A	N/A	101 knowledge and expertise contributions
AAFC-led knowledge transfer		# of individuals participating in knowledge transfer events organized by AAFC	N/A	N/A	Data Not Available
N/A: Not available Source: Stream B Program Performance and Risk Management Strategy and AAFC Databases

According to interviews with Agri-science cluster leads, different factors impact the ease with which knowledge and technology is transferred. For example, knowledge transfer systems are in place for Intellectual Property-related innovations through breeding programs or projects involving private sector partners. Though a major role for Agri-science clusters is to undertake knowledge transfer, capacity to support broader-based knowledge transfer activities varies by cluster. In some cases, an Agri-science cluster lead organization may be a provincial industry association with limited mandate or capacity to undertake widespread industry knowledge transfer. Since much of the R&D activities are contracted to universities, it can be challenging for some Agri-science clusters to take ownership of the research results and effectively translate and disseminate findings to industry stakeholders. Some Agri-science clusters are employing innovative techniques to disseminate knowledge such as a mobile-friendly interactive research hub, webinars, and conferences with closed-door sessions to discuss confidential technologies.

AAFC-led Knowledge Transfer projects resulted in 101 knowledge and expertise contributions, most frequently consisting of speaking or presentation activities and acting as a scientific or technical authority.

Outcome: The sector has additional capacity to develop knowledge and technologies

The program has made progress towards its target of newly trained highly qualified personnel in the sector. As indicated in Table 9, as of March 31, 2016, 103 persons completed a MSc or PhD during program activities, about 40% of the 260 targeted by March 2018. Since the number of graduates per year is increasing year over year, the program is on track to meet its target by March 31, 2018.

**Table 9: Achievement of Stream B Outcomes – Improved Sector Innovation Capacity - Industry-led R&D**
Program outcome	Indicator	Growing Forward results	Target for March 31, 2018	Achieved as of March 31, 2016
The sector has additional capacity to develop knowledge and technologies	# of newly trained, highly qualified personnel in the sector	260	260	103 persons who completed a MSc or PhD during project/activity
Source: Stream B Program Performance and Risk Management Strategy and AAFC Databases

Agri-science cluster leads advise that the program has helped to build sector highly qualified personnel capacity in various areas. For example, during a Field Crops Agri-science Cluster activity, an AAFC scientist who is one of the few Canadian specialists in corn inbred development was able to train a graduate student researcher at the University of Manitoba. A Swine Agri-science Cluster activity to develop effective feeding strategies for piglets involved nine graduate students and resulted in two participating students completing their PhD and one participant completing their MSc.

4.2.5 End outcome

Outcome: The sector produces, adopts, and commercializes innovative products, processes, practices, services and technologies

The program has made some progress towards the achievement of its intended end outcome. As of March 31, 2016, a total of five completed Agri-science projects were reported to have resulted in 20 commercial products. One innovation developed has been adopted by sector stakeholders (Table 10). It is too early to expect that Stream B supported R&D activities have realized all impacts associated with the end outcome because the program is not yet complete. Recipients are only asked to report on end outcomes in their final performance report. As of March 31, 2016 the program had received final performance reports for 11 completed projects^{Footnote 35}. Due to the nature of the R&D funded and the location of these activities along the innovation continuum, it can take two to ten years following activity completion for sector stakeholders to further develop, adopt or commercialize products or practices resulting from Stream B innovations. There are also some limitations to using Total Factor Productivity Growth as an indicator since it is also impacted by many factors beyond the scope of the program, such as investments by other programs or private sector companies in agricultural R&D. Data was not available for this indicator at the time of the evaluation.

**Table 10: Achievement of Stream B Outcomes – Sector Production, Commercialization and Adoption of Innovations - Industry-led R&D**
Program outcome	Indicator	Growing Forward results	Target for March 31, 2018	Achieved as of March 31, 2016
The sector produces, adopts, and commercializes innovative products, processes, practices, services and technologies	Number of projects that resulted in a commercial product	N/A	N/A	5
	Products being used/adopted by the sector	N/A	5 case studies that demonstrate how products are used	1 innovation adopted by the sector; 2 case studies: Swath Grazing & Midge-tolerant wheat case studies (see Text Box 3)
	Total Factor Productivity Growth	100	108	Data Not Available
N/A: Not available

A cost-benefit analysis conducted by Programs Branch in 2015-16 of two successful innovations provided some evidence of high rates of return from long-term innovation investments. The analysis showed that the midge-tolerant wheat and swath grazing innovations benefited from AAFC, provincial government, and industry investments in R&D activities since the mid-1990s and are expected to result in benefit/cost ratios of 37:1 and 170:1 respectively by 2025 (see Text Box 3).^{Footnote 36}

Text Box 3. Case Study Examples of Return on Investment from AAFC R&D Investments

Midge-tolerant wheat – In 1997, research funded by AAFC, provinces, producer groups and industry began to develop midge tolerant wheat varieties. By 2010, the first blend was made commercially available. Since then, uptake has expanded rapidly. Benefits include: reduced yield losses; improved flexibility in crop rotation and seeding dates; lower use of insecticides; and higher producer revenues. Investments of $12.2 million over more than twenty years are expected to result in benefits of $456 million by 2025 (benefit/cost ratio of 37:1).
Swath grazing – Funding from AAFC, provinces and industry has been directed towards swath grazing research since 1995. Increases in the adoption of swath grazing have resulted in reduced labour, machinery and fuel expenses. Investments of $7.7 million over more than 20 years are expected to result in benefits of approximately $1.3 billion by 2025 (benefit/cost ratio of 170:1).

4.2.6 Economic impacts

Program activities have resulted in significant economic impacts. An economic impact assessment conducted as part of the evaluation found that AAFC funding is estimated to have facilitated $289.9 million in spending across the Canadian economy from 2013-14 to 2015-16 (Figure 9). The total facilitative economic impact represents the combined AAFC investment, leveraged funding from private sector and other government sources, and indirect output impacts (that is, the impact on sectors that supply goods and services to the sectors directly impacted by the R&D activities). When facilitative impacts are compared to the AAFC investment of $120.2 million during this time period, there is a ratio of $2.4 in total facilitative economic impacts generated for every dollar of AAFC investment.

AAFC funding	120.2 million
Leveraged funding	74.5 million
Indirect economic impact	95.2 million
Total facilitative economic impact over 3 years	289.9 million

The attributable economic impacts represent the portion of economic impacts which are attributable to the program (that is, would not have occurred in the absence of AAFC financial assistance). Based on interviews with Agri-science cluster leads, approximately 80% of the total facilitated economic impact is estimated to be directly attributable to AAFC financial assistance.^{Footnote 37} Table 11 outlines what this spending means in terms of Gross Domestic Product, wages and salaries, and jobs from 2013-14 to 2015-16. As an illustration, of the $175.6 million in Gross Domestic Product expenditures that were facilitated by Stream B activities from 2013-14 to 2015-16, $140.5 million in Gross Domestic Product expenditures would not have occurred in the absence of AAFC financial assistance.

**Table 11: Total economic impacts facilitated by and attributable to Stream B industry-led R&D expenditures, 2013-14 to 2015-16**
Economic impacts	Facilitative economic impacts	Attributable economic impacts
Spending (millions)	$289.9	$232.0
Gross Domestic Product (millions)	$175.6	$140.5
Wages and Salaries (millions)	$106.4	$85.1
Average annual employment (# of jobs[1])	570	456
[1] Includes full-time, part-time jobs, and temporary jobs. Source: Economic Impact Assessment

4.3 Performance – Efficiency and economy

4.3.1 Effectiveness of Industry-led R&D Stream B Design and Delivery^{Footnote 38}

Contributing factors to effective program design and delivery

The major factors contributing to the effective delivery of Stream B activities are the program funding mechanisms, which facilitate collaboration across various R&D funders and performers and encourage industry to support longer-term innovation activities. In addition, the continuous improvement approach employed by the program has led to increasingly efficient delivery over time.

Collaboration across R&D funders and performers

The funding mechanisms for the Industry-led R&D component of Stream B facilitate collaboration across R&D performers by encouraging researchers to work collaboratively across different regions and types of organizations (for example, industry, government, and universities). Almost 90% of all Industry-led R&D activities involved one or more collaborator type, most frequently involving AAFC research centres and post-secondary institutions. Research collaboration generates a variety of benefits, such as improved R&D capacity in areas that are relevant to agriculture and agri-food industries and strengthened research networks, which can leverage further investments for follow-on projects, leading to improved research excellence.^{Footnote 39} Collaboration leads to improved research results by bringing together different perspectives, experience, skills and knowledge and fostering cross-disciplinary interactions. Collaboration encourages skills and knowledge transfer, develops the new knowledge systems and social relationships required to maximize value and stimulate creativity. Collaboration with different research institutions leverages research investments, provides access to different sources of highly qualified personnel, and makes more efficient use of available resources.

Agri-science clusters and projects bring together provincial industry organizations to coordinate research funding at a national level. National Agri-science clusters have allowed agriculture and agri-food industry organizations to fund R&D projects across Canada that support common goals and priorities. This coordination optimizes the use of limited industry funding as it reduces the level of duplication in funding. Industry stakeholders also benefit from knowledge transfer across a broader range of projects.

The evaluation found some evidence that opportunities or mechanisms could be explored to facilitate increased collaboration between AAFC and other federal and provincial government organizations. According to Agri-science cluster lead interviewees, in some instances, federal and provincial government funding rules related to stacking and the use of Growing Forward 2 funding constrained collaboration. For instance, it was reported that an Agri-science cluster activity involving National Research Council of Canada and AAFC researchers did not move forward because funding could not flow from one federal department to another. A study of collaboration among thirteen federal government departments found that three of the four most productive organizations (National Research Council Canada, AAFC, and Natural Resources Canada) are also among the least engaged in inter-departmental collaborations.^{Footnote 40}

Cost-shared funding

Cost-shared funding mechanisms, which provide from 50% to 75% in AAFC matching funding for industry-led R&D, encourage industry stakeholders to fund larger scale and longer-term R&D activities. According to Agri-science cluster lead interviewees, in the absence of AAFC financial assistance, it is likely that industry stakeholders would continue to fund some provincial productivity-related research, but it is much less likely that they would fund national R&D related to market expansion. R&D activities that focus only on productivity improvements may not be sufficient to maintain the competitiveness of the sector due to its vulnerability to external threats such as shifts in input and output prices. Longer-term R&D activities tend to involve market expansion activities such as developing new uses for agricultural products or developing value-added products, which require long-term investments in R&D but have the potential to yield significant long-term returns to the Canadian agriculture and agri-food sector by capitalizing on expanding Canadian and international demand for food and agri-based products with particular attributes.^{Footnote 41}

Continuous program improvement

The ongoing efforts to refine the program under Growing Forward 2 have improved the efficiency of Stream B delivery processes. AAFC senior management seek feedback from funding recipients through annual meetings, regular communications, and in-person visits. In October 2016, a consultation session was held to discuss program design and delivery and included representatives of all 14 Agri-science clusters, five universities, AAFC representatives across 5 Branches, and the Treasury Board Secretariat. AAFC senior managers have responded to program delivery challenges reported through consultations by undertaking lean^{Footnote 42} exercises to enhance the efficiency of application and claims processes. Various refinements have been made to program delivery such as identifying the optimal number of project files per program officer (about 6 to 8 files per officer), moving from a staggered to a continuous intake of Agri-science project applications (though this process presents some challenges in terms of ensuring adequate resource allocation), accepting electronic signatures, and accepting a per diem allocation for travel expenditures as part of claims processing.

Stakeholder engagement and continuous improvement activities have led to improved performance against service standards and client satisfaction. The percentage of approval/rejection letters being sent within 100 business days of full receipt of complete applicant packages increased from 11% in Q3 2013-14 to 100% in Q4 2015-16, though it is important to note that 236 applications for clusters and projects were received in 2013-14, while 25 applications were received for projects in 2015-16. Recent reports indicate that service standards are being exceeded, which could warrant a review of the current targets (for example, 97% of transactions met the standard of payment sent within 30 days of receipt of a completed claim in Q2 2016-17, above the target of 80%). The Agri-science cluster leads interviewed were satisfied with most aspects of the program design and delivery, with many indicating that the processes have improved over time.

Design and delivery constraints

The main challenges associated with Stream B program design and delivery include the complex application process, delays in processing applications, requirements for claims processing, and limited ability to respond to emerging needs and sectors.

Complex application process and delays in processing applications

Though improvements have been made over time, the application and approval process for Industry-led R&D activities remains complex and time consuming. Agri-science cluster lead interviewees were least satisfied with the application process compared to other program design and delivery elements. Various factors contribute to the administrative complexity of the application process such as the number of collaborators involved and background documents required (for example, letters of approval from the managers of every proposed university collaborator) and the requirement for separate funding agreements for Vote 1 and Vote 10 elements.

Some steps in the review process are duplicative. For example, proposed Agri-science cluster activities are required to undergo technical peer reviews both by Agri-science cluster leads prior to the application (for example, by industry-led science advisory committees) as well as by AAFC scientists during the application review process. In some cases, industry-led science advisory committees are comprised of world leading scientific experts and technical advisors who review potential activities before they are approved by the Agri-science cluster lead organization to be included as part of the application to AAFC. Once the application is submitted, each of the activities undergoes subsequent reviews by AAFC scientists. Agri-science clusters include, on average, 35 activities, and each of these activities needs to be reviewed by different AAFC scientists. Some activities require more than one reviewer if they involve different areas of scientific expertise (for example, climate change and genomics). According to AAFC program delivery representatives, part of the AAFC internal review challenge is finding an AAFC scientist to review a particular activity who is both knowledgeable of the subject matter and not involved in the proposed research activity. Screening mechanisms could be refined so that fewer Agri-science clusters or projects are required to go through the full technical review process.

The complexity of the application process combined with the unexpectedly high number of applications at the outset of the program (292 Agri-science projects and 23 Agri-science cluster applications by Q3 2013-14) led to delays resulting in less than half of the available budget authority being spent in the first year of the program. AAFC representatives reported that the major factors contributing to delays included insufficient staff resources to manage high program demand, changes in program leveraging (during the second intake in October 2013, applicants were asked to re-submit with 50% cost matching ratios), and requirements for program applicants to provide audited financial statements. Some not-for-profit organizations withdrew their applications because the latter requirement was too costly. The application process limited the extent to which applicants could communicate with AAFC and modify proposals based on feedback from AAFC. Due to the amount of time that lapsed between completion of similar programming in Growing Forward and the commencement of Stream B program activities in Growing Forward 2, it was difficult for Agri-science clusters and projects to maintain research capacity since some specialists funded by the program moved on to other projects.

Requirements for claims processing

The level of documentation required for processing claims, including a recipient cover letter, financial update, and transaction listing is a burden to both recipients and research collaborators. Some recipients have required a level of detail from their partners beyond what is required for AAFC to process claims, which can strain the R&D collaborations facilitated by the program. AAFC has raised this issue with clusters recipients, to ensure that partners are no longer asked to provide more detail than is required by the program. Some Agri-science cluster lead interviewees expressed concern that documentation requirements could deter potential research collaborators from participating in Agri-science clusters and could impact the program's ability to leverage agricultural R&D highly qualified personnel and infrastructure capacity. A few AAFC representatives noted that the program's consistency in interpreting program guidelines with respect to the eligibility of expenses, sampling processes, and acceptable tolerance levels for discrepancies could be improved. These issues have been examined as part of the continuous improvement activities of the Innovation Programs Directorate, including through the establishment of a departmental, Director-level claims working group that reviews claims processes across a variety of AAFC programs in order to streamline processes and improve consistency.

Limited ability to respond to emerging needs and sectors

Several Agri-science cluster lead interviewees indicated their belief that the program does not allow sufficient flexibility to respond to emerging priorities or opportunities identified by private sector partners, since most activities are determined at the beginning of the Agri-science cluster or project and are relatively fixed. Reallocating funds between activities or towards new activities can take six months to one year. A few Agri-science cluster leads indicated that part of the challenge is due to the Vote 1 and Vote 10 funding structure which requires two separate funding agreements (collaborative research and development agreements and contribution agreements) and requires increased administration in terms of making amendments. AAFC representatives noted that the lack of flexibility may be a matter of perception among recipients about what is possible (that is, reallocating funding between activities) and what is not possible (that is, reallocating funding between years) within the program and that greater education of program guidelines could resolve misperceptions. It could also be possible to use the Agri-science projects rather than the clusters to address emerging issues.

According to Agri-science cluster lead interviewees and AAFC program delivery representatives, the program also does not adequately serve the needs of emerging agricultural sectors (for example, bioproducts), which tend to be more reliant on individual private sector partners and do not have access to industry check-off resources to fund sector-wide R&D activities. Larger, more established sectors are better equipped to raise industry funds and more likely to receive public funding due to the quality of their applications. Smaller, emerging sectors struggle to coordinate funding and have limited capacity to administer national research programs such as those administered through the Agri-science clusters.

4.3.2 Economy and efficiency

Most of the available budget authority for the first three years of the Industry-led R&D Stream B program has been spent and program spending has increased over time.^{Footnote 43} A total of $120.2 million in AAFC funding was spent on Stream B Industry-led R&D activities from 2013-14 to 2015-16, representing 80% of the available budget authority during this period (Table 12). Program spending as a percentage of the available budget authority increased from 49% in 2013-14 to 93% in 2015-16. AAFC program delivery representatives indicated that factors contributing to under-spending in 2013-14 were the large backlog in applications and subsequent delays in processing those applications. Program spending against authorities improved in following years as a result of continuous improvement activities implemented to address the backlog.

**Table 12: Stream B Industry-led R&D Expenditures and Available Budget Authority**
R&D expenditures and available budget authority	2013-14	2014-15	2015-16	Total
Available Budget Authority (millions)	$45.1	$47.7	$57.5	$150.4
Actual Expenditures (millions)	$22.1	$44.6	$53.5	$120.2
% of Budget Expended	49%	93%	93%	80%
Source: AAFC databases

To examine the efficiency of the delivery of Stream B activities, efficiency ratios were developed that compare program outcomes with program costs and commitments. These efficiency ratios were compared with overall program results for similar programs implemented during Growing Forward. Stream B, under Growing Forward 2, has achieved program outcomes more efficiently than similar programs implemented under Growing Forward (Table 13). AAFC commitments under Growing Forward 2 have leveraged $0.45 in private sector investment per $1 of AAFC investment, compared to $0.26 per $1 of AAFC investment under Growing Forward. Fewer program expenditures were required per innovation developed compared to previous programs ($138,161 versus $316,667).

**Table 13: Efficiency in Achieving Program Outcomes: Comparison between Growing Forward and Industry-led R&D Stream B AgriInnovation Programming**
Indicator	Growing Forward results	Growing Forward 2 results as of March 31, 2016
Private Sector Investment Committed	$44.7 million	$97.2 million
AAFC Funding Committed	$171.0 million	$217.0 million
Leverage for every $1 AAFC Committed	$0.26	$0.45
Number of Innovations Developed	540	870
AAFC Funding Expended	$171.0 million	$120.2 million
AAFC Cost Per Innovation	$316,667	$138,161
Source: AAFC databases and AIP Stream B Program Performance Measurement and Risk Management Strategy

4.3.3 Performance measurement and reporting systems

Several mechanisms are in place to track the performance of Stream B program activities. Agri-science cluster and project recipients are required to prepare annual performance reports that describe progress against targeted milestones and outcomes such as the number of new or improved products, peer reviewed publications, and individuals attending information events. The final performance reports submitted following project/cluster completion capture additional information on the commercial potential, commercialization, and adoption of the new or improved innovations. The AAFC Programs Branch tracks success stories and commercialization and adoption impacts in an achievement database spanning multiple policy frameworks. These systems have been successful in collecting a significant amount of data and information on program outputs, and identifying potential case studies and success stories. Nevertheless, some limitations were identified with respect to the performance measurement data, such as challenges aggregating data due to differences in the interpretation and categorization of innovation impacts by recipients.

There are opportunities to improve the utility of performance data in order to facilitate communication of program results across a broader range of stakeholders and enhance knowledge transfer and adoption among industry stakeholders. Different types of indicators that capture the significance of impacts and help tell success stories in a manner that is accessible and understandable could be considered for inclusion in the program's performance measurement strategy. Other indicators that measure the long-term returns of innovation program investments could also be considered. While further analysis is required to assess their applicability, advantages and disadvantages, some potential measures of program impacts are as follows:

Highly qualified personnel capacity. Indicators could examine how the program contributes to specific areas of expertise (for example, type of degrees or areas of specialization of MSc or PhD candidates participating in Stream B activities). Indicators could explore whether candidates obtained employment within the sector, through follow-up surveys as part of a program evaluation (for example, number of MSc or PhD candidates who obtained employment in an agricultural R&D field or were hired as a result of the Agri-science cluster or project).
Collaboration. Indicators could explore incremental impacts such as the number of new partnerships or research networks formed and types of organizations or regions of the collaborations. Broader indicators of agricultural R&D collaboration could also be considered that draw from Organization for Economic Co-operation and Development benchmarking data such as agriculture patents with co-authors as a share of total agriculture patents.^{Footnote 44}
Leverage. Indicators could focus more on business involvement in R&D in terms of the number of businesses participating in agricultural R&D activities or the change in business R&D expenditures as a result of the Agri-science cluster or project. An indicator could be used to examine changes in sector capacity for investment in agricultural R&D.
Innovation. Different indicators could capture the nature of innovation impacts such as the number of innovations that advanced towards commercialization (for example, according to technology readiness levels)^{Footnote 45} or more descriptive success stories and examples of innovations developed and how they have impacted the sector in terms of improved productivity and expanded markets.
Return on investment. Various measures could identify return on investment such as the total value of sales resulting from innovations developed or Innovation Sales Rate (for example, percentage of business sales that are attributable to new products or processes).^{Footnote 46} In-process innovation measures could be considered such as the Risk Adjusted Net Present Value of the innovation pipeline, which is often used by multinational companies involved in agricultural R&D. Broader measures, such as Innovation Balanced Scorecards, draw from a combination of leading, in-process and lagging innovation indicators (for example, public R&D, researchers engaged in R&D, connectivity, scientific articles, entrepreneurial ambition, venture capital investment, Business Enterprise R&D Spending, Information and communication technology investment, patents, and labour productivity)^{Footnote 47} and are often used to assess government or economy-wide innovation impacts.^{Footnote 48}

5.0 Evaluation conclusions