Evaluation of the AgriScience Program

Abbreviations

AAFC: Agriculture and Agri-Food Canada
FTE: Full-Time Equivalent
R&D: Research and Development

Executive summary

Purpose

The Office of Audit and Evaluation of Agriculture and Agri-Food Canada (AAFC) conducted an evaluation of the AgriScience Program to provide senior management with an assessment of the relevance, design, delivery, effectiveness and impact of the Program.

Scope and methodology

AgriScience Program activities were evaluated using multiple lines of evidence: program documents, program data and literature review; AAFC staff, researcher and industry stakeholder interviews; bibliometric analysis; economic analysis; and case studies. The evaluation focussed on AgriScience clusters and projects funded from 2016–17 to 2020–21 and impacts of clusters funded prior to 2016–17.

Background

The AgriScience Program is a five-year program (2018–19 to 2022–23) that provides support for industry-led research, development and knowledge transfer that should lead to innovative agriculture, agri-food and agri-based practices, processes and products. The AgriScience Program has a budget of almost $191 million and is delivered through two components: the AgriScience – Clusters Component (Clusters) and the AgriScience – Projects Component (Projects).

Clusters are national in scope and bring together scientific expertise from industry, academia and government to collaboratively address multiple industry priorities of a specific commodity (for example, wheat or beef) or cross-cutting issues (for example, bioproducts or food processing).
Projects aim to support shorter-term research activities to help industry overcome challenges and address fiscal barriers experienced by small and emerging sectors.

Findings

The AgriScience Program has addressed industry needs, which include resolving challenges such as growing demand for food and agricultural products; threats due to diseases and pests; and lack of infrastructure and capacity to support ongoing research and development (R&D).
The AgriScience Program is aligned with government’s role and priorities. Moving forward, the Program could do more to support the increased federal and departmental focus on climate change and environmental sustainability, as well as other key priorities, though the potential for such support is limited by the industry-led nature of the Program.
While the cluster model is regarded as an effective approach for carrying out R&D, the effectiveness of the current AgriScience cluster model is limited by focussing on individual activities within a cluster, rather than strategically leveraging the cluster as a whole.
AgriScience Program expenditures are $27 million less than budgeted in the last three years, mainly due to delays in the initial implementation of the Program.
The Program has increased the sector’s capacity to conduct R&D by approving $239 million in funding, which leveraged an additional $93 million in industry and other government funding. This has resulted in the establishment of new partnerships and the hiring of additional researchers, which constitute a portion of the 727 highly qualified personnel working on AgriScience clusters and projects.
Peer-reviewed publications resulting from clusters made important contributions to the knowledge base.
Although data exists on the nature and extent of knowledge transfer activities, data on the impacts of these activities and how effectively they were carried out is not captured.
The AgriScience Program has made considerable progress in achieving its five-year performance targets related to technology development. However, the impacts of commercialization and adoption of these technologies, and technologies produced under the previous policy frameworks, cannot be assessed because this data is not collected.
The direct and indirect economic impacts on the Canadian economy of AgriScience Program expenditures over three years are estimated to be $314 million, which is in line with estimates for the first three years of the previous iteration of the Program (AgriInnovation Stream B was $232 million).^{Footnote 1}
The Program logic model and performance measures are largely focussed on outputs and early outcomes of individual activities and do not present the Program’s intended economic, environmental and social impacts. Impact measurement of clusters, as a whole, is lacking.

Conclusion

The AgriScience Program has successfully supported industry-led research through funded clusters and projects that contributed to the advancement of agricultural R&D in Canada. The Program succeeded in bringing key players together; developing the next generation of scientists; enabling the development of sector-specific R&D portfolios; filling gaps in knowledge; and supporting the development of new technologies (for example, processes, practices, varieties and products).

Although Program-level priorities exist, they have not been used to strategically guide funding allocation decisions at the cluster level. Clusters were given the flexibility to choose the priorities on which they wanted to focus based on their sectors’ needs and interests. This has resulted in clusters and projects addressing priorities to various extents. For example, limited R&D has been conducted in the priority area of climate change and environmental sustainability. The role of agricultural R&D in addressing issues of importance such as climate change is recognized as being increasingly more urgent; however, the AgriScience Program has limited its role due to the industry-led nature of the program. Although clusters are recognized as an effective and efficient mechanism for conducting R&D, AAFC has not used the cluster model strategically to address current and emerging challenges affecting the sector, address government and department priorities, or define and achieve specific and strategic impacts at the economic, social and environmental levels.

The indicators identified in the Performance Information Profile are at the output and immediate outcome levels, which means that recipients are not required to collect intermediate to long-term outcomes and impacts, nor are they required to submit a follow-up report on impact a few years after funding ends. Consequently, there is limited information on the performance and impact of Program components (project and clusters) and the Program overall.

The AgriScience Program is jointly delivered by Programs Branch and the Science and Technology Branch. Presently, there is no single mechanism to monitor AAFC expenditures. The decentralized management structure used by the Science and Technology Branch has limited its ability to efficiently track and report on its financial and human resources at the Program level.

Recommendations

Recommendation 1: The Assistant Deputy Minister, Programs Branch, in collaboration with the Assistant Deputy Ministers, Science and Technology Branch and Strategic Policy Branch, should clarify the AgriScience Program’s strategic objectives, priorities and intended impacts.
Recommendation 2: The Assistant Deputy Minister, Programs Branch, should require project and cluster funding recipients to develop a performance measurement and evaluation framework to support reporting on outcomes and impacts, including future reporting on results after funding ends.
Recommendation 3: The Assistant Deputy Minister, Science and Technology Branch, in collaboration with the Assistant Deputy Minister, Corporate Management Branch, should clarify, document and communicate the Science and Technology Branch’s AgriScience financial expenditure tracking processes and systems.

Management Response and Action Plan

Management agrees with the evaluation recommendations and has developed an action plan to address them by April 2023.

1.0 Introduction

The Office of Audit and Evaluation conducted an evaluation of Agriculture and Agri-Food Canada's (AAFC) AgriScience Program as part of the 2020–21 to 2024–25 Integrated Audit and Evaluation Plan. This evaluation was conducted in accordance with the Treasury Board’s Policy on Results and fulfills the requirements of the Financial Administration Act. The evaluation is intended to inform current and future program and policy decisions.

2.0 Scope and methodology

The evaluation assessed the relevance, design, delivery, effectiveness and impact of the AgriScience Program. It focussed on clusters and projects funded from 2016–17 to 2020–21 and the impacts of clusters funded prior to 2016–17. The evaluation does not include an assessment of the policy activities undertaken to redesign the AgriScience Program for the next agricultural policy framework, which launches in 2023–24. Rather, the evaluation focusses on the AgriScience Program delivered under the Canadian Agricultural Partnership.

Throughout the evaluation process, early results were shared with AAFC management to inform the next agricultural policy framework.

3.0 Program profile

3.1 Program description

The AgriScience Program is a five-year program (2018–19 to 2022–23) that provides support for industry-led research, development and knowledge transfer, which should lead to innovative agriculture, agri-food and agri-based practices, processes and products. By accelerating the pace of innovation, the Program aims to create solutions to address industry challenges and increase market opportunities in the agricultural sector. The AgriScience Program is a federally-delivered program that is part of the Canadian Agricultural Partnership, a $3 billion investment by federal, provincial and territorial governments to strengthen the agriculture, agri-food and agri-based products sector. AgriScience aligns with the Canadian Agricultural Partnership’s priority area of science, research and innovation.

The AgriScience Program is delivered through two components: the AgriScience – Clusters Component (Clusters) and the AgriScience – Projects Component (Projects).

Clusters are national in scope and bring together scientific expertise from industry, academia and government to collaboratively address multiple industry priorities of a specific commodity (for example, wheat or beef) or cross-cutting issues (for example, bioproducts or food processing). This is intended to lead to the creation of medium- and long-term research strategies.
Projects aim to support shorter-term research activities to help industry overcome challenges and address fiscal barriers experienced by small and emerging sectors. These projects also seek to explore high-risk opportunities that have the potential to yield significant returns.

Both components were implemented under the Growing Forward 1 (2008–09 to 2012–13) and Growing Forward 2 (2013–14 to 2017–18) policy frameworks, alongside other programming to support pre-commercialization activities. As a grants and contributions program, previous iterations of the AgriScience Program have been evaluated regularly. The most recent evaluation titled Evaluation of the AgriInnovation Program – Stream B: Research, Development and Knowledge Transfer assessed the first three years (2013–14 to 2015–16) of the five-year AgriInnovation Stream B Program.

3.2 Governance and Program delivery

The AgriScience Program is a jointly delivered AAFC program. It is managed by the Industry Development Division within the Innovation Programs Directorate of AAFC’s Programs Branch, while the Science and Technology Branch engages with industry to set research and development (R&D) priorities, negotiates Collaborative Research and Development Agreements and provides R&D support to AgriScience funding recipients. Units within other divisions of Programs Branch provide support in areas such as finance and performance reporting.

AgriScience clusters and projects are selected through an application process. Applications are reviewed by program and subject matter experts from multiple AAFC branches, including Programs Branch, Science and Technology Branch, Strategic Policy Branch, and Market and Industry Services Branch. The application review includes an assessment of the quality, eligibility and viability of the cluster and project proposals and ensures alignment with the objectives and priorities of the Program. A committee composed of Directors General from several AAFC branches, including Programs, Science and Technology, Market and Industry Services, Strategic Policy and Corporate Management branches, reviews and selects clusters and projects to be funded.

To be eligible to receive funding, project applicants must address at least one of the following priorities:

Improve support for minor commodities, emerging and transformative areas
Invest in discovery and applied science for major commodity sectors
Enhance efforts in clean growth, environment and climate change
Accelerate growth of Canada’s food and beverage processing or value-added sector
Strengthen knowledge transfer and adoption

Cluster applicants must address two of the above priorities, one of which must be to strengthen knowledge transfer and adoption.

Selection process – clusters

Prior to submitting a formal application, cluster applicants have their R&D activities assessed by three independent peer reviewers with technical expertise in the subject matter.

A science review is then conducted by scientists from the Science and Technology Branch, taking into account the comments from the peer review process and how feedback from that process has been addressed. Following the science review, program officers from the Programs Branch coordinate a principles-based review to ensure clusters:

present a clear strategic vision for research in their sector and build science collaborations amongst key partners,
align with multiple program priorities,
leverage increased funding from applicant, industry and other government sources,
identify how the expected research results will generate economic and societal benefits (for example, increased production, diminished environmental impacts, diminished threats, response to market demands), and
address broader emerging issues that have increasing potential impacts on the agricultural sector (for example, anti-microbial use/resistance, data management, artificial intelligence, automation).

Each proposal is also assessed on the technical, financial and managerial capacity of the applicant managing the cluster research. Reviews/assessments are conducted by representatives from multiple AAFC branches as outlined above.

Selection process – projects

Prior to submitting a formal application, project applicants submit a one-page project summary form, which is used to determine eligibility and alignment with program criteria and priorities. Projects do not require an external peer review.

When a formal application is submitted, a science review is then conducted by scientists from the Science and Technology Branch. Following the science review, program officers from the Programs Branch coordinate a principles-based review to ensure projects:

align with government and program priorities,
leverage increased funding from applicant, industry and other government sources,
identify how the expected research results will generate economic and societal benefits (for example, increased production, diminished environmental impacts, diminished threats, response to market demands), and
address broader emerging issues that have increasing potential impacts on the agricultural sector (for example, anti-microbial use/resistance, data management, artificial intelligence, automation).

Each proposal is also assessed on the technical, financial and managerial capacity of the applicant managing the project research. Reviews/assessments are conducted by representatives from multiple AAFC branches as outlined above.

Other considerations

A condition of cluster funding is that eligible project costs are shared between AAFC and industry with the industry applicant providing a minimum of 30%. For projects, for-profit organizations must fund at least 50% of eligible project costs, while not-for-profit organizations must fund at least 30% of eligible costs.

Depending on what type of research is being proposed, applicants can seek funding through Vote 1, Vote 10 or both. In cases where an approved cluster or project is funded by non-repayable contribution funding (Vote 10) for research activities conducted by industry, a Contribution Agreement is signed by AAFC and the funding recipient. In cases where non-monetary collaborative support (Vote 1), such as research capacity, equipment and facilities, are received from AAFC, a Collaborative Research and Development Agreement is signed.

3.3 Resources

AAFC’s AgriScience Program budget amounted to $190.73 million in departmental spending from 2018–19 to 2020–21 (see Table 1). The number of Full-Time Equivalent (FTE) employees whose salaries are paid through federal AgriScience funding ranged from 62.8 to 83.2 during the first three years of the Program.

**Table 1. AgriScience Program budget**
Type of Expenditure	2018–19	2019–20	2020–21	Total
Grants and Contributions (Vote 10) ($ million)	36.76	36.76	36.76	110.28
Science and Technology Branch Salary and Non-Pay Operating (Vote 1) ($ million)	20.58	20.58	22.58	63.74
Programs Branch Administrative Costs ($ million)	3.29	3.29	3.29	9.87
Capital Expenditures (Vote 5) ($ million)	2.28	2.28	2.28	6.84
Total ($ million)	62.91	62.91	64.91	190.73
Notes: Totals were calculated using the rounded values for each fiscal year. Programs Branch Administrative Costs exclude Employee Benefit Plan costs. Source: Program budget, AAFC Programs Branch and Science and Technology Branch

3.4. Intended outcomes

The AgriScience Program aims to achieve five outcomes:

Immediate outcome: The sector’s capacity to conduct innovative research and development is increased.
Intermediate outcome: The sector’s knowledge base is increased.
Intermediate outcome: Knowledge of innovative technologies, products, practices, processes and services is transferred to the sector.
Intermediate outcome: Research and development supported innovations are generated.
Long-term outcome: Innovations supported by research and development are commercialized.

The AgriScience Program Logic Model in Annex B provides further detail on the Program's key activities, outputs and outcomes.

4.0 Relevance

4.1 Alignment of Program with industry needs

The AgriScience Program has addressed several industry needs; however, limited R&D has been conducted in the program priority area of climate change and environmental sustainability.

Current and emerging global challenges in the sector include growing demand for food and agricultural products, changing consumption patterns and climate change, the latter of which is projected to have severe and costly effects on the sector in the coming decades.^{Footnote 2}^{Footnote 3} Sector activities (for example, policies, R&D) should be centered around related priorities, such as environmental stewardship in production and processing, food security and sustainability, food quality and safety and enhanced nutrition, among others.^{Footnote 4}^{Footnote 5}^{Footnote 6}

The Program meets funding recipient needs through its provision of long-term funding that supports the continuity of R&D initiatives; its ability to support large, complex activities; and its flexibility to support a range of sectors, priorities and activities. The needs and priority areas that were most frequently mentioned by interviewed funding recipients include: climate change and sustainability (43% of recipients); increased quantity and quality of food production (32%); greater investment in agricultural R&D (25%); addressing technology advancements and automation (25%); and addressing disease or illness (21%).

An analysis of each project and cluster activity’s cross-cutting areas of focus (primary and secondary) showed that 76% of AgriScience Program funding supports research related to enhancing primary production, including genetic improvement (30%), pest management (20 %) and agronomic improvement (20%). Research activities also focus on other cross-cutting areas, such as bioproducts, agri-food processing, environmental sustainability, automation and precision-focussed technologies, biosecurity, organics and food safety and traceability.^{Footnote 7}

External interviewees stated that funded AgriScience projects and clusters were well-aligned with industry needs given that industry partners determine the research activities to be included in funding applications to the AgriScience Program. As evidence of alignment with industry needs, the proportion of funding contributed by industry has steadily increased over the last three policy frameworks from 25% in Growing Forward 1, to 31% in Growing Forward 2 and to 33% in the AgriScience Program under the Canadian Agricultural Partnership.

Need for public-private collaboration in R&D

Literature reviewed during the evaluation describes an increasing need for public-private collaboration and investment in R&D to drive innovation along the value chain. Such collaboration enables various groups to make unique contributions and can help to prevent duplication of effort and siloing of operations.

Innovation projects involve high risk and uncertainty. Multiple sources suggest that private firms often underinvest in agricultural R&D,^{Footnote 8}^{Footnote 9}^{Footnote 10} which may be due to the limited potential to obtain sufficient benefits from their investment and the considerable time lag for benefits to accrue. ^{Footnote 11} As a result, much of the related literature suggests that government has a key role to play in the implementation of agricultural R&D. Some sector experts indicated that government support is most appropriate when conducting basic research (rather than applied research), conducting R&D with high social returns and investing in areas in which industry lacks incentive to invest (for example, R&D in areas where intellectual property rights are not well-defined or enforceable). ^Footnote12^{Footnote 13} The AgriScience Program is designed to provide the suggested government support, such as funding basic research (discovery science) and projects with high social returns (for example, clean growth, climate change and environment). The design of the Program also enables it to fund applied research given that industry engagement is at the centre of the Program.

Feedback from external interviewees highlighted that AgriScience funding effectively supported many recipients in carrying out R&D activities. Without this support, they note that R&D activities may not have occurred or would have taken place on a smaller scale or over a longer time period, thus reducing their overall impact. Cluster and project funding recipients estimated there is a 33% likelihood that their clusters or projects would have proceeded in the absence of AAFC funding. For activities that would have proceeded, other funding sources were in place and/or the activities were deemed as high priority, which necessitated their completion. However, most AgriScience funding recipients reported that it would have been difficult to secure the same level and nature of support from other sources.

4.2 Alignment with AAFC and government role and priorities

The AgriScience Program is aligned with government roles and priorities; however, it could do more to support key government priorities such as climate change and environmental sustainability.

Alignment with AAFC role and priorities

Science and Innovation is one of the three core responsibilities of AAFC. As stated in the 2021–22 Departmental Plan, this responsibility focusses on increasing knowledge through supported scientific and innovative research and transforming ideas into new products, processes or practices. These responsibilities align with the 2019 AAFC Minister’s Mandate Letter, which states that the Minister should support the agriculture and agri-food sector so it becomes a leader in innovation. The 2021 Supplementary Mandate letter adds that AAFC should support sustainable growth of the sector.

The AgriScience Program’s objective is to accelerate the pace of innovation by supporting commercial science activities and research that benefit the agriculture and agri-food sector. This is consistent with AAFC’s priority to advance and modernize sector research and its mission to provide leadership in the growth and development of a competitive and innovative sector, as stated in the 2021–22 Departmental Plan. By enhancing efforts in clean growth, the AgriScience Program aligns with the Minister’s role in supporting the efforts of farmers and ranchers in reducing emissions and building resilience.

Alignment with federal government role and priorities

The AgriScience Program is aligned with the federal government's roles, responsibilities and priorities, as it contributes to national and international commitments related to innovation, climate change, sustainability and collaborations/partnerships in R&D. As stated in the 2016 Inclusive Innovation Agenda and in the 2017 Innovation and Skills Plan, Canada should foster more partnerships and collaborations between businesses, post-secondary institutions, research institutions and government. These new collaborations have the potential to help bridge the commercialization gap and strengthen the value chains. Through its cluster component, the AgriScience Program aims to fund activities that increase collaboration between private sector organizations, post-secondary institutions, governments and other stakeholders. The AgriScience Program is also consistent with the Inclusive Innovation Agenda “Global Science Excellence” area of action, as it promotes partnerships with business and aims to strengthen basic and applied research capabilities.^{Footnote 14}

Through national level frameworks and strategy, the Government of Canada is committed to supporting sustainable development. The Federal Sustainable Development Strategy for Canada, for example, includes goals related to clean growth, sustainable food and sustainably managed lands. By supporting projects and activities addressing environmental challenges and adaptation to changing climate, the AgriScience Program aims to enhance efforts in clean growth, environment and climate change, which is consistent with the federal government's commitments.

The AgriScience Program also supports the Government of Canada's international commitments, such as the United Nations 2030 Agenda and the Sustainable Development Goals, specifically Goal 9: building resilient infrastructure, promoting inclusive and sustainable industrialization and fostering innovation.

Distribution of AgriScience activities across departmental and government priority areas

A majority of funded cluster activities is related to primary agriculture productivity research. The funding allocated to emerging priority areas in the AgriScience Program has increased compared to the previous iteration of the Program, AgriInnovation Program: Stream B (see Figure 1). These emerging priority areas include:

improved support for minor commodities, emerging and transformative areas
enhancing efforts in clean growth, environment and climate change
accelerating growth of Canada’s food and beverage processing sector

This shift in cluster funding is due primarily to five new AgriScience clusters that are heavily weighted towards minor commodities, as well as emerging and transformative areas, namely: Diverse Field Crops, Grape and Wine, Agronomy, Biomass and Automation. These new clusters received $40.4 million combined in Program funding versus $135.2 million for the 14 established clusters.

Description of this image follows — **Figure 1. AgriScience cluster funding distribution by priority areas**

AgriScience cluster funding distribution by priority areas
	Investing in discovery and applied science for major commodity sectors	Improved support for minor commodities, emerging and transformative areas - Emerging priority area	Enhancing efforts in clean growth, environment and climate change - Emerging priority area	Accelerating growth of Canada's food and beverage processing sector - Emerging priority area	Strengthening knowledge transfer and adoption
AgriInnovation Stream B	74%	20%	5%	5%	3%
AgriScience	53%	37%	18%	9	6%

There is uneven priority coverage among funded clusters and projects. While there are no specific funding level targets, combined clusters and projects funding remains relatively low in the environmental and value-added priority areas, such as food and beverage processing (see Figure 2).

**Clusters/projects funding by priority areas ($ millions)**
	Investing in discovery and applied science for major commodity sectors	Improved support for minor commodities, emerging and transformative areas	Enhancing efforts in clean growth, environment and climate change	Accelerating growth of Canada's food and beverage processing sector	Strengthening knowledge transfer and adoption
Primary priority area	$110.4	$75.7	$10.0	$11.9	$7.7
Secondary priority area	$5.3	$19.4	$28.3	$8.2	$4.4
Note: Funding for activities/projects that fit under multiple priority areas is counted under the activity/project's primary and secondary priority area; therefore, the funding allocated to the priority areas does not add to total Program funding. The figure reflects all cluster funding, and project funding ($58.2 million) allocated as of June 26, 2019. Source: Canadian Agricultural Partnership AgriScience Funding Analysis (December 12, 2019)

Climate change and environmental sustainability have been constant federal priorities for several years and have increased significantly in focus for the federal government and AAFC in 2020–21. They were commonly noted as priorities in the sector by both internal and external interviewees. Given this emphasis, greater support could be allocated to climate change and environmental sustainability under the next agricultural policy framework. While the Program could do more to support AAFC and federal priorities such as climate change and environmental sustainability, the potential for such support is dependent upon the industry-led nature of the Program.

5.0 Program design and delivery

This section presents the evaluation findings on the effectiveness of Program design and delivery, the cluster approach and Gender-based Analysis Plus considerations.

5.1 Effectiveness of cluster model

Clusters are regarded as an effective, systematic, coordinated approach for carrying out R&D; however, the effectiveness of AgriScience’s cluster model is limited by focussing on individual activities within a cluster, rather than strategically leveraging the cluster as a whole.

The evaluation found key benefits to a cluster approach, including the ability to address broad strategic national and industry-level priorities, the enhancement of effective stakeholder collaboration and the potential to leverage the strengths of key cluster participants to increase sector innovation in a more systematic way. Studies demonstrated that firms participating in clusters were more likely to be innovative and experience stronger growth than firms that did not participate in a cluster.^{Footnote 15}^{Footnote 16}^{Footnote 17} In addition, strong clusters helped to grow employment, wages, income, new businesses, entrepreneurial activity and patenting at the industry level.^{Footnote 18}^{Footnote 19}^{Footnote 20}^{Footnote 21} Cluster participation also increased the likelihood of R&D collaboration and, in some cases, helped small and medium-sized enterprises to access public funding and conduct R&D for the first time.^{Footnote 22}^{Footnote 23}

The evaluation found that the Program has not taken full advantage of the cluster model. AAFC monitors the performance of individual activities within each cluster rather than the clusters’ performance as a whole. This, along with the absence of targets to increase alignment with AAFC and government priorities, reduces the clusters’ potential benefit to AAFC. Clusters were able to choose the priorities they wanted to focus on based on their sectors’ needs and interests.

Although AAFC requires clusters to provide projected economic impacts as part of the application process, that information is neither verified nor monitored, and important elements of an R&D cluster model, such as employment growth, are not considered. In addition, as clusters are not required to submit strategic plans to AAFC, the work plans submitted lack strategic considerations or are not framed to illustrate how cluster activities will advance specific priorities overall.

Extent of collaboration

Evidence suggests that effective collaboration has occurred in commodity-focussed clusters. Collaboration is more complex for horizontal/cross-cutting clusters (for example, bioproducts, biomass) because they focus on the secondary sector. Internal and external interviewees indicated that horizontal clusters play an important role in supporting and facilitating collaboration among industry groups that do not benefit from a check-off system (that is, levies collected on sales of agricultural products and commodities that are used to advance the sector) or the structure of a producer association.

There is limited collaboration between clusters. Clusters engage in discussions about administrative matters, as well as AAFC-led strategic discussions about policy issues. However, there have been limited opportunities for clusters to share R&D methods and results or to leverage one another’s work. A few clusters expressed interest in collaborating, but experienced logistical and financial barriers, such as an inability to provide matching funds.

AAFC staff with strong knowledge of the Program noted that funded clusters are too focussed on commodities with little involvement from other parts of the value chain, which is an area of interest for AAFC. It was noted that the priorities put forward by the clusters are short-term focussed and that AAFC is not taking a strategic approach in cluster management. For example, certain factors were not sufficiently considered in the cluster selection process, such as the effectiveness of cluster management and continuous funding of certain clusters over multiple frameworks. The decision on whether or not to continue funding a cluster is hampered by the lack of data on the overall impact of that cluster.

5.2 Effectiveness of Program design and delivery

Changes to the application and reporting processes increased Program effectiveness.

In 2017, several modifications were made to Program performance and financial reporting and to the application process. These changes included a transition to an online application, the introduction of an early project review process and a simplified cluster peer review process whereby the applicant was responsible for obtaining peer reviews. The evaluation found that the Program’s effectiveness increased as a result of these changes, and that the administrative burden for applicants and recipients was reduced.

As part of the application, clusters are asked to identify how the research results are expected to generate economic and societal benefits or impacts. However, the applications reviewed as part of the cluster case studies demonstrated that minimal quantitative information on impacts was provided by cluster applicants. This may be attributed to a lack of sufficient guidance from AAFC on impacts and targets.

The Director General Innovation Committee, a multi-stakeholder committee, is responsible for reviewing and assessing cluster and project applications. The Committee met eight times between January and March 2018 to make cluster and project funding decisions for the funding cycle, providing the review and oversight function as intended.

Interviewed funding recipients confirmed that their administrative burden was reduced as a result of changes to the application and reporting processes. While changes to the peer review process are perceived positively, funding recipients continued to have difficulties securing peer reviewers due to the limited pool of eligible reviewers and the time commitment required by the reviewers. Most respondents agreed that the process changes reduced duplication of effort and acknowledged the peer review process as a well-recognized, valid and important practice.

Many improvements were made to reduce the time required to review and approve applications, including the sharing of informal information on potential Program priorities with applicants in advance (in some cases 18 to 24 months before Program launch).

Role of AAFC scientists and R&D staff

AAFC scientists and other Science and Technology Branch staff are an integral part of the AgriScience Program. Program data shows that all but one of the AgriScience clusters, and half of the AgriScience projects, received in-kind R&D support from scientists and staff from the Science and Technology Branch. Some clusters depend highly on the Branch (wheat, barley, agronomy and grape/wine), while others have worked less with the Branch (automation, biomass, bioproducts, green plant innovation) as AAFC has limited expertise in these areas.

Cluster and project funding recipients reported that the collaboration between AAFC and funding recipients is effective, citing factors such as strong communication with AAFC staff (for example, program officers) and scientists. Some funding recipients spoke to the value of receiving both Vote 1 and Vote 10 support as the two components complement one another, facilitate collaboration between researchers and fill a gap that cannot be addressed by industry. Despite some internal interviewees’ view that scientists working on industry-led projects may not be contributing to public good/fundamental research, the evaluation confirmed through consultations with scientists from the Science and Technology Branch that opportunities offered through AgriScience enable scientists to align their research programs, which are in the realm of public good, with those of industry.

The Program logic model (see Annex B) does not show the Science and Technology Branch’s role and contribution to AgriScience. Internal documents on the AgriScience Program’s way forward under the next agricultural policy framework also omit the Science and Technology Branch’s role.

5.3 Program administration

AgriScience Program expenditures are $27 million less than budgeted for the first three years due primarily to delays during the initial implementation of the Program.

AgriScience Program expenditures, as reported, are $27.06 million less than budgeted for the three-year period from 2018–19 to 2020–21. This apparent underspending occurred primarily in 2018–19 and is mainly a result of delays during the initial implementation of the Program (see Table 2). Similar delays occurred in the first year of AgriScience’s predecessor, AgriInnovation Stream B. The AgriInnovation Stream B Program spent 49% of its first-year budget, while the AgriScience Program spent 69% of its first-year budget. This pattern of underspending could have been addressed by systemically reducing the program budget in the first year of implementation.

**Table 2. AgriScience Program expenditures**
Type of expenditure		2018–19	2019–20	2020–21	Total
Grants and Contributions (Vote 10) ($ million)	Actual	24.35	41.70	36.40	102.45
	Budget	36.76	36.76	36.76	110.28
	Variance	(12.41)	4.94	(0.36)	(7.83)
Science and Technology Branch Salary and Non-Pay Operating (Vote 1) ($ million)	Actual	13.81	17.54	15.65	47.00
	Budget	20.58	20.58	22.58	63.74
	Variance	(6.77)	(3.04)	(6.93)	(16.74)
Programs Branch Administrative Costs ($ million)	Actual	4.57	4.33	3.60	12.50
	Budget	3.29	3.29	3.29	9.87
	Variance	1.28	1.04	0.31	2.63
Capital Expenditures (Vote 5) ($ million)	Actual	0.92	0.43	0.37	1.72
	Budget	2.28	2.28	2.28	6.84
	Variance	(1.36)	(1.85)	(1.91)	(5.12)
Total ($ million)	Actual	43.65	64.00	56.02	163.67
	Budget	62.91	62.91	64.91	190.73
	Variance	(19.26)	1.09	(8.89)	(27.06)
Notes: Totals were calculated using the rounded values for each fiscal year. Programs Branch Administrative Costs exclude Employee Benefit Plan costs. Source: Program expenditures, AAFC Programs Branch and Science and Technology Branch

Another factor that contributed to the variance is that some AgriScience Program Vote 1 salaries were erroneously coded under another Canadian Agricultural Partnership R&D budget (Foundational Science and Research, a budget used for AAFC-led R&D), which saw an increase in spending during this time.

The FTEs devoted to the AgriScience Program increased from 72.5 FTEs in 2018–19 to 83.2 FTEs in 2020–21 (see Table 3). Programs Branch delivered the Program with 42.0 FTEs in the first year and continued to do so in 2020–21 with 32.0 FTEs. The reduction in Programs Branch staff during this period is partially due to the reassignment of some staff to new AAFC programs, including emergency food security programs in response to the COVID-19 pandemic. The Science and Technology Branch supported the Program with 30.5 staff in 2018–19 and 51.2 staff in 2020–21 – the increase in Science and Technology Branch staff during this period reflects Branch correction of administrative coding errors.

Science and Technology Branch FTE numbers represent the Branch staff whose salaries are paid through federal funding dedicated to AgriScience. These numbers do not include all FTEs contributing to the Program. Estimates developed at the start of the Program predicted that just under 800 Science and Technology Branch scientists and staff would support AgriScience clusters and projects to various extents during the five-year Program period.

**Table 3. Number of full-time equivalents devoted to the AgriScience Program**
Fiscal Year	Programs Branch	Science and Technology Branch	Total
2018–19	42.0	30.5	72.5
2019–20	37.0	25.8	62.8
2020–21	32.0	51.2	83.2
Source: Program expenditures, AAFC Programs Branch and Science and Technology Branch

Ratio of AAFC program delivery costs to total program costs

Programs Branch program delivery costs ($12.50 million) as a percentage of total program costs appear to be higher than the program delivery costs of similar grants and contributions programs.^{Footnote 24} During the first three years, Programs Branch program delivery costs for AgriScience averaged 10.9% of total Program costs. As a comparison, the average program delivery costs as a percentage of total costs for similar programs examined is 8.1% (see Table 4).^{Footnote 25} The AgriScience administrative cost ratio, however, does not represent actual program delivery costs because it excludes some of the work conducted by the Science and Technology Branch (for example, science review, development of Collaborative Research and Development Agreements), as well as the work of other branches involved in the application review process. Therefore, total AgriScience Program delivery costs are expected to be more than $12.50 million, but Program financial information does not include sufficient detail to determine the exact value.

**Table 4. AgriScience Program delivery costs versus other similar programs**
Organization	Program	Years	Program delivery costs as percentage of total costs
AAFC	AgriScience Program	2018–19 to 2020–21	10.9
AAFC	AgriInnovation Program (Stream C: Enabling Commercialization and Adoption)	2013–14 to 2016–17	7.5
AAFC	Agri-Opportunities Program	2006–07 to 2010–11	8.3
AAFC	Agri-based Processing Initiative	2009–10 to 2013–14	8.4
Atlantic Canada Opportunities Agency	Innovation and Commercialization Sub-Program	2007–08 to 2011–12	8.3
Sources: AAFC. Evaluation of AgriInnovation Program Stream C: Enabling Commercialization and Adoption. 2018; AAFC. Evaluation of the Agri-Opportunities Program. 2010; AAFC. Evaluation of the Agri-based Processing Initiative. 2015; Atlantic Canada Opportunities Agency. Evaluation of ACOA’s Innovation and Commercialization Sub-program. 2015.

Oversight of expenditures

There is no single mechanism for tracking and reporting on AgriScience expenditures incurred by AAFC. The process for tracking expenditures is disaggregated across separate branches and systems.

Programs Branch, outside of grants and contributions funding, holds a small administrative budget, which incurs minimal expenses. Science and Technology Branch, however, incurs significant expenses in the conduct of R&D activities/projects. The process used to track expenses is not centralized and accountability of funding is shared by all Directors of Research, Development and Technology.

A financial coding structure is used to track the funds received and spent for each AgriScience activity/project to which Science and Technology Branch contributes. The evaluation found that coding of expenditures to the appropriate R&D activity/project can vary depending on the research centre and/or the individual entering the information in the system. This model has not prevented AAFC from tracking expenditures but has impeded efficient and effective tracking and reporting.

The invoices and financial reports that Science and Technology Branch submits to funding recipients are attested by Corporate Management Branch and are based on high-level templates agreed-upon within AAFC and with industry at the onset of the Canadian Agricultural Partnership. However, external funding recipients have noted that they would benefit from increased details in the Science and Technology Branch reporting. A review of Program financial documents has indicated that more detailed expenditure information is available.

5.4 Integration of Gender-Based Analysis Plus and official languages

Gender-based Analysis Plus and official languages considerations have not been included as a Program funding criterion.

Collection of Gender-based Analysis Plus and official languages data commenced in 2020–21 and data are available on the demographic characteristics of Program beneficiaries and personnel for some clusters and projects. However, Program documents indicate that the involvement of underrepresented groups in the Program is not a funding criterion.

6.0 Effectiveness

COVID-19 has had a considerable impact on the conduct of funded AgriScience activities, including an inability to access laboratories to carry out tests, the cancellation of planned knowledge transfer/training events and delays in contracting research activities. Other constraints include difficulties hiring and retaining research staff. The following paragraphs present the extent to which the AgriScience Program has achieved its expected outcomes.

6.1 Increase in sector capacity to conduct R&D

The AgriScience Program increased the sector’s capacity to conduct R&D by approving $239 million in funding, which leveraged an additional $93 million in industry and other government funding, facilitated the hiring of additional researchers and increased collaborations.

The AgriScience Program has made considerable progress in accomplishing its immediate intended outcome of increasing the sector’s capacity to conduct innovative R&D. Program data indicates that funding recipients’ capacity to conduct R&D increased due to $239 million in approved Program funding over five years, which leveraged over $93 million in committed industry and provincial government funding. During its first three years, the AgriScience Program funded 19 clusters and 58 projects, and was continuing to accept project applications. The previous iteration of the Program, AgriInnovation Stream B, funded 14 clusters and 86 projects. R&D collaboration was involved in 95% of AgriScience clusters and 30% of AgriScience projects.

Funding for almost 30% of AgriScience cluster activities spans multiple policy frameworks. Some of the information reported in the following section will include information on activities that started under a previous framework.

Funding recipients noted that the Program has helped to establish networks of organizations across Canada and new partnerships that increase R&D capacity, some of which would not have been formed in the absence of AAFC support. The evaluation found that Program activities complement and enhance existing R&D initiatives and Program funding is integral to maintaining R&D activities.

The Program has increased human resource capacity for agricultural R&D. During the first three years of the Program, 727 highly qualified personnel (known as HQP, that is individuals with university degrees at the bachelor’s level and above) worked on AgriScience clusters and projects,^{Footnote 26} exceeding the five-year target of 524 and helping to establish the next generation of researchers. Funding recipients noted that the Program also helped to increase human resource capacity for R&D by incenting retired researchers to re-enter the workforce to undertake key research activities.

6.2 Increase in knowledge base of the sector

Funded clusters and projects made strong contributions to the knowledge base. Although data exists on the nature and extent of knowledge transfer activities, data on the impacts of these activities and how effectively they were delivered is not captured.

The AgriScience Program has made considerable progress in accomplishing its intermediate intended outcome of increasing the knowledge base of the sector. The number of articles published in peer-reviewed journals or gray literature and the number of presentations at workshops and conferences—both facilitators of knowledge transfer—are on track to meet or exceed initial targets. Articles published in peer-reviewed journals totaled 895 (812 through clusters and 83 through projects^{Footnote 27}) in the first three years of the AgriScience Program, which is 81% of the five-year target of 1,100. This result is only slightly lower than what was accomplished during the first three years of the AgriInnovation Program Stream B (872 peer-reviewed publications).

Data has not been collected on the impact of peer-reviewed articles on the scientific field. As part of the evaluation, bibliometric analyses were conducted to determine whether articles related to R&D achievements from selected clusters filled gaps in knowledge in their field(s). The evaluation found that some articles were published in journals of high impact and were cited proportionally higher than comparable articles in their fields. Examples of articles produced by clusters included in the case studies are presented in Table 5. Of note, all of the articles below were published in journals with an impact factor^{Footnote 28} ranging from 2.1 to 5.6, which indicates that they mostly have a ‘fair’ to ‘good’ reputation.

**Table 5. Bibliometrics analysis of selected clusters’ peer-reviewed articles**
Citations	Field-Weighted Citation Impact ^[1]
Beef cluster - Estimation of phenotypic and genetic parameters for growth, efficiency and measures of feeding behavior in steers fed a finisher diet in fall versus winter feeding regimes (2011)
66 (41 more than AAFC average at that time)	4.53 (more than four times more impact relative to articles in the same field)
Bioproducts cluster - Flexural and axial behaviour of sandwich panels with bio-based flax-FRP skins and various foam core densities (2016)
21 (four more than AAFC average at that time)	2.09 (two times more impact relative to articles in the same field)
Canola cluster - Replacing dietary soybean meal with canola meal improves production and efficiency of lactating dairy cows (2015)
39 (22 more than AAFC average at that time)	2.64 (more than 2.5 times more impact relative to articles in the same field)
Poultry cluster - The successful experimental induction of necrotic enteritis in chickens by Clostridium perfringens: a critical review (2012)
131 (106 more than AAFC average at that time)	3.98 (four times more impact relative to articles in the same field)
Wheat cluster - Identification of candidate genes, regions and markers for pre-harvest sprouting resistance in wheat (2014)
60 (43 more than AAFC average at that time)	5.06 (five times more impact relative to articles in the same field)
[1] Field-weighted citation impact indicates how the number of citations received by a publication compares to the average number of citations received by similar publications. A value of more than one means that the publication is cited more than expected according to the global average. Source: Bibliometric analysis, Canadian Agriculture Library and AAFC Office of Audit and Evaluation

Program indicators provide evidence of the extent to which knowledge is shared. Bibliometric analyses could build on existing measures to demonstrate the relative value and uptake of knowledge developed through the Program, thereby demonstrating AAFC’s contribution to the national and international scientific community’s knowledge base. The Canadian Agriculture Library has the capability to support bibliometric analyses of the Program’s most promising publications.

Funding recipients indicated that other types of knowledge transfer activities included online communications (for example, social media, blogs, webinars, websites), written communications (for example, newsletters, flyers, fact sheets and publications in industry magazines) and presentations at national and international conferences, trade shows and on-farm demonstrations (for example, field days). These types of knowledge transfer activities are important in disseminating information, particularly to farmers and ranchers, about how to improve production techniques and practices. However, the value and uptake of the knowledge disseminated through these channels are not measured.

Overall, the AgriScience Program appears to be on track to meet most of its knowledge transfer targets. Table 6 illustrates the results achieved in the first three years of the Program.

**Table 6. Knowledge transfer by clusters and projects (2018–19 to 2020-21)**
	Knowledge transfer products developed	Training/ knowledge transfer events	Participants at training/ knowledge transfer events	Presentations
Clusters	2,630 (average of 138 per cluster)	620 (average of 33 per cluster)	405,860 (average of 21,361 per cluster)	2,902 (average of 153 per cluster)
Projects^[1]	205 (average of 3.5 per project)	213 (average of 3.7 per project)	62,127 (average of 1,071 per project)	500 (average of 8.6 per project)
Total	2,835 (75% of target)	833 (35% of target)	467,987 (93% of target)	3,402 (no target set)
[1] 2020-21 data from two projects is not included due to delays in reporting. Source: Performance data, AAFC Programs Branch

While all clusters are required to conduct knowledge transfer activities, no metrics are collected on the transfer of knowledge at learning events beyond attendance numbers. As a result, it is not possible to determine whether participants or attendees read and understood the material; whether the information was relevant, useful and adopted into practice; and whether the channels and platforms used were effective for reaching the target populations.

Funding recipients do not require learning event attendees to complete exit or follow-up surveys. Therefore, they are not measuring the effectiveness of the event – the relevance, usefulness and uptake of the information shared. AAFC could encourage funding recipients to take a strategic approach and select a few key events for which this information would be valuable.

6.3 R&D innovations are generated

The AgriScience Program has made considerable progress toward the achievement of its five-year performance targets related to technology development.

As shown in Table 7, considerable progress has been made by clusters and projects considering that there are almost two years remaining in the Program and that technologies are only reported at the planned final stage on the innovation continuum. For example, if a technology is targeted for utilization, it will not be reported when developed or assessed, but will be reported when utilized.

**Table 7. Generation of technologies by clusters and projects (2018–19 to 2020–21)**
	Technologies developed	Technologies assessed	Technologies demonstrated	Technologies utilized
Clusters	130 (average of 6.8 per cluster)	232 (average of 12.2 per cluster)	63 (average of 3.3 per cluster)	101 (average of 5.3 per cluster)
Projects^[1]	27 (average of 0.5 per project)	33 (average of 0.6 per project)	21 (average of 0.4 per project)	63 (average of 1.1 per project)
Total	157 (84% of target)	265 (68% of target)	84 (41% of target)	164 (47% of target)
[1] 2020–21 data from two projects is not included due to delays in reporting. Source: Performance data, AAFC Programs Branch

In the first three years of the Program, 154 technologies attained intellectual property protection,^{Footnote 29} which is 48% of the five-year target. Examples of intellectual property protection include, but are not limited to, plant breeder rights, patents filed, registered trademarks and copyrights and registered or released varieties.

The case studies indicated that most of the technologies developed by the selected clusters involve improved farm management practices, development of seed varieties, enhanced disease and pest management and improved animal housing. See Table 8 for examples of technologies developed by the selected clusters.

**Table 8: Technologies developed by clusters**
Cluster	Technology Developed	Purpose of Technology
Beef	New barley varieties	Improve feed barley yields
Beef	New forage varieties	Increase pasture productivity
Wheat	AAC Brandon wheat variety	Improve wheat yields through greater disease resistance and harvesting efficiency
Wheat	New durum wheat variety	Increase disease resistance and use in Italian pasta products
Bioproducts	EcoSynthetix’ DuraBind™ binder	Natural substitute for harmful petrochemical-based resins used in particleboard production
Canola	Rapid qPCR test	Detect presence and reduce incidence of clubroot, a disease of the canola plant
Canola	Additional use of canola meal	Substitute for soybean meal to improve milk production of dairy cattle
Poultry	Use of ramps in chicken aviaries	Reduce falls, collisions and keel fractures of egg-laying hens
Poultry	Spectrum LED bulb	Provide a higher amount of red light, which is more effective than other types of light in stimulating egg production of laying hens
Organic	Pelletized and green manure	Increase organic wheat yield and grain quality in an environmentally friendly manner
Organic	Nutrient management tool	Enable organic farmers to assess if sufficient nutrients exist in the soil to maximize production
Source: Case studies data collection tool, AAFC Office of Audit and Evaluation

6.4 Commercialization of R&D innovations

The commercialization and adoption of technologies supported by the Program cannot be assessed because limited data is collected.

As shown in Table 7, 164 new technologies (new products, practices, processes and systems) were utilized in the first three years of the Program, which is 47% of the five-year target. The researchers and funding recipients interviewed as part of the cluster case studies indicated they do not follow the technologies after completion of the research project nor do they conduct surveys to determine adoption rates. They indicated they did not have the resources or expertise to determine the adoption rates or economic benefits of the technologies developed. The collection of this data can be challenging where adoption occurs outside of the funding period or the technology is sold to a private company. In most instances, several generations of AAFC R&D funding were required to develop a new technology. The Innovation Programs Directorate's results team reported that almost 30% of the cluster activities funded by the AgriScience Program are a continuation of work carried out under previous policy frameworks. The proportion of activities that are a continuation is higher in some clusters than others. For example, 75% of barley cluster and 57% of wheat cluster activities are a continuation.

To obtain data outside of the funding period, funding agreements could require the collection of data after the funding period has ended. This is a practice of other federal government departments. For example, some departments hold back a small amount of funding and require funding recipients to report back on impact (and other matters) a few years following project completion to receive the remaining funding.

Only half of funding recipients interviewed were able to speak to technology adoption. Of those, the majority indicated that there has been significant uptake of technologies developed by their projects or clusters. Funding recipients stated that their primary barriers to commercialization included demonstrating proof of concept (that is, demonstrating effectiveness of technology, particularly to risk-averse users), cost and ensuring product availability. The long timeframe required for the commercialization of many of the technologies was noted as a barrier, as many projects span the course of more than one policy framework. Internally, staff identified knowledge translation and transfer activities as a key factor in facilitating uptake of technologies.

7.0 Impact

This section describes the impact of the AgriScience Program on the Canadian economy and Canada’s R&D landscape.

7.1 Economic impact during the Program lifespan

The economic impact of AgriScience Program expenditures over three years is estimated to be $314 million, including the creation of 442 jobs annually and incremental wages and salaries of $81 million.^{Footnote 30}

The first three years of Program expenditures were applied to Statistics Canada’s national input-output model, which estimates the overall impact of spending on output, gross domestic product, employment and other variables in the Canadian economy. The model found that AgriScience Program expenditures of $141.0 million from April 1, 2018 to March 31, 2021 resulted in an estimated $314.4 million in spending across the Canadian economy during the same time period (see Table 9).^{Footnote 31} This estimate includes AAFC investments, leveraged investments from industry and other government sources and indirect spending impacts. This amounts to $2.20 in total facilitative economic impacts for every dollar spent by AAFC on AgriScience. Approximately 67%, or $210.6 million, of the total facilitative economic impact is estimated to be directly attributable to the AgriScience Program (would not have occurred in the absence of AAFC assistance).

The estimates for the AgriScience Program expenditures are in line with estimates for the first three years of AgriInnovation Stream B under the previous policy framework.

**Table 9. Economic impacts of AgriScience expenditures from 2018–19 to 2020–21 compared to AgriInnovation Stream B**
	AgriScience	Stream B	AgriScience	Stream B	AgriScience	Stream B
	Total Economic Impact		Incrementality Factor		Attributable Impact
Spending ($ million)	314.4	289.9	0.67	0.80	210.6	232.0
Gross domestic product at basic prices ($ million)	200.3	175.6	0.67	0.80	134.2	140.5
Wages and salaries ($ million)	120.9	106.4	0.67	0.80	81.0	85.1
Average annual number of jobs	659	570	0.67	0.80	442	456
Note: The incrementality factor applied to the calculation of attributable impact differs because it is based on interviewees’ estimations. Source: Economic impact analysis, AAFC Office of Audit and Evaluation

The economic impacts resulting from commercialization or adoption of the technologies supported by the AgriScience Program are not presented in Table 9 because this information is not collected. These estimates could have been higher if that data were available.

7.2 Impact of R&D activities

AgriScience Program activities have improved productivity and facilitated future R&D in the sector. However, intended impact is not specified and no data is collected on impacts.

The Program has not documented its intended economic, social and environmental impacts. While Program objectives and intended outcomes are stated, impacts of the Program are not identified, and impact data is not collected. The evaluation conducted case studies on a selection of clusters to determine the impacts achieved by technologies developed under the Growing Forward frameworks.

It can be challenging to find measurable criteria to demonstrate the broad societal impacts of R&D. Additionally, the achievement of R&D outcomes can take time and may be influenced by several contributors, thus making it difficult to determine the impact of a particular project. Nonetheless, several other government departments have required grants and contributions recipients to collect, analyze and provide data on longer-term outcomes and impact.

Impact of clusters’ technologies^{Footnote 32}

Some important technological advancements have taken place as a result of R&D conducted by the clusters. The most significant impacts have been in improved productivity of the primary sector as demonstrated by the examples in Table 10.

**Table 10: Examples of impacts of clusters R&D**
Technology	Cluster	Impacts
Feeding canola meal to dairy cattle	Canola	The development of canola meal to feed dairy cattle received cluster funding in Growing Forward 1 and Growing Forward 2. This research concluded that canola meal fed to dairy cattle leads to more milk production per day than other types of feed. The research, funded by the AgriScience Program and conducted by the U.S. Dairy Forage Research Centre, focussed on early lactation and found an increase in milk yield of 3.9 to 9.8 pounds per day for cows consuming diets supplemented with canola meal compared to soybean meal. Canola meal has subsequently been promoted as a replacement for other feeds and is now the most commonly used feed formulation for dairy cows. The high demand for canola meal has resulted in an increase in canola meal production in Canada from 2 million metric tonnes in 2010 to 4 million metric tonnes in 2019.
Improved cattle nutritional and breeding management	Beef	The research results led to a higher proportion of cows calving in the first 21 days, which resulted in heavier weaning weights and subsequently higher economic returns. Annual economic impact is projected to reach over $15 million after five years. Twenty-five years later, it is anticipated to result in over $80 million in increased annual returns to the rancher.
Development of AAC Brandon wheat variety	Wheat	The development of the AAC Brandon wheat variety occurred over a 13-year period. The original cross for AAC Brandon wheat was made in 2003. The development of AAC Brandon wheat variety was undertaken by the breeding team at the AAFC Swift Current Research and Development Centre. It involved 10 scientists and 100 support staff that collected data from approximately 1 million test plants. The variety was initially produced prior to Growing Forward 1, but the disease resistance testing and selection process occurred through both Growing Forward frameworks. The first certified sales of AAC Brandon occurred in 2015. The Brandon wheat variety was the most commonly grown wheat from 2016 to 2021 and accounted for close to half of all wheat acres planted in Western Canada in 2020. The value/benefit of the Brandon wheat is that it is more resistant to disease and more effective in harvesting than any other comparable wheat. It is estimated that AAC Brandon has produced enough wheat to make 8.7 loaves of bread for every person on earth.
LED Spectrum Lighting	Poultry	Research carried out under Growing Forward 2 led to the approval of a patent for the Spectrum LED Bulb, which is commercialized under the AgriLux trademark name in Canada. According to AgriLux, the lightbulb increases the thickness of eggshells; results in less breakage; increases average egg production up to five more eggs per hen per production cycle; decreases feed costs by as much as three grams per day per hen; and reduces electrical energy costs by 85%. Anecdotally, customers (egg production operators) have informed AgriLux that they see a return on investment within nine to 12 months. The product has been sold throughout North America. However, the rate of adoption has been slow as only 45 of the more than 1,200 poultry farms in Canada currently use the bulb. Penetration into the United States market has also been limited, but the company is currently looking for potential distributors in this market.
Source: Case studies data collection tool and interviews, AAFC Office of Audit and Evaluation

Broad impact

Several recipients established new partnerships, ranging from two to over 200 partnerships per project/cluster, as a result of their involvement in the Program. Some recipients reported subsequent R&D partnerships (with organizations not involved in the AgriScience cluster or project) with values ranging from $250,000 up to $37 million, which feed value back into the agriculture sector. These partnerships increase the capacity of the sector and support the continuation and evolution of agricultural R&D in Canada. This data is not tracked through AgriScience’s performance measurement system.

In the first three years of the Program, 727 highly qualified personnel worked on funded AgriScience clusters or projects. Four of the eleven cluster funding recipients interviewed indicated that researchers began their careers as a result of AgriScience initiatives. Training the next generation of researchers is an important benefit of engaging highly qualified personnel in the Program. It is expected that highly qualified personnel who have been engaged in the AgriScience Program (or its previous iterations under Growing Forward 1 and 2) have and will continue to contribute to agricultural R&D, thus benefitting the Canadian sector; however, this cannot be confirmed as it is not a measure tracked in the current performance measurement framework.

Comparison with other countries

Canada is ranked eighth compared to other countries in terms of the percentage of agricultural value added by the agricultural knowledge and innovation system (R&D).^{Footnote 33} The Organisation for Economic Co-operation and Development defines value added as the value generated by producing goods and services. It is measured as the value of output minus the value of intermediate consumption. Canada has achieved more value added by agricultural R&D than New Zealand, Japan, the United States and South Africa, but less than the United Kingdom, Israel, Australia, Brazil, European Union, Korea and Switzerland (see Figure 3). This research demonstrated the high benefit-cost ratios and rates of return of agricultural R&D and highlighted areas for consideration.^{Footnote 34}

8.0 Performance measurement

Gaps in the AgriScience Program’s logic model and performance indicators result in the collection of insufficient data to measure the Program’s outcomes and impact.

The evaluation found that much of the Program performance measurement gaps stem from an overly simplified logic model in the Performance Information Profile. This model largely measures outputs and early outcomes with indicators that are not all applicable across all types of research. In the case of clusters, data is collected and reported at the activity-level only. The resultant AgriScience performance data has limited utility as the Program is unable to assess the achievement of all outcomes and the Program’s impact.

Gaps – Program-level

The AgriScience Program Performance Information Profile is aligned with the AAFC Departmental Results Framework, which includes limited outcome-level indicators.

The profile was based on AAFC-level guidance and templates, which did not include appropriate intermediate-level outcomes or impact-level statements.

At the time of writing this evaluation report, AAFC efforts are focussed on improving the outcome measures and determining the expected impact of the next agricultural policy framework. These decisions will influence AgriScience’s performance measurement approach.

In the logic model, all R&D activities are required to track performance using the same indicators and follow the same results chain. Therefore, the logic model and performance measurement framework do not capture different types of funded R&D, such as the development of new varieties, beneficial management practices and innovations that may result in intellectual property, such as vaccines. There is value in ensuring that the outputs and outcomes of funded R&D are appropriately measured. For example, a strong knowledge transfer measurement strategy is crucial for beneficial management practices research, but not for other types of R&D such as the development of a new variety.

The AgriScience Program’s Performance Information Profile presents outcomes and indicators at the output and immediate outcome level. For example, activities focussed on knowledge transfer only measure the number of events held, number of attendees (at those events), number of presentations, number of knowledge transfer products developed and number of articles published. If an activity does not result in a technology, then the performance of that activity is not measured beyond outputs.

The logic model does not specify the economic, social and environmental impacts that result from project outcomes. As a result, the performance measurement strategy does not outline what indicators should be collected at the impact level. Contribution Agreements do not require funding recipients to collect and report on the impacts of their AgriScience-funded activities a few years after funding ends.

Similar data gaps were highlighted in a recommendation from the 2017 Evaluation of the AgriInnovation Program: Stream B. At that time, AAFC made a commitment to organize expert panel workshops to help develop an analytical framework for assessing impact. From those workshops came a 2019 report titled Pathways to Impact: Opportunities and Strategies for Evaluating AAFC Investments in Agri-food Research, which presented recommendations for improvement. The report stated that the performance measurement system “does not assess the public good impacts of research and for the most part ignores the knowledge flows beyond the easily traceable transfers through citations and payments for intellectual property”.

Gaps – Cluster-level

Funding recipients are required to submit annual performance reports, as well as a final performance report for each of their activities when those activities are completed. AAFC does not require clusters to submit cluster-level performance reports. Some funding recipients recognize that they have not been able to report on impact at the cluster level; one cluster reported being in the process of reviewing its cluster-level performance to address this shortfall. The evaluation found that this lack of information at the cluster level makes it difficult to assess cluster performance and impact.

9.0 Conclusions and recommendations

In terms of program delivery, AAFC has made many improvements to the application process, which enables the Program to run more effectively. The role played by the Science and Technology Branch was found to be critical as many clusters and projects depend on the capabilities and expertise of the Branch scientists.

The Program focussed more on industry priorities than government priorities, leading to differing coverage of priorities at the cluster and program levels. The AgriScience Program could have a role to play in ensuring that agricultural R&D addresses challenges in the public good realm, such as climate change, environmental sustainability, rising price of food and food security. While the industry-led nature of the Program has not focussed on these priorities, AgriScience could develop targets to direct funding in government priority areas, while maintaining some flexibility to fund R&D that addresses emerging priorities.

While clusters are recognized as an effective and efficient tool for conducting R&D, AAFC has not been able to demonstrate the value of clusters and is not using them strategically to address current and emerging challenges affecting the sector as well as government and department priorities. AAFC has focussed on monitoring cluster activities instead of monitoring each cluster as a whole. Although AAFC has been investing in clusters (and projects) over several policy frameworks, it has not measured the impact of the R&D funded at the economic, social and environmental level. The gaps in performance measurement hamper AAFC’s ability to report on AgriScience’s performance and impact in areas such as uptake of knowledge products and utilization of technologies, or the economic impact of new technologies.

There lacks a single mechanism for tracking and reporting on AgriScience expenditures incurred by AAFC. The decentralized nature of the financial management structure used within the Science and Technology Branch has limited its ability to track and report efficiently on financial and human resources at the Program level.

Recommendations

Recommendation 1: The Assistant Deputy Minister, Programs Branch, in collaboration with the Assistant Deputy Ministers, Science and Technology Branch and Strategic Policy Branch, should clarify the AgriScience Program’s strategic objectives, priorities and intended impacts.
Recommendation 2: The Assistant Deputy Minister, Programs Branch, should require project and cluster funding recipients to develop a performance measurement and evaluation framework to support reporting on outcomes and impacts, including future reporting on results after funding ends.
Recommendation 3: The Assistant Deputy Minister, Science and Technology Branch, in collaboration with the Assistant Deputy Minister, Corporate Management Branch, should clarify, document and communicate the financial expenditure tracking processes and systems.

Management Response and Action Plan

Management from the Programs and Science and Technology branches are supportive of the recommendations and have developed an action plan to address them by April 2023. Specifically, work is underway to refine priorities and objectives for the next iteration of the program under the Next Policy Framework. Also, for all returning clusters, Innovation Programs Directorate will require recipients to conduct an impact assessment of a selection of the cluster’s prior work completed under previous frameworks; there will be new reporting requirements for clusters to capture intended impacts and achievements at the cluster level, and, at the end of the program cycle. Lastly, Science and Technology Branch Partnerships and Planning Directorate will work with Corporate Management Branch Financial Management Advisory Division to clarify the coding structure of projects under the AgriScience Program and to communicate the appropriate coding expectations to Science and Technology Branch and Corporate Management Branch personnel at research centres.

Annex A - Evaluation methodology

Document, data, and literature review

To assess program relevance, design, delivery and effectiveness, the evaluation reviewed internal program documents and performance data. The evaluation also examined select literature to support the assessment of relevance and the effectiveness of the cluster approach.

Bibliometric analysis

Eighty-five articles were reviewed as part of the bibliometric analysis, which was carried out in partnership with the Canadian Agriculture Library. To allow sufficient time for the intended impacts to occur, the bibliometric analysis examined a sample of articles published by researchers involved in the two previous iterations of the Program (AgriInnovation Stream B and the Canadian AgriInnovation Program). The sample of articles was selected from the six case study clusters. The number of articles selected for bibliometric analysis from each cluster is shown in Table 11.

**Table 11. Number of articles from each cluster included in bibliometric analysis**
	Growing Forward 1	Growing Forward 2
Beef cluster	9	9
Bioproducts cluster	N/A^[1]	7
Canola cluster	3	12
Organic cluster	8	7
Poultry cluster	6	9
Wheat cluster	7	8
Source: AAFC Office of Audit and Evaluation [1] The Bioproducts Cluster was developed in Growing Forward 2.

Interviews

Interviews were conducted with internal and external stakeholders to assess program relevance, design, delivery and effectiveness. External stakeholder interviews were conducted with representatives of a sample of organizations and companies that received cluster or project funding through the AgriScience Program, as well as a sample of external funding partners that were involved in the clusters and two subject matter experts. In total, 28 interviews with external stakeholders were completed as part of the evaluation. Interviews were also conducted with 18 AAFC Program staff and senior management.

The following scale was used to describe the significance of the qualitative findings in terms of the relative proportion of responses:

“All/almost all” – findings reflect the views and opinions of 90% or more of the interviewees in the group.
“Large majority/most” – findings reflect the views and opinions of at least 75% but less than 90% of interviewees in the group.
“Majority” – findings reflect the views and opinions of at least 51% but less than 75% of interviewees in the group.
“Half” – findings reflect the views and opinions of 50% of interviewees in the group.
“Some” – findings reflect the views and opinions of at least 25% but less than 50% of interviewees in the group.
“A few” – findings reflect the views and opinions of at least two interviewees but less than 25% of interviewees in the group.

Case studies

The case studies focussed on the Clusters Component of the AgriScience Program because it is of interest to senior management. In selecting the sample of clusters, consideration was given to the sector, type of activities, number of technologies developed and budget to ensure representation across the range of clusters funded. Consideration was also given to clusters with success stories identified by AAFC Program staff, particularly those with identified successes in the area of economic growth. The following six clusters were selected:

Canada’s Sustainable Beef & Forage Science Cluster
Bioproducts Agri-Science Cluster
Canola AgriScience Cluster
Organic Science Cluster III: Connecting Environmental Sustainability
Poultry Science Cluster III: Enhancing Value to Canadians through Research
Canadian National Wheat Cluster

Representatives of the selected clusters were asked to complete a data collection tool, which requested information on the adoption rate and economic impact of five leading technologies developed by the cluster during either Growing Forward 1, Growing Forward 2 and/or the Canadian Agricultural Partnership. One or two of the leading technologies from each cluster were selected for further investigation of the extent of utilization. Investigation of the other technologies described in the Data Collection Tool was limited to discussions with cluster funding recipients regarding the degree of utilization and adoption rate of these technologies.

The methodology employed for the case studies is as follows:

Reviewed documentation on the activities of each sample cluster, including that available from AAFC and the websites maintained by cluster funding recipients.
Conducted interviews with cluster funding recipients and reviewed the completed Data Collection Tools.
Conducted follow-up interviews with cluster funding recipients to discuss each of the leading technologies identified in each Data Collection Tool.
Selected one or two of the leading technologies identified in the completed Data Collection Tools for further investigation.
Reviewed the available documentation on the leading technologies selected for further investigation.
Examined the results of the bibliometric analysis for articles published by researchers involved in the case study clusters.
Conducted interviews with researchers, industry associations and industry partners to obtain more detailed information on the impacts of the leading technologies selected for further investigation.

Twenty-six interviews were completed as part of the case studies.

Economic impact analysis

In August 2021, expenditures during the period from 2018–19 to 2020–21 were applied to Statistics Canada’s national input-output model. The model provides estimates of the overall impact on output, Gross Domestic Product, employment and other variables in the Canadian economy. The same analysis was conducted for the previous evaluation of AgriInnovation Stream B.

The analysis was undertaken as follows:

Obtained AAFC expenditure data from the Programs Branch and the Science and Technology Branch.
Obtained a database of total planned project expenditures by funding source from Contribution Agreements and Collaborative Research and Development Agreements for all AgriScience clusters and projects.
Extrapolated estimates for total project expenditures using both the actual AAFC expenditures and planned project expenditures. The planned project expenditure data was used to develop the percentage breakdown of funding sources (AAFC, other government, and industry) for each year. These percentages were applied to validated actual expenditures for each respective year to develop an estimate for the total project and cluster expenditures per year. Actual AAFC Programs Branch administrative and Vote 5 capital expenditures were added to the total separately.
Developed estimates of the percentage of expenditures taking place within AAFC research centres and external to AAFC (for example, universities or private sector R&D facilities). The percentages were developed based on actual Vote 1 (internal to AAFC) and Vote 10 (external) expenditures.
Determined the percentage breakdown of expenditures in terms of major cost categories. Detailed project budgets were reviewed from all cluster and project Contribution Agreements using the latest amendments available and a sample of five cluster and 18 project Collaborative Research and Development Agreements work plan budgets to estimate overall project cost breakdown in terms of major cost categories, including administration, salaries/benefits, contracted services, travel, capital/assets and other direct project costs. Clusters and projects with the highest level of Vote 1 funding were selected to ensure the cost categories most accurately reflected the major expenditures in these types of projects. A separate breakdown was identified for Vote 10 (Contribution Agreements) and Vote 1 (Collaborative Research and Development Agreements) activities.
Estimates of further sub-categories of costs that were developed during the evaluation of the AgriInnovation Program Stream B were used to match the data with an appropriate industry code since it was assumed that the sub-categories of costs would be similar to those incurred during the AgriScience Program. The estimates were developed by coding and tabulating detailed cost descriptions for a sub-sample of project budgets. Investment analysis data on collaboration was also reviewed to estimate the proportion of external research that likely took place at a university versus a private sector or independent R&D lab. North American Industry Classification System based Input-Output Industry Classification codes were assigned to each expenditure sub-category. Further assumptions were made as to the likely percentage of the expenditure that is attributable to Canadian industries (for example, only 10% of computer purchase expenditures were included since most computer manufacturing takes place outside of Canada). The following industry categories were used in the analysis:
- Scientific R&D services
- Universities
- Other federal government services (except defence)
- Business, professional and other membership organizations
- Machinery, equipment and supplies wholesaler-distributors
- Basic chemical manufacturing
- Pesticide, fertilizer and other agricultural chemical manufacturing
- Agricultural, construction and mining machinery manufacturing
- Ventilation, heating, air-conditioning and commercial refrigeration equipment manufacturing
- Other electronic product manufacturing
- Traveller accommodation
- Other professional, scientific and technical services
- Air transportation
- Food services and drinking place
- Automotive equipment rental and leasing
- Software publishers
- Computer and peripheral equipment manufacturing
Developed a table of total expenditures per industry per year to provide input for the input-output analysis model.
Developed estimates for the direct and indirect economic impacts (for example, output, Gross Domestic Product, employment and wages and salaries) using the Statistics Canada input-output multipliers and simulations.
Estimated the total facilitative economic impacts (combined direct and indirect economic impacts).
Incorporated the incrementality factor to calculate the total attributable economic impacts. An incrementality factor of 67% was applied to all impacts based on cluster and project interviewee estimates that there is a 33% likelihood that their respective clusters and projects would have proceeded as planned in the absence of AAFC assistance.

Methodological limitations

The following methodological limitations were considered in interpreting the data:

Limitation	Mitigation Strategy	Impact on Evaluation
Long-term nature of Program outcomes. It can take many years before scientific projects generate the expected intermediate and long-term outcomes, which could limit the evaluation’s capacity to assess program impact.	To mitigate this challenge, the evaluation included six longitudinal case studies of clusters that received funding across two or three policy frameworks.	Longitudinal case studies provided a more accurate assessment of the long-term nature of Program outcomes.
Lack of impact data on technologies developed. Some cluster funding recipients had difficulty completing the Data Collection Tool. They were unfamiliar with the specific impacts (for example, adoption rates, financial benefits) of technologies developed due to being too far removed from the technologies after initial development or not collecting impact-related data.	The data provided in the Data Collection Tool was supplemented with interviews with funding recipients, project researchers and industry partners to determine the impacts of leading technologies identified in the Data Collection Tool.	The case studies included the information collected through the additional interviews and provided useful information on the impacts of a sample of the leading technologies funded by the Program.
Lack of data on impact of peer-reviewed published articles. Clusters were successful in publishing in peer-reviewed journals; however, the number of published articles was too high to allow for a fulsome bibliometrics analysis.	A number of articles were selected from the case study clusters from previous iterations of the Program.	The evaluation confirmed whether project-level work contributed to science excellence but was not able to do so at the cluster or Program level.

Annex B – AgriScience program logic model

Long-Term Program Outcomes	Innovations supported by research and development are commercialized.
Medium-Term Program Outcomes	The sector’s knowledge base is increased.
	Knowledge of innovative technologies, products, practices, processes and services is transferred to the sector.
	Research and development supported innovations are generated.
Short-Term Program Outcomes	The sector’s capacity to conduct innovative research and development is increased.
Outputs	Program outputs Applications received Approval/rejection letters sent out Contribution Agreements signed Collaborative Research Development Agreements signed Financial claims processed AAFC investment Documents/records indicating performance information On-going communication with industry
Outputs	Recipient outputs Program recipients are increasing innovation and research knowledge base Knowledge transfer products are developed
Activities	(Vote 1 and Vote 10) Activities to facilitate research and development: Program activities Receive applications Review and propose projects/applications for approval or rejection Negotiate and prepare Contribution Agreements Negotiate and prepare Collaborative Research Development Agreements Process financial claims Collect and analyze performance information Monitor projects Communication with industry
Activities	Recipient activities Conduct scientific research and development Test, develop and/or improve innovative agri-products, practices and processes Training and knowledge transfer Engage AAFC researchers to assist with research and development Manage projects Prepare project reports
Source: AgriScience Performance Information Profile

Footnotes

Footnote 1

The economic analysis for the AgriScience Program was conducted in August 2021. The estimate was calculated using preliminary reports of actual expenditures.

Return to footnote 1 referrer

Footnote 2

Boye, J.I., & Y. Arcand. 2013. Current trends in green technologies in food production and processing. Food Eng. Rev., 5, 1-17.

Return to footnote 2 referrer

Footnote 3

Organisation for Economic Co-operation and Development. 2015. Innovation, Agricultural Productivity and Sustainability in Canada, OECD Food and Agricultural Reviews. Paris: OECD Publishing.

Return to footnote 3 referrer

Footnote 4

Ibid.

Return to footnote 4 referrer

Footnote 5

Organisation for Economic Co-operation and Development. 2021. Agricultural policy monitoring and evaluation 2021: Addressing the challenges facing food systems. Paris: OECD Publishing.

Return to footnote 5 referrer

Footnote 6

Smeets Kristkova, Z., M. Van Dijk, & H. Van Meijl. 2016. Projections of long-term food security with R&D driven technical change — A CGE analysis. NJAS – Wagening. J. Life Sci., 77(1), 39-51.

Return to footnote 6 referrer

Footnote 7

This analysis included each project/activity’s primary and secondary cross-cutting areas; some projects/activities are counted twice and the percentages do not add to 100%.

Return to footnote 7 referrer

Footnote 8

Malla, S., & D.G. Brewin. 2019. Crop research, biotech canola, and innovation policy in Canada: Challenges, opportunities, and evolution. Can. J. Agric. Econ., 67(2): 135-150.

Return to footnote 8 referrer

Footnote 9

Organisation for Economic Co-operation and Development. 2015. Innovation, Agricultural Productivity and Sustainability in Canada, OECD Food and Agricultural Reviews. Paris: OECD Publishing.

Return to footnote 9 referrer

Footnote 10

Smeets Kristkova, Z., M. Van Dijk, & H. Van Meijl. 2016. Projections of long-term food security with R&D driven technical change—A CGE analysis. NJAS – Wagening. J. Life Sci., 77(1), 39-51.

Return to footnote 10 referrer

Footnote 11

Ibid.

Return to footnote 11 referrer

Footnote 12

Malla, S., & D.G. Brewin. 2019. Crop research, biotech canola, and innovation policy in Canada: Challenges, opportunities, and evolution. Can. J. Agric. Econ., 67(2): 135-150.

Return to footnote 12 referrer

Footnote 13

Yost, M. A., K.A. Sudduth, C.L. Walthall, & N.R. Kitchen. 2019. Public–private collaboration toward research, education and innovation opportunities in precision agriculture. Precis. Agric., 20(5), 4-18.

Return to footnote 13 referrer

Footnote 14

Government of Canada. 2016. Canada: A nation of innovators.

Return to footnote 14 referrer

Footnote 15

Seddon, M., and S. Usmani. April 2017. Superclusters! Lessons + opportunities for Canada.

Return to footnote 15 referrer

Footnote 16

Broekel, T., D. Fornahl, & A. Morrison. 2015. Another cluster premium: Innovation subsidies and R&D collaboration networks. Res. Policy, 44(8), 1431-1444.

Return to footnote 16 referrer

Footnote 17

Vernay, A-L., B. D'Ippolito, & J. Pinkse. 2018. Can the government create a vibrant cluster? Understanding the impact of cluster policy on the development of a cluster. Entrep. Reg. Dev., 30(7-8), 901-19.

Return to footnote 17 referrer

Footnote 18

The Royal Society. July 2020. Research and innovation clusters: Policy briefing.

Return to footnote 18 referrer

Footnote 19

Kiese, M. 2017. Regional cluster policies in Germany: challenges, impacts and evaluation practices. J. Technol. Transf., 44(6), 1698-1719.

Return to footnote 19 referrer

Footnote 20

Kircher, M., R. Breves, A. Taden, & D. Herzberg. 2018. How to capture the bioeconomy's industrial and regional potential through professional cluster management. N. Biotechnol., 40(Part A), 119-128.

Return to footnote 20 referrer

Footnote 21

Mazur, V., K. Barmuta, S.S. Demin, & E. Tikhomirov. 2016. Innovation clusters: Advantages and disadvantages. Int. J. Econ. Financial Issues, 6(S1), 270-274.

Return to footnote 21 referrer

Footnote 22

Seddon, M., and S. Usmani. April 2017. Superclusters! Lessons + opportunities for Canada.

Return to footnote 22 referrer

Footnote 23

Engel, D., V. Eckle, & M. Rothgang. 2017. R&D funding and private R&D: Empirical evidence on the impact of the leading-edge cluster competition. J. Technol. Transf., 44, 1720-1743.

Return to footnote 23 referrer

Footnote 24

Programs Branch program delivery costs exclude Employee Benefit Plan costs.

Return to footnote 24 referrer

Footnote 25

Program delivery costs for similar programs include Employee Benefit Plan costs.

Return to footnote 25 referrer

Footnote 26

2020–21 data from two projects is not included due to delays in reporting.

Return to footnote 26 referrer

Footnote 27

2020–21 data from two projects is not included due to delays in reporting.

Return to footnote 27 referrer

Footnote 28

Journal Impact Factor is a measure of the frequency with which the average article in a journal has been cited in a particular year. It is used to measure the importance or rank of a journal.

Return to footnote 28 referrer

Footnote 29

2020–21 data from two projects is not included due to delays in reporting.

Return to footnote 29 referrer

Footnote 30

The economic analysis was conducted in August 2021. The estimate was calculated using preliminary reports of actual expenditures.

Return to footnote 30 referrer

Footnote 31

The economic analysis was conducted in August 2021. The estimate was calculated using preliminary reports of actual expenditures.

Return to footnote 31 referrer

Footnote 32

The Innovation Programs Directorate’s results team initiated the study of clusters’ success stories, which the evaluation completed with further interviews and the completion of a data collection tool through case studies.

Return to footnote 32 referrer

Footnote 33

Organisation for Economic Co-operation and Development. 2021. Agricultural Policy Monitoring and Evaluation 2021: Addressing the Challenges Facing Food Systems. Paris: OECD Publishing.

Return to footnote 33 referrer

Footnote 34

Alston, J.M. 2010. The benefits from agricultural research and development, innovation, and productivity growth, OECD Food, Agriculture and Fisheries Papers, No. 31. Paris: OECD Publishing.

Return to footnote 34 referrer

Evaluation of the AgriScience Program

Abbreviations

Executive summary

Purpose

Scope and methodology

Background

Findings

Conclusion

Recommendations

Management Response and Action Plan

1.0 Introduction

2.0 Scope and methodology

3.0 Program profile

3.1 Program description

3.2 Governance and Program delivery

Selection process – clusters

Selection process – projects

Other considerations

3.3 Resources

3.4. Intended outcomes

4.0 Relevance

4.1 Alignment of Program with industry needs

Need for public-private collaboration in R&D

4.2 Alignment with AAFC and government role and priorities

Alignment with AAFC role and priorities

Alignment with federal government role and priorities

Distribution of AgriScience activities across departmental and government priority areas

5.0 Program design and delivery

5.1 Effectiveness of cluster model

Extent of collaboration

5.2 Effectiveness of Program design and delivery

Role of AAFC scientists and R&D staff

5.3 Program administration

Ratio of AAFC program delivery costs to total program costs

Oversight of expenditures

5.4 Integration of Gender-Based Analysis Plus and official languages

6.0 Effectiveness

6.1 Increase in sector capacity to conduct R&D

6.2 Increase in knowledge base of the sector

6.3 R&D innovations are generated

6.4 Commercialization of R&D innovations

7.0 Impact

7.1 Economic impact during the Program lifespan

7.2 Impact of R&D activities

Impact of clusters’ technologiesFootnote 32

Broad impact

Comparison with other countries

8.0 Performance measurement

Gaps – Program-level

Gaps – Cluster-level

9.0 Conclusions and recommendations

Recommendations

Management Response and Action Plan

Annex A - Evaluation methodology

Document, data, and literature review

Bibliometric analysis

Interviews

Case studies

Economic impact analysis

Methodological limitations

Annex B – AgriScience program logic model

Impact of clusters’ technologies^{Footnote 32}