Evaluation of the Collaborative Framework

On this page

Abbreviations

AAFC:

Agriculture and Agri-Food Canada

R&D:

research and development

RDT:

research, development and technology

SMS:

Science Management Solution

Executive summary

Purpose

The Office of Audit and Evaluation of Agriculture and Agri-Food Canada (AAFC) conducted an evaluation of the Collaborative Framework to provide senior management with an assessment of the relevance, design, delivery, economy and performance of the Framework.

Scope and methodology

Collaborative Framework activities were evaluated using multiple lines of evidence: review of documents, data and literature; interviews with AAFC staff and management, AAFC scientists, project funders and project collaborators; bibliometric analysis; and case studies. The evaluation focused on Collaborative Framework activities carried out from 2016–17 to 2020–21. Case studies examined longer-term performance by reviewing projects that began before 2016–17.

Background

The Collaborative Framework is an AAFC policy that enables external organizations to collaborate with, or support, AAFC science capacity outside of formal AAFC science programs, such as the AgriScience Program. External organizations provide funding, through cash or in-kind contributions, to Science and Technology Branch to support Research and development (R&D) projects while leveraging federal in-kind investments (people, resources and/or infrastructure) for research. External funders or collaborators may include provincial funding agencies, producer commissions/associations, universities/educational institutions and the private sector.

The Office of Audit and Evaluation’s 2019–20 preliminary assessment of Foundational Science and Research recommended a Collaborative Framework evaluation, noting that it is a significant source of R&D project funding with a complex delivery process and associated operational risks.

Findings

• The Collaborative Framework is aligned with AAFC and government roles and responsibilities, including commitments to engage in science, innovation and collaboration.
• The Collaborative Framework addresses industry R&D needs and priorities, enabling funders and collaborators to benefit from AAFC’s extensive network of experts and other tangible R&D resources.
• Strategic oversight is lacking within the Collaborative Framework. The project review and approval process focuses on singular project approvals, lacking a broader view of all projects within the Framework and AAFC's science portfolio. The Framework also lacks a strategy to guide and enable assessment of resource distribution. There is potential for gaps in addressing priorities and risk of duplication of project objectives.
• Despite the availability of communication resources and increased corporate support and engagement of research and development centres, there is inconsistent implementation of Collaborative Framework processes. Output data is not captured consistently or completely.
• Coordination and communication between AAFC and funders or collaborators is generally effective.
• AAFC’s total investment in the Collaborative Framework is not known because there is limited tracking of the AAFC resources used to conduct Framework projects, as well as the costs of managing and administering the Framework.
• The Framework does not have a performance measurement strategy.
• Collaborative Framework projects have increased the sector’s knowledge base and generated products, processes and practices, but the extent to which this has occurred is not fully captured.
• Framework operation has been constrained by factors such as inconsistent promotion to external stakeholders and the COVID-19 pandemic. Despite pandemic-related challenges, AAFC was able to adapt in certain situations to ensure research continued.
• The Collaborative Framework is an important source of funding for developing AAFC and sector R&D capacity. The Framework facilitates new or existing AAFC research projects and strengthens the capabilities and expertise of AAFC scientists.

Conclusion

The Collaborative Framework facilitates R&D collaboration between AAFC and external stakeholders, thereby increasing industry access to AAFC science capacity and supporting the enhancement and continuity of AAFC research. There are opportunities to better position the Collaborative Framework to achieve good governance and respond to AAFC’s collaborative research needs in a manner that aligns with Science and Technology Branch’s other research activities and future direction for the management of science. The efficiency and effectiveness of AAFC's collaborations could be enhanced with increased strategic oversight during proposal review. The Framework would also benefit from a performance measurement strategy that outlines specific outcomes and impacts. Furthermore, the effectiveness of the Framework could be improved by more consistently capturing project data to obtain a better understanding of project outputs and total AAFC investment.

Recommendations

Recommendation 1: The Assistant Deputy Minister, Science and Technology Branch, should incorporate additional considerations into the project selection and approval process to ensure that projects are strategically coordinated and aligned with Branch objectives and existing and emerging priorities.

Recommendation 2: The Assistant Deputy Minister, Science and Technology Branch, should develop a performance measurement strategy for the Collaborative Framework.

Recommendation 3: The Assistant Deputy Minister, Science and Technology Branch, should implement additional controls to ensure project data is captured consistently and completely.

Management response and action plan

Management agrees with the evaluation recommendations and has developed an action plan to address them by April 2023.

1.0 Introduction

The Office of Audit and Evaluation conducted an evaluation of Agriculture and Agri-Food Canada’s (AAFC) Collaborative Framework as part of the 2020–21 to 2024–25 Integrated Audit and Evaluation Plan. This evaluation was conducted in accordance with the Treasury Board’s Policy on Results and fulfills the requirements of the Financial Administration Act. The evaluation is intended to inform current and future program and policy decisions.

2.0 Scope and methodology

The evaluation assessed the relevance, design, delivery, economy and performance of the Collaborative Framework. It focused primarily on Collaborative Framework activities carried out from 2016–17 to 2020–21; however, case studies assessed longer-term performance by reviewing projects that began prior to 2016–17. The evaluation excluded projects:

• that did not receive external funding (that is, projects that received only in-kind contributions from collaborators);
• covered by international Memorandums of Understanding and agreements;
• with co-location agreements; and
• reviewed by the Office of the Auditor General of Canada.

Collaborative Framework activities were evaluated using multiple lines of evidence: review of documents, data and literature; interviews with AAFC staff and management, AAFC scientists, project funders and project collaborators; bibliometric analysis; and case studies. For the detailed evaluation methodology, see Annex A.

Throughout the evaluation process, early substantive findings were shared with AAFC management and staff involved in the Framework to inform ongoing planning.

3.0 Framework profile

3.1 Framework rationale

2010 audit by the Office of the Auditor General

The Collaborative Framework was developed in response to a recommendation from the 2010 Spring Report of the Auditor General of Canada. The 2010 audit examined whether AAFC managed its research activities in a manner that enabled it to achieve its strategic direction. This strategic direction was based partly on AAFC’s 2006 Science and Innovation Strategy, of which one of the goals was to build science and innovation partnerships.

The audit concluded that:

• AAFC had not developed and implemented appropriate plans to achieve its strategic direction through its research partnerships;
• project monitoring processes were not consistently followed or applied across research projects; and,
• little analysis of AAFC’s research portfolio as a whole was completed during project selection.

In response, AAFC committed to:

• developing and implementing a Collaborative Framework and guidelines to support partnership selection, management and monitoring;
• identifying approval processes, required resources and timelines for partnership arrangements, developing corresponding implementation plans, and planning and reporting on related expenditures; and
• improving processes for monitoring and reporting on project performance, and strengthening its Performance Measurement Framework to include clear and measurable targets for addressing national priorities through research to facilitate a more integrated project management approach.

3.2 Framework description

The Collaborative Framework is an AAFC policy that enables external organizations to collaborate with, or support, AAFC science capacity outside of formal science programs, such as the AgriScience Program. External organizations provide funding, through cash or in-kind contributions, to Science and Technology Branch to support R&D projects while leveraging federal in-kind investments (people, resources and/or infrastructure) for research. External funders or collaborators may include provincial funding agencies, producer commissions/associations, universities/educational institutions and the private sector. There is a 10% science service charge on cash funding received by AAFC to help support the research and development centre’s science operations, and a Collaborative Research and Development Agreement between AAFC and the external partner(s) outlines Intellectual Property considerations and requirements.

The Collaborative Framework is a component of the Foundational Science and Research Program inventory, which encompasses core research conducted by AAFC. Collaborative Framework projects provide significant funding to the Branch, assist in establishing and strengthening Science and Technology Branch’s partnerships and enable AAFC to be responsive to industry priorities. Guiding principles of the Collaborative Framework help to ensure that AAFC and its partners meet the terms of their agreements, and facilitate mitigation of any potential financial, political, legal, environmental and strategic risks.

Collaborative Framework projects may result from competitive calls for proposals, whereby a scientist submits a proposal to an external funding agency, or through discussions initiated by AAFC scientists, collaborators or funders.

Collaborative Framework projects may be short-term or strategic. Short-term projects are carried out over a period of more than two weeks but less than five years, and are led by an AAFC principal investigator. Alternatively, strategic projects are carried over five years or more and are usually guided by Director General champions, Directors of Research, Development and Technology (RDT) and/or AAFC’s International Engagement division, with support from the Science Policy and Partnerships Division. Strategic projects can include co-locations and international Memorandums of Understanding and agreements, and are intended to contribute to building the capacity of the Canadian innovation system.

The Collaborative Framework was not previously evaluated as a separate entity, but was discussed briefly in the 2016–17 Evaluation of AAFC Science 2.1.1. The Office of Audit and Evaluation’s 2019–20 preliminary assessment of Foundational Science and Research recommended a Collaborative Framework evaluation, noting that it is a significant source of R&D project funding with a complex delivery process and associated operational risks.

3.3 Proposal review and approval process

The Collaborative Framework is implemented by AAFC research and development centres (that is, at the local level) and supported by the Partnerships and Planning Directorate of AAFC’s Science and Technology Branch. Collaborative Framework projects are proposed, reviewed and approved through an established internal process prior to implementation (see Figure 1). Certain parts of the process vary depending on the project budget, project length and international involvement in the project.

Source: Created by the Office of Audit and Evaluation in support of the Evaluation of the Collaborative Framework

Description of above image

Figure 1: Collaborative Framework proposal review and approval process

• Development and discussion of proposal concept
• Proposal entry into SMS
• Proposal assessment by Project Review Committee
• Proposal assessment by Branch Executive Committee (if needed)
• Confirmation of support from external stakeholders
• Development of Collaborative Research and Development Agreement
• Submission of final proposal
• Project implementation and reporting

Project leads for short-term projects are directed to discuss proposal concepts with the local Director of RDT and others, as recommended by the Director of RDT. The proposal is then reviewed and approved by the project lead’s Director of RDT.

Proposals for all Framework projects are submitted in the Science and Technology Branch’s Science Management Solution (SMS), which is a suite of digital tools used to facilitate the internal development, approval and monitoring of R&D projects. Most proposals that require less than 5% of a scientist’s time are not entered in SMS.

Proposals are then assessed by the Project Review Committee, which consists of the local Director of RDT (Assessment Chair), the local Associate Director of RDT, the Integrated Services Manager, a commercialization officer from the Office of Intellectual Property and Commercialization and others as required (for example, International Engagement Division, if the project is international). Certain parts of the review process (for example, information sharing method and frequency) can be customized at the research and development centre level. Projects with a collaborator contribution of over $500,000 and projects of higher risk must undergo additional review by the Branch Executive Committee. Strategic projects must also undergo review by the Branch Executive Committee and must have a business plan.

Upon completion of required reviews, the project lead is permitted to confirm to external stakeholders that AAFC is interested in collaborating, which may involve submitting a research proposal to a competitive funding process. Once external funding has been approved, a Collaborative Research and Development Agreement is developed and executed by the Office of Intellectual Property and Commercialization. Financial validation of the agreement is provided by a local financial officer.

In the review of short-term projects, the local Director of RDT or a delegate is responsible for documenting funding decisions, approving proposal amendments and providing science quality assurance. The project lead then implements, updates and reports on the project in accordance with the requirements of the collaborative agreement, with oversight from the local Director of RDT. Oversight of strategic projects varies based on the scope of the project.

3.4 Resources

Collaborative Framework projects are funded through Vote-Netted Revenue and Special Purpose Allotments, which are funds AAFC collects from external stakeholders (for example, industry, government departments) on an annual basis for collaborative projects. These funds are supplemented with in-kind contributions from AAFC; however, the total cost of conducting Collaborative Framework projects for AAFC is not known.

AAFC received almost $45 million from external funders between 2018–19 and 2020–21 to carry out R&D projects under the Collaborative Framework (see Table 1).

Table 1: Collaborative Framework expenditures by external funders

Fiscal year	Expenditures by external funders ($)
2018–19	17,438,000
2019–20	17,857,000
2020–21	9,689,000
Total	44,984,000
Notes Totals were calculated using the rounded values for each fiscal year. As noted in the evaluation of the AgriScience Program, some AgriScience Program expenditures were erroneously coded and may thus be included in the values above. Source: Financial Data, Science and Technology Branch

Collaborative Framework expenditures by external funders in 2020–21 were lower than previous fiscal years due to delays and cancellations of R&D activities caused by the COVID-19 pandemic. See section 9.1 for additional impacts of the pandemic on collaborative R&D activities.

3.5 Intended outcomes

The Collaborative Framework was conceived as an instrument to support a wide range of research, development and technology transfer outcomes in Canada’s agriculture and agri-food sector. It is not a program and does not have a performance measurement strategy. There are no specific outcomes, indicators or targets set for research carried out under the Framework in Science and Technology Branch’s broader Foundational Science and Research Program Performance Information Profile.

AAFC’s Foundational Science and Research Program aims to achieve four outcomes:

1. Immediate outcome: The Canadian agriculture and agri-food sector knowledge base is increased through AAFC supported scientific and innovative research
2. Intermediate outcome: A strengthened national and international network of agricultural science
3. Intermediate outcome: Scientific knowledge (for example, products, technologies, beneficial management practices) is transferred or available for transfer to the agriculture sector and the science community
4. Ultimate outcome: The Canadian agriculture and agri-food sector is effective in transforming ideas into new products, processes or practices

4.0 Relevance

4.1 Alignment with AAFC and government roles and responsibilities

The Collaborative Framework helps to advance one of the three core responsibilities of AAFC: Science and Innovation. As stated in the 2022–23 Departmental Plan, this responsibility focuses on increasing knowledge through supported scientific and innovative research and transforming ideas into new products, processes or practices.

The Collaborative Framework is aligned with the federal government's roles, responsibilities and priorities, as it contributes to national and international commitments related to innovation and collaborations/partnerships in R&D. As stated in the 2016 Inclusive Innovation Agenda and the 2017 Innovation and Skills Plan, Canada should foster more partnerships and collaborations between businesses, post-secondary institutions, research institutions and government. These new collaborations have the potential to help bridge the commercialization gap and strengthen the value chains. The Collaborative Framework also supports the Inclusive Innovation Agenda “Global Science Excellence” area of action, as it promotes partnerships with business and aims to strengthen basic and applied research capabilities.^{Footnote 1} According to the literature reviewed in this evaluation, there is an increasing need for public-private collaboration and investment in research and development to prevent siloing of operations and drive innovation along the value chain.^{Footnote 2Footnote 3}

4.2 Alignment with industry needs and priorities

The Collaborative Framework facilitates industry access to AAFC R&D capacity, which serves to increase knowledge and resolve current and emerging issues in the sector. Current and emerging sector needs and gaps, as identified by funders, include sustainability and climate change, illness/disease, food quality and safety, soil health, and profitability or competitiveness of the sector. The Collaborative Framework also enables AAFC to gain a greater awareness of industry research needs, thus ensuring that its research is relevant to industry. Interviewees from all stakeholder groups confirmed to various degrees that the Framework addresses emerging research needs.

Furthermore, the majority of funders and all collaborators noted that certain projects would not have proceeded or would have proceeded on a smaller scale in the absence of AAFC support. All interviewed funders and collaborators valued AAFC's expertise, capacity to take on research projects, infrastructure and equipment. Funders acknowledged that AAFC’s network of scientists and staff have expertise in a wide range of subject areas, some of which cannot be obtained elsewhere. AAFC’s contributions helped collaborators obtain greater financial support from other sources, build and expand their organization’s network, and conduct higher quality projects (that is, projects that use more advanced technologies and achieve enhanced research performance).

4.3 Objective of the Collaborative Framework

The Collaboration Policy Framework 2.0 (2015) is a guidance deck that describes the types of R&D collaborations in which AAFC participates, including related processes and the roles and responsibilities of those involved. According to this deck, the Collaborative Framework outlines a process to help AAFC manage its collaborations within a good governance model – that is, a model that ensures projects meet government and departmental priorities, align with Science and Technology Branch strategies and effectively leverage government resources.

As part of the evaluation, interviewed management and staff were asked about the objective of the Collaborative Framework. Several different objectives were identified:

• mobilizing AAFC capacity to address industry’s R&D needs (reported by half of interviewed scientists and the majority of interviewed Directors and Associate Directors of RDT)
• ensuring alignment of collaborative R&D with AAFC roles and priorities (reported by half of other interviewed AAFC management and staff)
• monitoring AAFC’s collaborative R&D projects
• supporting AAFC resource management
• helping AAFC leverage broader stakeholder relationships and work toward common objectives with industry

The lack of integration of the Framework's objective into the overall AAFC portfolio of R&D programming may inhibit consistent understanding of the Framework’s objective, and makes it difficult to determine where the Framework fits within broader Branch and departmental planning and how to report on its achievements. The Collaborative Policy Framework 2.0 deck and other Framework documents do not reflect the evolution of the management of science, collaboration and AAFC’s role in sector R&D.

5.0 Governance

5.1 Good governance model

The Collaborative Framework’s design is intended to follow a “good governance” model by ensuring collaborations meet government and department priorities, align with Science and Technology Branch strategies and effectively leverage government resources. As part of this process, the Framework is intended to support appropriate management of risks (that is, financial, political, legal, environmental and strategic risks) and ensure compliance with departmental policies. In an effort to achieve good governance, the Project Review Committee assesses the following factors for each Framework proposal:

• alignment with federal government roles, departmental priorities and branch objectives;
• project management factors, including technical capacity of the applicant, knowledge and technology transfer planning, anticipated project impacts/benefits and leveraging of federal investment;
• resource and research and development centre requirements for conduct of research; and 
• intellectual property risk.

An executive assistant fulfills the role of the Committee secretary, thereby ensuring the completion of Committee assessment forms. In cases where significant concerns are identified during proposal review, the Committee formally convenes for discussion. As previously mentioned, risk management is further supported by Branch Executive Committee review of projects of high risk or with a collaborator contribution of over $500,000.

The Framework policy 2.0 also suggests good governance involves coordination, responsible use of resources, and operational oversight, as well as efficient execution of and adherence to agreements. It requires clearly defined roles and responsibilities, timely decision-making, regular review of project milestones and budgets, and management of variances in project deliverables. Documents from other federal government departments concur that these practices, along with a detailed approval process and regular reporting standards and performance measurements, are critical to the success of collaborative partnerships.^{Footnote 4Footnote 5}

5.2 Priority and objective alignment

During the proposal review and approval process, the Project Review Committee is tasked with ensuring that each project aligns with the federal government role (that is, addresses market access or generates public benefits) and a minimum of one departmental priority and one Science and Technology Branch objective.^{Footnote 6Footnote 7} Each Project Review Committee evaluation form assesses alignment with scores from zero to 20.

A sample of approved proposals from 2019–21 was reviewed to assess the extent to which Collaborative Framework projects aligned with federal and departmental priorities. The evaluation concluded that most reviewed projects demonstrated alignment. However, given priorities are very broad, this enables most projects to meet alignment criteria but does not ensure projects are advancing specific departmental objectives.

The evaluation found that strategic oversight is lacking within the Collaborative Framework. The proposal review and approval process focuses on singular project approvals, lacking a broader view of all projects within the Framework and AAFC's science portfolio. As discussed in section 3.1, the same finding was noted in the 2010 Office of the Auditor General report prior to the development of the Framework. The Framework also lacks a strategy to guide and enable assessment of resource distribution.

Due to the gaps in strategic oversight within the Framework, there is limited consideration for the extent to which projects address each AAFC priority and Science and Technology Branch objective. The majority (57%) of Collaborative Framework projects funded from 2016–17 to 2020–21 serve to “increase agricultural productivity”, while 14% of projects address each of the following objectives: “address threats to the agriculture and agri-food value chain”, “improve attributes for food and non-food uses” and “enhance environmental performance”. There was a gradual shift in priority coverage from 2017–18 to 2019–20, leading to a decrease in the percentage of projects funded to increase agricultural productivity and a slight increase in projects funded to address other objectives (see Figure 2). The observed trends were not maintained for most objectives in 2020–21, nor was there evidence to suggest the changing priority coverage was an intended outcome.

Source: SMS data, Science and Technology Branch

Description of above image

Figure 2: Collaborative Framework project distribution across Science and Technology Branch objectives

	Percentage of projects addressing objective
	2016–17	2017–18	2018–19	2019–20	2020–21
Address threats to the agriculture and agri-food value chain	9	8	13	24	22
Improve attributes for food an non-food uses	12	12	14	19	12
Enhance environmental performance	14	11	14	16	17
Increase agricultural productivity	64	68	57	39	50
Notes: Some Collaborative Framework projects from 2016–17 to 2019–20 were assigned to a fifth Science and Technology Branch objective titled “Related science activities.” This objective and the corresponding projects are not represented in the graph; therefore, the percentages for 2016–17 to 2019–20 do not add to 100%. The percentages for 2020–21 do not add to 100% due to rounding. The distribution of Collaborative Framework expenditures across objectives cannot be calculated because the project cost data in SMS is not reliable. Source: SMS data, Science and Technology Branch

The percentage of projects that serve to “enhance environmental performance” increased slightly from 14% in 2016–17 to 16% in 2019–20, then to 17% in 2020–21. Climate change and environmental sustainability have been constant federal priorities for several years and have increased significantly in focus for the federal government and AAFC. Given this emphasis, greater support could be given to projects that address climate change and environmental sustainability.

It was expected that the Framework would ensure alignment between industry and AAFC research goals and link to a formalized strategy to guide investment of AAFC resources in collaborative research. In the absence of this strategic approach, the effectiveness of project and resource distribution across key priorities and objectives cannot be assessed. The ability to adjust the current distribution would be dependent on the needs and interests of project funders and collaborators, given the industry-led nature of the current funding model.

5.3 Other considerations for proposal review

Due to the disaggregated nature of the proposal selection and approval process, there is also a risk of duplication of project objectives. This issue was also noted more broadly, in reference to all of AAFC’s collaborative projects, in a 2019 review of the Collaborative Research and Development Agreements process. The implementation of strategic oversight during the proposal selection and approval process would enable reviewers to systematically identify duplicative planned or ongoing projects. Duplication can also be prevented, in part, through less structured mechanisms, such as collaboration or informal discussion between AAFC scientists. Some AAFC scientists have taken measures to avoid unintentional overlap; however, this effort has been limited by their inability to access pertinent information on projects conducted by others within SMS.

Under the current proposal selection and approval process, there is limited consideration of whether the most valuable research opportunities are being pursued, and limited consideration of project distribution across research and development centres, research topics (for example, commodities, practices) and types of funders and collaborators.

Proposals are also approved with limited assessment of resource capacity as it relates to certain infrastructure (for example, equipment, data storage) and the human resources not directly involved in conducting research (for example, local financial officers, commercialization officers). This is likely contributing to the heavy workloads experienced by many of those staff.

The evaluation found there are no Gender-based Analysis Plus considerations defined in the Framework. Science and Technology Branch staff have reviewed data on gender representation across project leads. However, there is no available data on the demographic characteristics of external funders and collaborators, and thus no data on the involvement of underrepresented groups in Framework R&D activities.

Evaluation observations related to governance will be further reviewed by the Office of Audit and Evaluation’s Internal Audit Division in a forthcoming audit engagement.

6.0 Framework implementation

6.1 Coordination and communication

According to the literature reviewed for this evaluation, comprehensive internal and external communication is an important management practice for governments to follow when entering collaborative partnerships. The literature recommends identifying a key communicator for each collaborator group, and communicating in a timely and concise manner with an agreed upon frequency and method.^{Footnote 8Footnote 9Footnote 10} Interviewed funders suggested that a key factor to successful collaboration was to maintain open lines of communication to provide updates and resolve issues. The evaluation found that these practices have been implemented in several Collaborative Framework projects and determined that coordination and communication between AAFC and funders or collaborators is generally effective during the agreement development and project conduct phases.

Minimal communication challenges were identified and were primarily related to disagreements with funders and collaborators about intellectual property control and the extent to which publication should be prioritized. These challenges are acknowledged in the literature on collaborative research. Other potential challenges identified in the literature include divergent goals and interests, and differences of opinion regarding project timeframes and the scope of work and final deliverables.^{Footnote 11Footnote 12Footnote 13} The reviewed articles recommend assessing and discussing these factors prior to developing a collaborative agreement. Additionally, the evaluation found that a few funders experienced challenges due to AAFC’s unique payment schedule.

6.2 Adherence to Collaborative Framework processes

Following recommendations in 2018–19 from a Collaborative Framework working group, corporate support and engagement of research and development centres were increased to improve the consistency of Framework implementation. For example, Partnerships and Planning Directorate staff meet with executive assistants on a monthly basis to ensure adherence to Framework policy and they support the management and timely review of proposals. Despite increased support, evidence from SMS outputs and interviews with AAFC management and staff indicate that coordination processes are not consistently followed within the Framework. Interviewed AAFC management and staff reported several potential reasons for this inconsistency, including unfamiliarity with the process, high workloads, employee turnover, poor compliance, and a lack of knowledge in staff roles and responsibilities or department rules and policies for collaborative research. Process challenges were identified more frequently outside of the prairie region.

Science Management Solution (SMS)

SMS provides a platform for internal development, approval and monitoring of collaborations, and an auditable trail of corporate records. It is intended to serve as a mechanism for reporting to management and should increase accountability. However, the evaluation found that output data is not captured consistently or completely in SMS. Output data is available for only half of the 480 Collaborative Framework projects in SMS initiated between 2016–17 and 2020–21. The remaining half of the projects, particularly those conducted in the earlier part of this timeframe, may have had outputs that were not entered in SMS or that were not linked to the specific Framework project in SMS. In 2018, in an effort to resolve these issues, a Collaborative Framework working group recommended that Directors of RDT prioritize and ensure that good governance requirements are followed and documented within SMS. However, according to some interviewed staff, consistency issues remain.

The evaluation found that inconsistent implementation of SMS may be due, in part, to lack of familiarity with the system, particularly for new scientists and staff or those who do not use the system frequently. Interview evidence suggests experienced AAFC management and staff may also lack awareness or experience challenges with certain standard SMS processes within the Collaborative Framework. The SMS system is updated frequently, making it difficult to provide an up-to-date SMS manual that staff can refer to when they are unsure of the process. The evaluation also found that editing is prohibited during proposal review and following the final stage of the proposal process, unless a change request is submitted. A few interviewed staff suggested that project information may not be appropriately updated in SMS when project changes occur at these times. AAFC staff acknowledge that SMS is a useful tool for reporting to senior management; however, staff remain critical of SMS functionality. In all four case studies, AAFC staff reported challenges with SMS, describing it as “cumbersome” and difficult to navigate.

Project initiation

The evaluation also noted inconsistencies in project initiation. A few interviewed AAFC staff members reported that projects occasionally begin without a formal agreement in place, which leads to implementation challenges, such as the inability for the Office of Intellectual Property and Commercialization to conduct an Intellectual Property risk assessment. These AAFC staff members reported that projects are occasionally approved without scientists’ consideration of the potential commercial value of their research. In such cases, scientists may also be unaware of the actions required to transform their results into products, processes or practices with proper patents and licenses.

Framework resources

Some of the consistency issues described above could be improved using Framework-specific communication resources and training. There are communication resources and training available to all Collaborative Framework users, including:

• presentations on roles and responsibilities;
• internal guidelines for preparing and submitting research proposals;
• process maps; and
• SMS training.

The presentations were developed in response to a recommendation from a Collaborative Framework working group to improve communication of roles and responsibilities, and they were delivered to project leads and reviewers at AAFC’s research and development centres.

However, at the branch level, documents and training information were not disseminated in a systematic manner, and the use and effectiveness of these resources is not tracked. A few interviewed scientists indicated that the Collaborative Framework process is not well-communicated and that it would be useful to have resources available for new scientists to help them understand what is required of them within the Framework.

6.3 Project timeliness

There was consensus across most interviewees that Collaborative Framework projects are generally conducted on schedule. Project delays were primarily related to the following factors: COVID-19, lengthy AAFC staffing processes, untimely submission of information or approval in SMS, negotiation and signing of complex Collaborative Research and Development Agreements, and limited resource capacity. As previously noted, certain resource capacity challenges may be the result of insufficient consideration during the proposal review and approval process. In addition to impacting timelines, insufficient human resources can impact the well-being of staff.

AAFC is compliant with service standards for issuing draft Collaborative Research and Development Agreements, but service standards do not exist for other aspects of the Collaborative Framework proposal process. The lack of service standards was not commonly reported as a reason for project delays; only one staff member suggested that introducing service standards would reduce delays in the signing of Collaborative Research and Development Agreements. The implementation of additional service standards would be challenging for various reasons. For example, service standard adherence could be impacted by internal factors, such as additional review requirements for higher budget projects, or external factors, including competitive call due dates, award announcement dates and timelines set by principal project leadership from different institutions.

The efficiency of Collaborative Framework projects may be affected by certain required reporting processes in SMS. Most AAFC scientists reported that SMS is time consuming and inefficient, due, in part, to the requirement to create two separate project reports: one in SMS for AAFC management and another for funders. Other interviewed AAFC staff, including all AAFC case study interviewees, agreed that SMS is inefficient and presents reporting challenges.

7.0 Economy

There was a consensus among interviewed AAFC scientists that AAFC resources, human and other, are leveraged effectively when conducting Collaborative Framework projects. However, at the branch level, there is limited tracking of the AAFC resources invested in these projects. Therefore, the total cost of conducting Collaborative Framework projects for AAFC is not known. This cost includes the salaries of AAFC’s indeterminate employees (full time or seasonal) that are working on Framework projects, as well as the costs associated with the maintenance and use of facilities, equipment and information technology, and other common research costs.

AAFC does not have a costing model in place to determine the cost of Collaborative Framework management and administration. This information is valuable as it can be compared to total Framework costs; the resulting ratio can then be evaluated against those of similar initiatives.

8.0 Performance

There are no specific indicators or targets set for research carried out under the Framework in the Science and Technology Branch’s broader Foundational Science and Research Performance Information Profile.

Despite the absence of a performance measurement strategy, Collaborative Framework project output data is collected and recorded in SMS. Outputs include:

• scientific publications (for example, articles, book chapters, conference posters or proceedings) and other publications (for example, bulletins/newsletters, trade journal publications, web pages, guides/manuals);
• new or improved products (for example, genetic material, processes/systems, datasets) and licenses (for example, plant varieties); and
• knowledge and expertise contributions (for example, presentations, awards).

However, as previously noted, the evaluation found that output data is not captured consistently or completely in SMS. Therefore, the extent to which knowledge is transferred and innovations are generated under the Collaborative Framework is not fully captured.

8.1 Knowledge transfer to the sector

The knowledge gained from Collaborative Framework projects is transferred to stakeholders using a variety of mechanisms, including peer-reviewed and non-peer-reviewed publications, presentations, conferences, field days, workshops/webinars, meetings with producers or trade organizations, informal networking, funder websites and social media. The SMS data indicates that 726 scientific publications and 134 other publications were produced through Collaborative Framework projects initiated between 2016–17 and 2020–21. Of the scientific publications, 421 publications (including 360 journal articles) were peer-reviewed. Due to the inconsistency and incompleteness of the SMS data, the actual outputs may be higher.

Data has not been collected on the impact of Collaborative Framework publications on the scientific field; therefore, a bibliometric analysis of 80 peer-reviewed articles was conducted as part of the evaluation. The analysis found that:

• 42% of the publications were published in well-known journals (Current Journal Impact Factor greater than 3.0);
• 26% were cited more than the AAFC average for the published year;
• 17% had at least a moderate impact in the field compared to similar publications (Field-Weighted Citation Impact greater than 2.0);^{Footnote 14} and
• 83% of all reviewed articles were considered to be collaborations, with an external institution listed as one or more of the authors’ affiliates.

“Improving attributes for food and non-food uses” was the strategic objective with the highest percentage of publications that had at least a moderate impact in the field (27%) and were cited above the AAFC average (36%). Given the larger percentage of Framework projects dedicated to “Increasing agricultural productivity”, this strategic objective had a larger sample of articles (47 articles), a larger number of impactful articles (7) and a larger number of articles cited above the AAFC average (13) than the other strategic objectives.

There was a consensus among most interviewed external stakeholders and AAFC management, staff and scientists that Collaborative Framework projects were successful in transferring knowledge and increasing the sector's knowledge base. Evidence of knowledge transfer was further reported in the case studies, with two of the three completed case study projects producing peer-reviewed publications and a book chapter.^{Footnote 15} In three of the case study projects funded by producer groups, AAFC scientists reported their findings regularly during grower meetings, and in two of those projects, AAFC scientists presented their findings at academic conferences.

However, a few interviewed funders suggested that there was room for improvement in knowledge transfer, particularly as it relates to understanding the target audience. They emphasized the importance of conveying research findings to producers in an understandable manner (that is, with minimal technical reporting), and using the information channels that are most accessible to them.

Engagement in knowledge transfer varies by scientist, and in some cases, knowledge transfer is facilitated by the funder and/or AAFC extension staff. A few scientists indicated that knowledge transfer could be improved by publicly sharing project findings in laymen’s terms on AAFC’s website and with additional AAFC support. During the early stages of a project, scientists are asked to indicate whether knowledge translation and transfer support will be needed; however, this can be a difficult prediction to make at that time. Publication limitations and trade secrets were noted by a few scientists as occasional barriers to knowledge transfer.

8.2 Innovation development, availability and uptake in the sector

SMS data indicates that 11 new or improved products, processes and technologies were produced through Collaborative Framework projects initiated between 2016–17 and 2020–21. The data also indicates that four inventions (for example, technologies and registered varieties) were disclosed and three licenses were created. Due to the inconsistency and incompleteness of the SMS data, the actual outputs may be higher. The Office of Intellectual Property and Commercialization collects data on invention disclosure and licenses but cannot provide a breakdown by funding source. Therefore, they cannot distinguish Collaborative Framework project outputs from those funded through other sources, such as the AgriScience Program.

The long-term outcomes of Collaborative Framework projects are not known since they are not tracked or documented. The achievement of R&D outcomes can take time and may be influenced by several contributors, thus making it difficult to determine the impact of a particular project. There is value in ensuring that the outputs and outcomes of funded R&D are appropriately measured for assessments of impact, accountability and future funding distribution.

Most interviewees who were involved in completed or near-completed projects provided examples of cases in which Collaborative Framework projects generated new products, processes or practices. Two of the case study projects led to the adoption of new farming practices, while two led to the development of new products or processes. For example:

• One project resulted in farmers from Manitoba, Saskatchewan and Alberta adopting the practice of early seeding of spring wheat. The guidance for this farming practice was incorporated into a wheat production manual from the Alberta Wheat Commission and gained international recognition through the International Wheat Initiative. An invention disclosure was filed by the scientists and AAFC is examining the potential to commercialize a zero till plot seeder for seeding at the appropriate depth for cold soils.
• Another case study project led organic farmers to introduce new crops, such as buckwheat, into their crop rotations for controlling weeds and pests.
• The final completed case study project resulted in the development of a wheat breeder chip that validates DNA markers for multiple agronomic, disease and quality traits by growing plants with those traits. The product has the potential to lower the cost and accelerate the selection of the plant lines wheat breeders use to develop new varieties.

Some interviewed project funders indicated that research carried out through the Framework is often incremental (that is, it may not result in a product, but rather, gradual advances in the sector).

However, perceptions on the availability of products, processes and practices to the sector are mixed. In certain cases, innovations, such as beneficial management practices, are immediately made available and adopted by the sector. In other cases, adoption can take time, as innovations may not easily fit into existing systems, or producers may need more time to learn about and assess an innovation prior to implementation. Interviewed scientists indicated that Collaborative Framework projects provide opportunities to develop relationships with producers and industry that facilitate the adoption of new practices.

9.0 Factors constraining or contributing to framework operation

9.1 COVID-19 pandemic

The COVID-19 pandemic had a considerable impact on the conduct of Collaborative Framework projects. The pandemic led to restricted access to AAFC facilities, cancelled knowledge transfer events, missed opportunities to interact with potential funders and collaborators, travel restrictions and delivery challenges.

According to the majority of interviewed funders, project objectives were also constrained by AAFC’s approach to the resumption of research activities. Some funders indicated that AAFC was slower to develop protocols and resume in-person activities than others (for example, universities), and some suggested that a different approach should have been applied to each AAFC research and development centre, depending on regional circumstances.

Despite these challenges, some funders and collaborators praised AAFC’s efforts to ensure that research continued. For example, adaptation by scientists and agreement amendments contributed to the achievement of outcomes. A few funders specifically referenced the value of AAFC’s regular calls with industry to discuss the impacts of COVID-19 on research programs. Additionally, although the pandemic prevented certain knowledge transfer activities, many presentations were delivered virtually and were therefore more accessible to interested stakeholders.

9.2 Framework accessibility and promotion

Potential funders and collaborators are often made aware of the expertise, capabilities and capacity of AAFC research scientists through funding competitions, networking, presentations, existing relationships with scientists, direct outreach from AAFC scientists and AAFC online communications (for example, AAFC’s Fields of Science webpage, which includes AAFC scientists’ profiles). Certain AAFC scientists, such as those with unique capabilities or expertise, are often invited to participate in R&D projects. However, other AAFC scientists, such as those with less experience or those in regions with limited external funding opportunities, must individually promote their expertise and build relationships with potential funders and collaborators.

Although some promotional strategies have been used by AAFC, more can be done to facilitate connections between AAFC scientists and external stakeholders. Some interviewed AAFC management, staff and scientists suggested that AAFC could create mentorship opportunities in this area for scientists, and more effectively market scientists’ capabilities. For example, AAFC could explore organizing roundtables that enable industry to learn about AAFC scientists, as was already done in an AAFC research and development centre in the Eastern region. Another AAFC research and development centre promotes the expertise of their new scientists by sharing their profiles with existing and potential external collaborators by email, which was deemed a beneficial practice by a collaborator. Improving the transparency and accessibility of information on the Framework by developing a narrative to use when seeking funding and demonstrating the impact of Framework projects is another best practice, along with creating a resource for AAFC scientists that outlines available funding sources and alignment with AAFC priorities.

AAFC’s expertise, capacity and resources make AAFC an attractive collaborator in the sector. AAFC may also have a competitive advantage in funding competitions because it has A-base funded scientists and technicians; therefore, AAFC requires less external funding for scientist and staff salaries, making its services more accessible to external funders.

10.0 Impact for AAFC

The Collaborative Framework confers many benefits to AAFC and its scientists. While establishing criteria and a structured process for the assessment of externally-funded, collaborative research, the Framework maintains flexibility of proposal timing and project length. This flexibility enables AAFC scientists to address immediate research needs and take advantage of long-term opportunities.

Framework projects facilitate the continuity of research by building on existing R&D projects and initiating new research, some of which has led to subsequent R&D projects (for example, those funded through A-base or AgriScience). In all four case studies, the Collaborative Framework projects set the stage for further research. For example, in Prince Edward Island, findings from an organic crop rotation project led to further research on sorghum sudangrass as a biopesticide. The Swift Current research and development centre is considering expanding its research on early planting of spring wheat to barley and other classes of wheat.

AAFC scientists report that participation in Collaborative Framework projects has strengthened their capabilities and expertise and increased the promotion and recognition of their capabilities. SMS data indicates that AAFC scientists were recognized for their work on the Collaborative Framework (for example, through research grants, awards/honours/prizes) 62 times from 2016‑17 to 2020–21. Due to the inconsistency and incompleteness of the SMS data, the actual outputs may be higher.

Project funding has been used to hire students and term technicians, which has enabled the Science and Technology Branch to build larger teams with a more diverse skill set, expand team members’ professional networks, and contribute to the development of the next generation of scientists. These hired students and term technicians have supported other AAFC R&D projects as well.

Additionally, the Collaborative Framework is an important source of funding for AAFC scientists; over half of interviewed scientists and Directors and Associate Directors of RDT indicated that the Framework provides certain AAFC scientists with the funding needed to carry out their research. These interviewees suggest that Collaborative Framework funding is particularly important for new scientists, scientists whose research areas do not match A-base priorities, and scientists in regions with a substantial amount of available industry funding.

11.0 Conclusions and recommendations

The Collaborative Framework facilitates R&D collaboration between AAFC and external stakeholders, thereby increasing industry access to AAFC science capacity and supporting the enhancement and continuity of AAFC research. There are opportunities to better position the Collaborative Framework to respond to AAFC’s collaborative research needs in a manner that aligns with Science and Technology Branch’s other research activities and future direction for the management of science.

Although the Collaborative Framework provides a mechanism for managing individual collaborative projects, it has not been linked to a strategic plan or broader decision-making framework and thus has not achieved good governance as defined in key documents.

The implementation of increased strategic oversight during proposal review could enable projects to respond to research questions in a coordinated manner, prevent unintended concentration of priority coverage, eliminate potential project duplication and improve resource management, thus enhancing the efficiency and effectiveness of AAFC's collaborations. 

The effectiveness of the Framework could be improved by more consistently capturing project data to ensure project outputs are reported accurately in SMS and AAFC investment in the Framework is better understood. The Framework would also benefit from a performance measurement strategy that outlines specific outcomes and impacts of the Framework.

Recommendations

Recommendation 1: The Assistant Deputy Minister, Science and Technology Branch, should incorporate additional considerations into the project selection and approval process to ensure that projects are strategically coordinated, and aligned with Branch objectives and existing and emerging priorities.

Recommendation 2: The Assistant Deputy Minister, Science and Technology Branch, should develop a performance measurement strategy for the Collaborative Framework.

Recommendation 3: The Assistant Deputy Minister, Science and Technology Branch, should implement additional controls to ensure project data is captured consistently and completely.

Management Response and Action Plan

Management from the Science and Technology Branch is supportive of the recommendations and has developed an action plan to address them by April 2023.

Annex A: Evaluation methodology

Document, data and literature review

To assess Framework relevance, design, delivery, economy and performance, the evaluation reviewed internal documents and performance data. The evaluation also examined select literature to support the assessment of relevance.

Assessment on federal government and department priority alignment

The evaluation team was provided with two types of documents to assess the extent to which Collaborative Framework projects aligned with federal and departmental priorities: the Project Review Committee evaluation form and a SMS-generated proposal information report. Documents were provided for 39 projects; however, only funded proposals (that is, 29 projects, 12 from 2020–21 and 17 from 2019–20) were assessed.

The evaluation team reviewed the comments provided by each member of the Project Review Committee, as well as the scores provided by the Committee for Section A: Alignment Review. Section A includes a score of zero to 20 (with zero indicating no alignment and 20 indicating complete alignment) for each of the following criteria:

• Appropriate role for the Government of Canada
• Responds to departmental priorities
• Aligns with Science and Technology Branch objectives
• Builds on synergies (that is, works builds on and complements the expertise of other agencies, is appropriate for AAFC and its partners, addresses capacity limitations and the needs of the sector).

Additionally, the evaluation team reviewed the executive summary in the SMS-generated proposal information report, which included valuable information for determining alignment with federal government and department priorities.

Using these three information sources (Committee alignment scores, Committee comments and project executive summaries), the evaluation team determined whether each project had an excellent, good, weak or very weak amount of evidence demonstrating alignment with federal and departmental priorities.

Bibliometric analysis

Eighty peer-reviewed articles (20 per year from 2016–17 to 2019–20) were randomly selected from SMS for incorporation in the bibliometric analysis, which was carried out in partnership with the Canadian Agriculture Library. The Canadian Agriculture Library provided analysis in three main areas:

• Number of citations generated by the articles, with comparison to other AAFC and other Canadian agriculture publications (using Field Weighted Citation Impact), by subject area (when possible)
• Journal Impact Factor of the journals in which the articles were published, with comparison to other AAFC and other Canadian agriculture publications
• Percentage of articles produced through collaboration (that is, an external institution was listed as one or more of the authors’ affiliations), and identification of top collaborators and their location

Interviews

Interviews were conducted with internal and external stakeholders from various regions to assess Framework relevance, design, delivery, economy and performance. External stakeholder interviews were conducted with eight collaborators and eight funders of Collaborative Framework projects. Interviews were also conducted with eight AAFC management or staff members involved in delivery and oversight of the Collaborative Framework, eight AAFC scientists, five Directors or Associate Directors of RDT, five RDT finance or administration staff, and seven personnel from the Office of Intellectual Property and Commercialization.

The following scale was used to describe the significance of the qualitative findings in terms of the relative proportion of responses:

• All/almost all” – findings reflect the views and opinions of 90% or more of interviewees in the group.
• “Large majority/most” – findings reflect the views and opinions of at least 75% but less than 90% of interviewees in the group.
• “Majority” – findings reflect the views and opinions of at least 51% but less than 75% of interviewees in the group.
• “Half” – findings reflect the views and opinions of 50% of interviewees in the group.
• “Some” – findings reflect the views and opinions of at least 25% but less than 50% of interviewees in the group.
• “A few” – findings reflect the views and opinions of at least two interviewees but less than 25% of interviewees in the group.

Case studies

The case studies focused on four Collaborative Framework projects undertaken between 2014 and 2020. The projects selected for case studies were conducted by AAFC research and development centres in Western, Central and Eastern Canada and funded by a variety of producer groups and private companies.

The chart below provides an overview of the projects. These projects were selected based on budget, durations, regions, sectors, and types of collaborators (for example, industry, government, international, and other stakeholders).

Table 2: Projects selected for case studies

AAFC Research and Development Centre	Funding partner	Research focus	Total project value ($)
Swift Current, Saskatchewan	Western Grains Research Foundation	Advancing the use of genetics in wheat breeding	2,700,000
Lethbridge, Alberta	Alberta Innovates – Bio Solutions	Early planting of spring wheat	1,360,000
Sherbrooke, Quebec	Bayer Animal Health through University of Sherbrooke	Mastitis vaccine for dairy cows	358,000
Charlottetown, Prince Edward Island	PEI Certified Organic Producers Cooperative	Crop rotation for organic produce	213,000
Note: The third project listed (Mastitis vaccine for dairy cows) was cancelled by the funder after AAFC successfully completed its first phase of work.

Each case study included a review of project documents, reporting and publications, along with three interviews, two with AAFC staff and one with a funder or collaborator.

Methodological limitations

The following methodological limitations were considered when interpreting the data.

Limitation	Mitigation strategy	Impact on evaluation
Data limitation: Lack of reliable output and outcome data on knowledge transfer and innovation development, availability and uptake. The Collaborative Framework does not have a performance measurement strategy. Data is not captured consistently or completely in SMS.	To mitigate this challenge, the available data was supplemented with bibliometric analysis; interviews with external stakeholders and AAFC management, staff and scientists; and case studies.	The evaluation was not able to determine the extent to which knowledge was transferred and innovations were developed, available and used. The evaluation identified obstacles and some potential solutions in these areas.
Bibliometric analysis: The bibliometric analysis did not include all peer-reviewed publications from 2016–20 in SMS, nor did it include publications that may not have been captured in SMS.	A sample of 80 publications (20 per year) were randomly selected from SMS. The bibliometric analysis was triangulated with other lines of evidence.	The evaluation was not able to determine to the full extent of the impact that Collaborative Framework publications had on the scientific field or how these publications compare to other AAFC publications.