Evaluation of the Collaborative Framework — summary

About the evaluation

  • The Office of Audit and Evaluation of Agriculture and Agri-Food Canada (AAFC) conducted an evaluation of the Collaborative Framework to assess the relevance, design, delivery, economy and performance of the Framework.
  • The evaluation focused on Collaborative Framework activities carried out from 2016–17 to 2020–21; however, case studies assessed longer-term performance by reviewing projects that began prior to 2016–17. The following methods were used: review of documents, data and literature; interviews with AAFC staff and management, AAFC scientists, project funders and project collaborators; bibliometric analysis; and case studies.

What we found

Collaborative Framework summary

  • The Collaborative Framework is an AAFC policy that enables external organizations to collaborate with, or support, AAFC science capacity outside of formal AAFC science programs, such as the AgriScience Program.
  • External organizations provide funding, through cash or in-kind contributions, to support research and development (R&D) projects while leveraging federal in-kind investments (people, resources and/or infrastructure) for research.
  • External funders or collaborators may include provincial funding agencies, producer commissions/associations, universities/ educational institutions and the private sector.

Relevance

  • The Collaborative Framework is aligned with AAFC and government roles and responsibilities, including commitments to engage in science, innovation and collaboration.
  • The Collaborative Framework addresses industry research needs and priorities, enabling funders and collaborators to benefit from AAFC’s extensive network of experts and other tangible R&D resources.

Design and delivery

  • Strategic oversight is lacking within the Collaborative Framework. The project review and approval process focuses on singular project approvals, lacking a broader view of all projects within the Framework and AAFC's science portfolio. The Framework also lacks a strategy to guide and enable assessment of resource distribution. There is potential for gaps in addressing priorities and risk of duplication of project objectives.
  • Despite the availability of communication resources and increased corporate support and engagement of research and development centres, there is inconsistent implementation of Collaborative Framework processes. Output data is not captured consistently or completely.
  • Coordination and communication between AAFC and funders or collaborators is generally effective.

Economy

AAFC’s total investment in the Collaborative Framework is not known because there is limited tracking of the AAFC resources used to conduct Framework projects, as well as the costs of managing and administering the Framework.

Performance

  • The Collaborative Framework does not have a performance measurement strategy.
  • Framework projects have increased the sector’s knowledge base and generated products, processes and practices, but the extent to which this has occurred is not fully captured.
  • Framework operation has been constrained by factors such as inconsistent promotion to external stakeholders and the COVID-19 pandemic. Despite pandemic-related challenges, AAFC was able to adapt in certain situations to ensure research continued. 
  • The Collaborative Framework is an important source of funding for developing AAFC and sector R&D capacity. The Framework facilitates new or existing AAFC research projects and strengthens the capabilities and expertise of AAFC scientists.

Recommendations

Recommendation 1: The Assistant Deputy Minister, Science and Technology Branch, should incorporate additional considerations into the project selection and approval process to ensure that projects are strategically coordinated and aligned with Branch objectives and existing and emerging priorities.

Recommendation 2: The Assistant Deputy Minister, Science and Technology Branch, should develop a performance measurement strategy for the Collaborative Framework.

Recommendation 3: The Assistant Deputy Minister, Science and Technology Branch, should implement additional controls to ensure project data is captured consistently and completely.

Management response

Management agrees with the evaluation recommendations and has developed an action plan to address them by April 2023.