Evaluation of Agriculture and Agri-Food Canada's Innovation and Adaptation Programs

April 3, 2014

Note: Some of the content is wider than usual.

Canadian Agri-Science Clusters
Developing Innovative Agri-Products
Canadian Agricultural Adaptation Program (National)

Office of Audit and Evaluation


List of Acronyms

AAFC
Agriculture and Agri-Food Canada
ACAAF
Advancing Canadian Agriculture and Agri-Food Program
AIP
Agricultural Innovation Program
CA
Contribution Agreement
CAAP
Canadian Agricultural Adaptation Program
CRDA
Collaborative Research and Development Agreements
DIAP
Developing Innovative Agri-Products
FFPB
Farm Financials Program Branch
FPT
Federal, Provincial and Territorial
GMO's
Genetically Modified Organisms
IRAP
Industrial Research Assistance Program
OAE
Office of Audit and Evaluation
OECD
Organization for Economic Co-operation and Development
PAA
Program Activity Architecture
PB
Programs Branch
PMS
Performance Measurement Strategy
S&T
Science & Technology

Definitions

Vote 1 (Collaborative Research and Development Agreements): A legal agreement which sets out the parameters pursuant to which the parties (collaborators and AAFC) will both invest resources, efforts and share in benefits of the collaborative venture. The collaborator provides cash contributions and AAFC provides a Non-Pay Operating (NPO) financial allocation towards research and development support from AAFC scientists and AAFC's share of the costs incurred by a collaborative initiative.

Vote 10 (Contribution Agreements): A legal funding agreement where the recipient is allocated AAFC funding to conduct research. The recipient provides a financial (cash) contribution and in some cases an in-kind contribution. AAFC provides a funding contribution towards the eligible costs of the project.

Executive Summary

This evaluation examines the relevance and performance of Agriculture and Agri-Food Canada's (AAFC's) Innovation and Adaptation Programs: Canadian Agri-Science Clusters (Clusters), Developing Innovative Agri-Products (DIAP) and the national component of the Canadian Agricultural Adaptation Program (CAAP).

The evaluation was conducted by AAFC's Office of Audit and Evaluation (OAE) in accordance with the Treasury Board Policy, Directives and Standardson Evaluation (2009). The results are intended to inform planning for the next phase of policy and program development.

Background and Profile

Agriculture and Agri-Food Canada (AAFC) has traditionally contributed to sector innovation through its research and development activities, knowledge transfer, and marketing of agricultural and agri-food products, processes and services. Clusters, DIAP and CAAP (National) support innovation and adaptation in the pre-commercialization phase of the innovation continuum. For these innovation and adaptation programs, AAFC has committed $119.96M through different funding envelopes (Vote 1: Operation and Vote 10: Grants and Contributions).

AAFC's innovation and adaptation programs align with AAFC's strategic outcome: An innovative agriculture and agri-food and agri-based products sector.

Evaluation Methodology

The evaluation encompasses three programs: Clusters, DIAP and CAAP (National) and is based on multiple lines of evidence including: a program administrative and operational data file review, a document review, a comparative literature review, key informant interviews, and case studies.

Key Findings

AAFC innovation programs are aligned with federal priorities, contribute directly to departmental strategic outcomes for a competitive and innovative sector, and remain relevant as the sector continues to require government support to address barriers to innovation.

There is a clear federal role to facilitate innovation in the agricultural sector.

AAFC's innovation and adaptation programs are making progress toward achieving outcomes but this requires follow-up evaluations with further analysis and monitoring since sector adoption of innovation is a long-term process taking several years before it is possible to fully assess impacts.

There were some challenges that need to be addressed concerning the design and delivery of the three programs. These include the need for:

  • distinctive guidelines around program objectives and eligibility requirements;
  • streamlining of administrative processes;
  • expansion of communication strategy plans for proponents;
  • improved performance reporting and data management processes; and
  • enhanced coordination and communication between the Science and Technology Branch and the Programs Branch to provide consistent performance reporting and project monitoring systems.

Recommendations

The evaluation includes the following five recommendations:

Recommendation #1:

AAFC's Programs Branch with Science and Technology Branch should work together to:

Provide clear guidelines regarding program objectives, eligibility criteria and requirements for each program to ensure that applicants understand what funding, research and activities each program will support.

Recommendation #2:

AAFC's Programs Branch with Corporate Management Branch and Science and Technology Branch should:

Review funding mechanisms and administrative processes to gain efficiencies and economies where possible.

Recommendation # 3:

AAFC's Programs Branch with Science and Technology Branch should ensure that:

Program recipients have a more comprehensive communications strategy to ensure the dissemination of results to a wider audience taking in consideration any restrictions.

Recommendation # 4:

AAFC's Programs Branch and Science and Technology Branch should ensure that:

Clusters, DIAP and CAAP (and other innovation programs) improve performance reporting and data management relative to AAFC's use of Vote 1 monies and the Recipient's use of Vote 10 contribution funding. Performance reporting information should have comparable performance documents with similar reporting processes and formats. Both operational and performance data should be collected, managed and reported routinely throughout the program cycle rather than at the end to enhance monitoring and comparison of the programs and their individual projects.

Coordination and communications between the Science and Technology Branch and Programs Branch could be improved to enhance AAFC team members' understanding of the components and requirements of the programs to support the dissemination of performance results and data management.

Recommendation # 5:

AAFC's Programs Branch and Science and Technology Branch should ensure that:

The AAFC Expert Panel Review committee that evaluates Clusters, DIAP and CAAP application proposals should also consider economic benefits.

1.0 Introduction

Agriculture and Agri-Food Canada's (AAFC's) Office of Audit and Evaluation (OAE) conducted an evaluation of AAFC's Innovation and Adaptation Programs in accordance with the Treasury Board Policy, Directives and Standards on Evaluation (2009) and the Financial Administrative Act (FAA). The evaluation is part of AAFC's Five-Year Departmental Evaluation Plan. The evaluation results are intended to inform planning for the next phase of policy and program development.

The evaluation objective was to examine the relevance and performance with respect to the Growing Forward agricultural policy framework which expired in March 2013 and A-base funding for CAAP which expires in March 2014. The evaluation includes the Canadian Agri-Science Clusters (Clusters), Developing Innovative Agri-Products (DIAP) and the national component of the Canadian Agricultural Adaptation Program (CAAP). The three programs were evaluated together as they are all designed to function primarily within the pre-commercialization phase of the innovation continuum.

1.1 Program Context

While Canada performs well at science and technology discovery, it has particular challenges related to pre-commercialization efforts and support systems. The private sector often faces difficulty innovating within the pre-commercialization phase due to the necessary funding and support needed before new technologies can be marketed and generate profit. AAFC's innovation and adaptation programs support the agriculture sector at all points along the innovation continuum from basic research and development to commercialization.

The Clusters, DIAP and CAAP programs were part of a suite of initiatives designed to address the gap in the pre-commercialization phase of the innovation continuum. All three programs supported AAFC's Science and Innovation Strategic Outcome. The Clusters and DIAP programs focused on the development of new collaborations with industry to accelerate agri-innovation and the provision of funding for new industry stakeholder projects whose purpose was also to accelerate agri-innovation.Footnote 1 The CAAP program, which included both national and regional components, was designed to enable the sector to adapt by addressing emerging issues and challenges in order for the sector to remain competitive. This evaluation assesses only the national component of CAAP since the regional component ends in 2014.

As with previous adaptation programming, CAAP was industry-led enabling industry to develop and address approaches to resolving their own issues. Adaptation funding has been in effect since 1995 with the introduction of the Canadian Adaptation and Rural Development (CARD) Fund and is intended to act as a catalyst to stimulate change and new and innovative approaches in the agriculture, agri-food, and agri-based products sector. Adaptation programming has always been, and continues to be, driven by the focus on improving the industry's competitiveness. CARD II's (1999-2004) forward looking orientation contributed to the development of comprehensive programming under the Agricultural Policy Framework (APF) from 2003-2008. Following CARD II, the Advancing Canadian Agriculture and Agri-Food (ACAAF) program (2004-2009) was designed to focus on industry-led solutions and capture market opportunities and this contributed to the focus of the policy objectives of Growing Forward.

Clusters and DIAP, which were within the “Growing Canadian Agri-Innovations Program”, were designed to deliver the Growing Forward framework. The Growing Forward framework succeeded the Agricultural Policy Framework (APF). The APF was introduced in 2003 as a five year Federal, Provincial and Territorial (FPT) relationship. It provided funding for the Broker and Agri-Innovation programs aimed at accelerating the identification of new strategic opportunities and the rate of identification, assessment, development and adoption of innovation-based products thereby providing new opportunities for the agriculture sector. Growing Forward federal-only innovation initiatives were aimed at accelerating the pace of innovation and facilitating the adoption of new technologies supporting the competitive and innovative sector outcome.

1.2 Program Overview, Design and Delivery

1.2.1 Canadian Agri-Science Clusters (Clusters)

The purpose of Clusters was to encourage key agricultural organizations to create, plan and implement a national program of applied science and technology research and development activities by mobilizing and coordinating a critical mass of scientific and technical capacity in industry, government and academia.

The Clusters program had a mix of Vote 1 (Operating) support and Vote 10 (Grants and Contributions) funding. This included Contribution Agreements (CAs) and Collaborative Research and Development Agreements (CRDAs). CRDAs were only developed by AAFC at the request of the industry/sector organization leading the Cluster. The CRDAs were to support the Cluster's applied science work plan by engaging AAFC research scientists to conduct Cluster approved activities. Non-repayable contributions of up to $20 million over five years were to be provided to not-for-profit agricultural organizations that led approved national Agri-Science Clusters, pursuant to Vote 10-funded CAs. These organizations could use the funds for research, development and piloting activities through or with Canadian universities, and other R & D organizations to conduct a suite of applied research work. Funding of up to 85% of total eligible project costs was available for national Agri-Science Clusters. Non-repayable one-time contributions of up to $125,000 were made available to Clusters that needed assistance in developing their funding proposals.

Agricultural not-for-profit organizations leading the Agri-Science Clusters were required to establish science advisory bodies. These advisory groups had to be representative of the agricultural industry, academic and government scientific expertise related to the proposed applied science work plan.

1.2.2 Developing Innovative Agri-Products (DIAP)

DIAP intended to address the gap in the innovation continuum between when a new invention leaves the laboratory to when it is ready to enter the marketplace as a commercial product. It was designed to provide Canadian individuals, agri-entrepreneurs, firms and organizations greater access to government, university and other resources required to support successful transformation of innovative ideas to viable business ventures.

The specific objectives of the program included:

  • encouraging value-chain development and collaboration leading to new market opportunities for agricultural products; and
  • providing support to access government, university and other scientific resources that may be required for agricultural organizations and small to medium sized agri-entrepreneurs to address specific issues or opportunities.

The DIAP program had a mix of Vote 1 (Operating) support and Vote 10 (Grants and Contributions) spending. This included funding under Contribution Agreements (CAs) and resourcing and spending under Collaborative Research and Development Agreements (CRDAs). CRDAs were developed at the request of the sector organizations leading DIAP projects to engage AAFC research scientists through conducting applied science, technology development and piloting activities. Non-repayable contributions of up to $2 million were provided under CAs to not-for-profit organizations developing new or expanding opportunities for existing agricultural value-chains directly linked to primary agricultural production in Canada. In addition, non-repayable contributions up to $4 million (including approved funding for value-chain development under this initiative) were provided to for-profit and not-for-profit organizations to implement applied science, technology development and piloting projects that will expand the opportunities for competiveness of the agricultural sector in Canada.

Contribution funding was used to engage applied research, technology development and piloting activities with Canadian universities, and other R&D organizations. Eligible recipients were also able to use the funds to manage and administer approved activities, to develop intellectual property to access commercialization services and to prepare reports and communicate results to their stakeholders. Recipients eligible for funding under DIAP included individuals, universities, Canadian for-profit enterprises and agricultural, food and bio-based product organizations involved in developing agricultural value chains and the pre-commercialization of agri-practices, products and processes.

1.2.3 Canadian Agricultural Adaptation Program (CAAP), National Component:

CAAP was intended to facilitate the agriculture, agri-food, and agri-based products sector's ability to seize opportunities, respond to new and emerging issues and pilot solutions to new and ongoing issues to adapt and remain competitive. This Adaptation program was a successor to the Advancing Canadian Agriculture and Agri-Food (ACAAF) program.Footnote 2

The specific objectives included:

  • Seizing opportunities – to take advantage of a situation or circumstance to develop a new idea, product, niche, or market opportunity to the sector's benefit;
  • Responding to new and emerging issues – to address issues that were not of concern previously or were not known about at all; and
  • Pathfinding and piloting of solutions to new and ongoing issues – to test ways of dealing with new issues or find new ways to deal with existing issues.

The national component of CAAP was delivered only through Vote 10 contribution agreements. AAFC solicited, assessed and managed projects that were national in scope through contribution agreements. The maximum level of total government funding for projects under CAAP could not generally exceed 85 percent. The 15 percent industry contribution could be made up of cash and/or ultimate recipient in-kind contributions. The maximum amount payable to an ultimate recipient for a project was $5 million. Eligible applicants were defined as: any Canadian legal entity capable of entering into a contract including, but not limited to organizations and associations; cooperatives; marketing boards; aboriginal groups; for profit companies and individuals.Footnote 3

The national component applied to Canada-wide industries such as grains and livestock or to issues best addressed nationally (e.g., animal health and welfare). National projects were typically proposed by national organizations.

With respect to the national project approval process, AAFC had an established process under which projects were reviewed by teams of AAFC technical experts.

1.2.4 Delivery of Clusters, DIAP and CAAP

At the time of the evaluation, Clusters, DIAP and CAAP were being managed by the Programs Branch (PB). The Clusters and DIAP program were originally delivered through AAFC's Research Branch, Innovation Programs Directorate. In November 2011, the Agriculture Transformation Programs Directorate, Farm Financials Program Branch (FFPB) took over the delivery of DIAP as part of the harmonization process and administration of Agricultural Innovation Program (AIP). Subsequently in July 2012, Clusters was brought under the former FFPB in the Agriculture Transformation Programs Directorate. The Canadian Agricultural Adaptation Program (CAAP) was previously delivered through the former FFPB.

Scientific support for the management of the Vote 1 allocation was done by AAFC's Research Branch Science Partnership Directorate and more recently by the Cross-Sectoral Directorate in the Science and Technology Branch.

1.3 Program Resources

The initial financial allocation for the three programs was $168.70 million dollars. Specifically, Clusters was allocated $76.45 million over five years (2008-2013), DIAP was allocated $70.45 million over five years (2008-2013) and the national component of CAAP was allocated $21.8 million over five years (2009-2014).Footnote 4

Based on program data, the total commitment for the three programs was $119.96M through different funding envelopes (Vote 1: Operation and Vote 10: Grants and Contributions). The majority of this funding, $67.62M, was used by Clusters, while DIAP used $35.99M and CAAP $16.35M, respectively. The $16.35M in expenditures for CAAP included only Vote 10 funding. Clusters directed $24.48M to Vote 1 and $43.14M to Vote 10. DIAP directed $15.35M to Vote 1 and $20.64M to Vote 10 (See Table 1 below). For the three programs combined the overall variance between allocated and approved funding was about 29%, however some programs had much larger variances than others (Clusters 11.6%, CAAP 25%, DIAP 48.9%). This occurred primarily in the first fiscal year of the programs.

Table 1: Program Resources (in Millions of $) Authority and Approved
Program Vote 1 Authority Vote 1 Approved Vote 10 Authority Vote 10 Approved Total Authority Total Approved Variance in %
Clusters 33.46 24.48 42.99 43.14 76.45 67.62 11.6%
DIAP 42.76 15.35 27.69 20.64 70.45 35.99 48.9%
CAAP (national) 0 0 21.8 16.35 21.8 16.35 25.0%
Total 76.22 39.83 92.48 80.13 168.7 119.96 28.9%
Note: Does not include CAAP Vote 1 Regional and DIAP Vote 1 Amendments.

2.0 Methodology

2.1 Evaluation Approach

The evaluation was conducted by AAFC's Office of Audit and Evaluation (OAE) employing internal and external resources to complete the research and analysis. The evaluation collected and examined both primary and secondary data from multiple lines of evidence to address the evaluation issues and questions.

2.2 Evaluation Scope

As per the Treasury Board Directive on the Evaluation Function (2009), the evaluation examined the program's relevance and performance. Specifically, the evaluation examined: continued need for the program; alignment with government priorities; alignment with federal roles and responsibilities; achievement of intended outcomes and; the extent to which the program demonstrates efficiency and economy.

The evaluation examines the activities of the three programs between the fiscal years of 2008-2009 and 2012-2013. The evaluation is national in scope and includes an analysis of activities funded through Vote 1 (operating) and Vote 10 (grants and contributions) for Clusters and DIAP and Vote 10 for CAAP.

2.3. Data Collection Methods

At the beginning of the evaluation, the OAE held several sessions with members from each program to develop a joint logic model for Clusters, DIAP and CAAP (See Appendix B). The programs' performance measurement strategies and the new logic model helped facilitate the design of an evaluation matrix containing evaluation questions, indicators and appropriate data collection methods.

The evaluation used multiple lines of evidence including:

Program Administrative File Review and Operational Data File Review:
A project administrative file review was conducted for each of the three programs (Clusters, DIAP and CAAP). A checklist and template guide was used to assist in the review of files. The sample of files reviewed was based upon a number of factors including:

  • the characteristics of recipients (i.e. value-chain members or agri-sectors);
  • the project objectives: improved processes, product development, event mitigation, and capacity building; and
  • a range of funding dollar amounts.

Program records including project files, financial files, procedures manuals, and case-specific records were reviewed to assess program outputs for each of the three programs. An analysis of program performance information was used to provide information on overall program achievements.

A total of 28 files were reviewed including: 5 of 10 Clusters, 12 of 41 DIAP and 11 of 25 CAAP files. In addition, a more in-depth file review was conducted on two CAAP files and two DIAP files.

An operational data file review was also undertaken using available data from all three programs.

Document review:
A document review of foundational documents provided background information and context on the design and delivery of the programs and helped to assess questions related to relevance and performance.

Comparative Literature Review:
An extensive online search for similar programs and initiatives in Canada and abroad was undertaken including: the provinces of Saskatchewan and Manitoba, the National Research Council (NRC) of Canada's National Industrial Research Assistance Program (IRAP), Australia, the European Union, United Kingdom, New Zealand and the Netherlands. The review examined program profiles, evaluation reports and publications related to various aspects of research and development programs in agricultural and agri-food sectors. Interviews were conducted with representatives of five similar programs via email and telephone to obtain additional information.

Key Informant Interviews:
Interviews were conducted with a total of 40 key informants. These key informants included AAFC staff members (15) involved in the program design and delivery (the Programs Branch and the Science and Technology Branch), funding recipients of Clusters (8), DIAP (10) and CAAP (6) and one technical expert on agricultural design and delivery methods. Informants were chosen from a cross section of AAFC staff and recipients from the each of the program areas. Key informant interviews were conducted via telephone. Prior to each interview, the questionnaire was emailed to each key informant along with a letter that described the purpose of the evaluation, confidentiality of responses and importance of their participation.

Case Studies:
Two case studies were undertaken to provide comprehensive information and insights on the outcomes of projects funded through the programs. As well, one longitudinal case study of several ACAAF projects that had later evolved into a CAAP project was undertaken. Each of the three case studies included a detailed review of the documents and files associated with projects funded by the program. A cross-section of 17 respondents was interviewed for the case studies including: AAFC scientific researchers (5), university research scientists (2), program recipients (3), project partners (6) and a technical expert (1). Names of case study proponents were obtained through program officers. Case study proponents were then interviewed and asked to provide additional names of individuals involved in the projects such as AAFC researchers, partners, and experts.

2.4. Methodological Limitations

There were limitations to note when examining or interpreting the evaluation results and findings. Some project results were not available since they occurred after most of the evaluation data was collected. Project end dates for Clusters and DIAP tended to be in March 31, 2013 and for CAAP were as late as March 31, 2014. Therefore, the evaluation provides an assessment of immediate and some intermediate outcomes and does not examine end outcomes as it may take several years before it is possible to assess the extent of innovations resulting from program interventions.

3.0 Evaluation Findings

3.1 Relevance

3.1.1. Continued Need

Clusters, DIAP and CAAP address the needs (e.g. research, funding, and collaboration) of the Agriculture and Agri-Food sector within the pre-commercialization phase of the innovation continuum.

The agriculture and agri-food sectors face multiple challenges related to the pre-commercialization phase of innovation enhancing the need for these programs. In its assessment of the drivers of innovation, the Organization for Economic Co-operation and Development (OECD)Footnote 5 and the Department of Finance Footnote 6 emphasize the need for investment in innovation to spur economic performance. According to the OECD study, research and development (R & D) have “positive and significant effects on productivity growth” (OECD, 7).

The agricultural sector continues to face structural barriers to innovation. Various factors affect business commitment and involvement in innovation activities. Four important inputs that are necessary for private sector innovation to occur include: access to capital, existence of talented and educated people, knowledge and new ideas, and collaboration and linkages between businesses, academia and the public sector.Footnote 7 Low levels of collaboration between universities and industry has been noted as one of the competitive disadvantages for the Canadian economy.Footnote 8

Commercialization of innovation requires significant financial resources and is associated with very high risks. According to key informants, most organizations in the agricultural sector in Canada are not in a position to undertake major research and development projects without external support. The research necessary to improve the competitiveness of the agriculture sector requires significant funds and investment over a long period of time. Without support from the government, the agricultural sector does not have either the research capacity or infrastructure to conduct this level of research. Some informants further indicated that government involvement in the research improves the quality and timeliness of the research, makes it more accountable and reduces the risk of biased studies being published. These informants indicated a strong need for the federal government to provide funding to support the pre-commercialization phase of innovation programs.

Key informants indicated that Clusters, DIAP and CAAP are successful in addressing the needs of the agriculture and agri-food sector because they:

  • are industry-led and reflect the highest priority needs of the sector;
  • provide significant programming and funding for research that otherwise would not be funded by industry itself;
  • provide access to expertise only available within AAFC;
  • result in collaboration that ensures a better use of finite resources and accelerates innovation in the agriculture and agri-food sectors; and
  • have helped the sector to increase in maturity and research capacity enabling them to address new challenges and expectations as they emerge.

There is some minor overlap between Clusters, DIAP and CAAP on the AAFC Innovation Continuum. However, the distinctive characteristics of each of the programs' parameters enhance the rationale for having three separate programs.

Clusters, DIAP and CAAP complement each other on AAFC's Innovation Continuum by contributing to the Pre-Commercialization/ Pre-Adoption/Technology and Transfer Phase.

Examination of multiple lines of evidence indicates minor overlap between the activities of the three programs. The programs share the objective of developing new products, improved processes and practices through partnerships and collaborations within the pre-commercialization phase of the innovation continuum. They differ in their design and delivery and the scale and scope of the funded projects. Projects funded through the Clusters program were intended to be national in scope and include multiple research components. Research projects funded through the DIAP were intended to be project and region specific and usually have smaller budgets than Cluster budgets. CAAP industry-led projects were focussed on adaptation initiatives that respond and adapt to new emerging industry issues. CAAP was designed to have a broad scope and a forward looking orientation that would have the flexibility to respond to small or large project proposals through being delivered at the national, regional and multi-regional levels.

Some respondents indicated that it was important to clearly communicate the program objectives, application process and eligibility requirements to potential applicants to avoid any confusion. AAFC staff members reported that they worked closely with the other programs to ensure there was no duplication of research objectives and that the right program for each project was identified.

Clusters, DIAP and CAAP address the needs of the agriculture and agri-food sectors by providing coverage of most sectors through innovation and adaptation funding projects.

There were a total of 67 clients (61 or 91% of clients had 1 proposal approved, 3 clients had 2 proposals approved and 3 clients had 3 proposals approved). Seven organizations had over 50% of all program funding.

The three programs provided funding for a wide range of sectors within agriculture and agri-food. The sectors cover more traditional areas such as Beef, Dairy, Corn and Wheat as well as emerging sectors such as Organic, Sunflower, Herbs and Hemp. The largest amounts of funding dollars across the three programs were directed to Canola/Flax, Pork, Beef, Wheat, Dairy and Pulse, respectively.

For each of the three programs the Crop Production sector had the largest concentration of funding dollars and numbers of projects. Overall, 56 projects and $88 million (73%) of funding went to Crop Production. Sixteen projects and $31 million (26%) funding went to Animal Production and about 4 projects and (1%) funding went to Organic and other projects (See Figure 3 below).

With regard to the Clusters program, there were 5 Crop Production Clusters, 4 Animal Production Clusters and 1 Organic Production Cluster. DIAP had 42 instancesFootnote 9 of sector or commodity groups within Crop Production, 8 within Animal Production, and 1 that was 'other' agricultural activity. CAAP had 19 instances of sector or commodity groups within Crop Production, 7 within Animal Production and 3 that were 'other' agricultural activities. Neither DIAP nor CAAP had projects associated with Organic Production.

Description of this image follows
Desciption - Figure 1

Figure 1 presents the AAFC Cash Contribution Approved Amount (totalling $119,950,947) by the Production Sector. It indicates that Crop Production had the most cash approved with $81,180,728, Animal Production followed second with approximately $31,443,826, Organic Production was third with $6,439,396 and the category of 'Other' had the smallest amount of cash approved with $886,997.

While Crop Production had more projects and funding dollars than Animal or Organic Production, this was not an intentional design of the programs. The programs were not designed to direct more or less funding to any particular sector. The application based nature of the programs and the various capacities of the sectors or commodity groups to submit requests and manage these funds, may have played a role in the distribution of projects and funding dollars. Further analysis by Policy and Program Branches of the production sectors within agriculture and agri-food and their need for innovation and adaption funding is recommended to provide additional information on how funds could be directed to this kind of programming in the future.

3.1.2 Alignment with Government Priorities

The programs' objectives align with the federal government and AAFC's departmental strategic outcomes.

AAFC's Innovation and Adaptation Programs support the departmental strategic outcome: 'an innovative agricultural, agri-food and agri-based product sector' in AAFC's Program Activity Architecture (PAA) and align with Growing Forward policy objectives and A-base funding.

Previous evaluationsFootnote 10 and key publicationsFootnote 11 indicate the importance of the innovation mandate for the Federal Government. Innovation programs are also aligned with the Speech from the Throne (2011) which outlines the government's priority to bring innovative products to market, build on investments in Canada's Economic Action Plan (2012) and bolster its Science and Technology Strategy (2007).Footnote 12

Clusters, DIAP and CAAP support Canada's (2007) Science & Technology (S&T) Strategy which called for a focus on industry competitiveness and productivity and a need to explore new models for science with public-private partnering to increase industry investment and accelerate innovation.

3.1.3 Alignment with Federal Roles and Responsibilities

Clusters, DIAP and CAAP align with federal roles and responsibilities and have helped to support innovation related activities that enhance knowledge, introduce new processes and products and foster collaboration along the value chain.

The agricultural sector continues to face structural barriers to innovation and requires government support to overcome them. Innovation programs are aligned with federal priorities in the Government of Canada's Science and Technology Strategy (2007) which outlines objectives for innovation, including strengthening public-private sector research and commercialization and increasing the impact and efficiency of federal research and development assistance.Footnote 13 According to the Independent Panel on Federal Support for Research and Development (2011) federal government programs should invest resources where market forces are unlikely to operate effectively or efficiently and should address the full range of business innovation activities, including research, development, and commercialization. A special priority should be given to fostering collaboration with key actors in the innovation ecosystem — industries, provinces, postsecondary education institutions, civil society organizations and the relevant investor communities.Footnote 14

3.2. Performance:

3.2.1 Achievement of Expected Outcomes

Clusters, DIAP and CAAP have improved knowledge of potential innovative products, processes or practices and solutions/strategies to address agri-innovation issues and challenges.

The document and file review, case studies, and key informant interviews each provided evidence that Clusters, DIAP and CAAP programs have generally achieved the immediate outcome of improving knowledge of innovative products, processes or practices in agriculture and agri-food. As well, the programs were found to have improved knowledge of solutions/strategies to address agri-innovation issues and challenges. The three programs were making progress toward achieving their expected outcomes:

  • Between 2008 and 2013, Clusters, DIAP and CAAP had 76 approved projects which included hundreds of activities and sub-activities. Across the three programs (76 projects), the average project total funding was about $1.5M comprised of $1M vote 10 and $524K vote 1.
  • Clusters had 10 national Agri-Science Clusters formed with over 200 applied science and technology research and development activities using Vote 1 CRDA support and Vote 10 CA funding.
  • DIAP had 39 recipients of Vote 1 CRDA support and Vote 10 funding with 41 research projects funded and / or supported, each having on average four activities and four sub-activities.
  • CAAP had 25 projects and Vote 10 contribution funding agreements with 24 national organizations.
  • Of the total project spending for the three Programs, about 94% was provided to non-profit corporations over 59 projects and the remaining 6% was provided to profit corporations over 17 projects.
  • The average project spending for non-profit corporations was about $1.9M, while the average project spending for profit corporations was about $405K. All Cluster program projects were non-profit corporations.

The following are examples of how Clusters, DIAP and CAAP produced knowledge of innovative products, processes or practices and provided solutions/strategies for agri-innovation issues and challenges by:

  • Developing an innovative antifungal, antibacterial silver-based compound, “Agress®”, through a Pulse Science Cluster project to help the industry protect crops from bacterial and fungal infestations with low environmental impacts [Clusters];
  • Increasing market opportunities, through expanding information on uses and nutritional and health benefits of products such as mustard, oats, pulses, and maple syrup [DIAP, Clusters];
  • Increasing commodity quality, through the introduction of new varieties and genetic improvement of crops with better yield and higher resistance to herbicides and diseases such as pulse [Clusters], winter wheat, oats, apples, mustard [DIAP] and sunflower seeds [CAAP];
  • Developing and utilizing new disease resistant and pod shatter tolerant seeds to prevent crop loss and increase Canola yield and production [DIAP];
  • Designing and assembling measurement tool prototypes to automate authentication of maple syrup and fault detection of flavour [DIAP];
  • Developing the Vinealert website to provide cold weather alerts and updates to grape growers to help mitigate winter injury [DIAP];
  • Demonstrating the capabilities of natural fibre-based materials, such as flax and hemp, to replace fibreglass in vehicle applications, thus generating commercial opportunities for biomaterials [DIAP];
  • Developing an alternative oilseed that will be a viable choice for marginal lands and rotational cropping with the versatility to be modified for a range of applications and markets [DIAP];
  • Improving methods for crop irrigation systems [DIAP] and storing and shipping various agricultural crops and materials (e.g., apples [DIAP]);
  • Developing better practices to increase commodity production/productivity such as improving feeding practices for dairy cows [DIAP], creating databases to store and utilize information for swine production [CAAP], and new processing techniques and technology for pulses to be efficiently manufactured into flour [Clusters];
  • Creating sustainable production techniques, such as knowledge on rotational benefits, weed and disease management, crop management techniques and nutrient use efficiency [CAAP, DIAP, Clusters];
  • Developing an optimal market structure for barley growers with new market opportunities [CAAP];
  • Improving food quality and safety practices in commercial beef production [Clusters];
  • Developing and implementing a transportation certification program, for livestock (cattle, hog, horse, and sheep) and poultry, to ensure humane transportation of animals to address consumer concerns about animal welfare [CAAP];
  • Developing and utilizing a testing system to identify flax seeds contaminated with genetically modified organisms (GMO's) and reduce the amount of contamination to re-open access to the European markets [CAAP]; and
  • Increasing capacity building and knowledge transfer through benchmarking leadership work with other countries [CAAP].

Consistent with the pre-commercialization phase of the innovation continuum, most (92%) of the three programs' projects tended to be directed at 'process' innovation which entails evaluating new methods or products, such as new or improved ways of cultivating, planting, eliminating pests or breeding or feeding techniques or practices.Footnote 15 While each of the three programs had projects that were moving towards developing new products, the immediate emphasis of projects was improving knowledge around processes and practices that could lead to the development of new products in the future.

CAAP projects differed from Clusters and DIAP in that some of its projects focused on innovation directed at event responses to occasional or unusual events, such as outbreaks or diseases and assisting sectors to adapt through capacity building and knowledge transfer. This is consistent with the stated objectives and expectations of the CAAP program.

An examination of program files from Clusters, DIAP and CAAP indicated that innovation related activities most often tended to be directed at providing knowledge and information. The projects and activities also introduced new/improved processes, practices or products, increased commodity quality or production/productivity, developed new/improved genetic material and reduced disease or environmental damage, hazards or impacts. In a few cases, supplemental activities or by-products of these activities also occurred such as: increasing market opportunities, improving production flexibility, fulfilling regulations or standards, and reducing energy or material consumption to save costs. These activities were consistent with the findings of a study of innovation related activities in research organisations and centres in Australia.Footnote 16

Some detailed examples of projects that demonstrate innovation and adaptation activities that produced knowledge of potential innovative products, processes or practices and provided solutions/strategies to agri-innovation issues and challenges include:

Example 1: Clusters – Canadian Ornamental Horticulture Research and Innovation Cluster

Strategies were developed for more efficient regulation of greenhouse temperatures that could result in energy savings to producers of up to 41%. Technologies were also developed to reduce water consumption in nursery production that cut water usage by as much as 50%. Commercialization of water saving sensor technologies could lead to expansion of the bio-control industry.

Example 2: DIAP – Advancing Canada's Oat Industry through Collaborative Research - Prairie Oat Growers Association (POGA)

The project's genetic research developed two new oat varieties with excellent adaptation to western Canada, and contributed over 100 advanced breeding lines to experiments that created in-depth knowledge about germplasm performance, genotype-by-environment interaction, and molecular determinants of performance. As well, through collaboration between this project and the larger Collaborative Oat Research Enterprise (CORE) group, the largest and most useful set of collaborative oat germplasm was assembled.

The first comprehensive oat linkage map was developed that is resolved to 21 individual oat chromosomes. This revealed regions of similarities between the oat genome and other species such as rice. This new oat map will provide the foundation by which all future oat genomics discoveries can be linked together so that oat researchers have access to discoveries from other species.

Example 3: CAAP - Production of highly marbled Canadian pork by combining new technologies, quantitative selection and feeding - Canadian Centre for Swine Improvement (CCSI)

CCSI developed production of highly marbled Canadian pork by combining new technologies, quantitative selection and feeding methods. Pork marbling is a major quality trait for international markets, has an increasing value for domestic markets, and is the main component of sensory quality. One of the most innovative methods of pork marbling is to predict meat quality in live pigs. Such methods have been explored for many years, but accurate measurement was achieved only recently. The project confirmed that both genetics and feeding practices can influence pork marbling levels and quantified how the combination of the two can significantly raise marbling levels to points where consumer preferences are enhanced. This new method provided an opportunity to improve levels of marbling and quality without changing feeding procedures and diet. The results of this project opened new opportunities for Canadian breeders to select meat quality by using routine, non-invasive measures on live pigs. Canada is the first country to use this approach in a large-scale program. The results of the project were so promising that two breeding companies decided to adopt them and replicate the research immediately.

Clusters, DIAP and CAAP programs generally enhanced collaboration and partnerships within the sector.

An important requirement of the three programs was to enhance collaboration and partnerships of various stakeholders along the value chain – researchers, growers and producers and industry organizations. The three programs produced an estimated 500 partnerships (among industry stakeholders, through contribution agreements (vote 10)) and collaborations (via collaborative research development agreements (vote 1)). Enhancing collaboration and partnerships among stakeholders included leveraging industry participation and funding dollars to bring stakeholders together to work on the same initiatives. A review of the program files indicated that proponents of Clusters and DIAP were required to contribute at least 25% in-cash contribution. It was found that industry contributions tended to range from 15% to 30% while no in-kind contributions were documented. In the case of CAAP, recipients adhered to the requirement for a 15% in cash and/or in-kind contribution.

Findings indicate that the projects created an environment conducive to open communication, which helped to build trust and professional relationships among stakeholders. Industry was able to work on their priorities by partnering with growers and traders, universities and other groups, and by collaborating with such groups and with AAFC scientists. Vote 1 CRDAs provided access to AAFC researchers and research facilities and were felt to increase the quality of the research and reduce the investment risk for activities that were in the beginning stages. Interviews with key informants, case study projects and file reviews verified evidence of beneficial collaborations and partnerships. Some examples of such partnerships or collaborations included:

  • The Pulse Science [Clusters] project was led by representatives of six major universities in Canada, AAFC researchers and representatives of 13 different industry organizations;
  • The Innovative Mustard project [DIAP] was led by industry, growers and traders, four organizations, researchers from AAFC and 3 universities; and
  • The [CAAP] project on Pork Marbling was led by key associations in the Canadian pork industry, three different organizations, 1 university and 1 research centre.

Partnerships, collaborations and information sharing were occurring but a broader communication strategy was not present.

A review of similar programs implemented in Canada and other countries demonstrated that most research and development programs in agriculture and agri-food sectors encourage and facilitate collaborations or partnerships. Collaborations or partnerships are usually encouraged in the application forms and considered in the assessment process as they enhance the potential of projects to deliver planned deliverables. The Clusters and DIAP Programs appear to be unique in that they encourage the use of government facilities and researchers providing an expansive knowledge base that cannot be found elsewhere. However, it was found that with the three programs, partnerships, collaborations and information sharing could be improved between commodity groups, sector groups and value chain members. The three programs do not currently have a great deal of sector and cross-sector information sharing and collaboration within and across the three programs. Some information sharing is occurring on a national and international level within the sectors, but this seems to be limited to a few projects.

Informants felt that the agriculture and agri-food sector and university and AAFC researchers benefit from research collaboration but the impacts of information sharing were difficult to measure. Vote 1 support and Vote 10 funding components of Clusters and DIAP enabled shared funding and collaboration between the agriculture sector, university and AAFC researchers. Generally, respondents felt that this was beneficial. It was difficult however, to determine the extent and reach of the impacts of this collaborative research funding. Not all final performance reports have been received at time of reporting and stakeholders were not asked to comment or report on the impacts of increased research collaboration.

The number of publications, technology transfer and capacity building activities were found to vary across the three programs.

Fifty-eight percent of the total number of respondents interviewed felt that project results could be shared and communicated better. Of the project recipients interviewed, 86% of Clusters respondents, 55% of the DIAP respondents, and 33% of the CAAP respondents stated that results could be shared better. According to key informants, this could occur through developing and implementing a better communication plan at the outset, holding more face to face meetings, incorporating more technology transfer activities, and building better websites.

Evidence from performance data, interviews, case studies and file reviews showed that proponents of the three programs made efforts to disseminate the results of the research and innovation among industry representatives and the scientific community. The most common methods utilized were: progress and/or research reports, conference papers, oral communications at non-scientific events and scientific publications. A number of mediums also included: fact sheets, websites, field days, press releases, magazines, brochures, guides, abstracts, newsletters, poster presentations, books and book chapters, websites, presentations at national and international conferences, key note speaker invitations, steering committees, industry meetings and a few instances involved newspaper articles and radio interviews.

Project proposal plans containing communication plans reviewed as part of the evaluation noted most DIAP project applications contained very little information on how project results were to be disseminated. The project applications reviewed from Clusters contained more detailed communications strategies and the projects tended to result in a greater number of publications, and technology transfer and capacity building activities (see Table 2 and 3 below). CAAP results were not reviewed since they were not due to be released until 2014. While preliminary results indicated that about two-thirds of CAAP informants felt that project results were being shared it is too early support this finding. Performance information for the three programs does not show how successful the projects were in reaching their target audience and in achieving desired outcomes with these communications.

The intention of the programs was to increase collaboration and improve knowledge in Canada. Project applications reviewed made very little mention of sharing, expanding, including or obtaining research knowledge beyond North America. It may be beneficial in the future for international collaborations to be encouraged since agriculture and agri-food research is challenged with a finite number of researchers' available, costly infrastructure and budget constraints. Sharing of research progress and results would likely improve with the building of a network or community of knowledge to include global colleagues utilizing the wide array of online tools available such as digital academic repositories (both institutional and disciplinary), microblogs, multifunctional social networks, Wikipedia or Scribd.

Table 2 Summary of most frequent key activities Clusters*
Activity type Number of activities
Technology Transfer: Oral communication at non-scientific events 336
Technology Transfer: Reports 52
Publications: Scientific Publications 193
Publications: Conference papers 216
Capacity Building: Undergrads 83
Capacity: Summer students 83
Capacity: Master students 59
*Calculations based on five Final Performance Reports
Table 3 Summary of most frequent key activities DIAP*
Activity type Number of activities
Technology Transfer: Oral communication at non-scientific events 119
Technology Transfer: Reports 86
Publications: Scientific Publications 85
Publications: Conference papers 34
Capacity Building: Summer students 83
Capacity Building: Co-op students 19
Capacity Building: Industry professionals 15
*Calculations based on seven Final Performance Reports

Clusters, DIAP and CAAP programs have made progress towards conversion of applied research into commercial products.

Given that Clusters and DIAP projects were completed in March 2013 and CAAP will be completed in March 2014, more time will be needed for the research to be converted into products with commercial value. Key informants often felt that it was too early to tell if the projects have, or will result in the conversion of applied research into commercial products. When key informants were asked to indicate to what extent the three programs resulted in the conversion of applied research into commercial products, the average rating among all respondents was 3.3 out of 5. In general, responses did not vary amongst the three programs. A few respondents stated that they expect their project to be able to convert the research into commercial products within the next 2 to 3 years which is consistent with having a non-repayable contribution agreement.Footnote 17

Examples of research that have been or have the potential to be converted into commercial products include:

  • Genetic research on wheat has resulted in new wheat varieties [Clusters];
  • Research on the nutritional properties of canola oil resulted in the development of a canola based bread product [Clusters];
  • Maple water has been brought to the market [DIAP];
  • Research resulted in a new apple (Salish) [DIAP] and a new cultivar for roses [CAAP]; and
  • Knowledge of the content of waste products and poultry feeding techniques was used to develop new poultry food from waste materials [CAAP].

Three projects provide more detailed evidence of how research is being converted into commercial products:

Example 1: Clusters - Pulse Science Cluster – Pulse Crops (Canada) Association

The Pulse Science Cluster research helped to develop 6 bean varieties and 31 pea varieties, of which 1 bean and 11 pea varieties were officially registered with the Canadian Food Inspection Agency. Commercialization of these varieties has already started and the breeders are going through the 'seed increase' process. Research on the nutritional properties of pulse crops and clinical trials have been completed on a new lentil energy bar called GenkiBar® which was developed to be eaten between exercise activities to restore energy. The energy bar is being carried by retailers and has received some positive endorsements from athletes.

The use of lentils as a binder in meat products was studied since lentils can provide health-conscious consumers with 'non-GMO, gluten-free, low-fat, high-fibre' meat products. The ability to offer and sell value-added ingredients to the meat industry would help expand the market opportunity for the pulse industry. Several large meat processing companies are very interested in this work and are currently carrying out pre-commercialization evaluation trials.

Example 2: DIAP - Innovate Mustards - Mustard 21 Canada Inc.

The Innovative Mustard project produced industrial oil mustard that can be used in generating bio jet fuel. This bio jet fuel was tested in jet engines in 2012. Of the four test flights performed in Canada, three used 50/50 combination of regular aviation fuel and bio fuel and one used 100% bio fuel. The results demonstrated that biofuel produced by mustard has exactly the same characteristics of the regular fuel, and jet engines need no modification to use the fuel. Citing the results of the Innovative Mustard Project, Popular Science magazine listed the 100% biofuel jet flight as one of the “Top 25 scientific events of 2012”. The project negotiated with growers for a large application of industrial mustard across the Prairies as well as the development of a processing facility for extracting oil from the mustard. Currently, all processing has been conducted in laboratories and no commercial processing facility exists in Canada.

A second major development was the introduction of new varieties of yellow and brown mustard which produce a better yield. Over the next five years these varieties are expected to replace existing crops, which should increase mustard production in Canada by 30%. Negotiations with industry partners are complete and growers are ready to start growing the new seeds. The first variety of yellow mustard is expected to be commercialized in 2015 and the second is expected to be commercialized in 2017.

Another project provided a solution for addressing the shelf life of processed meat and other perishable foods through the use of mustard. Mustard can help to increase the shelf life of processed meat and perishable goods by killing off bacteria that contaminate the products. Negotiations are in process with meat producers and packagers and other companies to test the product. If the tests produce expected results, a large application is expected over the next 3 to 5 years.

Example 3: CAAP – Development of an Experimental Method and a Prototype to Transform and Enhance Downgraded Apples into Fruit Sugars– Les Vergers Cataphard et fils Inc.

The objective of this project was to develop a process that would add value to downgraded apples and improve profitability for apple producers since downgraded apples generate little revenue and account for about 40% of production. Fruit sugars are natural ingredients that are sought after in the organic and natural products sectors and could meet the demand of companies that use “product of Canada” or “product of Quebec” labelling when seeking products that are made exclusively in Canada.

This project developed and optimized a membrane filtration process to use downgraded apples by extracting and concentrating the sugars that they contain, without the use of expensive industrial equipment using vacuum principles. A production process was developed that allows the use of non-vacuum equipment, such as the equipment used in maple syrup production, for extracting, separating and concentrating the natural fruit sugars present in apples.

Les Vergers Cataphard et fils now produces a natural sugar syrup consisting predominantly of fructose. The product replaces processed sugars in food formulations and is being used by renowned Quebec chefs while it awaits distribution agreements.

While these examples provide some evidence of intermediate performance results for Clusters, DIAP and CAAP projects, it is too soon to measure longer-term impacts. A longitudinal case study of three ACAAF research projects that evolved into a CAAP project showed how the development, testing and adoption of a methodology for producing highly marbled pork through genetic selection of pigs and new ultrasound technology can result in an improved product for consumers. These multiple projects, however took years (from 2004 to 2013), millions of dollars ($2.4 million in AAFC investments) and significant contributions from the industry to produce innovations in the agriculture and agri-food sector. This provides an example of how some innovation research projects can take a number of years and extensive efforts to produce end outcomes.

3.2.2 Efficiency and Economy

Although thorough and rigorous, the application review process can be a time-consuming exercise.

Key informants were asked to indicate their level of satisfaction with application processing on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is very satisfied, CAAP (2.5) and Clusters' (3.0) recipients had the lowest average satisfaction rating for application processing activities, while the rating for DIAP (3.2) recipients was slightly higher. According to most key informants, the application process is complicated, confusing and time consuming. In particular, of the project proponents interviewed, 58% noted that their application approval was delayed, which resulted in loss of the significant project time (usually first growing season) and delayed the project implementation.

Between the 2008-2014 program period, there were 223 proposals received of which 76 (34%) were approved or completed. Over 61% of proposals were rejected and the remaining 5% were either transferred or withdrawn. CAAP had 25 proposals (26%) approved or completed and the proposal rejection rate was 74%.  DIAP had 41 proposals or (35%) approved or completed and the proposal rejection rate was 65%.  Although no data exists for Clusters, based on discussions with program staff, the rejection rate for Clusters is estimated to be around 29% - much lower than the other programs. Rejection rates are important because high rejection rates reduce program efficiency and economy.  This is because program objectives can only be achieved by approving applications, not by rejecting them. As the proportion of rejected applications increases the proportion of resources required to reject them also increases thereby increasing inefficiency. Taking into account the total project days (15,323) for CAAP from the application received to the decision letter, rejected applications account for 56% of the total project days. Moreover, the average duration from application to decision letter for rejected applications was 134 days. It is likely that significant resources were used to reject applications and the considerable differences in rejection rates between programs merits attention.

Based on the recipient budget expense claims analyzed, it was found that the claim processes were in-line with program expectations.

With the Clusters, DIAP and CAAP programs, claims are payments made to a client (recipient of a contribution agreement) based upon the submission and approval of legitimate expenses. Timely and efficient expense claim processes are an inherent part of AAFC client service standards. The evaluation examined the efficiency and economy of claim processes for Vote 10 funding only.

Clusters and DIAP recipients rated their satisfaction with claim processing as 2.5 and 2.6 respectively on a scale with 3 being somewhat satisfied and 5 being to a great extent satisfied. In contrast, CAAP recipients had a higher average satisfaction rating with 4.6 out of 5. AAFC staff indicated being less than somewhat satisfied with claim processing indicating 2.8 out of 5. According to key informants who provided a lower rating, it took very long time for AAFC to process claims ranging from at least three months to a year, which created financial problems. Some key informants also observed that AAFC changed the eligibility rules for expenses half way through the program implementation, which created administrative and financial problems for project proponents. The process was also noted to require significant paperwork and detailed descriptions of claimed budget items, which was felt by some informants to create an administrative burden.

Based on the recipient expense claims available for analysis, it was found that the claim process durations were in-line with program expectations. Claim processes were fairly efficient since the average number of days to process a claim was 27 days, excluding claims that had additional administrative events. Respondents that gave a higher rating on claims satisfaction indicated that they felt claim processing was very fast and efficient.

The claim process was the same for Clusters, DIAP and CAAP but the supporting documentation differed. The same process was used for large claim amounts as compared to small claim amounts. The average claim amount was $239,438 and the median claim amount was $108,256. Claim amounts varied considerably – from $3,832 to over $3M and Clusters processed fewer claims for greater amounts of money. The materiality (dollar amount) of claims was often not aligned with the claim process duration. For example, in the DIAP program, a claim for $38K was in process for 65 days, where a claim for $1.5M took only 18 days. Such examples are not unique and raise questions about the alignment of administrative processes/resources and project materiality in relation to efficiency and economy.

3.3 Design and Delivery

Design and delivery factors such as funding mechanisms and collaborative approaches were examined to determine their impact on achievement of results. Several factors were considered such as: selection criteria, timing of approvals, program reporting, funding mechanisms and organizational structure.

There are mixed opinions among key informants as to whether there are clearly defined and differentiated selection criteria for each program. While AAFC program staff members consider selection criteria as clear and differentiated, project proponents somewhat disagree.

According to AAFC staff members, the program had rigorous and clearly weighted selection criteria. While some project proponents noted that when they applied for funding, it was very clear which program their proposal fit into based on the program descriptions, others noted that the selection criteria did not seem very clear, and there was no explanation as to why projects (or some components of the projects) were not approved or why specific funding amounts were or were not allocated. Several key informants felt that the scope of the research was very similar across the three programs and that the only difference was the total amount of maximum funding allowed under each program with Clusters receiving the highest, and DIAP and CAAP receiving less, respectively.

Timing around the review and approval of the projects and restrictions on transfer of funds between fiscal years resulted in loss of a significant proportion of the project budget and the first year of the research activities.

By the time a majority of applicants received approval, they had already missed the first growing season and/or the first year of research. The proponents had to return a significant proportion of the project funding back to AAFC as they were not allowed to transfer the unspent budget to the following fiscal year. Consequently, the scope and scale of the some research activities had to be changed and/or proponents had to raise funds from alternative sources. Receiving approvals (e.g., ethics approval) from regulatory bodies (PMRA & Health Canada) to conduct research also took a very long time and slowed the progress of some research projects.

Data management systems, performance reporting and project monitoring could be coordinated and communicated better.

While some data information was available in a readable format, in general data management on program information and results tended to be incomplete and have little supporting documentation. In addition, performance reporting and project monitoring for the three programs did not have identical processes and formats and were difficult to compare.

In most cases each project under DIAP and Clusters managed both Vote 1 (Collaboration Research Development Agreement) and Vote 10 (Contribution Agreement). Some recipients noted that they would prefer to have one funding agreement, thereby reducing the administrative burden however, at the same time they were satisfied with the opportunity to access AAFC scientists and research through Vote 1. As the requirements under Vote 1 and Vote 10 differed, there were different approaches for reporting, filing expense claims and administering the project activities. Under Vote 1, the intention of the program was for industry (the collaborator) to provide cash contributions (at least 25%) however there were exceptions to this percentage amount which were lower. AAFC was intended to provide a Non Pay Operating (NPO) financial allocation of 75% but this amount varied. AAFC scientists provided work plans and progress reports on outcomes to industry who then directed them to the Programs Branch. Project funds were handled through the Science and Technology Branch of AAFC. In contrast, for Vote 10 funding, the recipient (industry) was responsible for submitting progress reports, claim expenditures and other financial information to the Programs Branch.

AAFC scientists involved in Vote 1 research indicated that the reporting required under the program can be extensive and require significant time, which reduces their time and effort in conducting the research. Nonetheless, there is increasing demand to keep track of activities and results to ensure program dollars are being used effectively.

The Science and Technology Branch and the Programs Branch did not appear to have an integrated approach to create and share information on performance results where regular communications ensured that the goals and achievements of the Innovation Programs were understood and managed in a co-ordinated system. Both AAFC scientists and program officers could improve information sharing so that they each can gain broader knowledge of research results and the need for performance reporting. For Clusters and DIAP, it was found that the Science and Technology Branch and the Programs Branch could enhance their communications concerning performance and data reporting related to program operations, administrative processes and scientific and research development.

AAFC organizational structure for the three programs has changed during the evaluation period.

According to AAFC staff members, the organizational structure has improved as a result of recent moves to the Programs Branch because it now has a larger team with financial, policy and claim experts and has more access to resources to deliver the programs. Some noted that the transfer to the Programs Branch has also improved the consistency between programs, improved communication and made program delivery more integrated and increased efficiencies. While these changes appear to be beneficial for the Programs Branch, it is important that the Programs Branch and Science and Technology Branch increase coordination with analysts, scientists and program officers, to ensure they do not operate in silos but create a horizontal approach for their shared programs and initiatives.

4.0 Conclusions and Recommendations

4.1 Conclusions

Relevance

  • There is a need for the federal government to support research and innovation in the pre-commercialization phase of the innovation continuum to maintain the competitiveness of the agriculture and agri-food sector in Canada. The Clusters, DIAP and CAAP programs have been addressing this need.
  • The objectives of the Clusters, DIAP and CAAP National programs each align with federal government roles and responsibilities and are consistent with current government priorities and AAFC strategic outcomes.

Performance

  • Clusters, DIAP and CAAP have improved knowledge of potential innovative products, processes or practices and have improved knowledge of solutions/strategies to address agri-innovation issues and challenges.
  • Clusters, DIAP and CAAP programs were generally successful in enhancing collaboration and partnerships within the sector for the individual programs, but a broader communication strategy is not present. Further work needs to be done to create a more comprehensive communications strategy within the agriculture and agri-food sector.
  • The Clusters, DIAP and CAAP programs have been successful in terms of generating intended outcomes. Early results suggest that the three programs will improve the effectiveness and efficiency of agriculture and agri-food. However, sector adoption of innovation and research results is often a long-term process requiring extensive efforts over many years before results can be converted into commercial products.
  • The extensive review and approval process for projects and restrictions on transfer of funds between fiscal years impacted on timing for research activities.
  • Performance and data management, project monitoring and reporting could be coordinated and communicated better between the Science and Technology Branch and the Programs Branch.

4.2 Recommendations

The evaluation includes the following five recommendations:

Recommendation #1:

AAFC's Programs Branch with Science and Technology Branch should work together to:

Provide clear guidelines regarding program objectives, eligibility criteria and requirements for each program to ensure that applicants understand what funding, research and activities each program will support.

Recommendation #2:

AAFC's Programs Branch with Corporate Management Branch and Science and Technology Branch should:

Review funding mechanisms and administrative processes to gain efficiencies and economies where possible.

Recommendation # 3:

AAFC's Programs Branch with Science and Technology Branch should ensure that:

Program recipients have a more comprehensive communications strategy to ensure the dissemination of results to a wider audience taking in consideration any restrictions.

Recommendation # 4:

AAFC's Programs Branch and Science and Technology Branch should ensure that:

Clusters, DIAP and CAAP (and other innovation programs) improve performance reporting and data management relative to AAFC's use of Vote 1 monies and the Recipient's use of Vote 10 contribution funding. Performance reporting information should have comparable performance documents with similar reporting processes and formats. Both operational and performance data should be collected, managed and reported routinely throughout the program cycle rather than at the end to enhance monitoring and comparison of the programs and their individual projects.

Coordination and communications between the Science and Technology Branch and Programs Branch could be improved to enhance AAFC team members' understanding of the components and requirements of the programs and thereby, support the dissemination of performance results and data management.

Recommendation # 5:

AAFC's Programs Branch and Science and Technology Branch should ensure that:

The AAFC Expert Panel Review committee that evaluates Clusters, DIAP and CAAP application proposals should also consider economic benefits.

Appendix A: Management Response and Action Plan (MRAP)

Recommendation #1:

Provide clear guidelines regarding program objectives, eligibility criteria and requirements for each program to ensure that applicants understand what funding, research and activities each program will support.

Management Response and Action Plan (MRAP) Target Date Responsible Position(s)
Agreed. The program streamlining and harmonization work going on at AAFC has allowed a thorough review of eligible activities and expenses before programs were launched under GF2. This work was done through collaborative efforts between PB and STB. Eligible activities and expenses are described in the applicant guide. Completed DG, Innovation Programs Directorate
PB is currently preparing for the launch of the successor program to CAAP. PB will clarify eligibility criteria and requirements in its communication materials before the end of March 31, 2014 Monday, March 31, 2014 DG, Innovation Programs Directorate

Recommendation #2:

Review funding mechanisms and administrative processes to gain efficiencies and economies where possible.

Management Response and Action Plan (MRAP) Target Date Responsible Position(s)
Agreed. The evaluation made recommendations on funding mechanisms and administrative processes related to claims. PB has made important changes to the claims process under GF2. PB has initiated a review of its processes using the LEAN process. The claims process will be looked at in order to identify possible efficiencies. Efforts will be made to implement next fiscal year. March 31, 2015 DG, Innovation Programs Directorate

Recommendation #3:

Program recipients have a more comprehensive communications strategy to ensure the dissemination of results to a wider audience taking in consideration any restrictions.

Management Response and Action Plan (MRAP) Target Date Responsible Position(s)
Agreed. Under GF2, applicants are asked to provide as part of their applications, not only their research and development work but also the knowledge transfer work they are planning on doing. PB will ask recipients to report annually on their knowledge transfer activities. As part of final project reports, recipients will be asked to report on the overall success of their knowledge transfer activities in terms of reaching their target clientele. AAFC scientists will continue to support collaborators and recipients in their communication strategies by providing reports. Friday, February 28, 2014. DG, Innovation Programs Directorate

Recommendation #4:

Performance reporting information should have comparable performance documents with similar reporting processes and formats. Both operational and performance data should be collected, managed and reported routinely throughout the program cycle rather than at the end to enhance monitoring and comparison of the programs and their individual projects.

Coordination and communications between the Science and Technology Branch and Programs Branch could be improved to enhance AAFC team members' understanding of the components and requirements of the programs and thereby, support the dissemination of performance results and data management.

Management Response and Action Plan (MRAP) Target Date Responsible Position(s)
Agreed. PB is currently working with STB on a common annual performance report template for Agri-Science Clusters and Agri-Science projects. This template follows the harmonized departmental template. A set of standardized performance indicators was developed that applies to both types of projects. This new template will be ready to send out to recipients before the end of March 31, 2014. Monday, March 31, 2014 DG, Innovation Programs Directorate
Coordination and communications between STB and PB is now very good. In addition to the collaboration on proposal review and recommendations, there are regular meetings to develop performance monitoring tools. A joint review of performance reports was established for the final reports for DIAP and the Agricultural Innovation Program. This process will be followed for the review of performance reports from AgriInnovation Stream B. Monday, March 31, 2014 DG, Cross Sectoral Directorate, Science and Technology Branch
For the new CAAP, the reporting format will be similar as it will follow the departmental templates. Both operational and performance data are now being collected. PB reports quarterly on operational data through an operational dash board. The implementation of GCDS in the coming months will facilitate this data collection on a common platform. Monday, March 31, 2014 DG, Innovation Programs Directorate

Recommendation # 5:

The AAFC Expert Panel Review committee that evaluates Clusters, DIAP and CAAP application proposals should also consider economic benefits.

Management Response and Action Plan (MRAP) Target Date Responsible Position(s)
Agreed. For future proposal intakes of Agri-Science Clusters and Agri-Science projects as well as for the new CAAP to be launched in April 2014, the AAFC Expert Review Panel process will include consideration of the potential long-term economic impacts of proposals on the agriculture sector and alternatively, the potential negative effects if the proposal was not funded. For Agri-Science Clusters and Agri-Science projects: pending on timing of next intake.
For new CAAP: April 30, 2014
DG, Innovation Programs Directorate

Appendix B: Evaluation Logic Model for AAFC's Innovation and Adaptation Programs (Clusters, DIAP and CAAP)

Description of this image follows.
Description - Annex B

Objective: Advance knowledge and facilitate the adoption of innovative products, processes or practices to enhance the competitiveness of the agriculture, agri-food and agri-products sector.

Activities: Program Management

  • Review project proposals
  • Develop Contribution Agreements (CAs) and Collaborative Research and Development Agreements (CRDAs)
  • Collect and analyze performance information
    Program Management leads to Outputs: Program Management

Activities: Science and Innovation/Adaptation

  • Develop Science Advisory Body
  • Conduct scientific research and development
  • Test, develop and/or improve innovative agri-products, practices and processes
  • Test/analyse solutions or strategies to address issues or opportunities
  • Technology and knowledge transfer
    Science and Innovation/Adaptation leads to Outputs: Science and Innovation Adaptation

Outputs: Program Management

  • Signed Contribution Agreements and Collaborative Research Development Agreements
  • Documents/ records indicating performance information
  • Claims document and financial records
    Program Management leads to Immediate Outcomes

Outputs: Science and Innovation/Adaptation

  • Results from science and technology research projects and piloting activities
  • Information on innovative agri-products, practices and processes tested, developed and/or improved
  • Information on potential solutions or strategies to address issues or opportunities
  • Document, communicate report on knowledge/results of innovative products, processes r practices

Immediate Outcomes:

  • Agriculture sector has access to improved knowledge of potential innovative products, processes or practices
  • Agriculture sector has access to improved knowledge of solutions/strategies to address agri-innovation issues and challenges
  • Sector enhanced collaboration and partnership
  • The sector shares information with target groups /project stakeholders
    Immediate Outcomes leads to Indeterminate Outcomes

Intermediate Outcomes:

  • Industry implements strategies to respond to emerging issues and opportunities
  • Sector conversion of applied research into commercial products
  • Sector adoption of innovative value-added products, processes, and practices
    Indeterminate Outcomes leads to End Outcomes

End Outcomes:

  • Sector improved competitiveness
  • Sector accelerated pace of innovation through new technologies adopted

Appendix C: References

Arumapperuma, Sudath (2006). Agricultural Innovation System in Australia. Journal of Business Systems, Governance and Ethics.

AAFC (2006) Science and Innovation Strategy. Web. http://www5.agr.gc.ca/resources/prod/doc/sci/cons/strat/pdf/sis-ssi_smmr_e.pdf.

AAFC (2010) The Way Forward: Summary of Agriculture and Agri-Food Canada's Science and Innovation Strategic Action Plan.

AAFC foundational documentation including PMS for CAAP and Clusters/DIAP, Terms and Conditions and other official documents.

Boland, W., Phillips, P., and Ryan, C. (undated) Public-Private Partnerships for the Management of National, Regional and International Innovation Systems.

Council of Canadian Academies (2009) Innovation and Business Strategy: Why Canada Falls Short. Report of the Expert Panel on Business Innovation. Ottawa: CCA.

Conference Board Canada. (2010, March). How Canada Performs: A Report Card on Canada 2009. http://www.conferenceboard.ca/e-Library/abstract.aspx?did=3526.

Council of Canadian Academies. (2009, April). Innovation and Business Strategy: Why Canada Falls Short. Report of the Expert Panel on Business Innovation. http://www.scienceadvice.ca/uploads/eng/assessments%20and%20publications%20and%20news%20releases/inno/(2009-04-29)%20report%20in%20focus%20-%20innovation.pdf

Government of Canada, Department of Finance. (2006). Advantage Canada, “Chapter 4: Investing for Sustainable Growth”. http://www.fin.gc.ca/ec2006/plan/plc4-eng.asp (Paragraph 3).

Government of Canada, Industry Canada. (2011). Innovation Canada: A Call to Action. Review of Federal Support to Research and Development – Expert Panel Report.http://rd-review.ca/eic/site/033.nsf/vwapj/R-D_InnovationCanada_Final-eng.pdf/$FILE/R-D_InnovationCanada_Final-eng.pdf

Government of Canada, Governor General. Here for all Canadians: stability, prosperity, security : Speech from the Throne, June 3, 2011.

Gray, R. and Weseen, S. (2007) The Economic Rational for Public Agricultural Research in Canada

House of Commons (2010) Competitiveness of Canadian Agriculture: Report of the Standing Committee on Agriculture and Agri-food.

OECD. (2009) OECD Work on Innovation – A Stocktaking of Existing Work. STI Working Paper 2009/2. Directorate for Science, Technology and Industry, Paris.

Office of Audit and Evaluation (2011) Meta-Evaluation of AAFC's Innovation Programs: Final Report, AAFC.

Office of Audit and Evaluation (2009) Evaluation of the Advancing Canadian Agriculture and Agri-Food Program. AAFC.

Parsons, M. and Phillips, N. (2007) An Evaluation of the Federal Tax Credit for Scientific Research and Experimental Development. Department of Finance, Working Paper 2007-08.

Phillips, P. (2011) Study: Growing Forward 2 (Science and Innovation), speaking notes. University of Saskatchewan.

Research and Analysis Directorate, Strategic Policy Branch. (Draft) Key performance indicators for innovation in the Canadian agri-food sector. AAFC.

Schwab, Klaus and Porter, Michael E. World Economic Forum. (2009) The Global Competitiveness Report 2008-2009. http://www3.weforum.org/docs/WEF_GlobalCompetitivenessReport_2008-09.pdf.

Science, Technology and Innovation Council. (2008) State of the Nation 2008 - Canada's Science Technology and Innovation System. Retrieved from http://www.stic-csti.ca/eic/site/stic-csti.nsf/eng/h_00011.html Section 4: Digest of Key Indicators, para. 43.

Sparling, D. and Thomspon, S. (2011) Competitiveness of the Canadian Agri-Food Sector. Canadian Agri-Food Policy Institute, Ottawa.

Tannenbaum, B. (2010) Draft Report AAFC Literature and Comparative Review (of innovation programming)