Evaluation of the AgriMarketing Program Stream C: Market Development

March 29, 2018

Abbreviations

AAFC

Agriculture and Agri-Food Canada

GAC

Global Affairs Canada

SME

Small and Medium-sized Enterprises

Executive Summary

Purpose

The evaluation of AgriMarketing Program, Stream C: Market Development was undertaken by Agriculture and Agri-food Canada’s (AAFC) Office of Audit and Evaluation as part of the 2016-17 to 2020-21 Departmental Evaluation Plan. The results of this evaluation are intended to inform future program and policy decisions.

Methodology and Scope

The evaluation assessed program activities and results achieved by the Program from the fiscal years of 2013-14 to 2016-17. To assess the Program's relevance and performance, the evaluation used multiple lines of evidence including document review, program file and database review, interviews, survey and extended file review.

Background

The AgriMarketing Program, Stream C: Market Development (hereafter referred to as Market Development or the Program) is a Growing Forward 2 program aimed at building and promoting Canada's ability to expand domestic and export markets by carrying out promotional activities to help position and differentiate Canadian products and producers ensuring industry's ability to meet market requirements.

Market Development is a five year (2013-14 to 2017-18) Grants and Contributions program delivered through the Business Development and Competitiveness Directorate in the Programs Branch. The Program is industry-led, providing total planned spending of just over $114-million to national non-profit industry organizations and small and medium-sized enterprises (SMEs) for long-term international strategies and market export plans, which include promotional and market development activities in traditional and new markets.

Findings

• Market Development fills a continued need to support the Canadian agriculture and agri-food sector in developing and maintaining domestic and international markets.
• There is an ongoing need for AAFC to support not-for-profit organizations and SMEs to enter and maintain access to export markets, to be able to gain and maintain access to highly competitive global markets.
• Market Development does not duplicate any other existing federal market development programs directed at addressing the needs of the Canadian agriculture and agri-food sector and is aligned with AAFC and federal government priorities.
• Market Development is on track and recipient outputs are consistently surpassing results.
• Market Development is supporting recipient sectors to maintain the visibility of Canadian sectors/products in targeted markets, is largely supporting recipient sectors to maintain or seize market opportunities, contributing to their trade competitiveness in targeted markets, and is on track to improving industry ability to access and respond to market opportunities and demands, enabling the sector to compete domestically and internationally.
• Some performance data and information is being aggregated into a database, but the quality and timeliness of that data limits its value for program performance management and decision-making.
• The Program improved and streamlined some decision-making processes and guidance documents; however, more work is needed to streamline remaining processes and develop tools to enable online access and performance reporting.
• Over the duration of the Program, Market Development has improved, and is now meeting service standards.
• The SME experiment with data linkage provided positive results and demonstrated the potential to further enhance performance data by more effectively measuring program outcomes.

Recommendations

Recommendation 1: The Assistant Deputy Minister, Programs Branch, should improve the cohesiveness of performance data and work with the Information Systems Branch towards implementing a system for managing and reporting on applications, projects, and performance information.

Recommendation 2: The Assistant Deputy Minister, Programs Branch, working with the Assistant Deputy Minister, Strategic Policy Branch, should make collaboration on data linkage for SMEs a regular part of the Program's performance measurement.

1.0 Introduction

The evaluation of the AgriMarketing Program, Stream C: Market Development (hereafter referred to as Market Development or the Program) was undertaken by Agriculture and Agri-Food Canada's (AAFC) Office of Audit and Evaluation as part of the 2016-17 to 2020-21 Departmental Evaluation Plan. Market Development is a Growing Forward 2 program aimed at building and promoting Canada's ability to expand domestic and export markets through undertaking promotional activities to help position and differentiate Canadian products and producers to ensure industry's ability to meet market requirements. The results of this evaluation are intended to inform future program and policy decisions.

2.0 Methodology and Scope

The evaluation was conducted in accordance with the Treasury Board of Canada Policy on Results and the Financial Administration Act, and reports on the core evaluation issues related to relevance and performance, focusing on results achieved by the Program from April 1, 2013 to March 31, 2017.

The evaluation used multiple lines of evidence including document review, program file and database review, interviews, survey and extended file review. The detailed evaluation methodology is in Annex A.

3.0 Market Development Program Background

3.1     Objectives

April 1, 2013 marked the official launch of Growing Forward 2, which is a five-year $3-billion framework that aligns federal, provincial and territorial policies and programming to support the agriculture sector and food industry. This framework includes three federally delivered programs:  AgriInnovation, AgriMarketing and AgriCompetitiveness. The objective of the AgriMarketing program is to enhance the agriculture, agri-food and agri-based products sector in domestic and international markets by supporting industry in gaining and maintaining access to markets and capitalizing on market opportunities. AgriMarketing consists of four streams that combine government initiatives and contribution funding for industry-led projects:

• Stream A: Breaking Down Trade Barriers (evaluated in 2015-16)
• Stream B: Building Market Success (evaluated in 2015-16)
• Stream C: Market Development (evaluated in 2012-13)
• Stream D: Supporting Assurance Systems (evaluated in 2016-17)

Stream C Market Development is a five-year, Grants and Contributions program (2013-14 to 2017-18). Funding is provided to industry to support promotional activities that help position and differentiate Canadian products and producers, and enable industry to meet market requirements.

Market Development consists of two components:

• Generic Component:  Provides funding support to non-profit organizations to develop and implement long-term market development strategies. The component focuses on promotional and market development activities in traditional and new emerging markets.
• Small and Medium-sized Enterprises (SME) Component:  Provides funding support to SMEs to implement international market export plans, including promotional and market development activities. SMEs are defined as legal entities capable of entering into legally binding agreements, with fewer than 250 employees and annual sales not exceeding $50-million.

Promotional projects are cost-shared between AAFC and recipients, generally to a maximum level of total government funding (federal-provincial-territorial) not exceeding 85 percent of eligible costs per Generic component project and 50 percent for SME projects, such as applicants are expected to contribute a specified level of eligible costs for each project.

The AAFC maximum contribution to Market Development's Generic component applicant normally does not exceed $2.5-million per year. Eligible applicants for this component include:

• Non-profit organizations operating on a national or sector-wide basis in the agriculture, agri-food, fish and seafood sectors.
• Non-profit industry organizations operating on a regional basis in the agriculture, agri-food, fish and seafood sectors that represent the majority of production within that sector and can demonstrate their ability to deliver a project from a national perspective.
• Alliances and technical marketing organizations.

Applications could be submitted until September 30, 2017, or until all funding was fully committed, and were considered on individual merit and assessment criteria. Funding was allocated to project applications that demonstrated a clear evolution of strategic industry development and the exploration of new markets.

Participants in Market Development could draw upon the technical and marketing support of AAFC's Agriculture and Food Trade Commissioner Service. The Agriculture and Food Trade Commissioner Service helps Canadian businesses succeed in markets around the world, with Commissioners present in 15 markets, with 35 agents offering services to help Canadian businesses. Agri-Food Trade Commissioners support Canadian agriculture and agri-food exporters by identifying and developing new opportunities for these exporters, ensuring products have access to markets and providing expertise on business problems and relationships. They work collaboratively with Global Affairs Canada (GAC) and the Canadian Food Inspection Agency (CFIA) to resolve market access issues, influence international technical trade-related discussions, policies and safeguards and, where necessary, challenge measures and policies advanced by other countries.

3.2 Logic Model

A logic model was developed by Programs Branch to provide a representation of the Market Development performance goals within the performance measurement strategy. The abbreviated logic model in Table 1 provides a summary of the key expected outputs and outcomes associated with the Program. A detailed logic model is in Annex B.

Table 1: Abbreviated Logic Model

Market Development Abbreviated Program Logic Model

Recipient Outputs	Program Outputs	Immediate Outcomes	Intermediate Outcomes	End Outcomes
Promotional products / events: 1- Media advertisements; media events, promotional events, print advertisements 2- Buyer workshops, buyer site visits, buyer seminars Market development products/ events: 1- Market reports, market studies, market visits for research purposes 2- Export preparedness, seminars, training 3- Industry to industry advocacy, participation in international meetings 4- Trade show participation, outgoing missions, incoming missions	Received applications Approval/rejection letters Contribution agreements prepared for review Financial claims processed Client Satisfaction Index	Recipient sectors are supported to maintain the visibility of Canadian sectors/products in targeted markets. Recipient sectors are supported to maintain and/or increase their capacity to identify and seize market development opportunities in targeted markets.	Recipient sectors are positioned to maintain or seize market opportunities that contribute to their trade competitiveness in targeted markets	Improved industry ability to access markets, respond to market opportunities and demands and compete domestically and internationally
Source: Business Development and Competitiveness Directorate, Programs Branch, 2017.

3.3 Application and Claims Process

The application process is outlined on AAFC's Market Development website, in the application process section. Applications are first reviewed by Program Officers who complete the initial screening. Once applications have successfully gone through the initial screening process and are approved by the Program Manager, they are then assessed by technical reviewers which include representatives from AAFC, the Agriculture and Food Trade Commissioner Service and where applicable, external representatives from departments, such as Fisheries and Oceans or GAC. Upon receiving an assessment from the technical reviewers, the Director General of the Business Development and Competitiveness Directorate is briefed on the recommendation by the Director. Applications received under the SME component are reviewed by a board comprised of Program Officers and Program Managers. Under the Generic component of the Program, once the sign-off has been obtained, the project is presented to a Director General Review Committee.

Recipients must submit supporting claims documentation such as receipts and performance reports to provide verification of expenses incurred throughout the project. Claims information is reviewed and verified for accuracy by the internal officials. The recipient will not receive the final (hold back) funding amount unless the performance report and other relevant documentation is submitted and approved by the Department.

3.4 Resources

Market Development was allocated just over $114-million for total planned spending from April 2013 to March 2018. The allocated budget for Vote 10 contribution funding is just over $106-million with the annual budget being just over $21-million, allocated over five fiscal years. Ten percent of allocated funding is Vote 1 operational funding to support program administration costs.

The actual expenditures, for the fiscal years 2013-14 to 2016-17, are indicated in Table 2. During the evaluation period, Market Development funded 268 projects (100 Generic and 168 SME).The actual expenditures are lower than anticipated in the first two fiscal years and increase in the last two fiscal years of available data.

Table 2: Market Development: Actual Expenditures, 2013-14 to 2017-18 ($)

Market Development: Actual Expenditures, 2013-14 to 2017-18

Fiscal Year	Personnel	Operating	Contribution	Total Actual Expenditure
2013-14	1,852,303	970,540	18,343,297	21,166,140
2014-15	1,924,517	1,251,220	18,894,223	22,069,960
2015-16	2,942,465	2,039,077	21,963,880	26,945,422
2016-17	2,933,251	861,871	25,705,301	29,500,423
2017-18	N/A	N/A	N/A	N/A
Total	9,652,536	5,122,708	84,906,701	99,681,945
Source: Business Development and Competitiveness Directorate, Programs Branch, Program Financial Data to March 2017.

3.5 Governance

Market Development is delivered through the Business Development and Competitiveness Directorate in Programs Branch at AAFC. Project applications for Market Development are internally reviewed by the Sector Development and Analysis Directorate, the Market Access Secretariat, the Agriculture and Food Trade Commissioner Service, and AAFC Regional Offices.

4.0 Program Relevance

4.1 Continued Need

Canada's relatively low population means that it relies on export markets for the continued growth of the agriculture and agri-food sector. Approximately 58 percent of the value of primary agriculture production in Canada is exported, either as primary commodities or as processed food and beverage products. In 2014, Canada was the world's fifth-largest exporter of agriculture and agri-food products after the European Union, the United States, Brazil and China. Canadian export sales grew by 12 percent to $51.5-billion in 2014, increasing Canada's share of total world agriculture and agri-food exports to 3.6 percent.

The need for Canadian agriculture and agri-food export products will continue to grow given that global demand for food is expected to rise significantly by 2050. The Government of Canada is encouraging Canada's agriculture sector to increase its agri-food exports to at least $75-billion annually by 2025 to address this demand and grow the national economy. Market Development aligns with this priority in that it supports the activities to expand the sector.

The agriculture sector in Canada is unique and disparate, being comprised of thousands of producers supplying a wide array of products both domestically and globally. Canadian not-for-profit organizations and SMEs face unique challenges in positioning and differentiating themselves and their products. SMEs, in particular, find it expensive and difficult to sufficiently organize and invest the resources needed to build new markets and increase market demand without government assistance. A lack of private sector investment in the development of market information is a barrier faced by SMEs for the growth of their export markets. This is due to the high costs of gathering information on foreign markets and the potential of information spillover with competitors, which would reduce the benefits of investment. Evaluation evidence supports that Market Development addresses these needs by providing a matching contribution program. This encourages cooperation and investment by industry members across Canada to gain information that is needed to identify and access foreign markets; develop distribution channels; find suitable and reliable suppliers; and deal with local regulations.

4.2 Alignment with Industry Needs and Priorities

Consultations on the Canadian Agricultural Partnership framework acknowledged the value of the AgriMarketing Program to help industry capitalize on market opportunities, gain access to foreign markets, and encourage private investment.

The evaluation found that government support was needed in the development of market intelligence and information; adaptation to foreign requirements for assessments and inspection of exports; responsiveness to potential outcomes of bilateral and multilateral negotiations; education, training and support for new exporters; and, traceability and crisis management. Market Development provided support in many of these areas.

To remain competitive and demonstrate that Canada is open for business, Canadian organizations and industry must maintain an international presence, especially at international trade meetings and events. Market Development activities, such as trade shows and missions, contribute to maintaining Canada's industry presence abroad.

Positive feedback from participants and program oversubscription demonstrate that Market Development is relevant to participants. The evaluation found that participants would not have engaged in the same level of market activities, accessed as many markets or had the same competitive advantage as other countries without the support of the Program. Qualitative evidence and performance data indicated that recipients would not have achieved the same level of success without support from the Program. Several recipients indicated they would not have been able to conduct their projects without the Program; one recipient stated: “We are a small company with a high-end product trying to break into the international market. It simply wouldn't have been feasible for us go abroad without the support of the Program”. There was unanimous agreement among recipients that the Program was highly relevant in addressing their needs.

The evaluation found that Market Development was particularly relevant for smaller companies and associations. Aspects of the Program that most contributed to the success of the Program were the 50-50 cost and risk sharing with AAFC, which encourages companies to engage in new markets and activities. Flexibility of the Program, which enables companies to engage in a wide range of activities, was highlighted as making the Program relevant.

4.3 Comparison to other Federal Programs

Global Affairs Canada (GAC) offers two programs that are comparable to Market Development:  CanExport and Global Opportunities for Associations.

The CanExport program provides funding to SMEs to support market development and export activities. While the CanExport supports similar types of market development activities, such as promoting international business development, its mandate specifically excludes the agriculture and agri-food sector.

The Global Opportunities for Associations program provides support to national associations to engage in market development and export activities. It is industry-led, like Market Development, but the eligibility requirements, dollar thresholds and eligible activities differ. The Global Opportunities program specifically excludes companies or associations that would be eligible for funding through Market Development. Under the GAC programming, applicants applying for funding to engage in activities in Government priority countries are awarded additional points. Comparatively, Market Development projects are funded on merit and whether they support priority markets by employing a technical review process with experts through the AAFC Market and Industry Services Branch and the Agriculture and Food Trade Commissioner Services. Market Development can cost-share projects up to a maximum of $2.5-million which is higher than the $250-thousand ceiling under the Global Opportunities for Associations.

The Market Development program addresses the need within the agriculture and agri-food sector. If it were not in place, sector applicants would be competing with other sectors for limited GAC resources. The evaluation found no significant duplication between Market Development and the GAC programs.

4.4 Alignment with Government and AAFC Priorities

Market Development supports the Government of Canada commitment to “help small and medium enterprises grow, become more innovative and export oriented”. The Program has encouraged and supported Canadian companies to seek access to emerging markets, reinforcing the Government's commitment to “promote trade and investment with emerging markets, including China and India.” The Program is aligned with the Minister's mandate letter which called on AAFC to help “improve economic opportunities and security for Canadians.”

By supporting and encouraging companies to access new markets, Market Development directly supports the achievement of the AAFC priority to "maintain and improve access to key markets."

5.0 Performance

5.1 Outputs

Market Development is on track to achieve its targets and expected results from recipient and program outputs in its performance measurement plan. The detailed Program Performance Plan is in Annex C. With respect to recipient outputs, the annual target for completed events (product launches and demonstrations) was realized each fiscal year from 2013-14 to 2016-17. The annual target for developing market development products (studies, trade shows, export missions and advocacy missions) was attained each fiscal year from 2013-14 to 2016-17.

Table 3: Program Performance Outputs

Program Performance Outputs

Indicator	Annual Target	2013-14 Results	2014-15 Results	2015-16 Results	2016-17 Results
Events: Product launches and demonstrations	750	912	1,158	892	246 Not all data was captured
Market development products: Studies, Trade shows, Export missions Advocacy missions.	1,400	2,305	1,503	1,514	1,452
Source: Business Development and Competitiveness Directorate, Programs Branch, Program Financial Data to March 2017.

5.2 Achievement of Outcomes

The evaluation found that project activities aligned with the criteria prescribed in program documentation and immediate outcomes were being achieved and targets surpassed. Both interviews and evidence obtained through performance data and file review supported the successful achievement of outcomes.

The extended file review provided examples of achievements of outcomes directly attributed to program funding. While examples are below, the full details are in Annex D.

• Groupe Export increased from 50 to 90 exporters in the association and from 25 to 46 companies exporting to Asia. (Generic)
• Canada Organic Trade Association engaged in an international awareness campaign, attending tradeshows, missions and international meetings, which helped it increase international sales by $22-million. (Generic)
• Pet Industry Joint Advisory Council of Canada estimates that 10-15 percent of its annual sales are a direct result of the support of the Program. (Generic)
• Fédération des producteurs acéricoles du Québec credits the Program with doubling its active export markets from two to four. (Generic)
• Tropical Link engaged in meetings and tradeshows in Malaysia and Australia, enabling them to connect with several new buyers, producing new leads in Malaysia that resulted in $300,000 in direct sales. (SME)
• Nutra Canada participated in a major tradeshow in Europe resulting in signing three new agreements with customers with an estimated value of $540,000 in the first year following the event. (SME)

Generic and SME program recipients noted that AAFC funding increased sales and enhanced market opportunities, helping their organizations to be well-positioned to maintain or seize market opportunities, and contributing to their trade competitiveness in targeted markets. Program recipients were very positive regarding the impact of the Program, with one recipient describing Market Development funding as a "life changer". While AAFC officials and Agri-Food Trade Commissioners largely agreed with these achievements and the positive impact of the Program, they also noted the impact of market and economic conditions that could influence the attainment of these goals.

The achievement of longer term outcomes was more difficult to assess as the Program had not reached completion and data regarding long-term impacts were not available. Directly attributing the impact of Market Development on Canada's agricultural exports will be challenging due to the many factors affecting Canadian agricultural exports, such as currency fluctuations, crop production levels, trade barriers, and international competition.

The often unstable characteristics of international markets and economic conditions play a large role in how recipients participate in the Program and in their ability to achieve immediate, intermediate and long-term outcomes. Evaluation evidence suggests that to some extent the Program is improving on industry's ability to respond to market opportunities and demands. Overall, recipients were positive but some indicated challenges with the length of time it took to amend contribution agreements to quickly respond to and seize opportunities. Recipients observed that they did not always fully understand amendment processes, indicating the importance of ensuring that information and advice on amendments is provided in a timely manner to meet this performance outcome.

Achievement of Results:  Generic Component

The Generic component funded a range of projects from several sectors. Nearly two-thirds of the funding was directed to livestock (32 percent) and field crops (31 percent). Recipients generally had projects involved in activities with more than one country.

Funding for Market Development requires that project activities be new activities in traditional and new markets. In the absence of a prescribed definition, Generic recipients defined new or established markets within their own context. While a market could be new for one association, it might be established for another. "New in emerging market" refers to a market that is completely new to the association, whereas "new in established market" refers to a new market that is inside an already established market (for example established in the United States, but not in Montana). The majority of project activities (58 percent) conducted through Generic projects were focused on deepening access inside established markets, rather than engaging in completely new markets, as outlined in Figure 1.

Description of above image

Figure 1 illustrates the distribution of Generic project activities by the type of market they focused on for the years 2013-14 to 2016-17. 58% of project activities focused on established markets, 19% of project activities focused on new markets in emerging markets, 3% of project activities focused on new markets in established markets, 1% of project activities focused on domestic markets, and 18% of project activities did not specify their focus market.

Market Development has the flexibility to respond to changes in market conditions by allowing for new activities in both established and emerging markets. Sustaining an established market can be equally important to growing new markets, as established markets can suddenly be under threat through new competition from other countries or changes in market behaviours or policies.

As performance information on target market countries where activities were occurring was not available in a cohesive, aggregated format, the evaluation findings cannot provide an analysis of specific target market countries for the Generic component.

Achievement of Results:  Small and Medium-sized Enterprises Component

The SME component is progressing towards achieving program outputs and outcomes.

The food and beverage processing sector had 62 percent of SME projects, accounting for over $2-million in funding. Fish and seafood had the second highest number of projects with 15 percent, accounting for approximately $600,000 in funding.

From 2013-14 to 2016-17, just under half (43 percent) of SMEs reported new sales as a result of the program funding and two-thirds (67 percent) reported new leads.

Success varied significantly across sectors, from 32 percent of new sales reported for the food and beverage processing sector to 63 percent for both the field crops sector and the livestock sector. SMEs in the field crops sector had the lowest reporting new leads with 38 percent, while the livestock sector had the highest with 75 percent.

As to the new sales generated due to funding from the Program, the food and beverage processing sector had the highest total sales during the first four years with just over $15-million. More SMEs were funded in the food and beverage processing sector than any other sector, and the distribution of sales was analogous to the funding provided to each sector.

The United States remains Canada's most important agriculture and agri-food export destination, accounting for more than half of Canadian exports. In 2014, Canadian exports to the United States increased by 13.1 percent to $26.5-billion, while exports to other markets grew by 10.1 percent to $25.0-billion. For the SME component of Market Development, activities in the United States became eligible in 2015-2016. The evaluation found that this enabled overall greater success with exports to the United States for program participants. New sales credited to the SME component rose dramatically in 2016-17, up from just over $2-million in 2015-16 to approximately $23-million. The United States was the most significant target of SME activity and approved funding in 2016-17.

Funding for SME activities in 2016-17 increased by $468,369 from the previous year (see Figure 2). The majority of new funding was allocated to activities in the United States, with an annual increase of $446,256. Although SME activities in the United States in 2016-17 ($460,168) received more funding than all other markets combined ($373,846), the increase in funding of activities in the United States does not appear to have come at the expense of funding to other markets, as funding to all other markets increased by $22,113 from 2015-16.

Description of above image

Figure 2 illustrates the distribution of SME project activities in dollars by the regional or national markets they focused on for the years 2013-14 to 2016-17.

• In 2013-14
  • $183,409 was allocated to East Asia
  • $48,884 was allocated to Southeast Asia
  • $22,484 was allocated to Latin America
  • $109,978 was allocated to Europe
  • $33,864 was allocated to the Middle East
  • $10,989 was allocated to miscellaneous markets
  This totalled to an overall $409,610.
• In 2014-15
  • $95,193 was allocated to East Asia
  • $25,250 was allocated to Southeast Asia
  • $33,400 was allocated to Latin America
  • $92,108 was allocated to Europe
  • $21,825 was allocated to the Middle East
  • $23,070 was allocated to miscellaneous markets
  This totalled to an overall $290,846.
• In 2015-16
  • $185,227 was allocated to East Asia
  • $57,834 was allocated to Southeast Asia
  • $0 was allocated to Latin America
  • $67,358 was allocated to Europe
  • $41,314 was allocated to the Middle East
  • $13,912 was allocated to the United States
  This totalled to an overall $290,846.
• In 2016-17
  • $232,174 was allocated to East Asia
  • $0 was allocated to Southeast Asia
  • $12,500 was allocated to Latin America
  • $71,006 was allocated to Europe
  • $32,666 was allocated to the Middle East
  • $58,166 was allocated to miscellaneous markets
  • $460,168 was allocated to the United States
  This totalled to an overall $ 834,014.

Impact of Small and Medium-sized Enterprises Investments over Time

Given the design of Market Development and limiting factors such as international trade rules which impact the type of support the Program may provide, program data can measure outputs but is not always as well suited for measuring outcomes. To examine outcomes more thoroughly for SMEs, a data linkage experiment was conducted by Programs Branch and the Research and Analysis Directorate in the Strategic Policy Branch, in partnership with Statistics Canada. The experiment used data from the SME component of Market Development, including program administrative data (i.e. from application forms, contribution agreements, financial and performance reporting) and linked them with information from Statistics Canada data, such as export registry and pay roll database information.

The experiment results were aggregated to assess the impact of investments over time and learn about SME program participants versus SME non-participants (SME non-funded applicants and SME non-applicants). The experiment indicated that SME program applicants who received funding reported more export sales and had a higher dollar amount of export sales compared to the two groups not receiving funding (SME program applicants who did not receive funding and non-applicants to the Program).

The evaluation cannot draw the conclusion that Market Development SMEs are necessarily more successful at achieving export sales since it is not apparent whether Market Development SME applicants and funded SME applicants are more successful because of the Program and its funding or due to other factors such as organization size. The evaluation found that gathering further information on the variables affecting performance outcomes may enable more extensive conclusions.

6.0     Program Design and Delivery

6.1     Performance Measurement

The 2013 AAFC Evaluation of Market Development recommended improvements to the Market Development performance measurement activities. For Growing Forward 2, Programs Branch implemented the recommendations to ensure more effective processes to gather performance measurement information.

For Growing Forward 2 funding, recipients are required to track the results of their project and report annually on the results. A performance report is required as part of the claim process and the recipient will not receive the final “hold back”, or final payment portion of funding, unless the performance report is submitted and approved by AAFC. These reports are compared to performance targets identified in the approved Project Work Plan.

The evaluation found that performance information could be gathered, documented, and reported in a more cohesive and accessible manner. Recipients appeared to have a lack of clarity as to the different requirements of the Performance Report, Progress Report, and the Project Work Plan documents. The evaluation found that performance information management was not always consistent and that performance information was sometimes missing, in several formats, in multiple electronic locations including AgriDoc and Knowledge Workspace, and in paper files.

Of the twenty files reviewed for the SME component, five files (25 percent) had missing or limited performance information, while of the fourteen Generic component files reviewed, only one file had incomplete performance information.

While many improvements have been made to performance data gathering for Market Development, there is still room for further enhancements. The Access database used to store performance information has limited functionality to track and generate reports. The information in the database is not always current since information has to be manually entered from print performance reports. As a result, comprehensive data is not readily available to enable an assessment of program activities.

Program Officers indicated challenges in obtaining compiled and aggregated performance results. The evaluation found that success stories and lessons learned were collected on an ad hoc basis, were mainly anecdotal, and that there was no systematic collection of lessons learned to enable information sharing and improving program activities.

6.2 Project Approvals and Processes

The evaluation observed changes implemented over the course of the Program to improve its design and delivery, including improving program documentation and processes through continuous improvement exercises such as a LEAN review. The changes addressed recommendations made in the 2015 AAFC Audit of Market Development, to improve and streamline decision-making processes and internal guidance documents.

The application review process was found to have clear and rigorous steps to determine acceptance or rejection. Since the inception of the Growing Forward 2 Program, the processes have been outlined on the AAFC website. The 2015 Audit recommended that the Program update and improve the technical review process to ensure sufficient information is gathered from other organizational areas. A revised technical assessment process has been developed by the Program to address underlying issues. Planned improvements include, among others, the creation of a Technical Officer Committee to enhance the assessments analysis, and the presentation of the assessment form and recommendation to the Director General Review Committee.

The Generic component received 124 applications with 81 percent of applicants approved and funded compared to 314 applicants with 54 percent of applicants approved and funded through the SME program. The higher percentage of rejected and cancelled or withdrawn SME applications suggests that SME applicants may benefit from additional assistance and training on completing the application process.

The initial design of Market Development had a number of delegated steps involved for projects to gain approval, which did not always enable potential recipients to gain approvals for funding in a timely manner, resulting in withdrawal from the Program. In 2013, the Program changed to a more responsive approvals process to ensure participants could move forward more quickly with their project activities. Delegation of authority for project approval was streamlined to gain greater efficiencies. At the start of the Program, projects as small as $1-thousand were subject to Ministerial approval. This changed mid-program to Director General project approval authority for funding that is $500-thousand or less; Associate Deputy Minister approval for $500-thousand to $2‑million and Ministerial approval is required for $2‑million or more. The evaluation found that the change in delegation of authority had a positive impact on project approval times.

Mid-program, internal processes started to be better documented, especially through the development of Program Officer Manuals, to ensure information sharing and to gain greater efficiencies. Recipients confirmed that they clearly understood the application review process and that they found flexibilities in the Program were useful, such as the ability to amend contribution agreements. Recipients indicated that website guidelines and support from Program Officers made the Program reasonably easy to navigate. While there had been some staff turnover, particularly in the SME component, recipients found Program Officers to be invaluable for providing guidance on the Program. The evaluation noted that applicants do not generally receive a rationale for not being granted funding.

The Program is working towards reducing documentation required from recipients and eliminating internal duplication in the claims processes. Specific areas of the application and claims process flagged for improvement through a 2015 LEAN review and recipient feedback included: more focus on outcomes and results of funding instead of unnecessary paperwork such as travel boarding pass receipts; increase clarity and guidance/training on processes for recipients; reduce duplication of internal efforts on verifying claim forms; and create a modernized application and claim system where the Program would move to more electronic records and less paper. The evaluation verified that internal staff and recipients supported implementing these changes.

The evaluation examined two comparative market development programs at GAC (CanExport and Global Opportunities for Associations) to determine if Market Development could derive any lessons or improvements from these programs. GAC program applicants use on-line applications and reporting, allowing them to easily access and track performance data. GAC Trade Commissioners have access to program databases where they input and analyze relevant information. AAFC has no comparable system that supports sharing of information or enabling continuity of service. At AAFC, Agri-Food Trade Commissioners do not have access to Market Development database information on applicants, and cannot easily follow-up on recipient information to ensure continuity and service.

6.3 Service Standards

Service standards for Market Development were being met, directly responding to recommendations presented in the 2015 Audit and the 2013 Evaluation. In examining data from 2014-15 to 2016-17, there was a general improvement each fiscal year in meeting service standards. In 2014-15 both the Generic and SME components were not fully meeting the service standard of a payment within 30 business days of receipt of a duly completed and documented claim. These standards showed improvement over the duration of the Program period so that by 2016-17, results were positive (Table 4).

Table 4: Market Development Service Standard

Market Development Service Standard

Service Standard Met	2014-15 Generic	2014-15 SME	2015-16 Generic	2015-16 SME	2016-17 Generic	2016-17 SME
Our goal is to send you a payment within 30 business days of receipt of a duly completed and documented claim	36%	50%	73%	76%	72%	80%
Source: Business Development and Competitiveness Directorate, Programs Branch, 2017.

7.0 Conclusions and Recommendations

7.1 Market Development Remains Relevant.

Market Development fills a continued need to support the Canadian agriculture and agri-food sector in developing and maintaining domestic and international markets. Agriculture and agri-food exports are vital for the Canadian economy and to supply the global demand for food and agriculture products. There is an ongoing need for AAFC to support not-for-profit organizations and SMEs to enter and maintain access to export markets, to be able to gain and maintain access to highly competitive global markets.

Market Development does not duplicate any other existing federal market development programs directed at addressing the needs of the Canadian agriculture and agri-food sector and is aligned with AAFC and federal government priorities.

7.2 Market Development is Making Progress towards Achieving Outputs and Outcomes

Market Development is progressing towards and often exceeding the achievement of targeted outputs. It has achieved its immediate outcomes by supporting recipient sectors to maintain the visibility of Canadian sectors/products in targeted markets; and, maintain and/or increase capacity to identify and seize market development opportunities in targeted markets.

Market Development is largely achieving its intermediate outcome of positioning recipients to maintain or seize market opportunities that contribute to their trade competitiveness in targeted markets.

Market Development is on track to achieve its long-term outcome of improving industry ability to access markets, responding to market opportunities and demands and competing domestically and internationally.

7.3 Collection and Reporting of Performance Information Could Be Improved

Program information, including performance data, continues to be collected through paper forms rather than by electronic collection methods. Performance data and program information is being aggregated into a repurposed database, but the collection process is resource-intensive and the database functionality is limited, constraining the ability to analyze and share information for timely decision-making. Recipient information could be better shared with Agri-Food Trade Commissioners since they are not provided with ongoing information on the status of approved Program applicants to assist them to support program recipients. While Programs Branch is working towards more efficient data collection methods, progress appears to be limited.

7.4 Service Standards are Improving for Application and Claims Processes

Market Development has improved upon its performance over the course of the program period and is meeting service standards. The steps for project approvals have been streamlined to successfully increase application approval response times, having a positive impact. Response timing for application review and claims processes could continue to be improved particularly on time-sensitive projects with lower materiality.

Recommendation 1: The Assistant Deputy Minister, Programs Branch, should improve the cohesiveness of performance data and work with the Information Systems Branch towards implementing a system for managing and reporting on applications, projects, and performance information.

7.5 Data Linkage Experiments are Providing Information on Outcomes

The SME experiment with data linkage provided positive results and demonstrated the potential to further enhance performance data by more effectively measuring program outcomes. Further work on data linkages may provide more information on the variables affecting performance outcomes and may enable more reliable conclusions as to the effectiveness of the Program, that is, to gain a better understanding of the impacts of the Program on SMEs over time.

Recommendation 2: The Assistant Deputy Minister, Programs Branch, working with the Assistant Deputy Minister, Strategic Policy Branch, should make collaboration on data linkage for SMEs a regular part of the Program's performance measurement.

8.0 Management Response and Action Plan

Evaluation of AgriMarketing Stream C: Market Development Program  (Market Development)

Recommendation	Management Response and Action Plan (MRAP)	Target Date	Responsible Position(s)
Recommendation 1: The Assistant Deputy Minister, Programs Branch, should improve the cohesiveness of performance data and work with the Information Systems Branch towards implementing a system for managing and reporting on applications, projects, and performance information.	Agreed. Under the Canadian Agricultural Partnership, performance measures and indicators for the various eligible activities have been standardized across the program. These performance indicators are collected through the Project Application Form and will be automatically transferred to a database and included in the final version of the Work Plan. Program Branch is collaborating with the Information Systems Branch to finalize the implementation of this database. In addition to enabling the collection and reporting of performance indicators, this database will enable reporting based on variables such as activities, markets, etc., as the results against the targets are entered when the performance reports are submitted by the program recipient.	April 2018	Director General, Business Development and Competitiveness Directorate
Recommendation 2: The Assistant Deputy Minister, Programs Branch, working with the Assistant Deputy Minister, Strategic Policy Branch, should make collaboration on data linkage for SMEs a regular part of the Program's performance measurement.	Agreed. Officials from Program Branch will engage with the Research and Analysis Directorate officials, in the Strategic Policy Branch, to determine how data linkage for SMEs can be expanded and automated. Program Branch officials will integrate this information into the regular SME Program performance reports.    A staged approach will be undertaken to collect SME program data through the new and improved application form as well as information collected through the database currently under development.	April 2020	Director General, Business Development and Competitiveness Directorate

Annex A:  Evaluation Methodology

The evaluation was conducted in accordance with the Treasury Board Policy on Results and Directive on Results.

The evaluation is based on multiple lines of evidence. Where possible, at least two sources were used to generate findings for each evaluation issue. Sources of evidence were as follows:

Document Review: A document review was conducted to determine program relevance and efficiency. This included an examination of foundational documents, previous evaluation and audit reports, AAFC annual publications and program renewal documents. SME and Generic Program Officer manuals were examined to gain information on the design and delivery of the Stream. SME client consultation interview notes were reviewed to supplement interview information on design and delivery and performance issues.

Review of program files and database: Program files (projects) were reviewed to determine the extent to which the Program has made progress towards expected outcomes and the extent to which it has been efficiently administered.

Project file reviews were conducted on 34 project files (14 from the Generic component and 20 from SME component) to determine the extent to which the Program has made progress towards expected outcomes (effectiveness), and the extent to which the Program has been efficiently administered. Project files were selected based on materiality, provincial and sector representation.

Several databases were used as sources of performance information for the evaluation. The evaluation reviewed: Access database information provided by the Program Design and Performance Division which consisted of aggregated data from performance forms (submitted by recipients); a data linkage experimentation excel chart from the Strategic Policy Branch which consisted of compiled results from SME data; and service standards were provided by the Service and Program Excellence Directorate database.

Extended File Review: Program files from each stream (five Generic & three SME) were selected for an extended file review to determine the extent to which outcomes have been achieved. The extended file review included a program file review and interviews with program recipients. Detailed information is provided in Annex D.

Interviews: To support an assessment of program relevance, efficiency and effectiveness, interviews were conducted to determine perceptions related to the continued need for the Program, its progress towards expected outcomes, internal and external factors that have facilitated or hindered progress and program efficiencies.

Fourteen interviews were conducted with AAFC Market Development staff which included Generic Program Officers, SME Program Officers, Operations Officials, and management level personnel (Director and Director General).

Eleven interviews were held with recipients including five Generic and four SME. Recipients selected represented a variety of sectors and included companies and associations based across the country. The distribution was as follows:

• New Brunswick-based national association (1);
• Quebec-based SME (1);
• Quebec-based national association (2);
• Ontario-based SME (3);
• Ontario-based national association (2); and
• British Columbia-based SME (2)

Interviews were conducted with two officials from GAC who are responsible for administering the Global Opportunities for Associations Program and the CanExport Program.

Three interviews were conducted with representatives from AAFC's Market and Industry Services Branch to gain information on market access conditions, comparison of GAC program and application review processes.

Survey: To capture the perspectives of Agri-Food Trade Commissioners, who are located globally, a survey was developed based on the key evaluation questions and administered through email. Eighteen Agri-Food Trade Commissioners were sent the survey and responses were received from thirteen, a 72 percent response rate. Out of thirteen respondents, there were ten Agri-Food Trade Commissioners and three agricultural counsellors.

Methodological Limitations

Methodological limitations were taken into account in interpreting the data.

Methodological Limitations

Limitation	Mitigation Strategy	Impact on Evaluation
During the evaluation period, some performance data was not available for the final year (2017-18).	Performance data was analyzed for all fiscal years other than the final year. Interviews supplemented this data by providing information on projected outcomes.	Limited capacity to fully measure end outcomes for Market Development.

Annex B: Market Development Logic Model

The logic model visually describes the linkages between the Program Alignment Architecture (PAA) and Market Development activities, outputs, immediate outcomes, intermediate outcomes, and end outcomes.

Market Development Logic Model

Objective	The Stream will enhance the competitiveness of the sector by building, promoting and expanding domestic and export markets through its support of industry-led promotional and market development activities aimed at differentiating Canadian products and positioning producers and exports to both maintain existing markets and identifying and seizing opportunities in emerging markets.
Activities	Recipient Activities: 1) Developing promotional initiatives: 1.1- Marketing & Advertising of Canadian sector in global markets 1.2- Delivering educational and technical training to buyers 2) Developing market development initiatives: 2.1- Conducting research on new and existing markets 2.2- Delivering training, preparing and supporting recipient members to export 2.3- Participating in international meetings advancing market access to existing and/or new markets 2.4- Participating or attending trade missions and shows AAFC Activities: 1) Program and project management oversight 1.1- Receiving Applications 1.2- Reviewing and proposing projects/applications for approval or rejection 1.3- Negotiating and preparing contribution agreements for approval 1.4- Processing financial claims
Outputs	Recipient Outputs: 1) Promotional products / events; 1.1- Media advertisements; media events; promotional events; print advertisements; 1.2- Buyer workshops; buyer site visits; buyer seminars 2) Market development products / events: 2.1- Market reports; market studies; market visits for research purposes 2.2- Export preparedness; seminars; training 2.3- Industry to industry advocacy; participation in international meetings 2.4- Trade show participation; outgoing missions; incoming missions AAFC Outputs: 1) Program and project management products 1.1- Received applications 1.2- Approval/rejection letters 1.3- Contribution agreements prepared for review 1.4- Financial claims processed
Immediate Outcomes	Recipient sectors are supported to maintain the visibility of Canadian sectors/products in targeted markets. Recipient sectors are supported to maintain and/or increase their capacity to identify and seize market development opportunities  in targeted markets.
Intermediate Outcomes	Recipient sectors are positioned to maintain or seize market opportunities that contribute to their trade competitiveness in targeted markets.
End Outcomes	Improved industry ability to access markets, respond to market opportunities and demands and compete domestically and internationally.
Link to PAA	1.2 Market Access, Negotiations, and Sector Competitiveness (PAA 2014-15)
AAFC Strategic Outcome	A competitive and market-oriented agriculture, agri-food and agri-based products sector that proactively manages risk
Source: AAFC, AgriMarketing Stream C: Market Development, Program Performance and Risk Management Strategy (PPRMS), September 2015.

Annex C:  Program Performance Plan

Program Performance Measurement Plan - Recipient Outputs

Indicator	Target	Date to Achieve Target	2013-14 Results	2014-15 Results	2015-16 Results	2016-17 Results	Remarks
Completed events (product launches & demonstrations)	750	Annually	912	1,158	892	246	Target surpassed in first three years. 2016-17. Not all 2016-17 data was captured.
Market development products, for example: Completed studies; Completed trade shows; Completed export missions; and Completed advocacy missions.	1,400	Annually	2,305	1,503	1,514	1,452	Target surpassed in all years
Source: Program Design and Performance Division, Business Development and Competitiveness Directorate, Programs Branch, November 30, 2017.

Program Performance Measurement Plan - Immediate Outcomes

Expected Results	Indicator	Target	Date to Achieve Target	2013-14 Results	2014-15 Results	2015-16 Results	2016-17 Results	Remarks
Recipient sectors are supported to maintain the visibility of Canadian sectors/products in targeted markets.	Number and type of participants at educational and training events.	3,000	March 31, 2018	617 buyers trained: 240 field crops, 377 livestock	3,074 buyers trained: 1,270 field crops, 1,804 livestock	20,294 buyers trained: 17,149 field crops, 5906 livestock, 204 fairs and exhibits.	31,063 buyers trained: 18,123 field crops, 15,701 livestock, 204 fairs and exhibits	Target surpassed
Recipient sectors are supported to maintain the visibility of Canadian sectors/products in targeted markets.	Number of consumers and/or demographics exposed to promotional material (websites hits, brochures, product tasting, information kits, etc.)	62,500,000	March 31, 2018	12,507,955	55,904,871	440 million	756 million	Target surpassed “Impressions/ distribution/reach” of social media or articles (print or online) are multiplied by 0,022, a conservative measure of who might actually be seeing the ad or article. One association is not captured in 2016-17.
Recipient sectors are supported to maintain and/or increase their capacity to identify and seize market development opportunities in targeted markets.	Number of new leads/sales/opportunities in markets as a result of activities	12,500 leads $3.25B in sales	March 31, 2018	2,603 Sales $674M	8,742 Sales $1.5B	18,619 Sales: $1.95B	33,365	Leads: Target surpassed Sales: Not all associations report this. Some associations may deem this information on their members confidential. Those who do report actual sales do so based on surveys of their recipients, though most quote export figures. The work needed to separate these two is not yet complete for 2015-16 and not started for 2016-17.
Recipient sectors are supported to maintain and/or increase their capacity to identify and seize market development opportunities in targeted markets.	Number of distribution agreements/contract signed	2,250	March 31, 2018	579	1,148	1,378	1,692	Not all associations report this. Some associations may deem this information on their members confidential. Target may be met. The target is based on the first year of data. However, as the nature of the Market Development activities shift annually based on market conditions, basing our target on an average of 3 years may be more representative of market activity.
Recipient sectors are supported to maintain and/or increase their capacity to identify and seize market development opportunities in targeted markets.	Number and type of participant members  trained, prepared, supported to export	7,500	March 31, 2018	1,698 (of which 1,351 field crops, 143 livestock, 204 food processing	2,076 (of which 1504 field crops,  143  livestock, and 429 food processing)	2,711 (of which 1750 field crops, 286 livestock, 675 food processing)	3,285 (2,188 field crops, 286 livestock, 811 food processing	Target may be met. The target is based on the first year of data. However, as the nature of the Market Development activities shift annually based on market conditions, basing our target on an average of 3 years may be more representative of market activity.
Source: Program Design and Performance Division, Business Development and Competitiveness Directorate, Programs Branch, November 30, 2017.

Annex D:  Extended Project File Review

Generic Projects

SME Projects