Evaluation of the Sector Engagement and Development Program

March 24, 2016

Report:
Office of Audit and Evaluation

List of acronyms

AAFC

Agriculture and Agri-Food Canada

CFIA

Canadian Food Inspection Agency

F/P/T

Federal/Provincial/Territorial

GF

Growing Forward

GF2

Growing Forward 2

MISB

Market and Industry Services Branch

OAE

Office of Audit and Evaluation

OECD

Organisation for Economic Co-operation and Development

P/T

Provincial/Territorial

PAA

Program Alignment Architecture

PMS

Performance Measurement Strategy

ROD

Regional Operations Directorate

SDAD

Sector Development and Analysis Directorate

SED

Sector Engagement and Development

STB

Science and Technology Branch

TME

Trade and Market Expansion

Executive summary

Introduction

Sector Engagement and Development (SED) is one sub-program within the Market Access, Negotiations, and Sector Competitiveness Program. SED includes most of the activities of the Sector Development and Analysis Directorate (SDAD) and the Regional Operations Directorate (ROD), both of which are part of the Market and Industry Services Branch (MISB). SED’s mandate is threefold: gathering and analyzing data pertaining to agriculture and agri-food markets and industries; maintaining relationships with industry and providing analysis of industry competitiveness; and, promoting sector interests.

The evaluation of SED was conducted between November 2014 and May 2015. This evaluation, which is the first of the program, complies with the requirements of the Treasury Board Policy on Evaluation (2009). In accordance with this policy, the evaluation addresses the relevance of SED, the achievement of its outcomes, its efficiency and economy, and design and delivery of SED. The evaluation covers the work of SED between fiscal years 2009-10 and 2013-14.

Methodology

The evaluation methodology consisted of a document and administrative review; literature and media coverage review; interviews with representatives of SED and of other areas of Agriculture and Agri-Food Canada (AAFC), and representatives of other federal departments, provinces and industry stakeholders; and case studies. Triangulation was used to verify and validate the findings obtained through these methods and to arrive at the overall evaluation findings.

Findings

Relevance

The evaluation found that there is an ongoing need within the sector for reliable information and analysis as well as for the facilitation of connections and engagement among stakeholders in support of industry competitiveness. SED addresses these needs, and the need for SED is, in fact, likely to grow based on trends projecting increases in technical trade barriers, and the continued emergence of new technology-based industries within the sector.

In Throne Speeches and Budgets that have occurred during the evaluation period, the Government emphasized the important role trade plays, including trade in the agriculture and agri-food sector, as an engine for economic growth. SED directly supports the sector’s economic growth by providing a wide-range of services, products and support, such as market analyses, relationship building with sector stakeholders and resolution of issues. As such, over the evaluation period, SED continued to meet a need, and was well aligned with government and departmental priorities.

The basis for the federal role in the agriculture and agri-food sector is found in the Department of Agriculture and Agri-Food Act which states that the powers, duties and functions of the Minister of Agriculture and Agri-Food include matters relating to agriculture; products derived from agriculture; and related research.

Achievement of expected outcomes

The evaluation found that SED is generating all of its expected outputs under each of the five major categories identified in the logic model (Market information analysis and intelligence; Industry analytical and technical reports; Value Chain Roundtable, International Market Engagement Team and other industry-government or Federal/Provincial/Territorial (F/P/T) engagement processes; Technical advice and services to the sector and to governments; and, Analysis of Provincial/Territorial (P/T) governments’ positions and activities). Over the past five years, SED has generated market information, analysis and intelligence; industry analytical and technical reports; Value Chain Roundtables, International Market Engagement Teams and other engagement processes; technical advice to industry and governments; and analysis of P/T government positions and activities.

Through the provision of its services, SED has met all three immediate outcomes: Canadian producers, companies, and industry organizations are better informed; the sector has the capacity and exercises leadership to enhance competitiveness and market performance within key domestic and international markets; and, government programs, policies, and regulations respond to sector priorities and competitiveness needs.

By ensuring better informed sector stakeholders, an enhanced leadership, and sound policy and regulation making, SED has been able to meet all three intermediate outcomes, namely, Canadian producers, companies and industry organizations make better decisions; Canadian agriculture and agri-food sector addresses priority issues, including trade-related issues to advance competitiveness; and, stakeholders implement strategies and use tools to manage changes associated with external forces.

Multiple lines of evidence suggest that SED has contributed to its end outcomes, which are improved producer, company and industry organization access to markets, responsiveness to opportunities and demands, and enhanced competitiveness domestically and internationally; and, agriculture and agri-food sector successfully adapts to changing and emerging global and domestic opportunities and issues. For example, SED has contributed to the emergence of new industries such as hemp and bioproducts by providing analysis and advice.

Efficiency and economy

The evaluation evidence suggests that SED was delivered economically and efficiently. During the evaluation period, SED was able to generate a high volume of outputs. Factors such as SED's reporting protocols, experience and expertise, relations with sector stakeholders, and organizational structure supported the generation of a high volume of outputs. The evidence also suggests that SED's value to the sector outweighs its expenditures.

Design and delivery

Although multiple lines of evidence show that SED is well designed, the evaluation found three risks that could impact SED's future capacity to deliver its mandate effectively and report on its outcomes. First, SED efficiently provides a wide range of services, many of which have been delivered in one form or another by AAFC for decades. Beneficiaries of services know that services come from AAFC and, perhaps, from MISB, but not that the work comes from SED. Having SED added to the Program Alignment Architecture in 2013-14 creates an opportunity for SED to continue to dialogue with other areas of the Department to forge a link between its activities and its new name, and report on activities that others within the Department may not be aware are under the responsibility of SED. Second, since SED owes much of its success to the knowledge and working relations nurtured by its officials, many of whom have been with the Department for many years and may be approaching retirement, the evaluation identified a need for a well formulated succession plan and knowledge management process. Lastly, the evaluation noted that while progress has been made, improvements are needed in terms of performance measurement.

1.0 Introduction

1.1 Context of the evaluation

The evaluation of the Sector Engagement and Development (SED) program was conducted by AAFC Office of Audit and Evaluation (OAE) as part of AAFC’s Five-year Departmental Evaluation Plan (2014-15 to 2018-19) and complies with the requirements of the Financial Administration Act (1985) and the Treasury Board Policy on Evaluation (2009). The evaluation focused on SED’s activities from 2009-10 to 2013-14 and the report consolidates all findings from each source of evidence.

1.2 Structure of the report

The evaluation addresses core evaluation issues related to relevance and performance, as defined in the treasury board directive on evaluation (2009), as well as additional questions and issues determined by OAE and AAFC senior management as being key for future program development. The evaluation questions, sub-questions and indicators are contained in the evaluation matrix (Annex A).

The report contains a profile of SED including a program logic model (Annex B), a description of the evaluation methodology and its limitations, findings organized by evaluation issue, and conclusions.

2.0 Profile of Sector Engagement and Development Program

2.1 Program activities, outputs and expected outcomes

Sector Engagement and Development (SED) is one sub-program within the market access, negotiations, and sector competitiveness program^{Footnote 1}. SED is not, strictly speaking, a distinct program with a dedicated budget and charter. Rather, SED is a cluster of activities funded through a combination of A-base resources and Growing Forward 2 (GF2) funding under streams A and C of the AgriCompetitiveness Program and streams B and C of the AgriMarketing Program. SED includes the activities of the Sector Development and Analysis Directorate (SDAD) and the Regional Operations Directorate (ROD), both of which are part of the Market and Industry Services Branch (MISB).

SED activities and outputs are designed to support Canada's agri-food and agri-based products industry^{Footnote 2}. In some instances SED personnel interact directly with, and SED products are provided directly to, industry representatives. In other instances SED personnel interact with, and SED products are provided to, "internal" clients within the Department and/or other intermediaries. More specifically, SED clientele are divided into four target groups:

• AAFC internal clients. SED's internal clients include the Minister's Office, the Deputy Minister and other senior departmental officials, Programs Branch, Science and Technology Branch, Strategic Policy Branch and other organizations within the Department.
• Portfolio partners and other federal government departments. SED personnel work with, and SED products are provided to, representatives of a number of federal government agencies and departments, including the Canadian Food Inspection Agency (CFIA, a "portfolio partner") and trade commissioners working for Global Affairs Canada^{Footnote 3}. SED clients in other federal government departments are typically those who work with and support Canada's agri-food and agri-based products industry.
• Provincial and territorial governments. SED officials work with, and SED products are provided to, representatives of all of Canada's provinces and territories^{Footnote 4}, particularly those working and associated with Canada's agri-food and agri-based products industry.
• Industry representatives. SED officials work with, and SED products are provided directly to, Canadian agri-food and agri-based products industry representatives. These include industry associations, commodity associations, small and medium-sized enterprises, and some of Canada's larger private companies within the industry.

SED activities fall into three broad categories, as follows:

• Data gathering from markets and industries, and analysis. The first category is comprised of activities involving data gathering and analysis related both to industries and markets. These activities are further divided into three sub-categories of data:
  1. Quantitative/empirical data (primary and secondary) gathering and analysis includes collecting and publishing information on production trends and outlooks for domestic and international policy development, program administration, regulatory development and implementation, within the Government of Canada and external stakeholders.
  2. Quantitative/empirical dataare gathered and analyzed: to forecast prices for the operations of the Advance Payments Program, Production Insurance, and Price Pooling Program; to confirm Canadian Wheat Board^{Footnote 5} initial prices for use in medium term forecasts by the Strategic Policy Branch; to provide input for policy development in general, and Memoranda to Cabinet.
  3. Qualitative data and intelligence are gathered and analyzed: to provide input for program development, Memoranda to Cabinet and Treasury Board submissions; and, to provide advice on industry consultation approaches and input into policy development.
• Industry relations and analysis. The second category of activities focuses on relations with industry representatives. These activities fall into two sub-categories:
  1. Relationship building with provincial and municipal governments, industry organizations and critical companies is done in part through the Value Chain Roundtables program which facilitates industry and government collaboration on the development of strategies to improve competitiveness and long-term market success in addressing sectoral issues. Value Chain Roundtables are co-chaired by industry and AAFC representatives. AAFC provides logistical support, shares expertise, and provides financial support. Value Chain Roundtables members include suppliers, producers, processors, food service industries, retailers, traders and associations. SED maintains relations with industry and provincial government representatives in the regions in order to proactively report to senior management on key regional issues and topics. SED also supports the Department in delivering its services with respect to market intelligence and export capacity building, supporting the promotion and delivery of national programs, and managing federal-provincial and federal-territorial bilateral agreements on agricultural policies and programs, including managing GF2 bilateral agreements with provinces. SED also maintains relations with regional industry stakeholders through regular interactions such as attendance at industry meetings and direct exchanges to assess supply capacity.
  2. Analysis of industry competitiveness including impacts of government policies, programs and regulations. The regulatory environment in which the agriculture and agri-food sector operates is complex and resource intensive for both government and industry. For the most part, federal regulations affecting food and agriculture are under the purview of departments and agencies other than Agriculture and Agri-Food Canada, such as Health Canada and the CFIA. The challenge is to balance the need to protect the public interest while enabling trade, industry innovation and growth. SED works with industry in evaluating the relevance of regulations, identifying alternatives, assessing the impact of changes, and increasing its understanding of regulatory processes and requirements. SED works with regulators to assess the impacts of potential approaches.
• Sector development. The third category is comprised of activities relating to the promotion of sector interests. These activities fall into four sub-categories:
  1. Analysis and resolution of sector issues including assistance in the resolution of industry-wide strategic and transactional issues.
  2. Oversight of federal/provincial/territorial (F/P/T) agreements including GF2 agreements, the administration of the Animal Pedigree Act; the Spirit Drinks Trade Act, and Age and Origin of Whisky regulations.
  3. Assistance with industry-led activities including sector-wide national dialogues and workshops.
  4. Assistance in identifying and resolving technical and trade issues including analysis and support provided to Trade and Market Expansion (TME) within MISB, primarily in a sector engagement and knowledge role.

Expected outputs stemming from data gathering and analysis, industry relations and analysis, and sector development are as follows:

• Market information, analysis and intelligence (including reports on market information and market analysis such as the Advance Payments Program, and the Price Pooling Program);
• Industry analytical and technical reports (including benchmarking, performance analysis, regulatory impact assessments, and organization position analysis);
• Value Chain Roundtables, International Market Engagement Teams and other industry-government or F/P/T engagement processes;
• Technical advice and services to the sector and to governments (including project technical reviews, business plan reviews); and,
• Analysis of provincial/territorial (P/T) governments’ positions and activities.

There are three expected immediate outcomes stemming from SED outputs, as follows:

1. Canadian producers, companies, and industry organizations are better informed;
2. The sector has the capacity and exercises leadership to enhance competitiveness and market performance within key domestic and international markets; and,
3. Government programs, policies, and regulations respond to sector priorities and competitiveness needs.

There are three expected intermediate outcomes stemming from SED immediate outcomes, as follows:

1. Canadian producers, companies, and industry organizations make better decisions;
2. Canadian agriculture and agri-food sector addresses priority issues, including trade-related issues, to advance competitiveness; and,
3. Stakeholders implement strategies and use tools to manage changes associated with external forces.

There are two expected end outcomes stemming from SED intermediate outcomes, as follows:

1. Improved producer, company and industry organization access to markets, responsiveness to opportunities and demands, and enhanced competitiveness domestically and internationally; and,
2. Agriculture and agri-food sector successfully adapts to changing and emerging global and domestic opportunities and issues.

Annex B contains a program logic model summarizing SED activities, outputs and outcomes, and the expected linkages among them.

2.2 Governance structure

SED's activities are conducted by two AAFC directorates both within MISB:

• Sector development and analysis directorate (SDAD). This directorate is headed by a director general and divided into five divisions each led by a director. The divisions are: the Horticultural and Cross Sectoral Division; the Animal Industry Division; the Grains and Oilseeds Division; the Industry Engagement Division; and, the Food Industry Division.
• Regional operations directorate (ROD). This directorate is headed by a director general and is comprised of five regional offices (Atlantic, Québec, Ontario, Mid-Western, and North-Western) each of which is headed by a regional director.

2.3 Program resources

The Sector Engagement and Development program is funded by existing departmental resources and GF2. SED expenditures have remained relatively stable. For the period from 2009-10 to 2013-14, SED expenditures totaled $112.9 million.

3.0 Methodology

3.1 Source of evidence

3.1.1 Document and administrative data review

A review of documents was undertaken to determine alignment with federal government priorities and departmental strategic outcomes, program performance, program economy and efficiency, and program design and delivery. The review established the context in which the program operated. Documents related to program design and activities that were reviewed included performance reports, Reports on plans and priorities, program profiles, financial information, business plans, policy frameworks, and annual reports. Government of Canada documentation included Throne Speeches, Budgets and government-wide priority statements.

3.1.2 Literature and media coverage review

A literature and media coverage review was conducted to assess the program's relevance, performance, efficiency and economy. The literature review examined best practices of other similar international programs, reports from Statistics Canada, and external peer-reviewed publications, articles and internet sources. The media coverage review involved a systematic analysis of a representative sample of public media (e.g. traditional news media, online coverage).

3.1.3 Interviews

Thirty-six interviews were completed as part of the evaluation. As shown in Table 2, interviewees included program representatives within Market and Industry Services Branch (MISB), internal Agriculture and Agri-Food Canada (AAFC) clients, representatives of other federal departments and agencies, and industry representatives.

Interview sub-group	Number of interviews
MISB Program Officials	8
AAFC Internal Clients	8
Representatives of other departments and agencies	4
Representatives of provincial governments	5
Industry Representatives	11
Total	36

3.1.4 Case studies

Case studies were used to provide context and to help understand how Sector Engagement and Development (SED) products and services are developed and used. Cases were selected to represent the spectrum of SED activities, and included "internal" case studies that examined the development and use of products within AAFC, and "external" case studies that examined the development of products and their use outside of the Department by representatives of other levels of government and by industry stakeholders. Case studies utilized both interviews and related documentation.

• Response to 2009 Triffid Flax Issue. In the summer of 2009, Canadian shipments of flax exported to the European Union tested positive for low-level presence of banned genetically modified material - FP967 or "CDC Triffid Flax". As a result some $320 million worth of Canadian exports of flax to the European Union were suspended. SED worked with AAFC's Market Access Secretariat, Programs Branch, legal counsel, senior management, other federal departments and industry partners to establish an action plan to eliminate the presence of CDC Triffid Flax in Canadian production and exports in order to regain the European Union export market.
• Response to 2014 problems for Prairie grain, oilseed and livestock producers due to excess moisture. In the summer of 2014, steady precipitation caused extensive overland flooding impacting millions of seeded acres in southeastern Saskatchewan and southern Manitoba. The flooding exacerbated an already wet season during which a significant number of acres had been left unseeded. The problem affected grain, oilseed and livestock producers. SED worked with AAFC's Programs Branch, Strategic Policy Branch, Policy, Planning and Emergency Management, and Deputy Minister's Office to engage industry, stakeholders, provincial officials and agencies to coordinate the development of an action plan to respond to the situation.
• Technical reviews of Industry applications to the Growing Forward 2 (GF2) AgriInnovation Program. AAFC offers programming to the agriculture and agri-food industry in support of the sector's efforts to remain competitive and profitable. Under Growing Forward (GF) and GF2, several programs were initiated including AgriInnovation which funds industry-led research and development and commercialization projects. From 2009 to 2014, AAFC received hundreds of applications under AgriInnovation. SED supported the project selection process by undertaking technical reviews of the applications.
• Consultations in support of the development of GF2. During the GF2 policy and program development process, SED engaged provinces, territories, and industry, at national and regional levels. Information collected from these consultations supported the development of GF2.
• Bee health forum. The Bee Health Forum is an industry-led, government-facilitated, multi-stakeholder forum, developed to address critical issues among agricultural sectors and agricultural suppliers affecting the health of honeybees. The Forum aims to identify measures to protect and improve honey bee health in Canada while ensuring the continued sustainability of crop production.

3.2 Methodological limitations

The evaluation faced a few methodological limitations, discussed below, including the mitigation strategies undertaken.

SED first appeared in AAFC’s Program Alignment Architecture (PAA) and Report on Plans and Priorities in 2014-15. The Report on Plans and Priorities presents SED projected expenditures for 2014-15 through 2016-17. Given the sub-program was created in 2014-15, there is no record of SED expenditures for the period prior to 2014-15.

The activities comprising the SED program have actually been conducted for decades. During the evaluation period, activities falling under what is now SED have been the responsibility of SDAD and ROD. Consequently, as a proxy for actual SED expenditures over the evaluation period, SDAD and ROD expenditure records were used.

The evaluation did not have access to robust, quantitative evidence for some performance indicators. For example, accurate records were unavailable for several outputs. Empirical evidence related to outcomes was limited to (a) anecdotal evidence in relation to particular incidents in which SED was involved and (b) macro-level economic indicators which were difficult to reliably attribute to SED program activities. In order to mitigate this limitation, interviews with a selected representative sample of stakeholders were used as a primary source of evidence. SED’s expected outcomes are in large part qualitative and observable; a systematic program of interviews provided a reasonable base of evidence from which to address outcome questions. This evidence base was bolstered by several complementary sources of evidence – document and administrative review, literature and media coverage review, and case studies – enabling the evaluation to corroborate and illustrate findings.

The lack of outcome data limited the evaluation’s capacity to conduct comprehensive cost-benefit or costs-per-outputs analyses. Even where data were available, an assessment of the relative contribution of SED to specific outcomes was not possible. The evaluation was only able to obtain access to export data; domestic production and sales data aggregated at a level that could have served the evaluation were not available. To address the question of efficiency, the evaluation opted for a descriptive approach drawing on macro-level outcome data plus anecdotal evidence, reinforced by interview data.

4.0 Evaluation findings

4.1 Relevance

4.1.1 Need for Sector Engagement and Development

The evaluation found that there is an ongoing need within the sector for reliable information and analysis, for the facilitation of connections and engagement among stakeholders in support of industry competitiveness, and a continued role for Sector Engagement and Development (SED) in providing these services.

Stakeholders in the Canadian agriculture and agri-food sector are operating in an environment that is increasingly complex. Factors such as increasing demand for specific product attributes, new industry standards, the proliferation of technical trade barriers, changing industry structure, low-cost competitors in agricultural markets, vertical integration of global supply chains and new multinational stakeholders on the domestic front are influencing the profitability, sustainability and success of Canadian farmers and other agricultural businesses. The sector's ability to remain competitive in the long-term depends on its profitability. Accordingly, the agriculture and agri-food system contributes significantly to the gross domestic product and employment in Canada. According to AAFC documents, the sector employs over 2.2 million people, accounts for 6.7 per cent of total Gross domestic product, and is the fifth-largest exporter and sixth-largest importer of agriculture and agri-food products in the world, with exports and imports valued at $46 billion and $34.3 billion, respectively^{Footnote 6}.

Beyond the complexity of its environment, the agriculture industry is presently facing significant volatility, and unpredictability which will be a long-term concern for the sector^{Footnote 7}. This volatility is characterized by larger-than-historical swings in market prices for agricultural products, foreign exchange rates, and input prices. Slow growing economies and high debt levels continue to limit prospects for trade growth for member countries of the Organisation for Economic Co-operation and Development (OECD). Meanwhile, the expansion of developing economies continues to outpace that of OECD countries, a trend that is expected to continue; Canada's trade relations with emerging economies are becoming increasingly important^{Footnote 8}.

Against this backdrop, there is a need for comprehensive and impartial analysis of agricultural and agri-food industries and markets that draw from the full range of available data (e.g., Statistics Canada and other high-level data including markets and sales information, producer/industry intelligence including supply information, provincial positions and data, government policies at all levels including internationally, and scientific and regulation-related information). Stakeholders requiring such analysis include industry stakeholders, both large and small. Large industry associations (e.g., beef, canola), while typically having internal research and analysis capacities, do not have access to a full range of government data, while small industry associations need help in understanding and entering markets.

Agriculture and Agri-Food Canada (AAFC) itself also requires analysis in support of policy development and decision-making related to regulations, programs, policies and other expenditure decisions.^{Footnote 9}. This also applies to other federal government departments and agencies involved in the promotion and/or the regulation of the agriculture and agri-food sector including Health Canada, Canadian Food Inspection Agency (CFIA), Global Affairs Canada, Atlantic Canada Opportunities Agency, Western Economic Diversification Canada, Canada Economic Development for Quebec Regions, and the Federal Economic Development Agency for Southern Ontario. Provincial governments, which play a significant role in Canada's agriculture and agri-food sector, need information and intelligence at a national level, from other provincial jurisdictions, and internationally. There is a need to connect the interests of, and facilitate relations among, sector stakeholders including producers and industry representatives, scientists and regulators, different levels of domestic governments, and international governments. This need relates to ongoing priority setting and other actions that require a full range of relevant perspectives, as well as one-time situations and crises requiring timely, joint planning and decision-making.

These needs relate directly to the competitiveness of the sector. No other program than SED, either within government or in the private sector, was identified as addressing these needs. All key interviewees for the evaluation remarked that the activities and outputs of SED were relevant to the sector and continued to meet the needs of internal and external stakeholders. Interviewees from across the sector stated that SED is effective in assembling, analyzing and adding value to data (i.e. comprehensively, in an unbiased fashion and directly applicable to sector concerns). Key interviewees see SED officials as suited to liaise with other stakeholders, and facilitate collaboration between government departments, industry, and Provincial/Territorial (P/T) governments to address issues facing the sector.

4.1.2 Alignment of Sector Engagement and Development with government and departmental priorities

The evaluation examined the extent to which SED activities align with the priorities of the department and the government as a whole. The evaluation found that the services and products that SED provides are aligned with government and departmental priorities.

Alignment with government-wide priorities: SED's objectives and activities are aligned with federal government priorities. The Economic action plan and 2013 Speech from the Throne^{Footnote 10} emphasize the important role trade plays as an engine of economic growth. The 2013 Speech from the Throne states: "[Canadian's] prosperity hinges on opening new markets for Canadian goods, services and investment."

Alignment is also reflected in federal budgets and related major government policy initiatives. For example, the 2014 Budget^{Footnote 11} notes that the federal government continues to take action to strengthen Canada's agricultural sector through the new Growing Forward 2 (GF2) policy framework, which came into effect on April 1, 2013. Under the framework, federal and provincial governments are providing more than $3 billion over five years for investments in innovation, competitiveness and market development.

Evidence supports the finding that SED is aligned with government priorities. Departmental representatives view AAFC as an economic department and see its role as helping established members of the sector to be more competitive while helping small organizations fully gain a foothold in the market. Key interviewees pointed out that SED has evolved to reflect changes in the government and AAFC priorities. For example, SED has shifted away from regional market development activities and branding to looking at the factors that influence competitiveness and towards working with industry to strengthen these fundamentals.

Alignment with departmental priorities: Within AAFC's Program Alignment Architecture (PAA), the SED sub-program 1.2.2 falls under Program 1.2: Market Access, Negotiations, Sector Competitiveness, and Assurance Systems. The AAFC 2014-15 Report on Plan and Priorities states that the aim of the SED Program is to support the AAFC Strategic Outcome 1: "A competitive and market-oriented agriculture, agri-food and agri-based products sector that proactively manages risk." This Strategic Outcome focuses on Canada's capacity to produce, process and distribute safe, healthy, high-quality and viable agriculture, agri-food and agri-based products, and to expand the sector's domestic and global markets. In this context, SED aims to strengthen the sector's capacity – collaboratively and individually (organizations/individual farmers) – and competitiveness to succeed in an ever evolving international and domestic context.

GF2 lays the groundwork for coordinated Federal/Provincial/Territorial (F/P/T) action over five years (2013 to 2018) to help the sector become more prosperous, competitive, and innovative. The agreement includes investments in strategic initiatives for innovation, competitiveness and market development. The intent is to achieve a profitable, sustainable, competitive and innovative agriculture, agri-food and agri-products industry that is market-responsive, and that anticipates and adapts to changing circumstances and is a contributor to the well-being of Canadians. SED is consistent with the competitiveness and market development focus of the GF2 policy framework.

The 2014-2017 AAFC Business Plan^{Footnote 12} identifies five departmental priorities, two of which align with SED activities:

1. Support and improve the competitiveness and adaptability of the agriculture, agri-food and agri-based product sector; and,
2. Maintain and improve access to targeted, key international markets.

4.1.3 Alignment of SED with the federal government's role

The evaluation found that SED activities are aligned with the federal government's role. AAFC's roles and responsibilities are mandated under the Department of Agriculture and Agri-Food Act (1985). As stated in the Act:

The powers, duties and functions of the Minister extend to and include…matters … relating to:

1. agriculture;
2. products derived from agriculture; and,
3. research related to agriculture and products derived from agriculture including the operation of experimental farm stations.

The role of AAFC has evolved, among other things, to help create the conditions for the long-term profitability, sustainability and adaptability of the Canadian agricultural sector^{Footnote 13}. Sector stakeholders expect AAFC to provide information, research, technology, policies and programs to help Canada's agriculture, agri-food and agri-based products sector compete in markets at home and abroad, manage risk and embrace innovation.

SED supports the Department in fulfilling its mandate through the provision of intelligence and advice to all sector stakeholders towards the goal of a more competitive and sustainable industry. SED plays a leading role in managing, on behalf of the Department, agreements between AAFC and the Department's provincial counterparts.

Both in terms of ongoing relations and the management of crises, there are many actions that can only be accomplished through F/P/T cooperation; SED is the federal government's interlocutor in this regard. By the same token, in the international arena, progress and solutions on many fronts require government-to-government interaction and, again, SED's responsibilities include leadership in these situations.

4.2 Performance - effectiveness

According to the Treasury Board's 2009 Policy on Evaluation, evaluating performance involves assessing effectiveness, as well as efficiency and economy. The subsections below discuss the effectiveness of SED, the extent to which SED has generated its expected outputs and is achieving its expected outcomes.

4.2.1 Generation of expected outputs

The SED Performance Measurement Strategy (PMS) lists the expected outputs of SED activities including, in many cases, expected quantities. The first step in evaluating the performance of the SED program was to determine which expected outputs were generated. In the context of the document review, internal reports detailing outputs along with samples of these outputs were sought from the program. Interviews with SED program personnel were conducted as a complement to the document review and to solicit explanations of documentation when needed. The evidence that was made available to the evaluation supports the observation that outputs – both those generated on a regular schedule for specific audiences as well as numerous ad hoc outputs – were produced by the program. Due to the lack of performance data on outputs produced, the evaluation was not able to provide a definite count as to whether SED met its PMS targets. However, when partial data is available and targets exist, the evaluation provides estimates on whether PMS targets were met.

Evaluation evidence found that outputs are being generated under each of the five major categories identified in the SED logic model:

1. market information, analysis and intelligence;
2. industry analytical and technical reports;
3. Value Chain Roundtables, International Market Engagement Team^{Footnote 14} and other engagement processes;
4. technical advice to industry and governments; and,
5. analysis of P/T government positions and activities.

• A) Market information, analysis and intelligence: SED is responsible for the production of market information. According to the documentation reviewed in support of this evaluation, SED produces three main types of reports: dynamic reports, static reports, and customized reports.
  • Dynamic reporting: Dynamic reports are published on the AAFC public website and customized by users to fit the user's information needs. Dynamic reports gather the most recent data available in a database and allow users to generate reports to their own specifications. This approach has the benefit of being both flexible and economical as it allows website users to tailor reports using the most up to date information, and at the same time frees up resources and decreases the amount of requests SED program officials have to manage.
  • Static reporting: Static reports are conventional reports that include information and analyses of markets for a specified time period. These reports are produced at regular intervals. Many are made available on the AAFC website. Others are disseminated to specific audiences or stakeholders.
  • Customized reporting:Customized reports are produced for, and delivered directly to, hundreds of clients on an ad hoc basis. These reports are customized for stakeholders including other federal government departments, P/T ministries of agriculture, industry organizations, producers, and other relevant stakeholders.

  Tallies containing the total number of outputs produced by SED were not universally available. One SED division, the Animal Industry Division, provided an accounting of their outputs for 2014-15^{Footnote 15}. As the other divisions do not systematically capture the outputs produced by fiscal year, the evaluation was only able to estimate if SED met the targets identified in its PMS.

  The Animal Industry Division produced 263 reports disseminated at pre-specified intervals - weekly, monthly, or annually - 149 via the AAFC website, and 114 reports via other electronic means. If the other divisions responsible for the production of market information and analysis produce a similar volume as that of the Animal Industry Division, it can be estimated that some 1,000 market information and analysis reports are produced and disseminated by SED yearly. This must be considered a rough, notional estimate to be used only in the context of the evaluation to provide a general idea in relation to set targets. Considering that the target set in the PMS is approximately 300, it can be suggested that SED meets its target for the production of market information and analyses.

  SED officials respond to ad hoc requests for information from the general public, other government departments, P/T government agencies, industry, and internal stakeholders. The Animal Industry Division estimates they received and responded to more than 700 requests last year. Using a similar estimation process as above would mean SED responds to 2800 ad hoc requests for information yearly; again viewed in the context of the evaluation as only a rough notion of size, not to be considered a true estimate or to be used for purposes beyond this report. SED officials gather and analyze empirical data to forecast and recommend prices for the operations of the Advance Payments Program and other programs. In 2014-15, SED responded to approximately 300 Advance Payments Program pricing requests, 300 requests for crop insurable values, and 400 requests for farm income price forecasts^{Footnote 16}. Note that no targets were set in the PMS for these outputs.

  Key interviewees described market analysis reports produced by SED as useful aggregations and syntheses of data and statistics emanating from across the sector and around the world. A number of interviewees representing different organizations explained that most sector stakeholders, including individual producers, processers and exporters, generate data. In addition, different internal groups within AAFC, such as the Research and Analysis Directorate in the Strategic Policy Branch generate data. Portions of this data is then collected and analyzed by a broad range of stakeholders, including: major national associations, AAFC, CFIA, Statistics Canada, Global Affairs Canada, provincial ministries of agricultural, multinational corporations, foreign governments, and industry associations. The key benefit for SED clients is that SED synthesizes and analyzes the data from all relevant sources.

• B) Industry analytical and technical reports: Industry analytical and technical reports are produced by SED with support from other experts in AAFC primarily for internal departmental purposes. Key interviewees indicated that these reports contribute to the Department's strategic thinking and are used to guide a variety of decisions.

  Since SED does not keep track of the number of these outputs, the evaluation was unable to measure them against the target set in the PMS. However, samples of the following kinds of industry analytical and technical reports were reviewed:

  • benchmarking of product costs in Canada and in other jurisdictions;
  • assessments of the impact of other countries' legislation on the Canadian agriculture and agri-food sector;
  • assessments of the impact of proposed regulation;
  • profiles of companies and organizations in the sector.
• C) Value Chain Roundtable, international market engagement team, and other engagement processes: Value Chain Roundtables are industry-led dialogue fora that bring together industry stakeholders from along the value chain with provincial and federal government members^{Footnote 17}. Value Chain Roundtables promote industry-led initiatives, in partnership with governments, on identified priorities with the goal to increase competitiveness, innovation, capacity and profitability of each component of the value chain. Value Chain Roundtables are formal structures supported financially by AAFC with regularly scheduled meetings. Cross-sectoral Working Groups function similarly and serve comparable purposes, but are less formally structured. There are currently 12 Value Chain Roundtables and seven Cross-sectoral Working Groups. Roundtables include: Bee Health; Beef; Food Processing; Grains; Horticulture; Organics; Pork; Pulses; Seafood; Seeds; Sheep; and Special Crops. Cross-sectoral Working Groups include: Bioproducts; Research Centre of Excellence; Crops Logistics; Grains Symposium; Agri-Subcommittee on Food Safety; Labour Task Force; and Regulatory Issues.

  SED participates in, supports, and/or leads other engagement processes, from ad hoc informal meetings to such fora as International Market Engagement Team meetings. All engagement fora were viewed by key interviewees as effective and valuable. Value Chain Roundtables in particular were considered by interviewees to be effective instruments in facilitating collective action that bring all stakeholders together 'from farm to fork' to consult and identify priorities. Value Chain Roundtables are both an intended SED output, as well as a consumer of other SED analytical outputs. SED co-facilitates Value Chain Roundtables meetings and provides supportive analysis and documentation, and arranges relevant guest attendees. Although there are no targets set for the number of Value Chain Roundtables, International Market Engagement Teams and other engagement processes in SED's PMS, the 2013-14 MISB year-end achievements report highlights the following Value Chain Roundtables outputs^{Footnote 18}:

  • created a cross-Value Chain Roundtables committee on regulatory issues;
  • developed 15 regulatory issue background documents to support the Value Chain Roundtables Regulatory Subcommittee;
  • completed 17 research projects (including statements of work, selection processes);
  • held 23 Value Chain Roundtables and 10 cross-sectoral face-to face working group meetings;
  • held more than 170 conference calls and virtual meetings; and,
  • established the Value Chain Roundtables Labour Task Force.
• D) Technical advice to industry and governments: Documentation shows SED provides technical reviews and conducts due diligence for activities under GF2 programs such as the AgriInnovation Program, and the AgriMarketing Program. The 2013-14 Market and Industry Services Branch (MISB) Year End Achievements Report reveals that SED conducted technical reviews and assessments for hundreds of proposals related to the AgriInnovation Program and the AgriMarketing Program. Reviews ensured a factual basis for each proposal in terms of market opportunities and that all reported data was consistent with verified sources^{Footnote 19}. Although precise tallies were not available, key interviewees estimated that over the past five years SED completed, on average, over 300 technical reviews per year, suggesting SED met its target of 268 technical reports, business plan reviews and related reports.

  SED officials spend time reviewing program applications. They draw on their "on the ground" knowledge of the industry to bring a regional/provincial/national and industry-wide perspective to the review of applications, including the history and capacity of an applicant, expected impacts that the project will have on the applicant, industry, region, and other stakeholders.

• E) Analysis of P/T government positions and activities: AAFC regional offices provide integrated access to AAFC programs and services and provide regional coordination on departmental initiatives^{Footnote 20}. Regional offices engage in regional strategic analysis including analysis of P/T governments' positions and activities. This engagement results in regional information and intelligence reports that provide analysis of regional stakeholder issues, or in the event of an emergency, emergency management intelligence reports that provide around-the-clock intelligence and implications for senior management. Although there is no target identified for this type of output, the 2013-14 MISB Year-End Achievements Report reveals that during the fiscal year, SED's regional offices prepared 122 regional information and intelligence reports^{Footnote 21}. Additionally, SED's regional offices prepared seven emergency management intelligence reports^{Footnote 22}.

  According to SED officials and users of this type of output (e.g., Programs Branch), SED officials in the regions are uniquely positioned to represent AAFC and to oversee due diligence based on their knowledge of industry priorities and their grasp of how provincial programs and services work. According to AAFC interviewees, consistency across the country on cost-shared programs is considered important. SED's contribution in this regard is significant.

4.2.2 Achievement of expected immediate outcomes

4.2.2.1 Expected immediate outcomes: Canadian producers, companies and industry organizations are better informed

The first immediate outcome expected to stem from SED's outputs is that SED's clients – from stakeholders in Canada's agriculture and agri-food industry to AAFC policymakers – are better informed with relevant knowledge. Based on all sources of evidence, the evaluation found that Canadian producers, companies and industry organizations are better informed.

Better informed industry: Representatives of large industry associations, which typically have internal data gathering and analytical capacity, emphasized the contribution of SED's market information, analysis and intelligence to the enhancement of their value chain's level of knowledge (e.g., technical issues, research, regulatory changes). Representatives of smaller industries, typically with limited resources and capacity, described their reliance on SED's market analysis as well as SED's advice, as "pathfinding" into AAFC.

The pricing forecasts produced by SED are used in a variety of ways. Producers provide lenders with pricing information in order to secure credit; provincial governments attract investments into the province; and price forecasts are fed into the Advance Payments Program and for use in crop insurance by AgriInsurance Canadian grain industry stakeholders noted during the interviews that the United States Department of Agriculture publishes daily port grain export bids while AAFC publishes port prices weekly^{Footnote 23}. The publication frequency of port prices confirms the importance industry places on this type of information.

The CDC Triffid Flax case study illustrates SED's analysis and advisory roles. As the crisis unfolded, industry stakeholders came to better understand the seriousness of the problem from the perspective of European Union regulations, and the need to resolve the issue through compliance. SED's role in the excess moisture flooding crisis in the Prairies in 2014 is another example. SED helped governments (both federal and provincial) and industry stakeholders develop a mutual understanding of the response required to address the numerous challenges that resulted from the flooding event.

Better informed AAFC internal clients: AAFC internal clients interviewed for the evaluation reported that they were better informed by the provision of SED-generated intelligence on the state of an industry or region. They stated that, this type of information fed into the development of programs and policies (e.g., GF2 cost-shared programs). SED information was found to help support those working in the international arena. According to interviewees, SED plays a role in supporting trade negotiations by providing analysis, information and expertise on the entire range of issues across the sector.

Better informed portfolio partners: Portfolio partners, such as CFIA and Health Canada's Pest Management Regulatory Agency, explained the importance of SED-produced information in helping them to better appreciate the perspectives, priorities, capacity and limitations of industry. Other partners, including Global Affairs Canada and the Canadian Dairy Commission, highlighted the role that SED analysis plays in the management of Canada's supply managed commodities. For example, the Canadian Dairy Information Centre uses SED data to inform the industry stakeholders in two ways. First, the information is used for the calibration of the supply and demand for milk. Second, and more broadly, the Canadian Dairy Information Centre serves as a unique and very useful information clearinghouse used by the key stakeholders involved in the Canadian dairy industry (e.g., CFIA, AAFC, Statistics Canada, industry members, provincial associations, provincial governments, and, the Canadian Dairy Commission).

4.2.2.2 Expected immediate outcomes: the sector has the capacity and exercises leadership to enhance competitiveness and market performance within key domestic and international markets

The second immediate outcome expected to stem from SED's outputs is that stakeholders in Canada's agriculture and agri-food industry are better equipped to exercise leadership in enhancing the sector's competitiveness. Across all sources of evidence, the evaluation found that, as a result of SED activity, the sector had greater capacity and exercised leadership in enhancing competitiveness and market performance domestically and internationally.

Working through Value Chain Roundtables: Value Chain Roundtables, co-chaired by AAFC and industry representatives, are widely perceived by those who are familiar with them to have contributed to enhancing industry's capacity to exercise leadership. The bringing together of representatives from across an industry value chain, including F/P/T governments, helps industry organize themselves to identify and address emerging issues, set strategic priorities, and develop long-term strategies to make progress towards meeting objectives (e.g., through the identification of action items which are followed-up at subsequent meetings). The presence of government at the table allows industry to provide feedback on policies, programs and regulations.

The 2012-13 MISB Year-End Achievements Report indicates that, at the request of industry stakeholders, a forum on labour (i.e., the Labour Task Force) was established through the Value Chain Roundtables program to facilitate discussions on labour issues affecting the competitiveness of the industry and to explore mechanisms to increase productivity and access to labour. The forum identified priority actions and fostered collaborative industry-government responses that led to a more sustainable and profitable agriculture sector, according to the report^{Footnote 24}. A Parliament of Canada report on Canada's food supply chain (Toward a Common Goal: Canada's Food Supply Chain) described Value Chain Roundtables as playing an important role in enabling industry to collectively and strategically build capacity and leadership.

SED is seen as supporting the Value Chain Roundtables in a number of ways. AAFC co-chairs are viewed by industry representatives and others as effective, particularly at providing strategic focus and advice. SED was described by interviewees as providing strong secretariat and research support. SED officials were also seen as effective in helping the Value Chain Roundtables access AAFC funding, particularly to fund research.

Working with emerging areas of the sector: The views of representatives of smaller industry associations suggest that SED helps such associations to bolster their capacity and to more effectively lead their membership. This is achieved through the provision of information, including customized reports, pathfinding within AAFC (e.g., to access programs, to meet officials), and direct advice and guidance.

This increased capacity to provide leadership and strategic direction demonstrates value to association members which, in turn, contributes to the sustainability of the industry. One small association representative, for example, explained how SED's assistance and guidance on the complex issue of traceability allowed him to assist his members to adopt traceability protocols. The CDC Triffid Flax case serves as an example of how a relatively small industry association can potentially be overwhelmed by a crisis. SED, working in partnership with others, was able to provide support throughout the crisis, including helping the industry associations to enable its producers to follow the sampling and testing protocol that was required by the European Union.

4.2.2.3 Expected immediate outcome: government programs, policies and regulations are responding to sector priorities and competitiveness needs

The third immediate outcome expected as a result of SED's outputs is that Canadian government – including AAFC as well as other federal departments and agencies – programs, policies and regulations are more responsive and relevant to the needs of the agriculture and agri-food industry in Canada. The evaluation found that government policies, programs and regulations are more responsive to sector priorities and competitiveness needs as a result of SED activities.

Industry input into AAFC policies and programs: From the perspective of industry representatives, greater responsiveness in government programming and policies is being achieved through SED interactions, including the Value Chain Roundtables Working Groups, and GF2 consultations, as well as on an ad hoc basis. According to AAFC's internal clients and portfolio partners, SED influences programs, policies and regulations by communicating industry priorities and challenges through various analyses that SED brings to AAFC committees and other discussions. As an example of this work, SED actively engaged the P/Ts and industry during the GF2 policy and program development process. The two primary engagement vehicles were regional engagement sessions (public, by invitation and theme-based), and specialized Value Chain Roundtables engagement sessions conducted by SED. The contributions from these engagement efforts helped to develop a mutual understanding among governments and industry on what is required to achieve a profitable and competitive sector. Information and feedback collected from these consultation and engagement sessions contributed to the development of the GF2 policy framework and programs. In a more specific example, SED merged priorities identified by the organics Value Chain Roundtables which led to the development of a program to help producers make the transition from traditional to organic farming.

Injecting a market perspective into AAFC's science and technology research: There was consensus among interviewees – both external and internal - that SED is effective in bringing a market perspective to the Department's research activities. At any given time, the AAFC Science and Technology Branch (STB) is working on some 200 research projects. By consulting with SED, STB has been able to shape its research programs and focus its research on subjects of greater relevance to the sector. For example, speaking of SED, one STB official noted: "We grow a lot of apples around here, all sorts of apples. One of their analysts put together this great report on the apple market: prices, demand, what different markets were looking for in terms of varieties, etc. So we looked at that and it really helped steer what we were doing in terms of research."

Reflecting industry's reality to portfolio partners: The evaluation found that the role SED plays in gathering and interpreting the industry perspective is respected and appreciated within AAFC, as well as by other departments and agencies. SED is seen as an interlocutor bringing industry's perspective to other government actors to help maximize the relevance of their actions. SED officials have made representations to regulators such as Health Canada and CFIA, as well as other departments such as Transport Canada with respect to backlogs (e.g., grain shipments) and Employment and Skills Development Canada concerning the sector's heavy use of temporary foreign workers (e.g., horticulture). The Border Measures Working Group, which is co-chaired by a SED official, with members representing the Canada Border Services Agency, has been described as an early warning monitoring system designed to keep items on the Import Restriction List from entering Canada without paying the required tariff.

Working with Industry, AAFC Clients, and other stakeholders to develop responses to events: SED works with a broad range of stakeholders, including industry, to help formulate a government response to specific challenges or events. The excess moisture event illustrates this type of response. SED officials played a central role in liaising between industry, provincial governments and AAFC facilitating dialogue and supplying timely intelligence reports. The result was a joint federal-provincial response that met the exigencies of the situation on the ground. Two other case studies, the Bee Health Forum and CDC Triffid Flax, also show how SED worked to help ensure that the industry perspective was properly reflected in government policies and programs.

In summary, the evaluation found that Canadian producers, companies and industry organizations are better informed, that the sector has greater capacity and exercises more leadership in enhancing competitiveness and market performance domestically and internationally, and that government policies, programs and regulations are more responsive to sector priorities and competitiveness needs. The next subsection describes how these immediate outcomes affected, in turn, intermediate outcomes.

4.2.3 Achievement of expected intermediate outcomes

4.2.3.1 Expected intermediate outcome: Canadian producers, companies and industry organizations make better decisions

By being better informed through SED interventions, sector stakeholders expect to make better decisions. The evaluation found that better decisions are made within the sector based on SED-supplied information.

Better industry decision-making: SED contributes to the enhancement of decision making among Canadian producers, companies, and industry organizations. This outcome extends to encompass a range of other stakeholders, including portfolio partners and AAFC internal clients. The case studies and interviews both highlighted a number of instances where decisions were made in whole or in part based on SED-generated information, advice or intelligence with support of different internal groups within AAFC. For example, producer associations and their members use pricing information provided by SED to make decisions about what to charge for their products. Producers have also used market analysis provided by SED to make strategic decisions about how to better match what they grow to potential market opportunities, both in Canada and abroad. Industry representatives also revealed that SED has played a role in supporting industry organizations in trade disputes: "SED was hugely helpful in providing an analysis that we are using in legal arguments which will lead to the removal of a barrier and automatically lead to increased revenues for the industry." Furthermore, SED has played a role in ushering innovation in certain industries: "As a result of the Value Chain Roundtable, we are looking at best practice processes to combine biomasses of different crops. We had been engaged for a while but the Value Chain Roundtables over the past two years has accelerated everything including the commercialization process."

Better AAFC internal client and portfolio partner decision-making: SED supports better decision-making on the part of AAFC personnel and portfolio partners through the provision of information in three basic forms: market analysis and related information; technical advice on specific issues; and, technical advice on research.

Examples of the first form include SED analysis, reports, and intelligence, related to regional markets, provincial positions, industries/commodity groups, and large individual private companies. These products are provided in some cases on a regular basis and in other cases on an ad hoc basis to AAFC decision-makers up to the level of deputy minister and minister, as well as to portfolio partners.

Technical advice on specific issues is provided on a regular basis and on an ad hoc basis in response to a particular crisis or situation. Regular advice is provided, for example, with respect to the tariff rate quota allocation system administered by the Minister of International Trade. Ad hoc advice is illustrated in the case of the CDC Triffid Flax response wherein SED helped shape the development of a sampling and testing protocol that was mutually acceptable to both Canadian industry, and European Union and Japanese regulators.

The main example of technical advice on research provided by SED comes from the technical reviews case study. It was estimated that SED's officials are requested to complete approximately 300 reviews per year of funding applications received from industry stakeholders under the GF2 AgriInnovation Program. These reviews are provided to STB; representatives of STB interviewed for the evaluation state that SED's input in this regard helped selecting projects that are relevant to the needs and interests of the sector.

4.2.3.2 Expected intermediate outcome: the canadian agriculture and agri-food sector addresses priority issues, including trade-related issues, to advance the sector's competitiveness

The second expected intermediate outcome is that the sector is better able to identify and address priority issues. The evaluation found that SED activities, outputs and related immediate outcomes led to enhanced sector capacity to identify and address priority issues.

Formal mechanisms to identify and address priorities: The views of interviewees suggest that SED's involvement and support of Value Chain Roundtables and other formal engagement fora constitute one of the main ways in which the program assists industry and other stakeholders to identify priority issues, and through which collective actions related to priorities are planned. Value Chain Roundtables bring people together in order to identify common impediments to competitiveness. From this, priorities are identified and action plans are developed.

By way of example, animal health and welfare emerged as a sector-wide priority several years ago. According to interviewees, animal health and welfare is a major competitiveness issue in both export and domestic markets as buyers are looking for certification that the animal has been raised according to established standards. SED has responded to this priority; a SED director chairs the National Farm Animal Welfare Council whose work has resulted in advances in animal health management as well as greater awareness of animal health issues among producers and other industry stakeholders and consumers. SED has made contributions to develop programs that respond to provincial industry priorities while respecting the overall GF2 Framework.

The Bee Health Forum case study also showed how SED was able to work with a broad range of stakeholders to help them define and address an emerging priority: to better understand the determinants of bee health and identify actions that could be taken to respond to risks and opportunities for bee health. Work by the forum led to a national strategy formulated around five objectives (as found in the National Bee Health Action Plan). The success of the forum led to the creation of the Bee Health Roundtable in November 2014.^{Footnote 25}

Informal mechanisms to identify and address priorities: Interviewees also described many instances in which analysis and advice contributed on an ad hoc basis to the identification of priorities, over and above the work accomplished through formal mechanisms. Interviewees commented on the level of trust and credibility that SED officials have established with industry and other stakeholders, and how this trust has allowed SED officials to help identify and shape industry priorities in discussions at industry committee meetings and other fora.

4.2.3.3 Expected intermediate outcome: stakeholders are implementing strategies and using tools to manage changes associated with external forces

The third expected intermediate outcome is greater sector capacity for dealing with changes resulting from external forces. The evaluation found that SED activities, outputs and related immediate outcomes supported stakeholder development and use of strategies and tools to more effectively manage change.

Assisting stakeholders in the implementation of strategies and use of tools to manage changes associated with external forces: There are several examples demonstrating that SED has supported constructive change management and met its outcome of using tools to manage change associated with external forces^{Footnote 26}. The work of SED officials in helping industry and other stakeholders implement strategies to manage non-tariff trade barriers and technical impediments to market access with support of TME was acknowledged and praised in interviews. A representative of a major industry that exports most of its product summarized SED's impact as being helpful in providing analyses used in legal arguments leading to the removal of a key export barrier. This representative saw an increase in revenues due to SED's involvement. Interviewees, including some representing major sector stakeholders, predicted that Canada would face an increasing number of non-tariff trade barriers. Multinational and bilateral trade agreements have dramatically reduced tariffs, leaving the imposition of technical barriers as the preferred method of protecting domestic industries. Canada's ability to quickly and effectively mobilize a response to such threats was described by a number of people as an important competitive advantage, and they saw SED's role in coordinating and/or supporting Canada's multi-disciplinary response as key to success. For example, Malaysia and the Philippines surprised Canada with the introduction of requirements for additional testing and certification of soy beans. SED brought together industry stakeholders and F/P/T governments, including trade experts and negotiators, and engaged the countries in a dialogue which caused them to first delay the introduction of the regulations and then abandon the regulations in the face of Canada's ability to demonstrate that the regulations were not based on sound science.

The CDC Triffid Flax case showed how SED played a key coordination and issue-management role in the rapid and successful development and implementation of a sampling and testing protocol. The protocol can be seen as both a strategy and a tool. Its success, and in particular the speed with which it was developed and accepted by the European Union, had a direct impact on the industry's competitiveness.

The evidence from the excess moisture case study supports the observation that as a result of a timely response on the part of governments, a significant number of farmers managed the disaster and were able to remain in the industry^{Footnote 27}.

In summary, the evaluation found that SED's activities and immediate outcomes resulted in better decisions within the sector, enhanced sector capacity to identify and address priority issues, and improved stakeholder development and use of strategies and tools to more effectively manage change. The next subsection describes the extent to which these intermediate outcomes led to expected end outcomes.

4.2.4 Achievement of expected end outcomes

4.2.4.1 Expected end outcome: improved producer, company and industry organization access to markets, responsiveness to opportunities and demands, and enhanced competitiveness domestically and internationally

An end outcome associated with SED is overall improved competitiveness and success of Canada's agriculture and agri-food sector. Despite difficulties related to empirically attributing macro-level sector performance directly to SED's activities, the evaluation found that SED has contributed to the sector's competitiveness. SED's contributions have been both indirect – i.e., creating conditions for industry action – and direct – providing analysis and advice leading to immediate increases in profitability.

SED helps create the conditions for enhanced competitiveness: As with almost any program's intended end outcomes, the downstream nature of such impacts makes them difficult to measure. The ability of the sector to enhance its competitiveness is based on many factors. SED's interventions constitute only one of these. Other factors include other federal and P/T government activities and policies, the actions and agriculture-related policies of other countries, non-agriculture related domestic and global human generated events that may affect consumption (e.g. geo-political conflicts, consumption and trade, changes in economies) and climate incidents. It is difficult, if not impossible, to attribute the direct causal effect of SED activities.

Sometimes success must be viewed in terms of maintaining or even slowing or mitigating the loss of market share as was the case for the CDC Triffid Flax incident. That said, the evidence produced by the evaluation suggests that SED has contributed to the sector's competitiveness potential. In terms of broad context, the evaluation finds that Canada's agriculture and agri-food sector continues to expand and remain competitive^{Footnote 28}. The 2013 overview of the Canadian agriculture and agri-food system by AAFC suggests that Canada has remained competitive in terms of long-term sales growth in domestic and international markets for agriculture and agri-food products^{Footnote 29}. This statement is supported by data from Statistics Canada illustrating a substantial, stable, positive agricultural trade balance ($9.2 billion in 2011 and $10.8 billion in 2013).

Direct contributions to competitiveness: The evaluation documented numerous direct examples of SED contributing to competitiveness at the firm, regional and industry levels. For instance, the CDC Triffid Flax case shows how SED, working in concert with other stakeholders, had a direct impact on the competitiveness of Canada's flax industry. The consensus view was that delays in finding a solution to the problem (i.e., the sampling and testing protocol) would have led to a much more significant drop in production and exports. In another situation, research and advice provided by SED respecting the use of product codes for exports into China led directly to increased profits for exporters. Another example involved SED officials working with Canadian producers, other AAFC officials and Canada's trade officials in Japan to build a direct relationship between Canadian buckwheat producers and Japanese buyers. The aim was to bypass United States brokers to have a direct entry into Japan's lucrative Soba noodle market. Similarly, SED was able to supply industry with market information and to identify potential sales leads in the wake of the Canada-South Korea Free Trade Agreement, which took effect on January 1, 2015.

4.2.4.2 Expected end outcome: the agriculture and agri-food sector is successfully adapting to changing and emerging global and domestic opportunities and issues

The second expected end outcome relates to the enhanced adaptability of the sector. All lines of evidence suggest that SED has contributed to a greater adaptability of the sector.

Responsiveness and adaptability: The agriculture and agri-food sector is complex, and subject to volatility. Natural disasters, trade disputes/sanctions, and disease can threaten the competitiveness, and even the viability, of an industry while trade agreements, weak competition, and emerging markets present opportunities to the sector. Interviewees suggested SED's activities and outputs have contributed to the sector's ability to successfully adapt to changing and emerging global and domestic challenges and opportunities. In most cases, it is not possible to quantify or disentangle the impact of the SED program from other factors that determine whether producers, companies and industries adapt. Nevertheless, interviewee responses suggest a positive contribution. For example, interviewees highlighted that SED contributed and continues to contribute to the emergence of new industries such as hemp and bioproducts, both of which are taking advantage of global trends and technological breakthroughs.

Interviewees pointed out instances of sudden and potentially disastrous impediments to Canadian exports requiring a rapid and coordinated technical response, one in which SED has frequently been involved. According to interviewees, the Canadian government has a positive reputation internationally for its ability to quickly and effectively pull together a collective response to issues as they arise. Three case studies highlight how SED contributed to the sector's ability to adapt to serious threats: Response to 2014 Problems for Prairie Grain, Oilseed and Livestock Producers due to Excess Moisture, Response to 2009 Triffid Flax Issue, and the Bee Health Forum.

In summary, the evaluation found that SED contributed to its end outcomes. SED contributed to the sector's competitiveness – by, for example, creating conditions for industry action and providing analysis and advice leading to increases in profitability – and to its adaptability.

4.3 Performance - efficiency and economy

4.3.1 SED economy

According to the 2009 Treasury Board Policy on Evaluation, economy is said to have been achieved when a program's activities and expected outputs have been delivered for the lowest possible cost, taking into consideration the program's context as well as requirements respecting output quality. A comprehensive, empirical assessment of SED's economy would require a detailed study of costs-per-outputs associated with comparison programs (e.g., in other countries) and/or a business process mapping exercise. Both of these methodological approaches were beyond the scope of the evaluation. The evaluation is limited to a descriptive approach, reinforced by interview data. The evaluation evidence suggests that SED is an economical program, although definitive conclusions in this regard cannot be drawn.

Over the period under consideration in the evaluation, SED's annual expenditures were $23.0 million in 2009-10, rose to $24.2 million in 2010-11, and dropped to $22.4 million in 2013-14. The expenditures covered primarily SED officials' salaries; SED had 222 Full-Time Equivalents in 2009-10, 240 in 2010-11, and 189 in 2013-14. As reported in sub-section 4.2.1, outputs generated for these expenditures included several hundred analytical products over five years including market information and analysis reports and responses to ad hoc requests, pricing reports and responses to pricing-related requests, technical reports and technical reviews, and analyses of P/T government positions and regional stakeholder issues. SED manages Value Chain Roundtables, International Market Engagement Team meetings and various ad hoc engagements and supports them with background research, summary reports and other reports. SED provides emergency management and crisis support services. SED also maintains a public website that includes user-generated report capabilities. However, costs-per-outputs are not possible to calculate.

According to interviewees, SED owes its ability to generate this volume of outputs to several factors. First, many of SED's outputs are generated on a regular basis, i.e., weekly, monthly or annually, according to a standardized format. In most cases reporting protocols are well-established; the generation of new reports is largely a matter of updating the previous analysis.

Secondly, the collective experience and expertise of SED officials combined with their unparalleled access to relevant data sources enables SED to respond to ad hoc requirements rapidly and without wasted effort; SED officials generally know what is needed, where to find the requisite data, and how to analyze it.

A third factor relates to the relationships SED officials maintain with sector stakeholders in the regions and nationally. According to interviewees, SED constitutes what may be the sector's most connected relationship hub. These relationships – knowing who's who, and having the ability to make timely contact with the right individual – further enhance SED's ability to respond to ad hoc requirements economically.

Lastly, SED is structured to be adaptable. If a new situation requires attention and one part of the organization is already occupied, another part is typically able to intervene. In this way, numerous requirements can be handled simultaneously without sacrificing quality or missing regular reporting deadlines.

4.3.2 SED efficiency

According to the 2009 Treasury Board Policy on Evaluation, program efficiency is a measure of the achievement of expected outcomes in relation to resources expended. For programs like SED, whose benefits can mostly be described in monetary terms (e.g., value of exports, value of domestic sales), cost-benefit analysis is the most appropriate vehicle for the assessment of program efficiency. However, as mentioned in section 3.2 of this report, the lack of data limited the evaluation in terms of producing a comprehensive cost-benefit analysis for the purpose of assessing SED's efficiency. Nevertheless, the available evidence suggests a high level of efficiency, although definitive conclusions in this regard cannot be drawn.

Macro-level outcomes: As highlighted in Table 3, Canadian agri-food and seafood exportsgrew by more than 40 per cent between 2009-10 and 2013-14, from $39 billion in 2009-10 to $56 billion in 2013-14. SED expenditures over the same period were generally stable with slight yearly fluctuations. From 2009-10 to 2013-14^{Footnote 30}, SED's expenditures totalled $112.9 million, while total sector exports over the same period were in excess of $238 billion. Over the five year period covered by this evaluation, the exports to expenditures ratio of the program was 2111:1. In other words, SED's expenditures represented 0.047 per cent of total sector exports^{Footnote 31}.

Fiscal Year	Sector Exports	Exports Growth	SED Expenditures	Exports to Expenditures Ratio
2009-10	$39,351,272,063		$23,010,443	1710:1
2010-11	$44,397,764,509	12.8 per cent	$24,207,693	1834:1
2011-12	$47,723,430,102	7.5 per cent	$21,953,177	2174:1
2012-13	$50,401,191,043	5.6 per cent	$21,358,564	2360:1
2013-14	$56,451,463,404	12.0 per cent	$22,370,141	2524:1
Total	$238,325,121,121	43.4 per cent	$112,900,018	2111:1

Given the size and complexity of the agriculture and agri-food sector, and given the influence of extraneous variables on the sector, attributing export sales increases or decreases to the SED program is difficult^{Footnote 32}. If SED activities and outputs led to improvements in the revenue generating capacity of the sector in excess of 0.048 per cent, or if the program contributed to maintaining or mitigating the loss of market shares as a result of external events by the same amount, SED would be considered cost-effective. Interviewees who commented on the question believed this to be the case.

The literature review supports the notion that SED's program activities are in line with what researchers have determined to be successful government strategies for promoting competitiveness in agriculture. Several studies have concluded that the highest performing farmers are typically "information rich"^{Footnote 33}. The literature also suggests that the provision of market data and intelligence is an effective method of investing in the sustainability and competitiveness of the agriculture sector and for reducing the effects of volatility^{Footnote 34}. Investments in market development and facilitation, in comparison to other types of agriculture supports (e.g., direct income payments), have been found to lead to benefits above the original expenditure^{Footnote 35}.

In summary, the evaluation evidence suggests that SED outputs are produced economically and that SED's value to the sector outweighs its cost to taxpayers.

4.4 Design and delivery

The evaluation question associated with design and delivery asks whether or not the SED program was delivered as intended. The evaluation’s analysis of the generation of expected outputs (Section 4.2.1) demonstrated that all expected outputs were produced, as well as outputs not included in the program logic model, generated in response to specific requests or other circumstantial requirements.

The evaluation assessed design and delivery with a particular focus on the role SED plays within AAFC and within Canada’s agriculture and agri-food sector as a whole. The evaluation also examined SED’s organizational structure and resourcing, and performance monitoring. Findings, as detailed below, relate to the unique features and attributes associated with SED as well as some of the challenges facing SED.

4.4.1 SED's role and identity

The evaluation found that SED plays a unique role within AAFC as well as within the agriculture and agri-food sector. According to interviewees, SED possesses the combined breadth of contacts and depth of knowledge and access to data. The melding of these two attributes is in large part what enables SED officials to respond economically and effectively to both regular information and intelligence expectations, and ad hoc and crisis situations within the sector. Interviewees emphasized the observation that the relationships SED officials enjoy with key sector stakeholders are long-standing and are characterized by a high degree of trust. By the same token, SED’s information and analysis products are viewed within the sector as both high value-added and highly credible. SED officials are able to translate the needs and concerns of one group (e.g., a regulator or foreign government) into the language of another group (e.g., Canadian industry).

SED’s information and intelligence is viewed to be reliable and unbiased. The activities undertaken by SDAD and ROD have been conducted for many years and, in some cases, decades. As of 2013-14, these activities were grouped under the name “Sector Engagement and Development” or “SED” and represented in the departmental Program Alignment Architecture (PAA). Interviews with AAFC officials stated they know of the activities and their importance for the sector, but they don’t know that these activities are grouped into “Sector Engagement and Development” or “SED”. A majority of interviewees, both internal and external to AAFC, had challenges to identify who was SED. They knew SDAD and ROD, but did not know that their activities were grouped into SED. This lack of awareness of SED could create potential vulnerabilities for SED in terms of reporting on the outputs, activities and results to SED, particularly as SED’s work is the underpinning of other work done within AAFC. The integration of SED into the PAA creates a greater obligation to report on performance and results (e.g., through the Departmental Performance Report). Having SED newly added to the PAA creates an opportunity for SED to continue to dialogue with other areas of the Department to forge a link between its activities and its new name, and report on activities that others within the Department may not be aware are under the responsibility of SED.

4.4.2 Organizational structure and human resources

For the five years covered by the evaluation, no evidence was found suggesting any problems in the organizational structure of SED; the organization of the program appeared to be working well. Importantly, the structure incorporated sufficient flexibility to enable SED officials to respond to ad hoc requestsand crisis situations.

SED and TME produce market data, intelligence and analyses reflecting different perspectives and audiences. SED’s reports aim to increase knowledge, competitiveness and adaptability within sector while TME’s reports aim to support trade and market expansion internationally. Evidence suggests coordination between both programs currently exists that avoids duplication of efforts. To maintain the effectiveness and efficiency of SED, continued coordination with TME is necessary to avoid duplication of future efforts.

SED is, in fact, a group of individuals, many of whom have worked in a similar capacity for many years. Internal and external clients know by name the SED officials that they need to engage. Interviewees spoke of relationships spanning 15 years or more. Industry representatives spoke of committees they sat on with SED officials. As one executive representing one of AAFC’s portfolio partners stated, “I think it works well because of the people, the officials work well together, they trust each other. They share information readily.”

Looking forward, interviewees expressed concern about the ability of SED to continue to serve its clientele and the sector as effectively once key people begin to retire. In terms of one of SED's core competencies, i.e., substantive expertise and knowledge – including corporate memory – interviewees noted the need for an extended ramping up period. Officers may need five to ten years before they possess the level of experience needed to fully respond to the sector's expectations. By the same token, a similar period of time is needed to develop the kind of relationships characterized by mutual trust needed to function optimally. This includes opportunities for informal encounters in addition to formal interaction. The concern, in practical terms, relates to succession: will SED be able to hire officers with the necessary skills and experience to maintain current service levels as existing officials retire? One AAFC interviewee stated, “succession planning is a key concern. We rely a lot on the knowledge of our people. We are the repository of knowledge. As people leave in the coming years we will have to make sure that their knowledge is transferred.”

At the moment, SED is producing outputs as expected and is achieving its outcomes. SED is able to produce high quality services and to adequately respond to ad hoc requests and crisis situations. However, the extent to which SED is successful in managing its human resources plan could have a significant impact on sustained program performance.

4.4.3 Performance monitoring and reporting

With the integration of SED as a sub-program of the PAA, a Performance Measurement Strategy (PMS) had to be developed. SED’s PMS has been in place for less than one year. Some monitoring activities were ongoing prior to the development of the PMS and the evaluation benefited from the products of these activities. However, the evaluation also found many evidence gaps. This is a sign that SED still has some distance to go in terms of fully implementing its PMS including collecting, analyzing and reporting on predetermined performance metrics.

The concern in relation to weaknesses in performance monitoring and reporting is twofold. First, the lack of performance data may hinder SED's ability to measure its success. Since the integration of SED into the PAA creates greater obligation to report on performance, SED could be unable to support its claims respecting the value of its activities, outputs and outcomes based on empirical evidence. Second, the lack of performance data could pose a risk for SED in its decision-making process, particularly when working in a complex environment that is subject to volatility. Readily available performance data is important in this context because decisions often need to be taken quickly and consider various factors. Overall, good performance data could contribute to an ongoing improvement in SED’s efficiency.

5.0 Evaluation conclusion

This section of the report presents conclusions based on the findings outlined in the previous sections. The information is structured along the main evaluation issues.

5.1 Relevance

Sector Engagement and Development (SED) is relevant. It continues to meet a significant need, and is well aligned with government and departmental priorities and roles. The need for SED is, in fact, likely to grow based on such trends as a projected increase in technical trade barriers, and the continued emergence of new technology-based industries within the sector.

5.2 Performance - effectiveness

The evaluation found that SED generated its expected outputs as well as outputs not included in the SED logic model; in particular, the latter group of outputs include SED's numerous and varied responses to ad hoc requests and crisis situations.

SED has met all three immediate outcomes, including:

• Canadian producers, companies and industry organizations are better informed;
• The sector has the capacity and exercises leadership to enhance competitiveness and market performance within key domestic and international markets; and,
• Government programs, policies and regulations respond to sector priorities and competitiveness needs.

Through the generation of its outputs, SED has been able to better inform Canadian producers, companies and industry organizations, and to increase the sector's capacity to exercise more leadership in enhancing competitiveness and market performance domestically and internationally. SED's outputs were also able to inform government policy-makers and regulators on the sector's priorities and competitiveness needs.

By ensuring better informed sector stakeholders, an enhanced leadership, and sound policy and regulation making, SED has been able to meet all three intermediate outcomes. As a result of SED's involvement in the sector, the following intermediate outcomes have been met:

• Canadian producers, companies and industry organizations make better decisions;
• Canadian agriculture and agri-food sector addresses priority issues, including trade-related issues to advance competitiveness; and,
• Stakeholders implement strategies and use tools to manage changes associated with external forces.

Despite the challenges to attribute macro-level performance directly to SED activities, the evaluation found that SED contributed to both of its end outcomes. The products, services and support provided by SED contributed to the end outcomes of:

• Improved producer, company and industry organization access to markets, responsiveness to opportunities and demands, and enhanced competitiveness domestically and internationally; and,
• Agriculture and agri-food sector successfully adapts to changing and emerging global and domestic opportunities and issues.

5.3 Performance - efficiency and economy

Based on the evidence reviewed, SED appears to be providing good value for money. Outputs appear to be generated economically, and the evaluation evidence suggests that SED's value to the sector outweighs its cost to taxpayers.

5.4 Design and delivery

SED is well organized and flexible enough to simultaneously meet its ongoing obligations to provide regular analyses and reports while addressing ad hoc requests and crisis situations as they arise. SED officials collectively possess a singularly comprehensive and deep knowledge of the sector combined with an ability to acquire and analyze data from a comprehensive range of sources. SED officials maintain active working relations with key stakeholders representing every aspect of the sector from across the country. These fundamental attributes have enabled SED to play a unique and valued role within Canada's agriculture and agri-food sector. SED, in coordination with Trade and Market Expansion (TME), is able to generate information and connect sector stakeholders to create value and solve problems for the sector.

SED efficiently provides a wide range of services, many of which have been delivered in one form or another by Agriculture and Agri-Food Canada (AAFC) for decades. Beneficiaries of services know that services come from AAFC and, perhaps, from Market and Industry Services Branch (MISB), but not that the work comes from SED. Having SED added to the Program Alignment Architecture (PAA) in 2013-14 creates an opportunity for SED to continue to dialogue with other areas of the Department to forge a link between its activities and its new name, and report on activities that others within the Department may not be aware are under the responsibility of SED.

The integration of SED as a sub-program in the PAA created the requirement to develop a performance measurement strategy for all of its activities. While SED has made progress in terms of performance monitoring, there is room for improvement in this regard. By strengthening its performance management strategy (PMS), SED will ensure its ability to support its claims with respect to its outputs and outcomes, and that its decision-making process relies on readily available information.

Although some efforts are needed to enhance SED's visibility within the Department as well as to strengthen its PMS, SED can count on knowledgeable and dedicated officials that are able to generate expected outputs and meet outcomes. However, the evidence suggests that the capacity of SED to serve its clientele effectively could be compromised once experienced officials begin to retire. In fact, the key role played by individual SED officials underlines challenges related to succession planning. The evaluation found that the extent to which SED is successful in managing its human resources plan could have a significant impact on its sustained program performance.

6.0 Issues and recommendations

The evaluation includes the following issues and recommendations:

Issue 1

Sector Engagement and Development (SED) counts on knowledgeable and dedicated officials that are able to generate expected outputs and meet outcomes. However, the evidence suggests that the capacity of SED to serve its clientele effectively could be compromised once experienced officials begin to retire.

Recommendation

As SED owes much of its success to the knowledge and working relations nurtured by its staff, many of whom have been with the Department for many years and may be approaching retirement, the Market and Industry Services Branch (MISB), in collaboration with the Corporate Management Branch, needs to develop and implement a succession and knowledge management strategy to maintain its reputation, quality of work and impact on industry.

Management response and action plan

Agreed: The Market and Industry Services Branch will develop an Integrated Human Resource Plan to ensure that it has the continued capacity to deliver quality programs and services, maintain expertise, equip its workforce with the appropriate competencies and achieve organizational excellence. The Integrated Human Resource Plan will be approved by MISB senior management and implemented by the end of August 2016.

Target date for completion: August 31, 2016.

The Market and Industry Services Branch will also develop succession plan to fill critical positions to ensure a continuous flow of qualified candidates. Succession plans will be implemented by October of next fiscal year (2016-17).

Target date for completion: October 31, 2016.

The Market and Industry Services Branch will fully implement the Government of Canada's Information Management initiatives (i.e. Knowledge Workspace and Directive on Recordkeeping) to ensure effective knowledge management by the end of March 2018.

Target date for completion: March 31, 2018.

Responsible positions

• Assistant Deputy Minister, Market and Industry Services Branch;
• Director General, Regional Operations Directorate, Market and Industry Services Branch; and,
• Director General, Sector Development and Analysis Directorate, Market and Industry Services Branch.

Issue 2

The integration of SED as a sub-program in the PAA created the requirement to develop a performance measurement strategy (PMS) for all of its activities. SED's PMS has been in place for less than one year. Some monitoring activities were ongoing prior to the development of the PMS and the evaluation benefited from the products of these activities. However, the evaluation also found many evidence gaps. This is a sign that SED still has some distance to go in terms of fully implementing its PMS including collecting, analyzing and reporting on predetermined performance metrics.

Recommendation

Although progress has been made, the evaluation found weaknesses in performance monitoring and reporting. It is recommended that the Market and Industry Services Branch review SED's current performance measurement strategy to ensure that meaningful performance measures are developed. These measures include indicators for program activities, outputs and outcomes so that future, more robust assessments of program effectiveness, efficiency and economy can be undertaken.

Management response and action plan

Agreed: The Market and Industry Services Branch will undertake a review of the SED's PMS with a view to refining its logic model and performance measures to ensure that performance is effectively monitored and reported. Updated PMS will be approved by MISB senior management by the end of the fiscal year (2015-16).

Target date for completion

March 31, 2016.

Responsible positions

• Assistant Deputy Minister, Market and Industry Services Branch;
• Director General, Regional Operations Directorate, Market and Industry Services Branch; and,
• Director General, Sector Development and Analysis Directorate, Market and Industry Services Branch.

Annex A: Evaluation Matrix

Issue: relevance

Question: Within the Canadian agriculture, agri-food and agri-based products sector, what are the needs addressed by the Sector Engagement and Development (SED) program? Within the sector, which population segments are targeted by the program, and is this the appropriate target group? See table below.

Indicators	Evidence Sources - Admin data	Evidence Sources - Doc review	Evidence Sources - Lit Review	Evidence Sources - Case Studies	Evidence Sources - Media Analysis	Evidence Sources - Interviews
Yes: Evidence source applies No: Evidence source does not apply Interviews types are coded as follows: 1 = SED Program Representatives 2 = AAFC Internal Clients 3 = Portfolio Partners and Other Federal Government Departments (OGDs) 4 = Provincial and Territorial Representatives 5= Industry Representatives
1.1. Demonstrated or expressed need for SED outputs on the part of industry representatives and other targeted population segments	No	Yes	Yes	No	No	Yes (1,2,3,4,5)
1.2. Presence or absence of other programs that complement or duplicate SED outputs	No	Yes	Yes	No	No	Yes (1,2,3,4,5)

Question: How is the SED program aligned with federal government priorities? See table below.

Indicators	Evidence Sources - Admin data	Evidence Sources - Doc review	Evidence Sources - Lit Review	Evidence Sources - Case Studies	Evidence Sources - Media Analysis	Evidence Sources - Interviews
Yes: Evidence source applies No: Evidence source does not apply Interviews types are coded as follows: 1 = SED Program Representatives 2 = AAFC Internal Clients
2.1. SED objectives correspond to recent/current federal government priorities as defined in the federal budget, the speech from the throne, and/or other relevant documents, including AAFC documentation	No	Yes	No	No	No	Yes (1,2)

Question: What is the nature of the federal government's role and mandate to deliver the SED program, and to what extent do the objectives of the SED program align with this role/mandate? See table below.

Indicators	Evidence Sources - Admin data	Evidence Sources - Doc review	Evidence Sources - Lit Review	Evidence Sources - Case Studies	Evidence Sources - Media Analysis	Evidence Sources - Interviews
Yes: Evidence source applies No: Evidence source does not apply Interviews types are coded as follows: 1 = SED Program Representatives 2 = AAFC Internal Clients 3 = Portfolio Partners and Other Federal Government Departments (OGDs) 4 = Provincial and Territorial Representatives 5= Industry Representatives
3.1. SED objectives correspond to the federal government's related role and responsibilities	No	Yes	No	No	No	Yes (1,2)

Issue: program performance

Question: To what extent has SED demonstrated and quantified generation of:

1. Market information, analysis and intelligence (including reports on market information and market analysis relevant to Advance Payments Program, Price Pooling Program, and the Canadian Wheat Board).
2. Industry analytical and technical reports (including benchmarking, performance analysis, regulatory impact assessments, and organization position analysis).
3. Value Chain Roundtables, International Market Engagement Teams and other Industry-government or F/P/T engagement processes.
4. Technical advice and services to the sector and to governments (including project technical reviews and business plan reviews).
5. Analysis of P/T governments' positions and activities.
6. To what extent is SED making progress or has the potential to achieve its expected outcomes:
   • 6.1 Immediate Outcomes
     • 6.1.1 Are Canadian producers companies and industry organizations being better informed?
     • 6.1.2 Is the sector having the capacity and exercising leadership to enhance competitiveness within key domestic and international markets?
     • 6.1.3 Are government programs, policies and regulations responding to sector priorities and competitiveness needs?
   • 6.2 Intermediate Outcomes
     • 6.2.1 Are Canadian producers, companies and industry organizations making better?
     • 6.2.2 Are stakeholders implementing strategies and using tools to manage changes associated with external forces?
   • 6.3 End Outcomes
     • 6.3.1 Improved producer, company and industry organization access to markets, responsiveness to opportunities and demands, and enhanced competitiveness domestically and internationally?
     • 6.3.2 The agriculture and agri-food sector is successfully adapting to changing and emerging global and domestic opportunities and issues?
7. Are there any unintended outcomes (positive of negative)? What is their net effect on the program?

See table below.

Indicators	Evidence Sources - Admin data	Evidence Sources - Doc review	Evidence Sources - Lit Review	Evidence Sources - Case Studies	Evidence Sources - Media Analysis	Evidence Sources - Interviews
Y: Evidence source applies N: Evidence source does not apply Interviews types are coded as follows: 1 = SED Program Representatives 2 = AAFC Internal Clients 3 = Portfolio Partners and Other Federal Government Departments (OGDs) 4 = Provincial and Territorial Representatives 5= Industry Representatives
4.1. Demonstrated and quantified generation of market information, analysis and intelligence (including reports on market information and market analyses including Advance Payments Program, Price Pooling Program and Canadian Wheat Board)	Yes	Yes	No	Yes	No	Yes (1)
4.2. Demonstrated and quantified generation of industry analytical and technical reports (including benchmarking, performance analyses, regulatory impact assessments, and organization position analyses)	Yes	Yes	No	Yes	No	Yes (1)
4.3. Demonstrated and quantified generation of Value Chain Roundtables, International Market Engagement Teams and other industry-government or F/P/T engagement processes	Yes	Yes	No	Yes	No	X (1)
4.4. Demonstrated and quantified generation of technical advice and services to the sector and to governments (including project technical reviews, business plan reviews)	Yes	Yes	No	Yes	No	Yes (1)
4.5. Demonstrated and quantified generation of analysis of P/T governments' positions and activities	Yes	Yes	No	Yes	No	Yes (1)
5.1. Performance data as outlined in the Performance Measurement Strategy Framework. 5.2. Evidence of views related to factors outside the SED program that have influenced the achievement of intended outcomes 5.3. Views on the extent to which intended outcomes have been achieved as a result of the SED program	Yes	Yes	Yes	Yes	Yes	Yes (1,2,3,4,5)
6.1. presence/ absence of unintended outcomes 6.2. Views on whether unintended outcomes occurred	Yes	Yes	No	Yes	Yes	Yes (1,2,3,4,5)

Issue: demonstration of efficiency and economy

Questions:

1. What were the costs of activities/outputs undertaken by the SED program, including Full-Time Equivalents costs, plus all other expenditures?
2. Does evidence exist that program resources were acquired at the lowest cost consistent with the required quality, quantity, appropriateness and timeliness?

See table below.

Indicators	Evidence Sources - Admin data	Evidence Sources - Doc review	Evidence Sources - Lit Review	Evidence Sources - Case Studies	Evidence Sources - Media Analysis	Evidence Sources - Interviews
Y: Evidence source applies N: Evidence source does not apply Interviews types are coded as follows: 1 = SED Program Representatives 2 = AAFC Internal Clients
7.1. Views on the appropriateness of program activities, processes and governance structures	N	N	N	Y	N	Y
7.2. Clearly defined program structure, and analysis of actual program operational costs in relation to the production of outputs	Y	Y	N	N	N	N
7.3. Views on whether the cost of producing program outputs is as low as possible	N	N	N	N	N	Y(1,2)
7.4. Analysis of program activities/components in order to identify which are critical, important and non-important to the achievement of the overall objectives of the program	Y	Y	N	N	N	N

Question: What was the value of outcomes achieved - immediate, intermediate, and end - and how does that compare to program costs? See table below.

Indicators	Evidence Sources - Admin data	Evidence Sources - Doc review	Evidence Sources - Lit Review	Evidence Sources - Case Studies	Evidence Sources - Media Analysis	Evidence Sources - Interviews
Y: Evidence source applies N: Evidence source does not apply Interviews types are coded as follows: 1 = SED Program Representatives 2 = AAFC Internal Clients 3 = Portfolio Partners and Other Federal Government Departments (OGDs) 4 = Provincial and Territorial Representatives 5= Industry Representatives
8.1. Extent to which program intended outcomes have been achieved at the least possible program cost	Y	Y	N	Y	N	Y
8.2. Views on whether good value is being obtained with respect to the use of public funds	Y	Y	Y	Y	N	Y(1,2,3,4,5)
8.3. Evidence of/views on whether there are alternative program models that would achieve the same expected outcomes at a lower-cost	N	N	Y	N	N	Y(1,2,3,4,5)
8.4. Views on how the efficiency of program activities could be improved	N	N	N	N	N	Y(1,2,3,4,5)

Issue: design and delivery

Question: Did SED program activities and outputs vary from the design of the program and, if so, what was the impact on outcomes? See table below.

Indicators	Evidence Sources - Admin data	Evidence Sources - Doc review	Evidence Sources - Lit Review	Evidence Sources - Case Studies	Evidence Sources - Media Analysis	Evidence Sources - Interviews
Y: Evidence source applies N: Evidence source does not apply Interviews types are coded as follows: 1 = SED Program Representatives 2 = AAFC Internal Clients 3 = Portfolio Partners and Other Federal Government Departments (OGDs) 4 = Provincial and Territorial Representatives 5= Industry Representatives
9.1. SED documented activities correspond to the program design	N	Y	N	Y	N	Y(1)

Annex B: Sector Engagement and Development Program logic model

Description - Figure 1

Annex B presents the logic model for the sector engagement and development program at AAFC. The contents of the logic model are as follows:

• The activities of SED are:
  • Data gathering from markets and industries, and analysis:
    • Quantitative/empirical data (primary and secondary)
    • Qualitative data and intelligence
  • Industry relations and analysis:
    • Relationship building with provincial and municipal governments, industry organizations and critical companies
    • Analysis of industry competitiveness including the impacts of government policies, programs and regulations
  • Sector development:
    • Analysis and resolution of sector issues
    • Oversight of F/P/T agreements
    • Assistance with industry-led activities
    • Assistance in identifying and resolving technical and trade issues
• The outputs of SED are as follows:
  • Markets and information, analysis and intelligence (including reports on market information and market analysis including APP, PPP and CWB)
  • Industry analytical and technical reports (including benchmarking, performance analysis, regulatory impact assessments, and organization position analysis)
  • VCRT, IMET and other industry-government or F/P/T engagement processes
  • Technical advice and services to the sector and to governments (including project technical reviews, business plan reviews)
  • Analysis of P/T governments' positions and activities
• The immediate outcomes of SED are as follows:
  • Canadian producers, companies and industry organizations are better informed
  • The sector has the capacity and exercises leadership to enhance competitiveness and market performance within key domestic and international markets
  • Government programs, policies and regulations respond to sector priorities and competitiveness needs
• The intermediate outcomes of SED are as follows:
  • Canadian producers, companies and industry organizations are better informed
  • Canadian agriculture and agri-food sector addresses priority issues, including trade-related issues to advance competitiveness
  • Stakeholders implement strategies and use tools to manage changes associated with external forces
• The end outcomes of SED are as follows:
  • Improved producer, company and industry organization access to markets, responsiveness to opportunities and demands, and enhanced competitiveness domestically and internationally
  • Agriculture and agri-food sector successfully adapts to changing and emerging global and domestic opportunities and issues