Evaluation of the Market Information Program (2014 to 2019)

Abbreviations

AAFC
Agriculture and Agri-Food Canada
BRM
Business Risk Management
CBSA
Canada Border Services Agency
CFIA
Canadian Food Inspection Agency
MIP
Market Information Program

Executive summary

Purpose

The Office of Audit and Evaluation of Agriculture and Agri-Food Canada (AAFC) undertook an evaluation of the Market Information Program (MIP) to assess program relevance, design, delivery, and effectiveness.

Scope and methodology

The evaluation assessed the MIP covering a five-year period between 2014–15 and 2018–19, using a variety of methods including: a document review; a literature review; a comparative review; case studies; and interviews with AAFC staff, other federal departments and agencies, and industry and provincial government stakeholders.

Background

While not a formal program, the MIP undertakes a number of activities that cover the collection, analysis, and dissemination of relevant market information on the agriculture and agri-food sector to support decision-making among AAFC internal stakeholders, as well as external stakeholders involved in the sector. To undertake its data collection and analysis activities, the MIP collaborates with other AAFC data specialists including those in the Research and Analysis Directorate, the Business Risk Management Programs Directorate, and the Market Access Secretariat, as well as with other federal departments and agencies, and with industry.

Findings

  • The MIP provides relevant information to AAFC and external stakeholders involved in the agriculture and agri-food sector, which would otherwise not be available. The MIP occupies a distinct niche, and its work does not duplicate the activities of other areas.
  • Supporting the growth and competitiveness of the agriculture and agri-food sector in Canada is at the core of the AAFC mission. The MIP supports a range of policies and programs that directly contribute to this mission.
  • The MIP has a solid track record in providing the market information needed by both AAFC and external stakeholders to support sound and strategic decisions. It provides a level of market intelligence that is not available from any other source.
  • The lack of a formal governance structure for the MIP may negatively impact the long term sustainability, relevance, and effectiveness of the MIP.
  • The MIP has yet to undertake a gender-based analysis plus review of its activities.

Conclusion

The evaluation found that that the MIP makes a significant contribution to building a comprehensive understanding of the agriculture and agri-food market, through its subject-matter expertise and its capacity to successfully collaborate with other key stakeholders. The MIP’s work is not duplicated elsewhere, and is directly aligned with the priority of AAFC and the federal government to support the growth and sustainability of the agriculture and agri-food sector in Canada.

The evaluation confirms that the information provided by the MIP is directly integrated in the decision-making and management processes of both internal and external stakeholders. The main challenge that the MIP faces is the absence of a formal governance and management structure, informed by a formal needs assessment, within which it can operate. This puts pressure on the MIP’s ability to maintain levels of operations including key data sources and sector expertise, while adapting to new business lines and product/markets (for example, alternative protein product). In the case of employee turnover, the MIP may be vulnerable in its ability to continue functioning at capacity for a sustainable period. Finally, the lack of a formal governance structure undermines the MIP’s ability to ensure adequate data management, documentation, and transparency processes.

The evaluation concluded that AAFC would benefit from establishing a formal governance structure to provide clarity about the MIP roles and responsibilities, within the wider AAFC context. AAFC would benefit from having a more coordinated approach to providing data collection and data analytics services to respond to stakeholder needs to ensure long-term sustainability, relevance, and effectiveness of the services AAFC provides.

Recommendations

  • Recommendation 1: The Assistant Deputy Minister of the Market and Industry Services Branch should develop a formalized governance structure for the Market Information Program, as part of a department-wide approach to data collection and data analytic services.
  • Recommendation 2: The Assistant Deputy Minister of the Strategic Policy Branch should lead on the development and implementation of a department-wide approach to data collection and data analytic services.
  • Recommendation 3: The Assistant Deputy Minister of the Public Affairs Branch should lead on the development of a strategy and changes to the AAFC’s digital channels, including its external website to ensure a user-centric, AAFC-wide approach to making relevant market information data and analysis accessible and useable.

1.0 Introduction

Following a request in 2019 by the Market and Industry Services Branch of AAFC, an evaluation of the Market Information Program (MIP) was included in the Agriculture and Agri-Food Canada (AAFC) Office of Audit and Evaluation 2018–19 to 2022–23 Integrated Audit and Evaluation Plan.

2.0 Scope and methodology

The evaluation focused on the relevance, design, delivery, and effectiveness of the MIP during a five-year period between 2014–15 and 2018–19. A particular focus was placed on the expected impact of the MIP on AAFC and external stakeholders (for example, farmers, producers, processors, associations, etc.) who use the MIP’s market information to support their decisions, while also considering the overall AAFC data and analytical continuum.

The evaluation assessed the MIP using a variety of methods including: a document review; literature review; comparative review; case studies; and interviews with AAFC staff, other federal departments and agencies, and industry and provincial government stakeholders. For the purposes of this evaluation, “information” includes numerical data and the analysis and reports derived from such data. Other data gathering and analysis functions within AAFC, including, but not limited to, those in the Research and Analysis Directorate, the Business Risk Management Programs Directorate, and the Market Access Secretariat were outside of the scope and only assessed for complementarity or as part of their interface with MIP. For a more detailed description of the evaluation methodology, see Annex A.

3.0 Program profile

3.1 Program context

As a multidimensional and trade-intensive sector of the economy, agriculture and agri-food activities require detailed and ongoing market data and information to operate successfully. Federal departments and agencies, as well as provincial ministries involved in agriculture and agri-food rely on this information to implement and manage their activities and programs in support of the overall sector. Farmers, producers, processors, and other related stakeholders use market data and information to make decisions for planning, monitoring, and sales purposes. Sector associations employ this information to monitor and plan, and make strategic choices on how best to support the growth and competitiveness of their sectors. Academics turn to this information to document and analyze trends, and support a range of research and development activities.

Canada, like every nation heavily involved in agriculture and agri-food activities, has built a strong repository of data and information to support the sector. Canada has conducted a census of agriculture every five years since 1956. Since the early 1900s, AAFC has produced market data and information that responds specifically to its needs and to the needs of the sector more generally. Statistics Canada and other stakeholders have supported the MIP in these efforts, and Statistics Canada has also gathered a wide range of non-market data and information on agriculture and agri-food activities to support environmental and public health programming.

3.2 Program objectives and activities

Within AAFC, there are several areas that have data gathering and analysis functions. One area is the MIP, which is part of the Sector Engagement and Development Program, and which is administered in the Sector Development and Analysis Directorate of the Market and Industry Services Branch. The MIP comprises market analysts from both the Animal Division and the Crops and Horticulture Division with support from sector specialists within the Sector Development and Analysis Directorate. In its work, the MIP also leverages departmental specialists including, but not limited to, those in the Research and Analysis Directorate, the Business Risk Management Programs Directorate, the Market Access Secretariat, to supplement market analysis for both internal and industry clients (see Annex B for the MIP Operational Map).

The MIP is not a “program” in the traditional sense of the word, and is not identified as a program in AAFC’s Departmental Results Framework and Program Inventory, as it is more a “service.”Footnote1 The MIP is a collection of activities undertaken to provide relevant market information to support the growth and competitiveness of the agriculture and agri-food sector in Canada. The MIP is identified on AAFC’s Priority Service Identification Tool Results – Service Inventory.

More specifically, the MIP’s goals are to

  • Provide data and market information products to internal and external clients that enable improved marketing and investment decisions, research, programming, and policy decisions;
  • Provide data and market information products that accurately reflect supply and demand;
  • Provide stakeholders with data and information in a timely manner that allows for enhanced decision-making; and
  • Ensure the delivery of impartial statistics and analysis while respecting the confidentiality of data providers.

To pursue these goals, the MIP undertakes a range of activities that fall under three broad categories: data gathering; analysis; and publication and distribution (See Annex C for the MIP Logic Model).

Data gathering

The MIP team is actively engaged in gathering the market information required for its analytical purposes. This is done by collaborating with a number of federal and provincial organizations and the industry itself. A total of 27 Memorandums of Understanding and Data Sharing Agreements signed by AAFC, are in place to provide the framework and conditions under which collaboration takes place with various federal and provincial departments and agencies, industry, and academia. For example, the 2014-2019 Memorandum of Understanding between AAFC and Statistics Canada on the Joint Agriculture and Agri-Food Statistical Program specifies AAFC’s role in consulting with industry and federal, provincial, and territorial partners to identify data requirements to support policy setting, program development, and industry priorities. This Memorandum of Understanding also outlines AAFC’s responsibility to undertake appropriate research and analysis, and provide survey and data verification activities. MIP and other data collection and analysis areas within AAFC use this specific MOU, and MIP accesses at least 77 percent of the data sets identified in this agreement.

The MIP also obtains data from other organizations which collect primary market data. These include the Canada Border Services Agency (CBSA) which collects data on imports and exports of commodities; the Canadian Food Inspection Agency (CFIA) which collects slaughter data; the Canadian Dairy Information Centre which provides information on dairy production and sales; some provincial ministries or agencies which provide market data (for example, Manitoba and Alberta); and private firms which provide a range of commodity prices.

Finally, the MIP itself collects primary data by collaborating with industry (for example, producers, processors, wholesalers, distributors, etc.), which directly shares commodity prices and other relevant commodity dataFootnote2, in accordance with data sharing agreements that protect the confidentiality of data providers. The sharing of information by industry is done entirely on a voluntary basis. The MIP team engages in ongoing dialogues with stakeholders, participates in conferences and meetings, and engages industry in discussions about its emerging data requirements. Once the MIP team has collected the data from industry, significant time and effort is needed to ensure data accuracy to be able to use it for analytical purposes.

The collaboration among all organizations involved in collecting the required market data and information is multi-faceted. In some cases, the MIP uses the data gathered by other organizations (Statistics Canada, CBSA, and CFIA). In other cases, the MIP works directly with industry to collect primary data, which it then uses and shares with partner organizations such as Statistics Canada to support the building of datasets that may, in turn, be used by the MIP. Data collected by the MIP is also shared internally within AAFC to be used by other data and analytical areas.

Analysis

The MIP provides analysis at various levels. In some cases, such as commodity prices, the MIP team limits its analytical work to gathering and validating the data and establishing the most appropriate publishing strategy (frequency, level of detail, regional distribution, etc.). In other cases, the MIP team proceeds with further analysis to provide forecasts, such as the outlooks for field crops, that the MIP team publishes regularly. The MIP team also addresses specific issues and produces one-time snapshots of segments of the industry, such as the 2018 statistical overview of the Canadian greenhouse vegetable industry.

The MIP team produces specific analyses in response to emerging situations such as trade disputes or market disruptions. These targeted analyses are done for AAFC’s decision-makers, and for stakeholders in the agriculture and agri-food sector, as applicable.

Publication and distribution

Reports that the MIP team prepares are typically published on the “Canadian agri-food sector intelligence” section of AAFC’s website, along with products and reports prepared by other data analytics teams within AAFC. Thus, this information and analysis is available without restriction and at no cost to users.

4.0 Relevance

This section summarizes the evaluation findings related to the relevance of the MIP. It focuses on the continued need for the MIP services and its alignment with federal and departmental roles, responsibilities, and priorities. The section explores the extent to which the MIP plays a complementary role to what other stakeholders may be producing in terms of market information.

4.1 Continued need for the program

The MIP provides relevant information to AAFC and external stakeholders involved in the agriculture and agri-food sector, which would otherwise not be available. The MIP occupies a distinct niche, and its work does not duplicate the activities of other stakeholders.

The agriculture and agri-food sector of the Canadian economy relies on strong access to national and international markets, which triggers the need for sustainable growth, innovation, and competitiveness. Market information constitutes a foundational component in supporting decision making within the agriculture and agri-food landscape.

The information that the MIP produces is aligned with the information that internal and external stakeholders need to make decisions and it is complementary, though not duplicative, of the information that other organizations provide. The information and products developed by the MIP complement information and products developed by other stakeholders.

MIP as high priority service

The strategic role of the MIP is recognized through AAFC’s designation of it as a priority service, in accordance with federal government’s Service Policy and the Guideline on Service Management. In reviewing its service inventory, AAFC attributed the highest ranking to the Domestic Statistics and Marketing Information service, which hosts MIP activities. Moreover, AAFC’s Market and Industry Services Branch has included the MIP in its Business Continuity Plan as requiring that services provided by the MIP be restored within two business days of a disruption.

The evaluation found no evidence of alternative services available to access the same breadth of centralized data and information or commodity expertise. At the time of the evaluation, evidence confirms that stakeholders considered the MIP to be a credible and trustworthy source of information. In the absence of the MIP, evidence suggests that both AAFC and external stakeholders would be challenged to access key data and information to support decision-making. Without the MIP, stakeholders would need to develop an alternative approach to finding data from a variety of sources and develop analytical activities to obtain comparable information. AAFC’s understanding of the sectors would be significantly hampered, negatively impacting policy and program development, and the absence of timely market intelligence would impede AAFC’s ability to address market disruptions. The evaluation found that an absence of the MIP could have negative impacts on the economic or financial well-being of industry sector clients, as well as other federal and provincial/territorial partners.

Unique service

To assemble the required market data and information, each country that is significantly involved in agriculture and agri-food activities has built a market information system that presents unique characteristics. In the United States the system is decentralized, whereas other countries, such as Australia, China, or Canada, have opted for more centralized systems. In all cases, these market information systems rely on multiple contributors. No country relies on a single organization for collecting, processing, analyzing, and distributing this information.

By far, the most significant source of primary data on agriculture and agri-food activities in Canada is Statistics Canada. Through its Agriculture and Agri-Food Statistical Program, Statistics Canada provides over 40 different data sets covering several dimensions of agriculture and agri-food activities, including but not limited to market data. This is in addition to all the data collected through the census of agriculture. Through the Memorandum of Understanding it has signed with Statistics Canada, AAFC provides just under $33-million over five years (2014–15 and 2018–19) to support the Agriculture and Agri-Food Statistical Program. The evaluation found that about two-thirds of the MIP reports use the data provided by Statistics Canada. Statistics Canada also relies on AAFC administrative data (i.e. slaughter, carcass weights, prices, etc.) for some of its agricultural reporting, which is provided to Statistics Canada from various data collection areas within AAFC.

Market information is multi-faceted and highly interdependent within Canada’s agriculture and agri-food landscape. The evaluation findings indicate that, within this landscape, the MIP occupies a distinct niche and its contribution does not duplicate other contributors. The MIP team makes a unique contribution to the market information available in Canada through its expertise in collecting some of the primary data, selecting the relevant data from all available datasets, nourishing a close and productive relationship with all key stakeholders (for example, Statistics Canada, Canada Border Services Agency, Canadian Food Inspection Agency, and industry), and offering relevant analysis. Interview evidence confirms that stakeholders assess the MIP’s provision of micro-level, sectoral business data and information as complementary to the economic analysis and forecasting performed by AAFC’s Research and Analysis Directorate. The MIP’s micro-level data feeds into Research and Analysis Directorate’s forecasting that supports Business Risk Management programs.

The evaluation confirms that the MIP provides relevant and needed market information to stakeholders, and the contribution of the MIP does not duplicate, to any meaningful extent, contributions from other organizations.

4.2 Alignment with AAFC and government roles and priorities

Supporting the growth and competitiveness of the agriculture and agri-food sector in Canada is at the core of the mission of AAFC. The MIP directly supports a range of policies and programs that directly contribute to this mission.

AAFC’s mission is to provide leadership in the growth and development of a competitive, innovative, and sustainable Canadian agriculture and agri-food sector. The Minister of Agriculture and Agri-Food is mandated to undertake key initiatives that include: a review of business risk management programs; the implementation of current trade negotiations with appropriate compensation measures as applicable; the expansion of new export markets; an enhanced capacity to react to trade disputes and provide faster short-term support to affected industries as applicable; and promote Canadian food at home and abroad.

The evaluation found that the MIP directly supports the policy and programming work undertaken by AAFC to uphold its mandate, and responds to the policy and programming goals assigned to the Minister of Agriculture and Agri-Food. It does this by providing expertise and market intelligence that would otherwise not be available.

Evidence indicates that the MIP activities fall within the Department’s legislated function under the Department of Agriculture and Agri-Food Act to undertake research as required on agriculture and agri-food activities and products.

5.0 Program design and delivery

The MIP has a solid track record in providing the market information needed by both AAFC and external stakeholders to support sound and strategic decisions. It provides a level of market intelligence that is not available from any other source.

The MIP generates a considerable level of activity. In 2018–19, the MIP team dealt with 2,800 email requests, produced 650 web-based reports, and 140 custom reports. The Program’s web site received over 2-million hits. Evaluation evidence confirms that the MIP team is effectively responding to information needs and requests, and is providing timely and credible information and analysis for AAFC and external stakeholders. The following sub-sections provide further insights on the impact of the MIP within AAFC, and its contribution to the work of stakeholders.

5.1 Beyond data: providing market intelligence

The Market Information Organization of the Americas defines an agricultural market information system as a structured set of units that collect, process, analyze, and disseminate agriculture market data. Such a system rests on two fundamental requirements: having access to the relevant data; and having the capacity to make sense of the data. This latter requirement entails the ability to contextualize the data, detect probable inaccuracies based on previous trends, and focus on the most relevant strategies to efficiently construct and communicate the resulting information.

While the MIP plays a critical role on both fronts, the evaluation found that it is the MIP team’s expertise to assess, gather, analyze, and make sense of the data that underlines its distinctive characteristic. Individuals from AAFC, other federal departments and agencies, provincial governments, and the industry who were consulted as part of this evaluation, repeatedly emphasized the knowledge and engagement of MIP team members, and the high level of credibility attached to their products. The evaluation did not identify other organizations outside of AAFC that possess the level and breadth of expertise found within the MIP team when it comes to understanding agriculture and agri-food market information.

It was found that MIP team members have a close working relationship with stakeholders. Stakeholders including other government departments and industry collaborate closely with the MIP team to verify their understanding of the MIP’s cleaned and aggregated data. Beyond the information published on the “Canadian agri-food sector intelligence” section of AAFC’s website, stakeholders noted they particularly valued being able to directly contact a member of the MIP team for further clarifications or insights. Building trusting and lasting relationships with industry was found to be a key component of the success of the MIP activities.

There is a largely held view among stakeholders consulted during the evaluation that the analysis provided by the MIP team is unbiased, and that MIP products are credible. This contrasts with stakeholder perception that analytical market publications produced by non-governmental organizations may reflect the organization’s specific interests. The evaluation found that this unique positioning of the MIP as a credible source is important considering the potential commercial and monetary stakes of decisions based on the information it produces.

Figure 1 Market Information Program Stakeholders and Information Uses, was created by the Office of Audit and Evaluation to map the range of the MIP’s internal and external stakeholders, and their use of the market information provided. Further illustrations are provided in the following sub-sections.

Figure 1: Market Information Program stakeholders and information uses

Description of this image follows

Description of above image

Figure 1 presents a diagram showing the flow of information from the Market Information Program to its internal and external stakeholders.

Source: Created by OAE in support of the MIP Evaluation, February 2020

5.2 Internal stakeholders

The MIP information is integrated in many of the policy and programming activities that AAFC undertakes, thereby providing support to AAFC internal stakeholders.

Senior management

The MIP provides ongoing sector information and snapshots to the AAFC Minister’s Office staff and the Deputy Minister’s Office staff so that Senior Management is better informed of current trends and to support them in anticipating emerging issues, which may range from trade disruptions to environmental or public health issues. For instance, while it is outside the period covered by the evaluation, Senior Management tasked the MIP to assist in the analysis of the impact of the coronavirus disease (COVID-19) outbreak on the Canadian supply chain. A case study, in the box to the right, notes an in-scope example. The MIP also supports the Deputy Minister’s Office to respond to public inquiries through INFO (AAFC/AAC).

The MIP team receives direct requests for information from Senior Management via telephone, emails, or in-person meetings. This is an ongoing working relationship that is both reactive to sudden shifts or issues, and pro-active in supporting long-term planning.

MIP supports senior management during trade disruptions

China is the largest consumer and producer of pork in the world, and constitutes the third-largest export market for Canadian pork ($514 million in 2018). In May 2019, citing labelling problems and banned feed additive allegedly found in Canadian shipments, China suspended imports from two Canadian pork exporters. By June 25, 2019, China had halted all imports of Canadian pork. On that day, the impact of the ban was unknown, and the pressure quickly mounted from industry for the federal government to provide compensation.

Within two days, the MIP team co-operatively with sector experts provided AAFC’s Senior Management with an initial impact analysis of the ban. During the following weeks, 19 additional analyses were provided, on weekly and bi-weekly bases.

The MIP information provided an accurate and fully contextualized assessment of the impact of the ban, and provided effective reporting to support constructive discussions between AAFC and industry. In light of the short-term nature of the impact and the resilience of the industry, it was determined that the losses incurred were manageable, and no government programs were deemed necessary to address this specific issue. China lifted the ban on November 5, 2019.

Market and Industry Services Branch and Strategic Policy Branch

All programming and policy branches of AAFC can use MIP data and information. However, the Market and Industry Services Branch and Strategic Policy Branch are particularly reliant on the products that the MIP provides.

As an illustration, the MIP data and information are used to support trade negotiations, including for instance gaining a better understanding of the expected impact of trade agreements on the applicable sectors, and the mitigation strategy that may be required (for example, the impact of the Canada-European Union Comprehensive Economic and Trade Agreement on the sector). Interviewees noted that during and following the implementation of any trade agreement, both Canada and its trading partners use the publically available MIP data for monitoring purposes. The use of the same information source by both trading partners helps to clarify any concerns either partner may have, and helps to prevent trade challenges from occurring.

When market-related issues or disruptions occur, the MIP uses its information and market intelligence to provide direct support to policy decision-makers. Recent examples of such support include the trade dispute between Canada and China on canola seeds, which triggered changes to the Advance Payment Program, and on pork products, the impact of which was assessed by AAFC Senior Management but no targeted measures were deemed necessary.

Program implementation

The MIP supports the implementation and management of key programs such as the Advance Payment Program and the Price Pooling Program. The MIP conducts short-term forecasts of commodity prices that are used by multiple Programs/Units. The short-term forecasts are used to set advance rates for the Advance Payment Program, which establishes government liability for defaulted advances. As well, these prices are used to forecast farm income, as well as the setting of budgets for AgriStability and AgriInsurance.

The MIP supports the Advance Payment Program (APP)

The APP provides farm operators with access to low-interest cash advances, increasing their marketing flexibility and cash flow. Each year, 22,000 operators (one fifth of all Canadian operators) access the APP, which provides $2.2 billion in advances.

The MIP information is used to set the level of advances provided. In 2018-19, the MIP provided 1,246 advance rates covering specific commodities and provinces.

Research and analysis directorate

The Research and Analysis Directorate integrates the MIP data in a number of its analyses, such as the farm income forecasts. Interviewees noted that MIP data saves considerable time and resources (estimated at 1 Full Time Equivalent for a full month for each Summer and Fall forecast) that would otherwise be required by the Research and Analysis Directorate to gather all the required information needed for the grain forecasts. This is the equivalent to two months of Full Time Equivalent expenditures to support Research and Analysis Directorate’s forecasting activities for the Business Risk Management programs.

Evaluation evidence indicates that AAFC’s suite of Business Risk Management Programs (for example, AgriInsurance and AgriStability) requires commodity prices forecasted by the MIP to establish producers’ coverage levels, payouts, and, ultimately, federal contributions. In particular, the MIP provides all the price forecasting for the AgriInsurance (crop) programs offered by the provinces. These are federal-provincial-territorial cost-shared programs.

Another example of MIP outputs supporting analytical work is the Farm and Agriculture Regional Model that is used to produce the projection for domestic crops and livestock markets. A range of data sources are incorporated as part of this modeling process, including the MIP data.

The MIP potentially saves AAFC money

The MIP potentially saves AAFC significant amounts of over-contributions each year. These savings are a result of the MIP providing accurate commodity price data required by the Business Risk Management Programs (BRM) which constitute almost 60% of the department’s budget.

AgriInsurance contributions are based on per-unit prices from the MIP. The 2020-21 planned expenditure for AgriInsurance is $632 million. An overestimation of 1% or 10% would translate into an additional $6.32 million or $63.2 million of AAFC contributions held in reserve and unavailable for other programs or paid out on overvalued claims.

5.3 External stakeholders

The evaluation confirmed that the MIP information is integrated in the operations of the agriculture and agri-food industry. Both at the producer and professional association levels, industry stakeholders use the MIP information to support pricing, sales, trade, and other related decisions and processes. For instance, the MIP is providing import levy invoicing services for the Canadian Beef Check-Off Agency and the MIP is examining the possibility of providing import levy invoicing services for the Canadian Pork Council.

Industry stakeholders interviewed as part of this evaluation indicated that the MIP provides clean, aggregated data that is presented in standardized and accessible formats. The MIP is particularly appreciated by stakeholders for its ability to provide a comprehensive national picture that integrates a wide range of data from different sources. In instances where custom data is required, industry stakeholders indicated that the MIP is capable of responding promptly, and MIP team members are considered to be easily accessible and responsive.

While acknowledging the important contribution of the MIP, industry stakeholders noted they hoped the MIP would remain flexible to continuously adapt to emerging trends in the agriculture and agri-food sector, in addition to addressing any data gaps that might be identified through consultations between AAFC and industry.

While industry remains the predominant external user of MIP information, the evaluation found that the MIP successfully responded to requests from other groups, including other federal agencies (such as CBSA and CFIA), provincial ministries and agencies and academia.

6.0 Program design and delivery

The lack of a formal governance for the MIP may negatively impact the long-term sustainability, relevance and effectiveness of the MIP.

6.1 MIP governance structure

The Office of the Comptroller General of Canada and the Institute of Internal Auditors define governance as the combination of processes and structures implemented by organizations to inform, direct, manage, and monitor the activities of the organization toward the achievement of its objectives.

Evaluators expected to find an appropriate MIP governance structure and activities in place. As noted in Section 3.2, a series of Memorandums of Understanding and Data Sharing Agreements are in place to guide the MIP’s data gathering activities. Beyond these agreements, a significant finding of the evaluation is that the MIP lacked key elements of a formal governance structure. Notably, there was no formal operational structure and coordination among MIP units, there was no designated budget, and no formal sustainability or succession plan. Roles and responsibilities were not clearly defined, and the approach to data management, documentation, and transparency was inadequate. Evidence suggested that, over time, MIP activities have developed in an ad hoc manner to meet stakeholder needs as they have arisen. The evaluation did not find evidence of any formal assessments or analysis of stakeholder needs to determine the relevance or effectiveness of the MIP’s products and services.

The lack of a formal governance structure for the MIP may create management challenges in terms of resource planning, sustainability, and performance monitoring. The evaluation found limited documentation regarding the structural organization of the MIP, or how its resources are assigned and used. Similarly, information defining the MIP objectives and intended results, activities, and outputs is scattered throughout formal and informal sources. In the absence of key documents such as organizational charts, or a detailed program profile, Figures 1 (in Section 5) and 2 (in Annex B) were developed by the Office of Audit and Evaluation to facilitate their analysis and understanding. Both figures were validated by MIP staff.

6.2 MIP pressures

The MIP has several pressure points in addition to its lack of a formal governance structure. The agriculture and agri-food sector must adapt constantly to emerging consumer trends (for example, emergence of plant-based meat substitutes); production shifts or restructuring (for example, trade deals, disruptive technologies, new markets or competition); and environmental drivers (for example, climate change, new environmental regulatory or policy frameworks). As a result, the market information needs of stakeholders also evolve. Without undergoing a formal assessment or analysis of stakeholder needs, addressing information gaps could continue to be ad hoc and not take into consideration a more strategic and departmental perspective. While acknowledging the significant contribution of the MIP in meeting their information needs, stakeholders who were interviewed as part of this evaluation identified the following gaps that they consider as not being fully addressed by the MIP or other data and information providers, both within AAFC and within the sector:

  • Local, and community-specific variances in market prices and commodity production;
  • Gaps in price data;
  • More detailed product lines (for example, product-specific information as part of trade data such as value of carcass cut-outs, or additional pork products in the Consumer Price Index basket to allow for a better analysis of pork products);
  • Additional data from some jurisdictions, such as Quebec;
  • Additional interprovincial trade data to enhance traceability of livestock movements between provinces; and
  • Additional market information on international markets, to enhance contextual information during market disruptions.

In the absence of a formal assessment of stakeholder needs, AAFC may be unable to accurately determine where there are gaps or modifications needed to adequately support the sector.

The MIP has added, or is in the process of adding, new business lines and activities to its operational duties, which include

  • Providing import levy invoicing services to the Canadian Beef Check Off Agency in exchange for micro-level export/import data;
  • Providing import data and analysis to the Market Information Services Branch for possible actions at the World Trade Organization (started in January 2020);
  • Exploring the possibility of providing import levy invoicing services for the Canadian Pork Council in exchange for micro-level export/import data; and
  • Exploring the possibility of collecting Quebec marketing data (a request made to the province is under consideration) in areas where Quebec has stopped collecting data.

Internal and external stakeholders consulted for this evaluation indicated that the MIP could explore options to provide market information in more user-friendly formats, for example dashboards, to enable stakeholders to better interact with the data.

6.3 Long-term sustainability

The Sector Development and Analysis Directorate, where the MIP is situated in AAFC, is guided by a five-year Market Information Strategic Plan that covers 2019 to 2023. This Plan provides a road map with strategic objectives and targets, covering all three subsectors (animal, horticulture, and grains and oilseeds).

Although the MIP is guided by the Market Information Strategic Plan, the evaluation identified that this Plan does not provide a governance and management structure specific to the MIP. The evaluation findings that the MIP lacks a formal governance structure and that it should undertake an assessment to determine where there are gaps or modifications needed to its products and services, underline the need to address these gaps to ensure a longer-term sustainability of the MIP’s ability to remain relevant and effective.

6.4 Visibility and coordination

During the evaluation interviews, both individuals within AAFC and external stakeholders were often unfamiliar with the “Market Information Program” as an entity. In all cases, these stakeholders knew individuals from the “red meat”, “horticulture” or “grains and oilseeds” areas; were very aware of the products and services they access on a regular basis on AAFC’s web site, and knew who to contact if they were seeking further insights or information.

The evaluation found that the MIP operates largely as three distinct units (animal, horticulture, and grains and oilseeds) and a collection of activities rather than a defined program. As an illustration, team members from each of these units who were interviewed often had a fairly limited knowledge of what the other two units were undertaking.

The evaluation found that external stakeholders remained unaware of the full range of products and services they could access and whether they were making the best use of all the available support. While internal stakeholders received the information they needed, there was at times a lack of clarity about which data collection or data analysis group was providing which products or services. The evaluation found that AAFC could benefit from a more coordinated approach to the provision of data/analytic services and products, and by making the services and products more clearly available on AAFC’s external website. From a stakeholder perspective, responses indicated that increased visibility and clarity of the products and services available, along with the key contacts, was their key interest, rather than a concern with which internal AAFC group was responsible.

7.0 Gender-based analysis plus

The MIP has yet to undertake a gender-based analysis plus review of its activities.

As per the Treasury Board of Canada’s Directive on Results, the evaluation examined the extent to which Gender-Based Analysis Plus has been integrated into the management of the MIP. Gender-Based Analysis Plus is an analytical process to assess the potential impacts of policies, programs, services, and other initiatives on diverse groups of women, men and non-binary individuals, taking into account a range of identity factors such as sex, gender, age, disability, education and income level, cultural and racial identification, as well as more external factors such as individuals’ geographical locations.

The evaluation has found no evidence that the MIP has proceeded with a Gender-Based Analysis Plus review as framed by Women and Gender Equality Canada (formally Status of Women Canada). AAFC introduced a Policy Statement on Gender-Based Analysis Plus in February 2015. Considering that the fundamental purpose of the MIP is to contribute to the establishment of a detailed profile of the agriculture and agri-food sector in Canada, it would appear particularly relevant to conduct such an analysis, particularly as part of a more formal management structure. Because participation in the sector is so dependent on access to robust market information, a Gender-Based Analysis Plus review would also be consistent with AAFC’s commitment in the 2019-2020 Departmental Plan to continue monitoring AgriDiversity, a program which aims, in part, to support more youth, women, Indigenous Peoples and persons with disabilities in accessing the resources to better participate in the sector.

8.0 Conclusions and recommendations

Relevance

The agriculture and agri-food sector is volume, sales and trade intensive, and as such, access to detailed market information is vital. The evaluation found that the MIP makes a significant contribution to building a comprehensive market understanding, through its unique subject-matter expertise and its capacity to successfully collaborate with key stakeholders. The work of the MIP does not duplicate what other stakeholders are undertaking. While the MIP and other stakeholders operate within a multi-faceted and highly interdependent agriculture and agri-foods landscape, each entity uses market information differently. The MIP responds to information needs from AAFC and external stakeholders to support decision-making with data and analysis that is not otherwise available.

The evaluation found that the work of the MIP is directly aligned with the long-standing priority of AAFC and the federal government to support the growth and sustainability of the agriculture and agri-food sector in Canada.

Program effectiveness and efficiency

The MIP demonstrated its capacity to provide the required market data and analysis that internal and external stakeholders are seeking. Within AAFC, the MIP successfully supports the implementation and management of key programs and policy initiatives. It demonstrated its capacity to react promptly to emerging issues, such as trade disruptions, and to provide the analysis and market intelligence that Senior Management within AAFC required. External stakeholders, particularly industry, indicated they have integrated the market information provided by the MIP in their ongoing management and decision-making processes.

The evaluation concluded that AAFC would benefit from establishing a formal governance structure to provide clarity about the MIP roles and responsibilities. This should take into consideration other AAFC areas that collect data and provide data-based and analytical services and products, to ensure an AAFC approach that provides an ease of access for users.

To address the evaluation findings regarding gaps in stakeholder awareness or access to available AAFC services, products, or points of contacts, the evaluation found that more could be done to provide ease of access to AAFC services and products posted on AAFC’s external website. As the AAFC Public Affairs Branch has already launched an update of the external site with regard to accessing AAFC services and products, a recommendation is provided to enable a more user-centric approach to accessing AAFC information overall, which will include access to services and products produced by the MIP.

Recommendations and management responses are phrased within the wider context of an AAFC-wide approach, so that users could benefit from having a more coordinated approach to providing data collection and data analytics services to respond to internal and external stakeholder needs. This would ensure long-term sustainability, relevance, and effectiveness of the services offered by AAFC.

Recommendations

The evaluation provides three recommendations.

  • Recommendation 1: The Assistant Deputy Minister of the Market and Industry Services Branch should develop a formalized governance structure for the Market Information Program, as part of a department-wide approach to data collection and data analytic services.
  • Recommendation 2: The Assistant Deputy Minister of the Strategic Policy Branch should lead on the development and implementation of a department-wide approach to data collection and data analytic services.
  • Recommendation 3: The Assistant Deputy Minister of the Public Affairs Branch should lead on the development of a strategy and changes to the AAFC’s digital channels, including its external, website to ensure a user-centric, AAFC-wide approach to making relevant market information data and analysis accessible and useable.

9.0 Management response and action plan

Management has agreed:

  • to change its title to “Market Information Service” to better reflect its role;
  • to implement an enhanced management and governance structure that will foster continued improved service and enhanced business outcomes, and formalize existing internal information sharing practices to support Chief Data Officer’s efforts to implement AAFC's Data Strategy;
  • to present a short-term option for certain data governance decisions to DGMC for decision, prepare an implementation plan for the Data Strategy, and prepare for the appointment of a Chief Data Officer;
  • to develop a multi-channel promotional plan to help brand the Market Information Service; and to all stakeholders (internal and external).
  • complete web renewal work focused on enhanced client experiences and improved product offerings.

Annex A: Evaluation methodology

Interviews and consultations

Interviews were conducted to develop an understanding of the role of the MIP staff, of major activities with deliverables, and of costs associated with the MIP activities, and to assess the MIP administration and delivery. The key informant interviews were used to support information obtained through other lines of evidence.

A total of 34 interviews were conducted with 48 individuals, as follows: 23 members of AAFC staff (Senior Management, program officials and members of other groups at AAFC who use and/or provide data to the MIP); 12 representatives of other federal departments or agencies; four representatives from provincial government; and nine industry stakeholders.

MIP file and administrative data review

To assess the relevance, design, delivery, and effectiveness of the MIP, a review of the MIP files was completed as part of this evaluation.

Literature review

To assess relevance, design and delivery, and program effectiveness, a literature review was conducted. The literature reviewed included documentation detailing the work of institutions conducting activities similar to the MIP, industry reports by Statistics Canada, and other economic and statistical documentation.

Comparative review

To assess the continued relevance, performance, and design of the MIP, a comparative review of similar programs was undertaken. The review of similar programs included the United States, Australia, China, and the Food and Agriculture Organization of the United Nations. The review explored best practices of other programs in order to assess the continued relevance of the program and inform policy directions.

Document review

To assess relevance, a review of the MIP-related and broader federal government documentation was undertaken. Documentation reviewed included MIP-related documentation for the years covered by the evaluation including, but not limited to, program, departmental planning and performance documents, such as Reports on Plans and Priorities and Departmental Performance Reports, and broader Government of Canada documents that may speak to relevance issues, such as Speeches from the Throne.

Case studies

To assess relevance and performance, two case studies were undertaken to assess the MIP’s support provided to Advance Payment Program on a continuous basis, and a market disruption with China involving pork products. The case studies provide insight into the MIP impact on the Department’s and industry’s decision-making during situations that require timely and accurate market information. Each case study focused on a different situation: trade (including disputes), and departmental programs. The approach to examine the case studies included program and departmental interviews, and documentation and literature review.

Methodological limitations

Because the MIP is not a formal program (but rather a set of activities), a number of different challenges and limitations arose in the evaluation. The key limitations are outlined below:

Limitation Mitigation strategy Impact on evaluation
Lack of information available about the program. Because the MIP is not a formal program, traditional sources of information defining the program, its objectives, and its results were not available. Available documentation was used, and data gaps were filled with information gathered through interviews. The evaluation placed higher reliance on interview data.
Lack of understanding of the program among external stakeholders. Because the MIP is not a formal program, stakeholders are not clear of what is included in the MIP. A program definition was included in the interview guide (circulated to key informants prior to the interview). Some elements raised by key informants were outside of the scope of the program as it was defined for the evaluation. These were excluded from final reporting.
Limited documentation on activities. Limited documentation exists to track MIP outputs. Effectiveness was explored primarily qualitatively, using examples of program outputs and their impact (primarily as discussed in interviews), though quantitative information was included as available. The evaluation placed higher reliance on interview data.

Annex B: Program governance and resources

While not a completely distinct entity, the MIP is situated within AAFC’s Market and Industry Services Branch, which provides critical industry services, including market information and sector expertise. The MIP’s core team members are located in the Crops and Horticulture Division and the Animal Industry Division of the Sector Development and Analysis Directorate, within the Market and Industry Services Branch.

As illustrated in the organizational chart (next page), the directors of these two divisions work with deputy directors, sector specialists, market analysts, and market information officers to undertake the various program activities that are associated with the MIP. Sector specialists located in other sections or groups of these two divisions also provide support to the MIP, particularly in the grains and oilseed, red meat, and dairy and poultry sectors. Finally, the MIP team works with multimedia technology analysts to transfer all the content online.

In the absence of a formal budget allocation, the MIP provides estimates of resources needed to support its activities, which are tracked and monitored on an ongoing basis. At the time of the evaluation, program representatives estimated that just under $2.5 million-was required to operate the MIP. These resources cover salaries and related operational expenses for 22 departmental resources. They do not cover any transfer of resources in accordance with Memorandums of Understanding or Data Sharing Agreements signed between AAFC and partner organizations, such as Statistics Canada, for the production of data used by the MIP.

Figure 2: Market Information Program operational mapping

Description of this image follows

Description of above image

Figure 2 provides an organizational chart of the Market Information Program within the Sector Development and Analysis Directorate of Market and Information Services Branch.

Source: Created by OAE in support of the MIP Evaluation, February 2020

Annex C: Market Information Program logic model

The MIP logic model describes its key activities and outputs and the sequence of immediate, intermediate and end outcomes.

Objective

The objective of the program is to provide reliable and timely data and market information to AAFC and industry clients that allow for improved marketing and investment decisions, research, programming and policy decisions.

Activities

  • Data-gathering, analysis, compiling and dissemination of
    • primary data collected by AAFC, and
    • secondary data collected by industry/partners
  • Relationship-building with federal, provincial/territorial governments, and industry organizations and companies
  • Securing information in support of market development needs.

Outputs

  • Market information
  • Analysis and intelligence
  • Market information reports and inputs
  • Recommended rates for the Advance Payments Program
  • Recommended rates for Price Pooling Program
  • Information for technical reports: benchmarking, performance analyses, regulatory and policy impacts
  • Technical advice and services to the sector and to governments (includes methodology decisions) Information for program assessments and planning

Immediate/intermediate outcomes

  • Industry and producers, organizations and governments make better (informed) decisions
  • Canadian industry and producers service domestic and international markets better
  • Government programs, policies and regulations respond to sector priorities and competitiveness needs
  • Government agriculture and agri-food sector address priority issues to advance competitiveness
  • Stakeholders implement strategies and use tools to manage changes associated with external forces

End outcomes

  • Improved producer and company access to markets
  • Responsiveness to opportunities and demands
  • Enhanced competitiveness domestically and internationally Agriculture and agri-food sector successfully adapts to changing and emerging global and domestic opportunities and issues

Link to Departmental Results Framework
1.2 Market Access, Negotiations, Sector Competitiveness and Assurance Systems. 1.2.2 Sector Engagement and Development.

AAFC Strategic Outcome
SO1: A competitive and market-oriented agriculture, agri-food and agri-based products sector that proactively manages risk.

Source: Adapted from the MIP (Animal, Horticulture, Grains and Oilseeds) Logic Model in Relation to the Sector Development and Analysis Directorate Logic Model

Date modified: