Evaluation of the Dairy Farm Investment Program and the Dairy Processing Investment Fund

Abbreviations

AAFC

Agriculture and Agri-Food Canada

CETA

Canada-European Union Comprehensive Economic and Trade Agreement

DFIP

Dairy Farm Investment Program

DPIF

Dairy Processing Investment Fund

Executive summary

Purpose

The Office of Audit and Evaluation of Agriculture and Agri-Food Canada (AAFC) conducted an evaluation of the Dairy Farm Investment Program (DFIP) and Dairy Processing Investment Fund (DPIF) to assess their relevance, efficiency and effectiveness.

Scope and methodology

DFIP and DPIF activities and outcomes from 2017–18 to 2022–23 were evaluated using multiple lines of evidence including: a literature review, a program document and file review and key informant interviews conducted with DFIP and DPIF program officials, representatives from dairy industry associations, a sample of DPIF recipients and departmental and academic experts.

Background

Canada's dairy industry has operated under a supply management system since 1972. Supply management is designed to match supply and demand by controlling the production and price of milk in Canada and imposing tariffs to limit the import of dairy products into Canada.

In 2016, Canada entered into the Canada-European Union Comprehensive Economic and Trade Agreement (CETA) permitting, as of 2017, the duty-free import to Canada of a limited quantity of cheese originating in the European Union. The authorized imports were phased in over a period of 5 years and as of 2022, 16,000 metric tonnes of all cheeses and 1,700 metric tonnes of industrial cheeses are permitted to be imported under the tariff rate quota structure. This represents about 4% of Canada's annual cheese consumption.

In 2016, the Canadian government announced that it would provide the industry with compensation to help mitigate the anticipated market losses from CETA by dedicating $350 million to programming to support and maintain dairy producers and processors productivity and competitiveness. That programming took the form of DFIP ($250 million) and DPIF ($100 million).

Findings

• DFIP and DPIF were aligned with the federal commitment to support the Canadian dairy sector for market concessions under CETA and AAFC's core responsibility for domestic and international markets.
• The evaluation is unable to determine whether either program mitigated anticipated future growth losses or other anticipated impacts from CETA for program participants.
• Funding levels and high program demand resulted in the oversubscription of DFIP and DPIF and a requirement to adjust program parameters during implementation. Some concerns related to DFIP were addressed with the changes made between Phases 1 and 2.
• Ongoing communications with industry stakeholders informed the design of both DFIP and DPIF and provided supports to DPIF applicants throughout the application and project management process. However, a lack of clarity in messaging about program intent led to challenges managing stakeholder expectations.
• Good operational practices including a two-step application process and use of a data management system enhanced efficiency and reduced burden for applicants and program officers. DFIP's use of specialized operations teams permitted quick application assessment and has been replicated in other AAFC programs.
• DFIP and DPIF collected voluntarily-provided data on the diversity of producers and processors applying for and benefiting from program funding, although Gender-based Analysis Plus considerations were not included in program design.
• DFIP funding proportions generally matched the provincial share of quota. Small and medium sized farms received slightly over half of DFIP funds. Youth and/or new dairy producers under age 35 may have faced more challenges matching the 50% cost share required to participate in the program.
• DPIF provided the majority of its funding to small and medium sized businesses.
• While DFIP has collected data to assess program performance, inconsistencies between units used in establishing targets and those used in collecting data to assess progress make it difficult to assess program outcomes.
• It is too early to determine whether DPIF's capital investment component has achieved its expected outcomes.

Conclusion

Both DFIP and DPIF were aligned with departmental and federal priorities to compensate the Canadian dairy sector for the anticipated impacts resulting from CETA. However, this evaluation was unable to determine whether either program mitigated anticipated future growth losses or other anticipated impacts from CETA for program participants.

Both DFIP and DPIF implemented good practices in program design and delivery that can and have informed other AAFC programs. However, oversubscription, insufficient funding to meet demand, a lack of clear communication about the intent of DFIP and changes to key DPIF program parameters resulted in challenges managing stakeholder expectations.

Both DFIP and DPIF were open to Canadian dairy producers and processors whose proposed projects met program eligibility criteria. One potential barrier to participation that was identified in the evaluation related to new producers or producers under the age of 35 who may have struggled to match the 50% funds required by DFIP.

DFIP achieved its outcomes to varying extents depending on the type and size of project implemented, with most recipients reporting at least moderate cash and labour cost savings. While it is too early to assess achievement of DPIF outcomes for capital investment projects as project results will continue to be reported until 2026, early results indicate progress toward the outcomes.

Performance measurement challenges in both programs make it difficult to assess program impacts or whether the programs met their stated objective of supporting dairy sector adaptation to CETA impacts. Given the lack of ultimate outcome indicators linking either program's activities to the objective of supporting the dairy sector to adapt to CETA impacts, the evaluation was not able to determine the extent to which either program met this objective.

Recommendations

Recommendation 1: The Assistant Deputy Minister, Programs Branch should document and share within AAFC the good practices used in DFIP and DPIF design and delivery.

Recommendation 2: The Assistant Deputy Minister, Programs Branch working with the Assistant Deputy Minister, Strategic Policy Branch should review current supply management programs to ensure that they do not present barriers for producers and processors who are members of underrepresented groups and that Gender-based Analysis Plus performance measures are in place.

Management response and action plan

Management agrees with the evaluation recommendations and has developed an action plan to address them by March 2024.

1.0 Introduction

The Office of Audit and Evaluation conducted an evaluation of the Dairy Farm Investment Program (DFIP) and the Dairy Processing Investment Fund (DPIF) as part of the 2022–23 to 2026–27 Office of Audit and Evaluation Plan. The evaluation complies with the Treasury Board of Canada's Policy on Results and fulfills the requirements of the Financial Administration Act. Findings from this evaluation are intended to inform current and future program and policy decisions.

2.0 Scope and methodology

This evaluation examined the relevance, efficiency and effectiveness of DFIP and DPIF. The evaluation used a variety of methods including literature review, program document and file review and interviews. The evaluation covers the period from 2017–18 to 2022–23.

For the detailed evaluation methodology, see Annex A.

3.0 Program profile

This section provides an overview of the context within which DFIP and DPIF were implemented.

3.1 Context: the Canadian dairy industry

Over the 10-year period from 2012 to 2022, the Canadian dairy industry has undergone a number of changes. The number of dairy farms fell from 12,762 in 2012 to 9,739 in 2022, an average annual decline of 2.4%. This decline suggests a consolidation of farms. Similarly, the number of dairy cows has fallen slightly, from 1.42 million head in 2012 to 1.38 million head in 2022. However, milk production increased by around 18% from around at 79.8 million hectolitres in 2012, to 94.5 million hectolitres in 2022. This increase in milk production is generally attributed to the continuous progress in farming practices, technological innovations, enhanced efficiency within the dairy industry and genetic improvement in dairy cattle. Cash receipts have also increased by about 39%, increasing from $5.9 billion in 2012 to $8.2 billion in 2022. This growth is generally attributed to increased milk production and higher farmgate milk prices.

The number of dairy processors increased from 451 in 2012 to 507 in 2022. The total value of manufactured shipments of milk and dairy products increased from $14.2 billion to $17.4 billion during the same period.

Canada's dairy industry has operated under a supply management system since 1972. Supply management is designed to match domestic supply to domestic demand by controlling the production and price of milk in Canada and imposing tariffs to limit the import of dairy products into Canada. Farmers purchase quota that allows them to produce and market a certain amount of milk (measured by quantity of butterfat) at prices set by their respective provincial milk marketing boards. Milk produced by a farmer in excess of this quota may not be sold. Under supply management, the benefits of increased efficiency such as through economies of scale and the corresponding incentives to do so, can sometime be limited as dairy producers cannot easily expand production beyond their allotted production quota without the necessary corresponding quota increase.

While production by dairy processors is not directly limited by a supply management structure, they are indirectly affected by the producers' quota system as access to domestic raw milk is influenced by the amount of milk produced by dairy farmers under the quota limits. Canada's dairy processors purchase their dairy inputs from Canadian farms and thus must pay the input costs determined by provincial milk marketing boards. However, processors can market any additional quantity of goods they are able to produce as a result of investments in their facilities as they are not bound by a specific level of quota. Additionally, under the import for re-export program, dairy processors can import milk and/or dairy ingredients from foreign sources and then process them into various products intended for export markets.

In 2016, Canada entered into the Canada-European Union Comprehensive Economic and Trade Agreement (CETA). Beginning in 2017, a limited quantity of cheese originating in the European Union could be imported into Canada duty-free, under a tariff rate quota system. The authorized imports were phased in over a period of 5 years and as of 2022, 16,000 metric tonnes of all cheeses and 1,700 metric tonnes of industrial cheeses can be imported under the tariff rate quota structure. This represents about 4% of Canada's annual cheese consumption. Cheese manufacturers, distributors and retailers are eligible to apply to all cheese tariff rate quota, while processors who use cheese as an ingredient in the production of further processed food products are eligible to apply for industrial cheese tariff rate quota. This gives them the right to import a specific quantity of the type of cheese that the quota corresponds to. The tariff rate quotas for cheese imports under CETA are administered by Global Affairs Canada. This market access for the European Union represented a significant change for the Canadian dairy sector as import controls had remained largely unchanged since 1995.

In 2016, the government announced that it would compensate the industry for the anticipated impacts of CETA by dedicating $350 million to programming to support and maintain the productivity and competitiveness of both dairy producers and processors. That programming took the form of the Dairy Farm Investment Program (DFIP) and the Dairy Processing Investment Fund (DPIF).

3.2 Dairy Farm Investment Program

DFIP was established to assist the dairy farming sector adapt to the market changes anticipated as a result of the CETA by supporting improved productivity and increased efficiency on Canadian cow milk farms. The program has now ended and a new Dairy Direct Payment Program has been implemented to compensate dairy producers for the anticipated impacts of CETA, the Canada-United States-Mexico Agreement and the Comprehensive and Progressive Agreement for Trans-Pacific Partnership.

For the DFIP logic model see Annex B.

Activities

All projects and activities supported by DFIP were required to relate to improving efficiency and productivity of milk production from cows. Projects could include a single, integrated system or any other investment in eligible equipment. Eligible equipment had to be new, commercially available and related to either barn operation, cow comfort, feeding, herd management or milking. All equipment was required to be installed or utilized in the dairy barn.

Eligible activities included:

• hiring external expertise to assess how the farm could improve efficiencies and productivity
• purchasing, shipping and installing eligible equipment
• training necessary to operate eligible equipment
• retrofits of facilities within the existing barn footprint related to the installation and operation of eligible equipment

Resources

DFIP was a 6-year non-repayable contribution program with an overall budget of $250 million from 2017–18 to 2022–23, including $25 million for program administration costs. The program was extended by 1 year to enable recipients to complete projects that were delayed by COVID-19.

The program was delivered in 2 phases. DFIP Phase 1 was announced November 10, 2016 and implemented from April 1, 2017 to March 31, 2020. Phase 1 had a funding allocation of $129 million. In this phase, funding up to $60,000 was available for small projects and funding up to $250,000 was available for large projects. Phase 1 used a first come-first served application process. Retroactive payments were available for projects activities that were undertaken as far back as the DFIP announcement date (November 10, 2016).

Phase 2 was implemented on April 1, 2020 with a total of $98 million funding allocated. Phase 2 introduced 2 key changes to the program's design: a single $100,000 funding cap for all projects and a two-step application process which used a randomized process to select those to be invited to submit detailed applications.

Eligible costs could be reimbursed up to 50% to a maximum of $100,000 with the maximum level of total government funding not exceeding 85% of eligible costs per project. Across both Phase 1 and Phase 2, funding was allocated by province in proportion to the provincial share of the total national milk quota.

Table 1: Dairy Farm Investment Program expenditures by fiscal year 2017–18 to 2022–23

Fiscal year	Expenditures ($) Expenditures as per public accounts	DFIP full-time employees
2017–18	19,217,845	17.081
2018–19	61,852,928	29.506
2019–20	61,686,884	28.705
2020–21	44,113,872	18.626
2021–22	25,089,852	5.516
2022–23	12,561,892	4.633
Total	224,523,273
Source: Programs Branch data

3.3 Dairy Processing Investment Fund

DPIF was established to provide funding to dairy processors for investments that would improve productivity and competitiveness and help them prepare for market changes resulting from CETA. The program ended on March 31, 2023 and a new Supply Management Processing Investment Fund has been implemented to compensate dairy processors for the anticipated impacts of CETA, the Canada-United States-Mexico Agreement and the Comprehensive and Progressive Agreement for Trans-Pacific Partnership.

For the DPIF logic model, see Annex C.

Activities

DPIF provided funding for projects to improve dairy processing. Eligible projects consisted of either capital investments in dairy processing equipment and infrastructure or the engagement of external expertise.

Eligible capital investment activities included:

• improvement of manufacturing technologies and processes (including the acquisition and installation of equipment), or the introduction of new or improved products
• construction, expansion, or modernization of dairy processing establishments in Canada
• engagement of external expertise related to the implementation, certification or validation of plan improvements

Eligible access to expertise activities included:

• business improvements and market development
• collaborative research with AAFC scientist(s) relating to improving existing products, practices, processes or technology

Resources

DPIF was a non-repayable contribution program with an overall budget of $100 million from 2017–18 to 2022–23. DPIF was extended by 2 years to enable funding recipients to complete their activities which were delayed due to COVID-19 and supply chain delays. The maximum funding available was $10 million per capital investment project and $250,000 per access to an expertise project. Complete applications received by March 31, 2018 could request reimbursement of eligible costs incurred from November 10, 2016, when DPIF was announced.

Eligible costs for capital investment projects were reimbursed up to 50% for the first $2 million and up to 25% for costs above $2 million. Eligible costs for access to expertise projects were reimbursed up to 50% for for-profit applicants and up to 75% for not-for-profit applicants.

The maximum level of total government funding to for-profit companies was not to exceed 75% of eligible project costs. Not-for-profit applicants could have all eligible project costs covered by government funding.

Table 2: Dairy Processing Investment Fund expenditures by fiscal year 2017–18 to 2022–23

Fiscal year	Expenditures − capital investments ($)	Expenditures − access to expertise ($)	DPIF full-time employees
2017–18	11,600,000	0	8.73
2018–19	32,537,244	15,960	11.56
2019–20	21,542,428	33,063	10.87
2020–21	13,599,396	0	10.63
2021–22	8,000,000	20,316	8.27
2022–23	0	0	0.00
Total	87,279,068	69,339
Source: Programs Branch data

4.0 Relevance

4.1 Alignment of programs with federal and departmental priorities

The evaluation found that both DFIP and DPIF were aligned with federal and departmental priorities. These programs provided the first round of compensation that was publicly presented as part of the federal government's commitment to support Canada's supply-managed sectors to adapt to anticipated market changes resulting from the implementation of CETA. Similarly, both DFIP and DPIF were aligned with AAFC's core responsibility for domestic and international markets, specifically to support the competitiveness of the Canadian agriculture and agri-food sector at home and abroad.

4.2 Alignment of dairy programs with dairy sector needs as a result of CETA

Supply management is the system that the dairy sector has chosen for themselves and the government respects and supports this choice. Supply management controls the amount of milk that can be sold so that producers receive a minimum price for their product. However, controlling production through a quota system can discourage on-farm investments in innovation and modernization since the ability of producers to market any increases in production is limited. Dairy processors in Canada, while not subject to production limits, purchase their dairy inputs from provincial milk marketing boards and must pay the prices set by their respective milk marketing boards. Imports under CETA could displace Canadian production, which could impact producer and processor profits. Following the implementation of CETA, dairy producers and processors expressed concerns about the reduction of future earnings due to the loss of market share. Producers were also concerned about the potential reduction in quota value and diminishing returns on investment as a result of increased imports.

The impacts from CETA for either producers or processors are difficult to isolate due to the interplay of several factors, such as:

• Tariff rate quota fill rates for CETA have been high, suggesting increased competition in the dairy sector;
• Structural constraints on producers due to supply management limit the impact that investments in production can have in certain jurisdictions;
• Both dairy production and dairy processing have increased in Canada, however the entry into force of other free trade agreements^{Endnote 1} since the implementation of CETA complicates economic analyses that would seek to isolate CETA's impact on the dairy sector.

Because neither programs' performance measurement strategy included an ultimate objective of supporting dairy producers and processors to adapt to market changes resulting from CETA, (that is, specific indicators measuring the adaptation to market changes were not established and associated data was not collected) the evaluation was unable to determine whether the support provided through DFIP and DPIF helped the sector adapt to these market changes. As a result, the evaluation is limited to reporting on the extent to which DFIP improved producer efficiency and DPIF increased processor competitiveness, as these are the only program outcomes described in their performance measurement strategies (see section 7).

5.0 Program design

5.1 Key program design elements

Funding levels and oversubscription

The most significant challenge for both DFIP and DPIF was the unanticipated high demand resulting in their oversubscription and a requirement to adjust the program parameters for both programs.

DFIP

Across both phases, DFIP funded 3,450 unique farms, representing approximately 33% of the 9,403 dairy farms in Canada (as of 2021 Census of Agriculture). Dairy sector organizations had stated that all dairy farms should have been able to receive funding to invest in efficiency enhancing upgrades, as all dairy farms were expected to be impacted by CETA.

During DFIP Phase 1, applications were accepted by mail or by email on a first come-first served basis during a single application period. This process resulted in complaints from applicants and industry representatives that the program was unfair. There was concern that producers with poor access to high speed internet, or who needed or preferred to use paper applications faced barriers in submitting their applications quickly enough to be considered. This application model, together with limited funding availability, meant that there were producers who spent time preparing comprehensive applications that were not considered for funding due to the rapid allocation of all available project funds.

In response to these concerns, several design changes were implemented in DFIP Phase 2. These included changing to a single funding cap of $100,000 per project, prioritizing applicants who didn't receive Phase 1 funding and limiting funding to one project per producer. These changes permitted the distribution of funds across more applicants. DFIP Phase 2 also implemented a two-step randomized selection application process which, together with changes to the funding cap, resulted in stakeholder perception that DFIP Phase 2 was more equitable than Phase 1, though the sector still expressed concerns about the adequacy of the program to meet their needs.

The design challenges noted above, in part, informed the development of AAFC's Dairy Direct Payment Program. Under this new program, every dairy farmer in Canada is eligible for direct payment, proportionate to the amount of quota they hold. This payment is intended and advertised as compensation for market access commitments made under CETA, the Comprehensive and Progressive Agreement for Trans-Pacific Partnership and Canada-United States-Mexico Agreement.

DPIF: Capital investments component

The non-repayable, retroactive payment eligibility and up to 50:50 cost share aspects of the contributions (for the first $2 million of eligible costs, with eligible costs over $2 million subject to a 25:75 cost-share ratio) were key factors in securing the support of the dairy processing industry. Applicants needed to have submitted a complete application by March 31, 2018 to be eligible to receive the retroactive payments to November 10, 2016. DPIF received 68% of its applications before March 31, 2018. This high volume of applications in the initial intake period had 2 effects on the program:

• DPIF was unable to meet its service standard for sending an approval or rejection notification letter 80% of the time within 100 business days of receiving a fully complete applicant package in fiscal years 2018–19 and 2019–20. In 2018–19, the letters were sent within 100 business days 62% of the time and in 2019–20 the letters were sent within 100 business days 56% of the time.
• DPIF developed and applied an oversubscription strategy that placed additional restrictions to the cost-share ratio. This led to some recipients receiving less funding than they anticipated based on the original published cost-share ratios.

DPIF: Access to expertise component

DPIF's access to expertise component provided dairy processing organizations access to AAFC scientists to conduct collaborative research to improve their existing products, practices, processes and/or technology. While AAFC had notionally allocated $4 million for this component, only $69,339 was distributed to 2 access to expertise projects.

The low uptake informed the development of AAFC's Supply Management Processing Investment Fund, which does not include funding for access to expertise. Instead, it funds only the acquisition and installation of new automated equipment and technology.

5.2 Communications

Consultations with producer and processor associations informed several elements of program design for both DFIP and DPIF Phase 1 and Phase 2. For example, both programs included retroactive payments to the date they were announced (November 10, 2016). For DFIP, stakeholder feedback and several discussions with and written submissions from industry representatives informed the change in application model and funding cap between Phase 1 and Phase 2 (see section 3.2 Resources). For DPIF, in response to consultation with the dairy processing industry, the program was open to all dairy processors rather than just the cheese processors directly impacted by the imports under CETA.

Effective communication within AAFC and between AAFC and applicants resulted in positive experiences for applicants throughout DFIP and DPIF's implementation and management. For example, DPIF program officers conducted proactive outreach and provided ongoing support to applicants throughout the application process. In addition, strong internal collaboration between DFIP program officers and the AAFC Contact Centre team in Winnipeg supported management of over 10,000 calls from producers.

However, a lack of clear communication about the intent of DFIP resulted in a misalignment of stakeholder expectations and the program as implemented. Both DFIP and DPIF were designed to provide funding for capital investments to improve producer productivity and efficiency and to enhance processor competitiveness. The stated objective of these programs in AAFC communications, including the public facing website, was to support the Canadian dairy industry to adjust to market changes resulting from CETA. Clearer messaging that DFIP intended to support dairy producers by helping enhance efficiency rather than through direct compensation may have helped to mitigate some complaints about DFIP from dairy producers.

5.3 Operational practices

DFIP and DPIF both employed a number of good operational practices. For example, the 2-step application process employed by DFIP Phase 2 and DPIF enhanced efficiency and reduced burden for applicants to these programs.

DFIP

During DFIP Phase 1, the program lacked an automated data management system. All applications, whether received by email or traditional mail, were managed manually. This lack of a data management system together with the higher than anticipated demand necessitated a large expansion of the number of full-time employees during periods of high application intake in 2018–19 and 2019–20. With the implementation of a data management system for DFIP Phase 2, applicants were encouraged to submit applications online. This enhanced the efficiency of program administration by ensuring data accuracy and facilitating application intake^{Endnote 2}, project monitoring and reporting and payment.

DFIP created specialized operations teams – one for program administration and a separate expert team dedicated to application assessment. The use of specialized teams is considered a good practice as it permits quicker application assessment in high volume programs. This approach has since been successfully replicated in at least 4 other AAFC programs.

DPIF

DPIF staff developed detailed administrative guidelines that clearly explained the procedures for staff to follow at each stage of the project lifecycle: screening, approval and monitoring. The guidelines also included other good practices such as a file naming convention, a list of required documents in the electronic applicant folders and email scripts for interactions with applicants and recipients. Detailed administrative guidelines provide many benefits, such as standardizing the steps in a process, providing a documented reference for employees before seeking guidance and assisting with the onboarding of employees.

DPIF staff developed and maintained an electronic database that contained comprehensive and easily accessible information for each project. This facilitated the monitoring and reporting on the results of the funded projects.

6.0 Program reach

6.1 Program reach – Common elements

At the time of program design, DFIP and DPIF were announced as having a target beneficiary group of all Canadian dairy producers and processors whose projects met the eligibility criteria. Demographically, this means the programs' target recipients were primarily male and over age 45.

DFIP and DPIF provided applicants the option to voluntarily self-identify as a member of one or more of the following underrepresented groups: women, Indigenous people, persons with disabilities, youth (under age 35) or official language minority communities. For successful applicants, DFIP and DPIF supported producers and processors operating in official language minority communities by including translation costs as eligible expenses.

6.2 Program reach – DFIP

By design across both Phase 1 and Phase 2, the allocation of DFIP funds across provinces was determined based on provincial share of milk production quota. These provincial funding proportions generally match the provincial share of quota, with Quebec and Ontario receiving the highest percentage of funding.

DFIP program data indicates that across Phase 1 and Phase 2, an average of 64% of projects (56% of funding) were undertaken by small and medium sized farms, with an average of 34% of projects (43% of funding) undertaken by large farms.

The evaluation identified one potential barrier to participation: new producers or producers under the age of 35 may struggle to match the 50% funds required by DFIP. DFIP program data indicates that of the 3,054 contribution agreements, 57 recipients (or 2%) voluntarily identified as youth (under age 35)^{Endnote 3}. According to the 2021 Census of Agriculture, youth under 35 comprise approximately 15% of Canadian dairy farm operators.

6.3 Program reach – DPIF

DPIF was an applicant-led program open to all Canadian dairy processing companies. There were no geographical representation requirements or allocations to consider when reviewing project applications. The evaluation found that the distribution of funds for processors who participated in DPIF generally reflected the geographic distribution of processors across the country, except for Quebec, which received slightly higher funding than its overall share of dairy processors (47% versus 43%). 54% of DPIF funding (over 88% of projects) was allocated to small and medium businesses.

7.0 Achievement of outcomes

7.1 Dairy Farm Investment Program

DFIP program data indicates there were 3,454 contribution agreements (1,903 in Phase 1 and 1,551 in Phase 2) with 3,450 individual farms. In Phase 2^{Endnote 4}, farms invested in 11,210 projects implementing equipment in the 5 eligible categories as presented in Figure 1 below:

Source: Program administrative data, May 2022

Description of figure 1

Figure 1 presents a pie chart showing the numbers of DFIP Phase 2 projects across 5 broad equipment categories.

Equipment type	Number of DFIP Phase 2 projects
Barn Operation	3,394
Cow Comfort	3,070
Feeding	2,583
Milking	1,404
Herd Management	759

Figure 2 presents the extent to which DFIP achieved its 2 outcomes of percentage of actual cash costs saved and percentage of actual direct labour costs saved over the course of the program. A majority of recipients reported at least moderate savings in both cash and labour costs in their final project report.

Source: AAFC DFIP administrative data, October, 2022.

Description of figure 2

Figure 2 is a bar graph presents the percentages of funding recipients who reported percentage ranges of actual direct labour costs saved, and actual cash costs saved. The X axis represents the percentage ranges of savings. The Y axis presents the percentages of recipients reporting labour cost savings (orange bars) and the percentages of recipients reporting cash cost savings (blue bar).

Percentage savings range (X axis)	Percentage of recipients reporting labour costs saved (Y axis-orange bar)	Percentage of recipients reporting cash costs saved (Y axis blue bar)
Unknown	5%	5%
Increased	2%	3%
No Savings	10%	6%
1-3%	17%	18%
4-6%	19%	25%
7-10%	16%	20%
11-15%	10%	11%
16-20%	8%	4%
21-25%	4%	2%
25+%	9%	4%

7.2 Dairy Processing Investment Fund

Capital investments

The evaluation was unable to determine the extent to which DPIF achieved its expected outcomes for its capital investment projects because the intermediate and long term results of the projects are not yet known. DPIF will continue to receive project results data for 3 to 5 years after their completion. The date to achieve the desired outcomes is March 2026. At the time of the evaluation, preliminary results had only been received for 71 of 105 funded projects. Early data from these projects indicates that recipient performance metrics are improving from their baselines. The following trends have been reported as of August 2022:

Table 3: Preliminary results from capital investment projects

Performance Indicator	Results (% change)	Target (% change)
Average decrease in cost of production	−6%	−11%
Average increase in production levels	105%	110%
Average increase in volume of milk and milk components used	75%	100%
Average decrease in water used	−12%	−50%
Average decrease in wastewater discharged	−29%	−1%
Average increase in revenue	112%	90%
Number of new to the organization products	52	No target set
Source: AAFC DPIF program data, August 2022

DPIF projects funded capital investments that involved the expansion and modernization of facilities and equipment for the processing of milk into products such as cheese, butter, cream, yogourt, milk powder and milk protein concentrates. DPIF recipients considered this funding important in helping them improve their productivity and competitiveness. Recipients reported that the DPIF funding helped their business grow through increased production and storage capacity, expanded product lines, ability to reach larger domestic and foreign markets, hiring more staff and enhanced their competitiveness. Recipients also reported that the funding served to speed up investments they would likely have made over a longer period of time.

In addition to the preliminary results from the projects, an impact assessment using AAFC and Statistics Canada data was conducted by AAFC in the fall 2022. This assessment did not find a statistical difference between DPIF recipient and non-recipients on their short-term financial performance. However, there were limitations to the assessment. For example, the data for the analysis was from 2020 and most projects funded by DPIF were not yet complete at that time. The assessment stated that it is possible that the impact of the capital investment projects funded by DPIF would not be evident in the data until a few years later. The assessment suggested that future research could be conducted when updated data is available to re examine the effectiveness of DPIF.

Access to expertise

In addition to the 105 capital investment projects, DPIF also funded 2 projects under the access to expertise component. The evaluation was unable to determine whether the 2 access to expertise projects met their desired outcomes because indicators relevant to the projects were neither established nor included in the DPIF Performance Information Profile. AAFC's Science and Technology Branch, which implemented the 2 projects, measures outputs of projects through metrics such as the number of scientific publications that result from projects and whether the Office of Intellectual Property and Commercialization receives invention disclosures from project recipients. Neither of the access to expertise projects resulted in any AAFC scientific publications or invention disclosures to AAFC's Office of Intellectual Property and Commercialization.

The evaluation found that for one of the projects in the access to expertise stream, it was expected that the recipient would report on this project's impact at the same time they reported on the funds received in the capital investment project stream. However the capital investment project does not include indicators that directly measure the results of the access to expertise project. Without direct measurement of the results of the access to expertise project and its impact on the results for the capital investment project, it is not clear how attribution of results to the access to expertise project will be possible.

Furthermore, one medium term outcome in DPIF's Performance Information Profile may have been originally intended to measure access to expertise results. However, these indicators were not applicable to either of the projects funded under access to expertise.

8.0 Performance measurement

8.1 Dairy Farm Investment Program

The DFIP Performance Information Profile identifies indicators, measures and targets for immediate, intermediate and ultimate outcomes and the program has successfully collected data for each of these. However, the absence of ultimate outcome indicators linking the program to its stated objective of supporting farmers to mitigate the impacts of anticipated market loss due to CETA makes it difficult to assess the extent to which DFIP meets this challenge.

The indicators and targets for the immediate and intermediate outcomes would benefit from clearer distinction to better support the logical flow and narrative of how DFIP activities will lead to successive levels of outcomes. Presently, the indicators measure the number of projects that are funded. This is an important output of the program, but is not an intermediate outcome.

Inconsistencies between the units used for targets and those in which data was collected make it difficult to assess DFIP outcomes against established measures. For example, the targets for ultimate outcomes are set as number of labour hours saved weekly and number of dollars saved annually, but results were reported in ranges of percentages of actual cost savings and actual direct labour savings as seen in Figure 2 above. However, data from these projects indicates overall improvements to labour and cost savings from their baselines.

8.2 Dairy Processing Investment Fund

There are some challenges identified in the performance measurement strategy for DPIF that will make it difficult to measure its effectiveness. For example, the access to expertise component was not included in the measurement of the program's outcomes or indicators.

The indicators in DPIF's Performance Information Profile do not link the program's outputs and expected outcomes to the purpose of DPIF, which was to help the dairy sector adapt to the market changes resulting from the implementation of CETA.

Additionally, one intermediate and 2 ultimate outcome indicators at the program level have targets that are lower than the baselines. The evaluation found that program targets were set in March 2020 based on workplans and that the program's baselines were updated in September 2021 based on results reported at the time.

Cumulatively, these limitations will make it difficult to demonstrate that DPIF achieved either its objective (to support Canadian dairy agri-food processors in becoming more productive, efficient and competitive) or its purpose (to help the dairy sector adapt to the market changes resulting from the implementation of CETA).

9.0 Conclusions and recommendations

This section presents the evaluation's conclusions and recommendations.

9.1 Conclusions

Both DFIP and DPIF were aligned with departmental and federal priorities to support the Canadian dairy sector adapt to market changes resulting from CETA. However, this evaluation was unable to determine whether either program mitigated anticipated future growth losses or other anticipated impacts from CETA for program participants.

Both DFIP and DPIF implemented good practices in program design and delivery that can and have informed other AAFC programs, including implementing a two-step application process, using a data management system and integrating specialized operations teams. However, oversubscription, insufficient funding to meet demand and a lack of clear communication about the intent of DFIP and changes to key DPIF program parameters resulted in challenges managing stakeholder expectations.

Both DFIP and DPIF were open to Canadian dairy producers and processors whose proposed projects met the eligibility criteria. The evaluation identified a potential barrier to participation for new producers or producers under the age of 35 who may have struggled to match the 50% funds required by DFIP.

DFIP achieved its outcomes to varying extents depending on the type and size of project implemented, with most recipients reporting at least moderate cash and labour cost savings. While it is too early to assess achievement of DPIF outcomes for capital investment projects as project results will continue to be reported until 2026, early results indicate progress toward the outcomes such as increases in production and revenue.

Performance measurement challenges in both programs make it difficult to assess program impacts or whether the programs met their stated objective of supporting the dairy sector's adaptation to CETA impacts. Given the lack of ultimate outcome indicators linking either program's activities to the objective of supporting the dairy sector to adapt to CETA impacts, the evaluation was not able to determine the extent to which either program met this objective.

9.2 Recommendations

Recommendation 1: The Assistant Deputy Minister, Programs Branch should document and share within AAFC the good practices used in DFIP and DPIF design and delivery.

Recommendation 2: The Assistant Deputy Minister, Programs Branch working with the Assistant Deputy Minister, Strategic Policy Branch should review current supply management programs to ensure that they do not present barriers for producers and processors who are members of underrepresented groups and that Gender-based Analysis Plus performance measures are in place.

Annex A: Evaluation methodology

Literature review

This line of evidence included a review of peer reviewed literature examining the need for and impact of DFIP and DPIF and government support for dairy producers and processors more broadly. The literature review contributed primarily to the assessment of program relevance. The evaluation team collaborated with the Canadian Agricultural Library to identify the peer reviewed literature.

Program document and file review

The evaluation included a review of DFIP and DPIF foundational documents, as well as departmental and federal government reports, to assess the relevance of DFIP and DPIF.

Files on DFIP and DPIF projects funded from 2017–18 to 2022–23 were reviewed. This review contributed to the assessment of program efficiency and effectiveness.

The evaluation team collaborated with AAFC's Research and Analysis Directorate (RAD) to identify any relevant analyses that they have conducted that have assessed the effectiveness of DFIP and DPIF. For example, AAFC's Research and Analysis Directorate concluded an impact assessment of DPIF in a linked file environment in the fall of 2022, the results of which were taken into consideration as part of the evaluation.

Key informant interviews

To support an assessment of DFIP and DPIF relevance, efficiency and effectiveness, interviews were conducted to assess sector risks or programming gaps addressed by DFIP and DPIF, their progress toward expected outcomes, internal and external factors that have facilitated or hindered progress and program efficiencies. The key informant interviews were used to support evidence obtained through other lines of evidence.

20 interviews were conducted. This included interviews with federal officials representing DFIP (n=2) and DPIF (n=4), a sample of DPIF program recipients based on funding distribution across provinces and territories (n=6), dairy industry associations (n=2), departmental experts (n=3) and academic experts (n=3). Where relevant, key informants were asked to provide insight into the relevance and performance of both DFIP and DPIF.

Interviews were conducted by telephone, videoconference, or email and were conducted by OAE.

Methodological limitations

Limitation	Mitigation strategy	Impact on evaluation
Data availability: Some data was missing or unavailable. DFIP and DPIF were extended by up to 2 years to provide recipients with additional time to complete their projects and demonstrate achievement of their targets. As a result, data on the achievement of outcomes was limited.	The evaluation assessed progress toward the achievement of outcomes and considered additional analyses completed within AAFC and by academia and think tanks.	Medium-High
Response bias: Interviewees may have been biased in their responses, based on their role and responsibility in relation to DFIP, DPIF and dairy supply management, as well as personal experiences and perspectives.	Interviewees represented a full and diversity of views, interview data was triangulated with other lines of evidence and findings were reviewed by multiple team members to identify elements of bias.	Low

Annex B: DFIP logic model

Activities

Program activities

• Receive applications
• Issue approval/rejection letters
• Prepare Contribution Agreements for review
• Process financial claims
• Issue AAFC funding
• Sign Contribution Agreements
• Performance data collection and analysis

Recipient activities

• Capital equipment purchases, hiring of required external expertise/consultants related to the implementation of dairy farm expansion and/or renovation.

Outputs

Program outputs

• Approval/rejections letters sent out
• Contribution Agreements signed
• Financial claims processed
• Documents/records indicating performance and financial information

Recipient outputs

• Dairy farms apply for and receive funding through Contribution Agreements to modernize and/or expand dairy production facilities

Immediate outcomes

• Participation – dairy farms participate in the program

Intermediate outcomes

• Canadian dairy farms invest in new equipment to enhance their productivity

Ultimate outcomes

• Increased farm efficiency reduces required level of effort to produce milk
• Increased farm efficiency reduces expenses for participating farms

Source: Supply Management Initiatives Performance Information Profile, Programs Branch, 2022

Annex C: DPIF logic model

Activities

Program activities

• Receive applications
• Issue approval/rejection letters
• Prepare Contribution Agreements for review
• Process financial claims
• Issue AAFC funding
• Sign Contribution Agreements
• Performance data collection and analysis

Recipient activities

• Capital equipment purchases, hiring of required external expertise/consultants related to the implementation, certification or validation of plant improvements including those related to federal registration and quality assurance systems; expansion and/or renovation of existing establishments
• Processor engagement of business consulting and advisory services
• Training/knowledge transfer activities led by not-for-profit organizations
• Engaging AAFC researchers to assist with product and/or process development or improvement

Outputs

Program outputs

• Approval/rejections letters sent out
• Contribution Agreements signed
• Financial claims processed
• Documents/records indicating performance and financial information

Recipient outputs

• Processing facilities apply for and receive funding through Contribution Agreements to modernize and/or expand dairy processing facilities and for technical expertise

Immediate outcomes

• Participation - dairy processing facilities and not-for-profit organizations participate in the program

Intermediate outcomes

• Processors' productivity is advanced
• Processors increase the use of fluid or industrial milk and milk components
• Environmental sustainability of processors' production is increased
• Technical expertise provided to processors improves production processes and product quality
• New or improved Canadian dairy products are introduced in the market

Ultimate outcomes

• Canadian dairy processors are competitive

Source: Supply Management Initiatives Performance Information Profile, Programs Branch, 2022