Part 1 |
Save page Remove page | Previous | 1 of 2 | Next |
|
|
small (250x250 max)
medium (500x500 max)
Large
Extra Large
Full Size
Full Resolution
|
This page
All
|
Ab« Associates Inc. Evaluation of the Nutrient Standard Menu Planning Demonstration 55 Wheeler Street Cambridge, Massachusetts 02138-1168 617 492-7100 telephone 617 492-5219 ftaimiU Final Report August 1998 Contract No. 53-3198-4-001 Hampden Square, Suite 500 4800 Montgomery Lane Bethesda, Maryland 20814-5341 301 913-0500 telephone 301 652-1618fusmik 640 North LaSalle Street Suite 400 Chicago, Illinois 60610-3781 312 867-4000 telephone 312 867 -4200 ftatmtU Prepared/or: JohnEndahl U.S. Dept. of Agriculture Food and Nutrition Service Office of Analysis and Evaluation 3101 Park Center Drive Alexandria, VA 22302 Prepared by. Mary Kay Fox Mary Kay Crepinsek Nancy Buratein Patty Connor Frederic Glantz Abt Associates Inc. 55 Wheeler Street Cambridge. MA 02138 Table of Contents Page List of Exhibits iii Acknowledgments xiv Chapter 1 Introduction 1-1 The National School Lunch and School Breakfast Programs 1-1 The NSMP Demonstration 1-5 The School Meals Initiative for Healthy Children 1-5 Organization of this Report 1-7 Chapter 2 Evaluation Design , 2-1 Research Objectives 2-1 Evaluation Design 2-2 Standards Used in Evaluating Nutrient Content 2-6 Estimation of Program Impact 2-8 Chapter 3 Implementation ofNSMP in Demonstration SFAs 3-1 Selection of Demonstration SFAs 3-1 Early Implementation Challenges 3-4 Status of Implementation in Spring 1997 3-4 Implementation ofNSMP in Demonstration SFAs 3-8 Characteristics That May Predict Successful NSMP Implementation 3-18 Chapter 4 Nutrient Content of Lunches Before and After NSMP 4-1 Overview of the Analysis 4-1 Frequency of Foods Offered in Lunch Menus 4-2 Nutrient Profile of Lunches Served Before and After NSMP 4-5 Differences Between Weighted and Unweighted Analyses 4-11 Chapter 5 Nutrient Content of Breakfasts Before and After NSMP 5-1 Frequency of Foods Offered in Breakfast Menus 5-1 Nutrient Profile of Breakfasts Served Before and After NSMP 5-3 Differences Between Weighted and Unweighted Analyses 5-10 Chapter 6 Impact ofNSMP on Program Operations and Costs 6-1 Program Participation 6-1 Meal Acceptability 6-3 Meal Costs 6-4 A la Carte Revenues 6-12 Use of USDA-Donated Commodities 6-12 Menu Planning Practices 6-16 Abt Associates Inc. U Table of Contents i Table of Contents (Continued) Page Chapter 7 Burden and Acceptability of the NSMP System 7-1 Start-up Costs 7-1 Maintenance Costs 7-6 SFA Directors' Perceptions About Burden of NSMP Tasks 7-9 Perceptions of School Food Service Staff 7-11 SFA Directors' Attitudes Toward NSMP 7-13 State Directors' Attitudes Toward NSMP 7-16 Chapter 8 State Monitoring of SFAs Using the NSMP Option 8-1 State Monitoring Requirements 8-1 State Agency Experiences with Demonstration SFAs 8-3 Changes in Monitoring Procedures to Accommodate SMI Reviews 8-6 Perceptions about SMI Monitoring Requirements 8-11 Chapter9 Accuracy of SFAs'Nutrient Analyses 9-1 Problems Detected in Nutrient Analyses Conducted by SFA Staff 9-1 Findings from On-Site Meal Observations 9-12 References Appendix A Supplementary Exhibits: Nutrient Content of Lunches A-1 Appendix B Supplementary Exhibits: Nutrient Content of Breakfasts B-1 Appendix C Study Implementation C-l Appendix D Methodology Used in Nutrient Analysis D-l Appendix E Methodology Used in Study of Meal Costs E-l Appendix F Methodology Used in Sampling and Weighting Surveys of School Food Service Staff F-l AM Associates Inc. Table of Contents ii 3 List of Exhibits Page Exhibit 1.1 Traditional Meal Pattern Requirements for the National School Lunch Program . 1 -3 Exhibit 1.2 Traditional Meal Pattern Requirements for the School Breakfast Program 1-4 Exhibit 2.1 Evaluation Design 2-3 Exhibit 2.2 Data Collected at Each Measurement Point 2-4 Exhibit 2.3 Nutrient Standards Used in the Evaluation 2-7 Exhibit 3.1 Characteristics of Demonstration SFAs 3-2 Exhibit 3.2 Final Implementation Status of Demonstration SFAs 3-5 Exhibit 3.3 Reasons for Withdrawal from the Demonstration 3-7 Exhibit 3.4 Software Systems Used by Demonstration SFAs 3-9 Exhibit 3.S Disclosure of Nutrition Information by Demonstration SFAs 3-12 Exhibit 3.6 Cafeteria Promotions Conducted by Demonstration SFAs 3-13 Exhibit 3.7 Use of Fortified Foods by Demonstration SFAs 3-14 Exhibit 3.8 Reported Successes and Problems in Meeting Nutrient Standards 3-16 Exhibit 3.9 Time Required by Demonstration SFAs to Achieve Full Implementation 3-17 Exhibit 3.10 Characteristics of Demonstration SFAs by Final Implementation Status 3-19 Exhibit 4.1 Foods Offered More and Less Often in NSMP Lunch Menus Compared to Pre-NSMP Lunch Menus 4-4 Exhibit 4.2 With the Exception of Calories in Secondary School, Lunches Served Before and After NSMP Provided More than One-third of the RDA 4-6 Exhibit 4.3 Lunches Served After NSMP Provided Fewer Calories from Fat and Saturated Fat than Lunches Served Before NSMP 4-8 Exhibit 4.4 More SFAs Met Standards for Calories from Fat and Saturated Fat in Lunches After Implementing NSMP 4-10 AM Associates Inc. List of Exhibits Si f List of Exhibits (Continued) Page Exhibit 4.5 Lunches Served Before and After NSMP Met the NRC Recommendation for Cholesterol but Not for Sodium 4-12 Exhibit 4.6 Weighted and Unweighted Analyses of 1997 Lunches Yield Slightly Different Mean Values for Calories and RDA Nutrients, but Conclusions About Whether Lunches Met NSMP Standards are Similar 4-14 Exhibit 4.7 Weighted and Unweighted Analyses of Lunches Yiel \ Similar Results for Calories from Fat, Saturated Fat, Protein, and Carbohydrate 4-16 Exhibit 4.8 Weighted and Unweighted Analyses of 1997 Lunches Yield Slightly Different Mean Values for Cholesterol, Sodium, and Fiber, but Conclusions About Whether Lunches Met NRC Recommendations are Similar 4-17 Exhibit S. 1 Foods Offered More and Less Often in NSMP Breakfast Menus Compared to Pre-NSMP Breakfast Menus 5-2 Exhibit 5.2 With the Exception of Calories, Breakfasts Served Before and After NSMP Provided More than One-Fourth of the RDA 5-4 Exhibit 5.3 Breakfasts Served After NSMP Provided Significantly Fewer Calories from Fat and Saturated Fat than Breakfasts Offered Before NSMP 5-7 Exhibit 5.4 More SFAs Met Standards for Calories from Fat and Saturated Fat in Breakfasts After Implementing NSMP 5-8 Exhibit 5.5 With the Exception of Sodium in Secondary Schools, Breakfasts Served Before and After NSMP Met the NRC Recommendations for Cholesterol and Sodium 5-9 Exhibit 5.6 Weighted and Unweighted Analyses of Breakfasts Yield Similar Mean Values for Calories and RDA Nutrients 5-11 Exhibit 5.7 Weighted and Unweighted Analyses of Breakfasts Yield Similar Results for Calories from Fat, Saturated Fat, Protein, and Carbohydrate 5-13 Exhibit 5.8 Weighted and Unweighted Analyses of 1997 Breakfast Menus Yield Comparable Results for Cholesterol. Sodium and Fiber Content 5-15 Exhibit 6.1 Average Daily NSLP Participation Was Essentially Constant Over the Course of the Demonstration 6-2 Abt Associates Inc. „ List of ExnlWts Iv r List of Exhibits (Continued) Page Exhibit 6.2 Mean SBP Participation Rates Before and After NSMP 6-3 Exhibit 6.3 Kitchen Managers' Perceptions About Changes in Lunch Plate Waste Since Implementation of NSMP 6-5 Exhibit 6.4 Kitchen Managers' Perceptions About Changes in Breakfast Plate Waste Since Implementation of NSMP 6-7 Exhibit 6.5 School Food Service Staff Perceptions About Students' Acceptance of School Meals Since NSMP 6-8 Exhibit 6.6 On Average, the Cost of Producing a Reimbursable Lunch Under NSMP Was Comparable to or Lower than the Cost Before NSMP 6-10 Exhibit 6.7 Costs of Producing a Reimbursable Lunch Before and After Implementation of NSMP 6-11 Exhibit 6.8 On Average, the Cost of Producing a Reimbursable Breakfast Under NSMP Was Comparable to or Lower than the Cost Before NSMP 6-13 Exhibit 6.9 Costs of Producing a Reimbursable Breakfast Before and After Implementation of NSMP 6-14 Exhibit 6.10 A la Carte Revenues Before and After Implementation of NSMP 6-15 Exhibit 6.11 Value of USDA-Donated Commodities Used Before and After Implementation of NSMP 6-16 Exhibit 6.12 On Average, Lunches Offered After NSMP Included as Many Options as Lunches Offered Before NSMP 6-18 Exhibit 6.13 The Amount of Variety in Weekly Lunch Menus Planned After NSMP Was Comparable to Menus Planned Before NSMP 6-19 Exhibit 6.14 On Average, Breakfasts Offered After NSMP Included as Many Options as Breakfasts Offered Before NSMP 6-21 Exhibit 6.15 The Amount of Variety in Weekly Breakfast Menus Planned After NSMP Was Comparable to Menus Planned Before NSMP 6-22 AM Associates Inc. \ List of Exhibits v List of Exhibits (Continued) Page Exhibit 7.1 Non-Labor Start-up Costs Reported by Demonstration SFAs 7-2 Exhibit 7.2 Labor Hours Devoted to NSMP Implementation, by Task 7-5 Exhibit 7.3 Most of the Time Devoted to NSMP Implementation Was Spent on Data Entry and Staff Training 7-7 Exhibit 7.4 Time Required for NSMP Maintenance Compared to Time Spent on Previous Menu Planning Practices 7-8 Exhibit 7.5 SFA Directors Found Obtaining Menu Production Data to be the Most Burdensome NSMP Task 7-10 Exhibit 7.6 School Food Service Staff Perceptions About Impact of NSMP on Overall Paperwork Burden 7-12 Exhibit 7.7 School Food Service Staff Perceptions About Effect of NSMP on Job 7-13 Exhibit 7.8 SFA Directors' Opinions About NSMP 7-14 Exhibit 7.9 SFA Directors' Perceptions About Advantages and Disadvantages of NSMP ... 7-15 Exhibit 7.10 State Directors'Opinions About NSMP 7-17 Exhibit 7.11 State Directors' Estimates of Non-Demonstration SFAs' Menu Planning Selections 7-18 Exhibit 7.12 State Directors' Perceptions About Why SFAs WiU Not Select NSMP 7-19 Exhibit 7.13 Recommendations of SFA and State Directors to Increase Likelihood of Successful NSMP Implementation 7-21 Exhibit 8.1 Reasons for State Visits to Demonstration SFAs 8-3 Exhibit 8.2 NSMP Topics Covered During State Visits to Demonstration SFAs 8-4 Exhibit 8.3 Methods Used by State Agency Staff to Assess Accuracy of SFAs' Nutrient Analyses 8-5 Exhibit 8.4 Summary of Problems Identified by State Monitoring Staff 8-7 Inc. U«t of Exhibits vi n List of Exhibits (Continued) Page Exhibit 8.5 Changes in State Schedules and Procedures to Accommodate SMI Monitoring .. 8-9 Exhibit 8.6 State Directors' Opinions About SMI Monitoring Requirements 8-12 Exhibit 9.1 Errors Found in Weekly Menu Analyses Completed by SFA Staff 9-3 Exhibit 9.2 Errors Found in Recipes Entered and Analyzed by SFA Staff 9-6 Exhibit 9.3 Data Entry Errors in Foods Added to the Nutrient Database by SFA Staff 9-8 Exhibit 9.4 Prevalence of Missing Data in Information Provided by Manufacturers and Data Entry Decisions Made by SFAs 9-9 Exhibit 9.5 Age Groups Analyzed by Predominant Grade Configuration 9-11 Exhibit 9.6 Adherence to Planned NSMP Menus 9-13 Exhibit 9.7 Items Omitted and Not Substituted in Observed Lunches 9-M Exhibit 9.8 Unanalyzed Items Offered During Observed Lunches 9-15 Exhibit 9.9 Adherence to Planned Portion Sizes 9-16 Exhibit A.1 Mean Calorie and Nutrient Content of Lunches Served Before and After NSMP A-1 ExhibitA.2 Mean Nutrient Profile of Lunches Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations A-2 Exhibit A.3 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Lunch Before and After NSMP A-3 Exhibit A.4 Distribution of Percentage of Calories from Fat and Saturated Fat in Lunches Served Before and After NSMP A-4 Exhibit A.5 Mean Calorie and Nutrient Content of Lunches Served Before and After NSMP: Unweighted Analysis A-6 Exhibit A.6 Mean Nutrient Profile of Spring 1997 Lunches: Comparison of Weighted and Unweighted Analyses A-7 Inc. I1 LMofExhMts vN List of Exhibit* (Continued) Pay* Exhibit A.7 Mean Nutrient Profile of Spring 1994 Lunches: Comparison of Weighted and Unweighted Analyses A-8 Exhibit A.8 Mean Nutrient Profile of Lunches Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Unweighted Nutrient Analysis A-9 Exhibit A.9 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Lunch Before and After NSMP: Unweighted Nutrient Analysis A-10 Exhibit A. 10 Distribution of Percentage of Calories from Fat and Saturated Fat in Lunches Served Before and After NSMP: Unweighted Nutrient Analysis A-l 1 Exhibit A. 11 Mean Calorie and Nutrient Content of Lunches Served Before and After NSMP: Middle Schools and High Schools A-12 Exhibit A.12 Mean Nutrient Profile of Lunches Served Before and After NSMP Compared to NSMP Standards arid NRC Recommendations: Middle Schools and High Schools A-13 Exhibit A. 13 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Lunches Before and After NSMP: Middle Schools and High Schools A-14 Exhibit A. 14 Mean Calorie and Nutrient Content of Lunches Served Before and After NSMP: Middle Schools and High Schools Unweighted Analysis A-13 Exhibit A.15 Mean Nutrient Profile of Lunches Served Before a.id After NSMP Compared to NSMP Standards and NRC Recomrnendaticns: Middle Schools and High Schools Unweighted Nutrient Analysis A-16 Exhibit A. 16 Number of SFAs That Satisfied Nutrient Standards and Recommendatior for Lunches Before and After NSMP: Middle Cchools and High Schooi- Unweighted Nutrient Analysis A-17 Exhibit A.17 Mean Calorie and Nutrient Content of Lunches Served Before and After NSMP Elementary Schools in Early-Implementing SFAs A-19 Exhibit A. 18 Mean Nutrient Profile of Lunches Served Before and After NSMP Compart 1 to NSMP Standards and NRC Recommendations: Elementary Schools in Early-Implementing SFAs A-20 Inc. ^ List of Exhibits vW 7 List of Exhibits (Continued) Exhibit A. 19 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Lunches Before and After NSMP: Elementary Schools in Eariy-Implementing SFAs A-21 Exhibit A.20 Mean Calorie and Nutrient Content of Lunches Served Before and After NSMP: Elementary Schools in Early-Implementing SFAs Unweighted Nutrient Analysis A-22 Exhibit A.21 Mean Nutrient Profile of Lunches Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Elementary Schools in Eariy-Implementing SFAs Unweighted Nutrient Analysis A-23 Exhibit A.22 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Lunches Before and After NSMP. Elementary Schools in Early-Implementing SFAs Unweighted Nutrient Analysis A-24 Exhibit A.23 Mean Calorie and Nutrient Content of Lunches Served Before and After NSMP: Secondary Schools in Early-Implementing SFAs A-25 Exhibit A. 24 Mean Nutrient Profile of Lunches Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Secondary Schools in Eariy-Implementing SFAs . A-26 Exhibit A.25 Number of SFAs That Satis*rd Nutrient Standards and Recommendations for Lunches Before and After NSMP: Secondary Schools in Early Implementing SFAs A-27 Exhibit A.26 Mean Calorie and Nutrient Content of Lunci.es Served Before and After NSMP: Secondary Schools in Early-Implementing SFAs Unweighted Nutrient Analysis A-28 Exhibit A.27 Mean Nutrient Profile of Lunches Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Secondary Schools in Eariy-Implementing SFAs Unweighted Nutrient Analysis A-29 Exhibit A.28 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Lunches Before and After NSMP Secondary Schools in Eariy-Implementing SFAs Unweighted Nutrient Analysis A-30 Inc. List of Exhibits ix /0 List of Exhibits (Continued) Page Exhibit A.29 Grade Groups Used by Demonstration SFAs and Associated Nutrient Standards for Lunches A-32 Exhibit A.30 Mean Percent FDA Provided by Lunches Served Before and After NSMP Compared to Minimum Nutrient Standards Defined in Regulations A-33 Exhibit A.31 Mean Percent RDA Provided by Lunches Served Before and After NSMP Compared to Minimum Nutrient Standards Defined in Program Regulations: Unweighted Nutrient Analysis A-34 Exhibit B. 1 Mean Calorie and Nutrient Content of Breakfasts Served Before and After NSMP B-l Exhibit B.2 Mean Nutrient Profile of Breakfasts Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations B-2 Exhibit B.3 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Breakfast Before and After NSMP B-3 Exhibit B.4 Distribution of Percent of Calcries from Fat and Saturated Fat in Breakfasts Served Before and After NSMP B-4 Exhibit B.5 Mean Calorie and Nutrient Content of Breakfasts Served Before and After NSMP: Unweighted Analysis B-5 Exhibit B.6 Mean Nutrient Profile of Spring 1997 Breakfasts: Comparison of Weighted and Unweighted Analyses B-6 Exhibit B.7 Mean Nutrient Profile of Spring 1994 Breakfasts: Comparison of Weighted and Unweighted Analyses B-7 Exhibit B.8 Mean Nutrient Profile of Breakfasts Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Unweighted Nutrient Analysis B-8 Exhibit B.9 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Breakfast Before and After NSMP: Unweighted Nutrient Analysis B-9 Exhibit B.10 Distribution of Percent of Calories from Fat and Saturated Fat in Breakfasts Served Before and After NSMP: Unweighted Nutrient Analysis B -10 tac UatofExMM* a // List of Exhibits (Continued) Page Exhibit B. 11 Mean Calorie and Nutrient Content of Breakfasts Served Before and After NSMP: Middle Schools and High Schools B-12 Exhibit B. 12 Mean Nutrient Profile of Breakfasts Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Middle Schools and High Schools B-13 Exhibit B. 13 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Breakfast Before and After NSMP: Middle Schools and High Schools B-14 Exhibit B. 14 Mean Calorie and Nutrient Content of Breakfasts Served Before and After NSMP: Middle Schools and High Schools Unweighted Analysis B-15 Exhibit B. 15 Mean Nutrient Profile of Middle and High School Breakfasts Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Middle Schocls and High Schools Unweighted Nutrient Analysis B-16 Exhibit B. 16 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Breakfast Before and After NSMP Middle Schools and High Schools: Unweighted Nutrient Analysis B-17 Exhibit B. 17 Mean Calorie and Nutrient Content of Breakfasts Served Before and After NSMP: Elementary Schocls in Early-Implementing SFAs B-19 Exhibit B. 18 Mean Nutrient Profile of Breakfasts Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Elementary Schools in Early-Implementing SFAs B-20 Exhibit B. 19 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Breakfast Before and After NSMP: Elementary Schools in Early- Implementing SFAs , B-21 Exhibit B.20 Mean Calorie and Nutrient Content of Breakfasts Served Before and After NSMP: Elementary Schools in Early-Implementing SFAs Unweighted Nutrient Analysis B-22 Exhibit B.21 Mean Nutrient Profile of Breakfasts Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Elementary Schools in Early- Implementing SFAs Unweighted Nutrient Analysis B-23 Abt Associates Inc. List of Exhibits xi ^ List of Exhibits (Continued) Page Exhibit B.22 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Breakfast Before and After NSMP: Elementary Schools in Early-Implementing SFAs Unweighted Nutrient Analysis B-24 Exhibit B.23 Mean Calorie and Nutrient Content of Breakfasts Served Before and After NSMP: Secondary Schools in Early-Implementing SFAs B-2S Exhibit B.24 Mean Nutrient Profile of Breakfasts Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Secondary Schools in Early- Implementing SFAs B-26 Exhibit B.25 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Breakfast Before and After NSMP: Secondary Schools in Early-Implementing SFAs B-27 Exhibit B.26 Mean Calorie and Nutrient Content of Breakfasts Served Before and After NSMP: Secondary Schools in Early-Implementing SFAs Unweighted Nutrient Analysis B-28 Exhibit B.27 Mean Nutrient Profile of Breakfasts Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Secondary Schools in Early- Implementing SFAs Unweighted Nutrient Analysis B-29 Exhibit B.28 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Breakfast Before and After NSMP: Secondary Schools in Early-Implementing SFAs Unweighted Nutrient Analysis B-30 Exhibit B.29 Grade Groups Used by Demonstration SFAs and Associated Nutrient Standards for Breakfasts B-32 Exhibit B.30 Mean Percent RDA Provided by Breakfasts Served Before and After NSMP Compared to Minimum Nutrient Standards Defined in Regulations B-33 Exhibit B.31 Mean Percent RDA Provided by Breakfasts Served Before and After NSMP Compared to Minimum Nutrient Standards Defined in Program Regulations: Unweighted Nutrient Analysis B-34 Exhibit D.l Weighted and Unweighted Coding for Sample Menu D-4 ExnibitEl Overview of Meal Coit Methodology Framework E-2 AW Associate* Inc. .1st of Exhibits xll /J List of Exhibits (Continued) Pag* Exhibit E.2 Calculation of the Total Food Costs Attributable to Breakfast and Lunch E-4 Exhibit E.3 Calculation of the Labor Costs Attributable to Breakfast and Lunch E-6 Exhibit E.4 Calculation of the Total Reported Cost of Producing Breakfast and Lunch E-7 Exhibit E.5 Estimation of the Percentage of Food Costs That Arc Reimbursable E-8 Exhibit F. 1 Number of Schools in Universe and Number of Schools Sampled by Stratum and Across All School Types: Spring 1996 and Spring 1997 F-l Exhibit F.2 Response Rates for Surveys of School Food Service Staff F-3 Abt Associates Inc. List of Exhibits xiii H Acknowledgments The NSMP Demonstration and its attendant evaluation are indebted to the staff in the 34 school districts who participated in the demonstration and supported the evaluation through its three-year time line. Special thanks are due to staff in the 23 districts that remained in the demonstration all three years. Their unfailing cooperation made this evaluation possible. Thanks are also due to staff in the 17 State offices who provided information on experiences with demonstration SFAs and on reactions to new regulations governing meal planning practices in Child Nutrition programs. Staff of the Office of Analysis and Evaluation, Food and Nutrition Service, of the U.S. Department of Agriculture had responsibility for overseeing the project. The Project Officer, John Endahl, guided all stages of project planning and implementation as well as development of study reports. Leslie Cristovich also provided valuable insights and direction throughout the project and in preparation of the final report. Finally, several staff members at Ab* Associates played important roles on the project including Patty Connor (Manager of On-site Data Collection and Nutrition Coding); Diane Greene (Survey Director); Gail Langeloh and Mary Kay Crepinsek (Senior Nutritionists); Nancy Burstein (Director of Analysis); Frederic Glantz (Task Manager for the Cost Analysis); K.P. Srinath (Sampling Statistician); Ellie Lee and Don Laliberty (Programmers); and Susan Altman (Editor). Eileen Fahey served as the contract secretary, responsible for production of all study reports, and was ably assisted by Eileen McEnaney and Judi Groves. AMAssociates Inc. ti Acknowledgments xiv Chapter 1 Introduction The National School Lunch Program (NSLP), operated by the Food and Nutrition Service (FNS) of the U.S. Department of Agriculture (USDA), has been providing meals to the Nation's school children since 1946. The School Breakfast Program (SBP) has been in operation since the early 1970s. Over the years, research has shown that meals offered in both the NSLP and SBP have provided children with the calories, vitamins, and minerals needed to sustain health and promote normal growth. More recent research has shown, however, that in comparison to current public health recommendations, school lunches are high in fat, saturated fat, and sodium, and low in carbohydrate. Since 1993, when this situation was first identified, FNS has been working on many fronts to enhance the nutritional quality of meals offered in the NSLP and SBP. As part of this ongoing initiative, FNS sponsored a demonstration project to evaluate the acceptability and impact of an alternative system for planning lunch and breakfast menus. This system, known as Nutrient Standard Menu Planning (NSMP), utilizes computerized nutrient analysis to monitor the nutrient content of meals served to students. Abt Associates Inc. of Cambridge, Massachusetts was awarded a contract to conduct an independent evaluation of the three-year demonstration project. This report presents results of that evaluation. The National School Lunch and School Breakfast Programs All public and private nonprofit schools are eligible to participate in both the NSLP and the SBP. The NSLP was established in 1946 "to safeguard the health and well-being of the Nation's children and to encourage the domestic consumption of nutritious agricultural commodities and other foods."1 Today the NSLP operates in over 94,000 schools and institutions. More than 26 million children receive meals through the program on any given day; about half of these meals are provided free of charge. The SBP began as a pilot project in the mid-1960s to support the provision of breakfast to children living in "poor areas and areas where children [had] to travel a great distance to school."2 The SBP was officially authorized as a permanent program in 1975, and the target population was expanded to include "all schools where [the program] is needed to provide adequate nutrition for all children in attendance."1 Currently, the SBP operates in approximately two-thirds of the schools and institutions that offer the NSLP, most commonly in schools that serve large numbers of economically disad vantaged children. On an average day, roughly seven million children receive breakfast through the SBP. The vast majority of these meals are provided free of charge. School Food Authorities (SFAs) participating in the NSLP and SBP receive two types of federal assistance: donated surplus commodities and cash reimbursements. Entitlement to surplus commodities 1 N«uon«l School Lunch Act oflM6. PL 79-396. 2 Child Nutrition Act of 1996. PL 19-642. 3 1975 Amendments to theChikJ Nutrition Act, PL 94-105 Abt Associates Inc. tf Introduction 1-1 is based on an established per-meal flat rate ($0,145 per meal in 1996-97) applied to the number of reimbursable lunches served the previous year. In addition, SFAs may receive bonus commodities which do not count against entitlements. Subject to availability, SFAs may request bonus commodities in amounts that can be used without waste. The type and amount of bonus commodities available varies from year to year depending on purchasing decisions made by USDA. Cash reimbursements for NSLP and SBP meals are based on the number of meals served, established per-meal reimbursement rates, and the poverty level of participating students. SFAs receive a base payment for each meal served, with substantially higher rates paid for meals served free or at a reduced price to income-eligible students. Childrens' household size and income determine their eligibility for free and reduced-price meal benefits. Nutrition Standards for School Meals Program regulations require that meals served in the NSLP and SBP meet defined nutrition standards in order to be eligible for Federal subsidies. Program regulations for the NSLP stipulate that "the nutrients of the lunch, averaged over a period of time, approximate one-third of the National Academy of Science's Recommended Dietary Allowances (RDAs) for children in specific age/grade groups."4 To ensure that these standards are met, program regulations have historically included food-based menu planning guidelines. These guidelines, originally known as the Type A meal pattern," define specific types of food iO be offered as well as minimum acceptable portion sizes (Exhibit 1.1). Although Lie authorizing legislation for the SBP did not include a specific RDA goal for breakfast meals, a meal pattern, modeled after the NSLP meal pattern, was developed to ensure that breakfasts would provide approximately one-fourth of the RDA (Exhibit 1.2). Over the years, research has shown that, with few exceptions, the NSLP has provided students the opportunity to satisfy one-third of their daily nutritional requirements (Wellisch, J., et al., 1983; St. Pierre, R., et al., 1992; and Burghardt, J., et al., 1993). Similarly, meals offered in the SBP have provided, on average, 20-25 percent of the RDAs (St. Pierre, R., et al., 1992; and Burghardt, J., et al., 1993). The most recent nationally representative study of school meals, the School Nutrition Dietary Assessment Study (SNDA), published in 1993, focused attention on another aspect of nutritional quality (Burghardt, J., et al., 1993). The SNDA study found that NSLP meals were high in fat, saturated fat, and sodium, and low in carbohydrate, in comparison to recommendations included in the Dietary Guidelines for Americans (U.S. Departments of Health and Human Services and Agriculture, 1995) and in the National Research Council's (NRC) Diet and Health report (National Research Council, 1989). At the time, programs were not required to meet these guidelines. In response to the SNDA findings, and as part of the national Healthy People 2000 initiative, USDA made a commitment to implement the Dietary Guidelines in the NSLP and SBP. The Department pledged to "increase to at least 90 percent [by the year 2000] the proportion of school lunch and breakfast services ... that are consistent with the nutrition principles in the Dietary Guidelinesfor Americans'' (U.S. Department of Health and Human Services, 1991). 4 7 CTR 245. Pwt 210 Abt Associate* Inc. Introduction 1-2 n Exhibit 1.1 Traditional Meal Pattern Requirements for the National School Lunch Program1 Meat or meat alternate Lean meat, poultry, or fish Cheese Large egg(s) Cooked dry beans or peas Peanut butter Peanuts, soy nuts, tree nuts, or seeds 1 serving per meal 2 OK. 2oz. 1 serving 1/2 cup 4 tbsp. 1 oz. s 1/2 the requirement Vegetables, fruits and/or full-strength jukes2 2 or more servings per meal, 3/4 cup total portion Bread or bread alternate Enriched or whole-grain bread Enriched or whole-grain biscuit, muffin, roll or equivalent Cooked enriched or whole grain rice, macaroni, noodles, or other cereal grains such as bulgur or corn grits 1 or more servings per meal/ 8 servings per week 1 slice 1 serving 1/2 cup Milk Fluid milk (whole milk and low fat milk must be offered daily) 1 serving per meal 1/2 pint (8 fluid oz.) 1 Requirement* were changed in June 1995 when the School Meals Initiative for Healthy Children wai implemented (tee text). 2 No more than one-half of the total requirement may be met with full-strength fruit or vegetable juice. a Introduction 1-3 Exhibit 12 Traditional Meal Pattern Requirements for the School Breakfast Program1 HMHRHHHHHBHBI 4 components must be offered: 1 serving of fluid milk 1 serving of vegetables, fruit, or juice 2 servings of bread/bread alternate, meat/meat alternate, or one serving of each Meat or meat alternate Lean meat, poultry, or fish Cheese Large egg Cooked dry beans or peas Peanut butter Peanuts, soy nuts, tree nuts, or seeds loz. loz. 1/2 4 tbsp. ■ 2 tbsp. loz. Vegetables, fruits, and/or full-strength juices 1/2 cup Bread or bread alternate Enriched or whole-grain bread Enriched or whole-grain biscuit, muffin, roll or equivalent Enriched, whole-grain, or fortified cereal 1 slice 1 serving 3/4 cup or 1 oz. Milk Fluid milk (whole milk and low fat milk must be offered daily) 1/2 pint (8 fluid oz.) Requirements were changed in June 1995 when the School Meals Initiative for Healthy Children was implemented (see text). AM Associate* Inc. n Introduction 1-4 The NSMP Demonstration As part of an ongoing initiative to enhance the nutritional quality of meals offered in the NSLP and SBP, FNS sponsored a demonstration project to evaluate the acceptability and impact of an alternative system for planning lunch and breakfast menus. This system, known as Nutrient Standard Menu Planning (NSMP), uses computerized nutrient analysis to plan menus rather than food-based menu planning guidelines. Under NSMP, SFAs are free to use whatever portions and combinations of food they wish in order to meet specific nutrient-based standards. Thus, in theory, NSMP provides more flexibility in menu planning while, at the same time, providing a greater degree of assurance that meals served to students meet nutrition standards. FNS selected 35 SFAs to participate in the demonstration, from a pool of 127 applicants. SFAs were purposefully selected to provide diversity in geographic location, district size, student participation rates, food service program characteristics, and staff experience with computerized nutrient analysis. One SFA dropped out shortly after selection because school board approval for the project was not obtained, and the district was not replaced. Thus, the demonstration began with 34 SFAs. Participating SFAs were expected to implement NSMP for both lunch and breakfast in all schools. Menus were to be planned using newly developed nutrient analysis software systems approved by FNS. Approval was granted only to systems that met detailed specifications set forth by the Agency. A key requirement was that the nutrient analysis be based on a weighted average of all foods served in reimbursable meals. A weighted analysis incorporates information about students' food selection patterns and gives more weight to foods that are selected more often. Approved NSMP software systems also incorporated the Child Nutrition Database (CN Database), which was developed by USDA specifically for use in the Child Nutrition programs. The CN Database includes nutrient information from the USDA standard nutrient database for foods routinely used in school food service programs. Initially, USDA planned to augment the main database with nutrient information for an extensive number of commercially prepared food items. USDA staff requested nutrition information from major food manufacturers that serve the school food service industry. As discussed later in this report, however, food manufacturers were slow to respond to this request. Menus planned under NSMP were expected to meet a defined set of nutrient standards. Lunches were expected to provide one-third of the RDA and breakfasts were expected to provide one-fourth of the RDA. Both meals were expected to be consistent with Dietary Guidelines goals for the percentage of calories from total fat and saturated fat. Chapter 2 includes detailed descriptions of the demonstration time line and implementation protocol as well as the research design used in the associated evaluation. The School Meals Initiative for Healthy Children USDA's commitment to implementing the Dietary Guidelines in school meals was formalized in the School Meals Initiative for Healthy Children (SMI),' which promoted expansive changes in nutrition S Federal RefM»er 60 113. 31188 31222. Jwt 13. 1995 to Introduction 1-6 standards for school meals. The original SMI proposal, issued in June 1994, required that all SFAs use NSMP (referred to as NuMenus) or an alternative known as Assisted NuMenus (ANSMP) to plan NSLP and SBP menus. (ANSMP allows SFAs to arrange or contract for menu development and nutrient analysis by an outside source such as a State agency, a consortium of SFAs, or a consultant.) Further, the initial SMI proposal required that school meals be consistent with the Dietary Guidelines by the start of SY 1998-99. The extended time line was built into the proposed regulation because comments received during public hearings and in response to an initial Federal Register notice indicated that some SFAs would need a considerable amount of time to implement NSMP or ANSMP and to develop menus consistent with the Dietary Guidelines. In November 1994, as part of the reauthorization of the Child Nutrition programs. Congress enacted The Healthy Meals for Healthy Americans Act.6 This law precipitated two important changes in the Department's initial SMI proposal. Fust, it required the Department to develop a modified food-based menu planning system, similar to those shown in Exhibits 1.1 and 1.2, that schools could use in lieu of NSMP or ANSMP. Second, the law shortened the time line for implcmc nation of the Dietary Guidelines, requiring that all SFAs be in compliance by tl * first day of the 1996-97 schooi year (two years earlier than the Department had suggested), unless a waiver was granted by the cognizant State agency. The final SMI rule, issued in June 1995, provided three options for menu planning: NSMP, ANSMP, and a modified food-based meal pattern (one for breakfasts and one for lunches).7 Menu planning options were further expanded in May 1996, when The Healthy Meals for Children Act allowed use of the traditional NSLP and SBP meal patterns, i.e., those in effect before the modified pattern defined in the SMI rule, or "any reasonable approach'' to satisfy the nutrient standards defined under SMI. SFAs that elect to use either of the food-based systems (the modified meal patterns defined in the SMI rule or the traditional meal patterns) or another approach approved by a cognizant State agency do not have to analyze the nutrient content of planned menus. However, every five years State tgency staff must analyze a representative weekly menu as part of a mandated SMI re view and compare results of the analysis to the same nutrient standards used in NSMP and ANSMP. The regulatory changes described above, which were implemented after the inception of the NSMP demonstration, have already incorporated NSMP as a menu planting option fcr the NSLP and SBP. Nonetheless, findings from the demonstration are important in understanding the challenges and benefits associated with use of NSMP. The collective experience of SFAs that participated in the demonstration will contribute to meaningful dialogue about the current NSMP system and may inform future decisions about program requirements, training, technical assistance, and monitoring. 6 PL I03-44S. 7 M. 104-149. Abt Aaaoclrt—Inc. Introduction- 1-6 *l Organization of this Report Eight additional chapters in this report present the following information: • Chapter 2 provides an overview of the design and objectives of the evaluation. • Chapter 3 describes the demonstration SFAs and the process of NSMP implementation. • Chapters 4 and 5 present information about the impact of NSMP on the nutrient content of lunches and breakfasts. These chapters also include comparisons of weighted and unweighted analyses of the same menus. • Chapter 6 reports on the impacts cf NSMP on food service program costs and operations. • Chapter 7 describes start-up costs associated with implementing NSMP It also summarizes the opinions and recommendations of key stakeholders, including SFA directors, school food service staff, and State agency staff. • Chapter 8 discusses monitoring and accountability under the new SMI regulations. Problems encountered by State agency staff who visited demonstration SFAs or completed SMI reviews in other SFAs are described. • Chapter 9 reports on problems encountered in the nutrient analyses completed by SCA staff. Findings from on-site observations of meal service are also included. Appendices provide supplementary exhibits as well as additional information about data collection methodologies, the development of sample weights for surveys of school food service staff, and methodologies used in assessing nutrient content and meal costs. Inc. u Introduction 1-7 Chapter 2 Evaluation Design This chapter outlines the research objectives defined for the evaluation and describes the research design. The sequence and timing of data collection are also discussed and resoondents and data sources are identified. Finally, the chapter identifies the nutrients that were examined in the analysis and the standards 'hat were usrd to evaluate the average nutrient content of school meals. Research Objectives The study included both process and impact evaluations, with the following research objectives: Process Evaluation To identify difficulties and successes experienced during NSMP implementation. To assess start-up and maintenance costs associated with NSMP. To determine which SFA characteristics are associated with successful implementation ofNoMP. To assess monitoring and accountability under NSMP, including the accuracy of nutrient analyses completed by SFAs. To assess the user-friendliness and efficiency of NSMP software systems. "o assess the opinions and recommendations of key stakeholders (SFA directors, school food service staff, and State agency staff) regarding the NSMP system. hacMCt Evahttfioa To compare the average nutrient content of meals served under NSMP to defined nutrient standards. To compare the average nutrient content of meals served before and after implementation of NSMP. To compare results of weighted and unweightr) nutrient analyses of menus planned under NSMP. To compare costs (food, labor, and other) associated with producing school meals before and after implementation of NSMP. To compare school food service operations, including participation rates and commod. ty usage, before and after implementation of NSMP. to a-1 Evaluation Design The demonstration spanned three academic years, from SY 1994-95 through SY 1996-97. The sequence and timing of major milestones in both the demonstration and its associated evaluation are summarized in Exhibit 2.1. FNS selected SFAs in January 1994. Shortly after that, while traditional meal patterns were still in use and before SFA staff received any training on NSMP, baseline data for the impact -valuation were collected. Key staff from demonstration SFAs attended a three-day training session in June 1994. Staff were expected to begin serving meals planned using NSMP the following fall, with the goal of achieving full implementation (i.e., NSMP-planned menus served for both breakfast and lunch in all schools in the district) by Fall 1995. The original design called for collection of initial process evaluation data in Spring 199S, when SFAs were expected to be approximately six months into NSMP implementation. As discussed in Chapter 3, however, work on NSMP implementation was substantially delayed in most SFAs for reasons that were beyond SFA control. Consequently, initial process evaluation data were collected about six months later, in the Fall of 1995 Follow-up data were collected in Spring 1996 and Spring 1997. Comparable data were collected each year, with the exception of data on meal costs, which were collected only in the Spring of 1997 and only from SFAs that h~d fully implemented NSMP. Data Collection This section provides an overview of the data collection methodologies used. Exhibit 2.2 provides a summary of the types of data collected at each measurement point. An expanded discussion of study implementation is provided in Appendix C. Interviews with SFA Directors SFA directors were interviewed at three points during the demonstration. In Fall 1995. directors were interviewed via telephone for purposes of the process evaluation. These interviews focused on experience with NSMP implementation to date, including progress made, difficulties encountered, and solutions employed. SFA directors were asked to assess advantages and disadvantages of the NSMP system and to offer recommendations for improving the demonstration as well as the NSMP system in general. SFA directors were also interviewed in person in Spring 19% and 1997. in conjunction with follow-up data collections.1 Interviews included questions about resources devoted to NSMP implementation. experiences with NSMP software, and perceptions about the burden and acceptability of NSMP. In districts where someone other than the SFA director had primary responsibility for actually working with NSMP software, this staff member (referred to as the data manager) was also interviewed. toe. Evaluation OMign 2-2 tn BLANK PAGE Exhibit 2.1 Evaluation Design Jan 1994 Jure 1994 Fall 1994 Fall 1995 NSMP ImplctTtcntotion FNS selects SFAs SFAs receive NSMP training SFAs expected to begin NSMP implementation SFAs expected to be fully implemented I i I I ■ 1 i SY 93-94 SY 94-95 SY 95-96 SY 96-97 Spring 1994 FaU 1995 Fo oring 1996 Soring 1997 Data Collection Baseline data collected Initial process evaluation data collected How-up data collected Follow-up data collected Notes: Process evaluation data were originally scheduled to be collected in Spring 1995. See Exhibit 2.2 for a summary of data collected at each measurement point. Inc. Evaluation Design Jf' Exhibit 2.2 Data Collected at Each Measurement Point 1 ! , —1 Measurement Point Data Codec Interviews wfthSFA Directors,: SFA Menus and Recipes Data on Meal Costs . -vcyscf On-S d Service Observation* | S|aff Interviews [ fdthStat Directors j Baseline Data: Spring 1994 / / / /3 Initial Process Evaluation Data: Fall I99S / / /2 Follow-up Data: Spring 1996 / / / / /2 Follow- «p Data: Spring 1997 / / / / / /' /2 /3 1 Interviews actually completed in Summer 1997 after the school year had ended. 2 SFAs were asked to submit start-up logs every six months beginning in January 1995. ' Administrative records were collected by mail in Fall 1994 (for SY 1993-94) and Fall 1997 (for SY 1996-97) after SFA staff had prepared year-end summaries for the preceding school year. Notes: Interviews were also conducted with FNS Regional office liaisons in Fall I99S. These interviews were not repeatcl in subsequent years, however, because it was determined that Regional office staff were not very involved with demonstration sites and were therefore unable to contribute new information. SFA directors who dropped out of the demonstration were interviewed by phone shortly after withdrawal. The interview focused on factors contributing to the decision to terminate participation in the demonstration. AM Associate* Inc. PZ£ Evaluation Design 2-4 SFA Menus and Recipes Detailed information on menus and recipes used in demonstration SFAs was collected in Spring 1994, 1996, and 1997. Each time, data were collected on the menus offered during a specified five-day period (referred to as the target week).2 Target weeks were spread between mid-April and mid-May. Data were collected for lunches and breakfasts served in elementary schools, middle schools, and high schools. The following information was collected for all reimbursable menu items: full name; type of food (e.g., white vs. whole wheat bread; fat content of milk or ground meat); form of food (e.g., fresh, frozen, canned; raw or cooked; convenience entree or prepared from scratch); brand name, if appropriate; preparation method; and planned portion sizc(s). Recipes were collected for all items prepared by combining two or more foods or ingredients. In addition, SFA staff were asked to provide, for each menu item, the total number ofservings anticipated for the district as a whole, i.e., the number of individual servings of each menu item expected to be served, exclusive of a la carte servings and servings sold to teachers or other adults. This information was needed to support a weighted nutrient analysis, as required under NSMP (see Chapter 3). In Spring 1997, when SFAs were using NSMP, additional data were collected to permit an assessment of the accuracy and completeness of nutrient analyses conducted by SFA staff. This included printouts of recipe and menu analyses for the target week (to assess the accuracy of data entered and analyzed); vendor-supplied nutrient information sheets (to assess the accuracy of data entry for foods added to the nutrient data base at the local level); and back-up menu production records (to assess the accuracy of serving projections used in weighted nutrient analyses). Data on Meat Costs To assess the costs of producing reimbursable meals before and after NSMP, a substudy of meal costs was conducted in three randomly selected schools (one elementary school, one middle school, and one high school) in each district. In Spring 1994 and again in Spring 1997, school food service staff in these schools were trained by Abt Associates staff to collect data for the cost substudy. Data were collected during the same target week covered in the menu analysis (see above) and included detailed menu production records and time studies. Menu production records included, for each food served during the target week (including a la carte foods and foods sold to adults), information on the number of servings prepared, the number of servings left over, and the number of servings actually served or sold. Daily time records were completed by all employees in the sampled schools for the full five-day period. The time records were used to allocate time spent on breakfast and time spent on lunch. A time allocation grid was also obtained for central office staff to identify time devoted to breakfast and lunch production. Data from administrative records (bid/inventory lists and salary rosters) were used to translate food production information and time re.ords into food and labor costs, respectively. These data were used to compute the average cost of producing a reimbursable lunch and breakfast, if offered, in each school. A detailed description of the methodology used in the study of meal costs is provided in Appendix E. 2 To obtain a reasonable aiataamrnr of nutrient content, it is necessary to examine meals offered over a period of time rather than a single meal. The National Research Council (SRC) recommends that group feeding programs plan menus so that nutrient standards are met over a five- lo ten-day period. A sample five-day period, equivalent to a full week in most school districts, is routinely used in USDA-sponsored evaluations of Child Nutrition programs. NSMP requirements specify that analyses be based on a typical school week, ranginc from three to seven days. AM Associate* Inc. h Evaluation Design 2-5 On-Site Observations On-site observations of meal preparation and service assessed compliance with planned menus, recipes, and portion sizes. One-day observations were completed during the target week in each of the three schools selected for the cost substudy. Observers documented whether items on the planned menu were actually offered and collected descriptive information on any additional items offered. They also observed food preparation and documented the nature and extent of deviations from planned recipes. In addition, observers weighed five random portions of all foods offered to children to determine average portion sizes.3 Finally, for purposes of the substudy of meal costs, staff observed students as they passed through servings lines and recorded which items students included in their reimbursable meals. Surveys ofSchool Food Service Staff In Spring 1996 and 1997, school food service staff in a random sample of schools in each SFA completed brief mail surveys. Surveys assessed staff perceptions about NSMP, particularly perceptions about burden and about the impact of NSMP on the acceptability of school meals. The methodology used in selecting schools and in developing weights for use in the analysis is described in Appendix F. Interviews with State Agency Staff Brief telephone interviews were completed with directors of State agencies and other relevant staff in Fall 1995 and Summer 1997. Interviews included questions about problems encountered in demonstration SFAs or other SFAs using NSMP as well as State agency directors' perceptions about monitoring under the new SMI guidelines (see Chapter 1). Start-Lip Logs SFA directors were asked to maintain detailed logs during the implementation period, documenting the number of staff hours devoted to a variety of different NSMP tasks. Directors were instructed to record only hours associated with work that would not have been done in the absence of NSMP. Respondents also provided information on the hourly wages of individual staff members so that costs associated with the reported labor could be calculated. SFAs were asked to submit start-up logs every six months, beginning in January 1995 and continuing until full implementation was achieved. Administrative Records Data on program participation, food service income and expenses, a la carte revenues, and commodity utilization (total dollar value of commodities received) were collected by mail in Fall 1994 (for SY 1993- 94) and Fall 1997 (for SY 1996-97), after year-end summaries had been prepared by SFA staff. Standards Used in Evaluating Nutrient Content Two sets of nutrient standards weie used in evaluating the nutrient content of NSLP and SBP meals (Exhibit 2.3). The first set is comprised of standards specified in the NSMP demonstration protocol (and, under SMI, carried over into current NSLP and SBP regulations). These include standards for calories and target nutrients for which RDAs have been established (protein, vitamin A, vitamin C, calcium, and iron) as well as for the percentage of calories from fat and saturated fat. 3 Commercially pre-park«|rrl item* such as cartons of mult and |wtas^ of cookie*, crackers, or chip* were excluded became previous experience has thown minor, if any. deviations front expectations due to quality control in commercial pack aftng processes AM Associate* Inc. Evaluation Design 24 & Exhibit 23 Nutrient Standards Used in the Evaluation Nutrient Standard Standards Defined for NSMP Nutrients with established RDAs:' Breakfast: One-fourth of the RDA Calories, protein, vitamin A, vitamin C, * calcium, and iron Lunch: One-third of the RDA Nutrients included in the Dietary Guidelines for Americans'} Total fat S 30% of total calories Saturated fat < 10% of total calories NRC Recommendations3 Carbohydrate > 55% of total calories Protein < 15 % of total calories4 Cholesterol Breakfast: £ 75 mg' Lunch: slOOmg6 Sodium Breakfast: soOOmg5 Lunch: sSOOmg6 1 National Research Council (1989). Recommended Dietary Allowances, 10th edition. Washington. DC: National Academy Press. 1 U.S. Departments of Health and Human Services and Agriculture (1995). Nutrition and Your Health: Dietary Guidelines for Americans, 4th edition. Washington, DC: U.S. Government Printing Office. 1 National Research Council (1989). Diet and Health: Implicationsfor Reducing Chronic Disease Risk. Washington DC: National Academy Press. 4 Diet and Health recommends a maximum protein intake equivalent to leu than twice the RDA. To achieve recommended levels of calories from fat and carbohydrate, the percentage of total calories from protein needs to be in this range. ' One-fourth of suggested maximum intake of cholesterol (300 mg) and sodium (2,400 mg). 6 One-third of suggested maximum intake of cholesterol (300 mg) and sodium (2,400 mg). Abt Associates Inc. * Evaluation Design 2-7 A second set of standards, based on the National Research Council (NRC) recommendations, was defined for nutrients and food components that are analyzed by NSMP software but not included in standards defined for NSMP. These include the percentage of energy from carbohydrate and protein as well as total cholesterol and sodium content. Neither demonstration SFAs, nor current NSMP users outside the demonstration, are required to meet these additional standards. They are used in this report solely for the purpose of facilitating interpretation of the data. Data on fiber content are also presented in this report. However, these data are not compared to a standard because there is no scientific consensus about an appropriate benchmark for daily consumption of fiber. Estimation of Program Impact The impact evaluation examined the effect of NSMP on nutrient content, meal costs, and key aspects of food service operations (e.g., participation rates, a la carte revenues, use of donated commodities, number of daily choices offered, and variety of foods offered over the course of a week). Because relatively few SFAs had implemented NSMP by Spring 1996, impact analyses focus on comparisons of baseline (Spring 1994) and final follow-up (Spring 1997) data. Throughout the text the terms 'pre- NSMP' or 'before NSMP* are used to refer to Spring 1994 data and 'after NSMP' or 'NSMP' are used to refer to Spring 1997 data. The basic measure of program effect is the difference between estimates made prior to NSMP implementation and those made afterward. Simple /-tests are used to test the statistical significance of differences in means or proportions. It is important to realize that impacts that are detected cannot necessarily be ascribed to NSMP. In the absence ot a control group, it is impossible to rule out the possibility that changes over time were influenced by factors external to the demonstration. Thus, information on program impacts should be considered suggestive rather than conclusive, and is not necessarily representative of what might happen in programs nationwide. Impact analyses are further limited by small sample sizes. Some of the SFAs selected to participate in the demonstration dropped out or never implemented NSMP. Others implemented NSMP only in elementary schools. Sample sizes vary from one analysis to the next depending on the number of SFAs included. The maximum sample size for any SFA-lcvel analysis is 19 (analysis of elementary lunch menus). These small samples provide adequate statistical power to detect only very large effects. Consequently, the absence of statistical significance cannot be interpreted with confidence as evidence that a meaningful difference does not exist. Rather, the conclusion must be that very large differences do not exist. AM AMoeif■ Inc. Evaluation Daaign 2-f Jd Chapter 3 Implementation of NSMP in Demonstration SFAs This chapter describes the SFAs that participated in the demonstration, the extent to which SFAs were successful in implementing NSMP, and difficulties and successes experienced by SFA staff. The first section of the chapter describes the SFAs selected to participate in the demonstration. The next two sections summarize challenges faced by SFAs during early stages of implementation and the final implementation status of all SFAs at the end of the demonstration (Spring 1997). The next section describes expectations of the original NSMP protocol and the extent to which SFAs were able to meet these expectations. The final section of the chapter identifies characteristics that may predict successful NSMP implementation. Selection of Demonstration SFAs SFAs were invited to volunteer for the NSMP demonstration in the Summer of 1993. Thirty-five SFAs were ultimately selected by FNS staff from a pool of 127 applicants. One SFA dropped out shortly after selection because school board approval for the project was not obtained. Thus, the demonstration began with a total of 34 SFAs. SFA selection was structured to ensure that the demonstration would provide an opportunity to assess the feasibility and impacts of NSMP in a variety of different school districts. As summarized in Exhibit 3.1, FNS staff purposefully structured the sample of SFAs to ensure diversity in geographic location, community type, school district size, and program participation rates. The nutrition and food service experience of the SFA director and the availability of nutrition expertise among other SFA staff were also considered in selecting sites. About two-thirds of the selected SFAs had a registered dietitian on staff. In about half of die SFAs, the director and/or NSMP coordinator was a registered dietitian. SFAs selected for the demonstration were managed both by veteran school food service professionals (up to 38 years of experience) and by relative newcomers (1 -2 years of experience). Characteristics of existing food service programs were also taken into account. Sites were selected to ensure variation in centralization of menu planning, use of cycle menus, the extent of centralized food production, use of standardized recipes, and prevalence of scratch food preparation (as opposed to use of pre-prepared convenience items). A final factor that influenced site selection was prior experience with computerized nutrient analysis. Because ready access to a computer is a prerequisite for NSMP implementation, all of the districts selected had some pre-existing computer capability. However, only 13 of the 34 selected districts had actually analyzed the nutrient content of their menus by computer prior to the demonstration. In about half of the selected districts, the SFA director and/or NSMP coordinator had prior experience with computerized nutrient analysis. Abt Associate* Inc. lmp4«m»nUtk>n of NSMP in Demonstration SFA» 3-1 Exhibit 3.1 Characteristics of Demonstration SFAs FNS Region Mid-Atlantic Mountain Plains Midwest Northeast Southeast Southwest Western Community Type Urban Suburban Rural Enrollment Under 2400 2400-9,999 10,000-24,999 25.000 or more Mean enrollment Average Daly NSLP Participation Rate 40 percent or less 41-55 percent 56-70 percent 71 percent or more Mean participation rate Percent of Lunches Served Free 25 percent or less 26-50 percent 51-74 percent 75 percent or more Mean percentage 4 4 5 5 5 6 5 9 16 9 6 9 t 11 33.984 8 9 13 4 53% 10 10 8 6 44% AM A«wocia«M Inc. ■pMMMM of NSMP In Demonstration SFAs 3-2 $SL Exhibit 3.1 (continued) Nutrition/Food Service Experience of Staff Registered dietitian or nutritionist on stiff SFA director or NSMP coordinator is registered dietitian SFA directors' total food service experience (years) Minimum Maximum Mean SFA directors' experience as director (years) Minimum Maximum Mean Characteristics of Food Service Program Prior to NSMP Fully centralized menu planning Cycle meniKs) used Some centralized food production1 Percentage of entree items prepared from scratch 25% or less 26-30% More than 50% Mean percentage Prior Experience with Computerized Nutrient Analysis District menus previously analyzed by computer SFA director or NSMP coordinator had prior experience with computerized nutrient analysis 23 It 1 38 15 1 19 8 21 10 17 7 15 12 51% 13 17 1 50% or rnore of schools in distrkt are served by base or central kitchens. Note: Some data may differ from application forms or from previous reports (Few. MK, 1997) because mformation for SFAs was clarified/updated in subsequent interviews. Source: Application forms for NSMP demonstration (Fall 1993). Abt Associates Inc. Imptemontetion of NSMP In Domooolintioo SI As 3-3 33 Early Implementation Challenges From the outset, NSMP implementation in all SFAs was complicated by a number of factors over which SFAs had no control. Most important was a significant delay in the approval of NSMP software systems. Software vendors were slow to submit products for USDA review and approval, and many of the programs submitted did not satisfy the functional criteria defined by USDA. The delay in the availability of approved software systems effectively postponed concerted implementation efforts in most SFAs. While most SFAs began working on NSMP at some level shortly after receiving training in June 1994, the majority did not begin to work in earnest until sometime afterward. By Jaruary 199S, six months after SFA staff were trained and the point at which process study data were originally scheduled to be collected, only two NSMP software packages had been approved. At that tune, FNS contacted all demonstration SFAs and asked that they select one of the two available systems, move forward with NSMP implementation during the remainder of SY 1994-95, and continue to work toward full implementation by Fall 1995. Several SFAs lobbied for approval of other softwait systems, and a third system was ultimately approved.1 While all but two of the SFAs that completed the demonstration eventually selected one of the three available software systems, many SFAs did not really begin working with NSMP software until well into the latter half of SY 1994-95 or the subsequent summer months. This delayed start-up had a substantial impact on the progress SFAs were able to make during the first year of the demonstration. Another problem that complicated implementation was the poor response USDA received from food manufacturers contacted to supply nutrient information for the CN Database. Very few manufacturers responded to the request, and much of the data submitted was found to be incomplete, inaccurate, or otherwise questionable. As a result, the database included in the approved software systems did not include many of the commercially purchased products used in school food service. This situation placed an unanticipated burden on demonstration SFAs. SFA staff had to assume responsibility for obtaining nutrient information tor nationally-available commercial products and for entering these data into the their own databases. Status of Implementation in Spring 1997 By Spring 1997, when final follow-up data collection began, less than half (16) of the original 34 SFAs were considered fully implemented (Exhibit 3.2). Ten of these SFAs had implemented NSMP in accordance with expectations—NSMP menus were being served in all schools and menus had been developed using approved NSMP software and district-wide menu production information. The other six SFAs were serving NSMP menus in all schools but used a modified approach to weighted nutrient analysis. Modifications included basing saving projections used in weighted analyses on data from a subset of schools rather than all schools in the district or on staff predictions rather than actual menu production data. One of these SFAs also used an unapproved software system one that had been in use prior to the demonstration. I Sawe January 1995. 1! other software sysetns have been approved AM Associates lr>c. ■nptewiHtJun or NSMP In DwwonsUaHon SFAs 3-4 Final Implementation Status of Demonstration SFAs Fully implemented Implemented without modifications Weighted analysis completed using approved software and district wide menu production data Implemented with modifications Weighted analysis completed using approved software but used menu production data from a subsample of schools1 Weighted analysis completed using approved software but used staff predictions rather than district-wide menu production data Weighted analysis completed using unapproved software and staff predictions rather than district-wide menu production data Partially implemented Implemented for elementary lunch; no analysis done for lunch at other levels or for breakfast at any level Implemented for elementary breakfast and lunch; no analysis done for other levels Implemented for breakfast at all levels; no analysis done for lunch Not implemented No menus analyzed fully Lunch menus analyzed, but with unapproved software and without weighting; breakfast menus not analyzed Breakfast and lunch menus analyzed but without weighting; analysis based on one "set" menu each day, without consideration of numerous other available choices Dropped out 10 i 2 4 2 1 1 3 1 1 11 l One SFA used district-wide menu production data for 'inch menus but based breakfast analysis on data from one elementary school one middle school and one high school Inc. hnphimentation of NSMP in Demonstration SFAs J-5 3^ Because the unapproved software did not have the ability to conduct a weighted analysis, SFA staff developed a companion spreadsheet program to compute weighted weekly averages (based on staff projections). Four SFAs achieved only partial implementation and three SFAs never implemented NSMP. Three of the partially implemented SFAs implemented NSMP only in elementary schools, and the other SFA implemented NSMP only for breakfast In all four of these SFAs, directors reported that they had not achieved full implementation because they were unable to dedicate the required number of staff labor hours. Among the three SFAs that had not implemented NSMP by Spring 1997, one was still in the process of developing and analyzing menus. The director was doing all the work herself and had experienced several false starts and multiple software problems. The other two non-implemented SFAs had analyzed breakfast and lunch menus, but the analyses deviated substantially from the NSMP protocol. One of these SFAs used unapproved software and used unweighted nutrient analysis. The other SFA based analyses on a single defined meal each day rather than incorporating all of the menu options that were actually available. For example, daily lunch analyses included only one entree, ignoring a second entree offered in elementary schools and numerous other entrees (more than ten on some dayv) offered in middle and high schools. Finally, a total of 11 SFAs withdrew from the demonstration sometime before Spring 1997. In most cases, several different factors contributed to the decision to terminate participation. Concern about the use of weighted nutrient analysis was the primary reason for withdrawal, cited by eight of 11 SFA directors (Exhibit 3.3). Most often, the concern centered around the amount of staff labor required to obtain district-wide menu production data. Some SFA directors were also concerned that weighted nutrient analysis would limit flexibility in menu planning and decrease the ability of individual schools to cater to students' preferences. A few directors were worried that a weighted analysis could only meet nutrient standards if popular high-fat food items were eliminated or offered much less frequently. These SFA directors believed that such changes would have a negative impact on lunch participation in middle and high schools. Problems with NSMP software contributed to the withdrawal of five SFAs. Specific problems included incompatibility of available NSMP software with other software currently in use for other aspects of program operations; the need for upgraded hardware in order to run NSMP software; and numerous problems with the prototype NSMP software received at the June 1994 training and/or with early releases) of approved systems. Five of the SFA directors who dropped out of the demonstration were concerned about the time line for implementation. Specifically, they were unhappy that they were expected to implement NSMP by the start of the 1993-% school year despite the lengthy delay in the software approval process. Directors reported that they could not dedicate the amount of labor required to implement NSMP within the condensed time frame. X Exhibit 33 Reason for Withdrawal from the Demonstration Concerns about weighted nutrient analysis Labor requirements Potential negative impact on flexibility in menu planning Potential negative impact on secondary school participation NSMP software problems Approved NSMP software systems not compatible with existing system Could not afforJ necessary hardware upgrade Numerous problems with NSMP software Demonstration time line too constrained Change in SFA director Concerns about NSLP SMI regulations issued prematurely NSLP may be block granted Note: Respondents could report more than one reason for withdrawal. Source: Interviews with directors of drop-out SFAs. November 1995-January 1997. Two SFAs terminated participation because there had been a change in SFA directors. The new directors were not prepared to assume responsibility for continuing with NSMP implementation. Finally, two SFA directors were influenced by situations that were not directly related to NSMP or to the demonstration. One director believed that issuance of the SMI regulations (see Chapter 1) prior to the conclusion of the evaluation undermined the intent of the demonstration. He was therefore unwilling to devote the time and resources necessary to continue with the project. The second SFA director was concerned that the NSLP was going to be block granted (an idea that was being debated by the Congress at the time), and that, as a consequence, her district would cease to participate in the program. AM of NSMP in Demonstration SFAs 3-7 il Implementation of NSMP in Demonstration SFAs The sections that follow describe key components of NSMP and summarize the difficulties and successes experienced by SFA staff in attempting to follow implementation guidelines in each area. All 23 of the SFAs that remained in the demonstration through Spring 1997 are included in these analyses. Additional information related to implementation is provided in Chapter 7 which reports on start-up and maintenance costs incurred by demonstration SFAs as well as SFA staff perceptions about the burden and acceptability of the NSMP system. Staff Responsibility To increase the probability of success, SFAs were required to identify one individual who would assume responsibility for overseeing NSMP implementation. In 17 of the 23 SFAs that remained in the demonstration through Spring 1997, SFA directors assumed direct responsibility for NSMP implementation. Ten of these directors did virtually everything themselves, including data preparation and entry. The other seven directors delegated data preparation and entry tasks or received substantial assistance from other staff or volunteer assistants, including dietetic interns and State office staff. For one SFA, the State office essentially took over these tasks because neither the SFA director nor his limited staff had time to do it. All of the SFAs in which responsibility for NSMP implementation was delegated were either very large (enrollments in excess of 25,000) or large (enrollments between 10,000 and 24,999). NSMP Software Systems As noted above, demonstration SFAs had three approved NSMP software systems from which to choose. These were School Nutrition Accountability Program (SNAP); Computer-Assisted Food Service (CAP'S); and NUTRIKIDS (developed and marketed by Lundifiyte Systems, Inc.). By the time the demonstration was over, many other software systems had been approved. Aoout half of the 23 SFAs that remained in the demonstration through Spring 1997 used NUTRIKIDS (Exhibit 3.4). Five of these SFAs originally started with SNAP, the prototype software distributed at training, but switched to NUTRIKIDS during the course of the demonstration. SNAP was the next most common software package, used by five SFAs. CAFS was used by three SFAs. Two of these SFAs already owned the CAFS food service management software system and simply purchased the NSMP software as an enhancement. The other SFA that elected to purchase CAFS did so because of the system's other features, i.e., modules that support a variety of food service administration and operations tasks. One SFA elected to use one of the NSMP systems that received approval later in the demonstration— PCS, a multi-module school food service management system offered hy PCS Revenue Control Systems, Two SFAs continued to use a software system that was in place prior to Lie demonstration (Practorcare), even though the software never received USDA approval. One of diese SFAs was in the fully-implemented- with-modifications group and the other SFA never implemented NSMP. Some SFAs experienced problems with NSMP software during the initial stages of implementation. Most of these problems were related to glitches in initial versions of the software, inexperience on the AM *IIOCIHM Inc. Impiomerrtaoon of NSMP in Demonstration SFAs 3-f 3? Exhibit 3.4 Software Systems Used by Demonstration SFAs Approved NSMP Systems NUTRIKIDS SNAP CAFS PCS Unspproved Software Systems Practorcarc 12 5 3 1 Note: Excludes 11 SFA» that dropped out of the demonstration. Source: SFA director interviews, Spring 1997. part of SFA staff, or problems with hardware. The frequency of software problems decreased as the demonstration continued and both software manufacturers and SFA staff gained more experience. Overall, SFAs using approved NSMP software systems were pleased with their selections. The majority found the software to be both easy to use and time efficient. For most SFAs, the only major problem with NSMP software was the fact that the nutrient database did not include information for brand name food products. This shortcoming placed the burden of acquiring and entering these data on the shoulders of SFA staff. In many SFAs the task of collecting, reviewing and entering data for literally hundreds of food items consumed a sizeable amount of SFA resources. Data for Weighted Nutrient Analyses To support development of serving projections for weighted nutrient analyses, SFAs were required to maintain menu production records which documented the number of individual portions actually served in reimbursable meals. A la carte and adult servings were not to be included in these counts. Unless an SFA planned to conduct separate analyses for individual schools within the district, weighted analyses were to be based on district-wide data. A sample menu production record was provided in the NSMP training manual, but SFAs were free to adapt the form for their own use. As noted in the preceding discussion of reasons for SFA withdrawal, many SFAs found it difficult to comply with this requirement. The amount of labor involved in collecting and synthesizing data from all schools in the district was the major problem Although virtually all SFAs maintained some form of menu production records, these records were maintained only at individual schools. Few SFAs dealt with Abt Associates Inc. Imptomsntatlon of NSMP in Demonstration SFAs 3-9 * these records at the district level. Most SFAs used purchasing records, ordering requisitions and inventory reports, rathet than detailed menu production records to track district-wide food usage. Consequently, the need to gather and synthesize menu production data required most SFAs to implement and manage an extensive stream of paperwork. SFA staff also identified more specific difficulties involved in obtaining these data. These included the need to convert food production information traditionally reported in volume or weight (e.g., ten gallons of gravy) to servings, the unit of measure needed for NSMP; separating foods sold a la carte from those sold in reimbursable meals; maintaining separate counts for similar items that were traditionally counted together, e.g., cookies, sandwiches, chips; and obtaining production information for self-serve foods, including, but not limited to, salad bars and other specialty food bars. While the sentiment regarding the requirement for district-wide menu production data was largely negative, seven SFA directors found this requirement, in combination with the requirement that all recipes be standardized, to be beneficial. These directors reported that the increase in accountability and standardization allowed them to identify and rectify inefficiencies and weaknesses in their overall operations. Standardized Recipes SFAs were expected to have a standardized recipe for every menu item that included more than one ingredient or involved preparation other than simple heating. A standardized recipe was defined as "one that has been tested to provide an established yield and quality through the use of ingredients that remain constant in both measurement and preparation methods." Districts were free to use standardized USDA recipes or to use/develop their own recipes. Approved NSMP software included modules to develop, analyze, and modify standardised recipes. SFAs that used salad bars or other theme bars were encouraged to develop recipes based on menu production information, i.e., the number of reimbursable meals served from the theme bar and the total quantity of food used. Several versions of a recipe might be needed if different types of schools offered different items or, for any pre-portioned foods, different portions. SFA staff were expected to reanalyze/update theme bar recipes if there were changes in the food items offered. This requirement proved to be more challenging than some SFA directors expected. When initial process evaluation data were collected, few SFA directors expected standardized recipes to be a problem. As implementation progressed, however, and SFAs gained experience in this area, difficulties increased. The level of effort required to modify and test existing recipes was greater than many SFA directors anticipated. Recipes often needed to be tested several times before a product was deemed acceptable. SFA directors complained that there were not enough "healthy" quantity recipes available. Although USDA issued an expanded set of recipes during the course of the demonstration, many SFAs incorporated only a few of these items, reporting that many of them were "inappropriate" for their populations (i.e., "my students will not eat that"). Resistance among SFA staff, particularly cooks, was another problem faced by SFA directors striving to standardize food production. Many cooks were unhappy with the increased emphasis on standardization, which required that they abandon some of their favorite recipes and refrain from making individual decisions about how a recipe might be improved. The need for a standardized approach to recipe preparation and food portioning was a recurrent theme in training sessions nek) for SFA staff. Abt Aaeoclatee Inc. " Implementation of NSMP In Demonstration SFA* 3-10 Y0 Age/Grade Groups Used In Analyzing Menus The demonstration protocol defined four different age/grade groupings for use in menu analysis: preschool and early elementary grades K-l (ages 3-6); upper elementary grades 2-5 (ages 7-10); middle school grades 6-8 (ages 11-13); and high school grades 9-12 (ages 14-17). SFAs were expected to plan and analyze separate menus for these four age groups. Because all school districts were not expected to have exactly identical grade configurations, the protocol allowed menu planners to incorporate one additional age group at either end of the defined span (e.g., 6-10 or 10-14). If the disparity in age/grade groupings included two or more years at either end of the spectrum, menu planners were required to develop customized standards using a specialized function available in approved software. Technical guidance issued after the training session stipulated that the calorie requirement in any customized standard be within ten percent of the calorie requirement for the oldest and youngest children in the group. This requirement effectively limited the number of ages that could be included in developing a customized standard and required that two separate menus be analyzed for most schools with wide grade spans, e.g., elementary schools covering grades K-6 or K-8 or combination schools covering grades K-l2. From the beginning, many SFAs had difficulty de.ermining which age groups to use in their nutrient analyses. This was particularly true for SFAs that had a number of different grade configurations for one type of school (e.g., elementary schools that covered grades K-6,1 -3, and K-12). SFAs' initial confusion about this issue was exacerbated by the issuance of the SMI regulations because the requirements specified in the regulations differed from those defined for the demonstration. First, although the SMI regulations encourage use of the four basic age/grade groups defined in the NSMP protocol, these are presented as an option rather than a requirement An alternative scheme presents minimum, less stringent, age/grade groupings. Second, the SMI regulations permit calculation of a combined middle/high school menu. Finally, although the -egulations allow for development of customized standards to meet individual district needs, no restriction is placed on the number of age groups that can be included in a combined, customized standard. In the end, the age/grade groups used by many SFAs were not consistent with expectations of the initial protocol. Only two of the 20 SFAs that planned elementary school menus (16 fully implemented SFAs and four partially implemented SFAs) planned two separate menus for lower and upper elementary grades. Six of the 16 SFAs that planned middle and high school menus (all of the fully implemented SFAs) used a combined middle and high school analysis rather than separate analyses for each school level. Finally, as discussed in Chapter 9, the age spans chosen for analysis, particularly for elementary menus, were not always appropriate. Alternative Offer-versus-Serve Rule NSMP menus, whether for breakfast or lunch, must include a minimum of three menu items, and one of the items must be milk. In addition, one of the three items offered in lunch menus must be an entree. Menu items are defined as any single food or combination of foods served together, e.g., a sandwich is considered one menu item. Demonstration SFAs were required to implement offer-versus-serve (OVS), a regulatory provision that allows students in secondary schools and, at the discretion of SFAs, elementary schools, to refuse some of the food items offered in reimbursable lunches. The OVS rule defined for the demonstration required that students take a minimum of two menu items. No restriction was placed on the items that could be refused. AbtAeaoclaOalnc. Implementation of BBS In Pemonetretlon SFAe i£TT HI Confusion about the OVS rule increased sharply when final SMI regulations were published in June 1995. The SNA regulations require that students select a minimum of two menu items and one of the selected items must be an entree. Most SFA directors incorrectly believed that the SMI regulations superseded the guidelines provided during the June 1994 NSMP training. Consequently, only five of the 20 fully or partially implemented SFAs followed the OVS rule defined for the demonstration. The remaining 15 SFAs followed the SMI version of the OVS rule or maintained the OVS rule used under the traditional meal pattern (students must take 3 of 5 meal components). This situation made it impossible for the evaluation to provide FNS with information about the acceptability or impact of the "any two items" OVS rule. Disclosure of Nutrition Information SFAs were not required to disclose nutrient information, but they were encouraged to do so as a means of increasing awareness of both NSMP and the SFA's efforts to improve the nutritional quality of school meals. Half of the SFAs that fully or partially implemented NSMP disseminated information about the nutrient content of school meals (Exhibit 3.5); some SFAs used more than one method. Eight SFAs used printed menus that were sent home to parents, three used posted information in cafeterias, and two published information in newspapers or local cable television bulletin boards. ExMbtt 3.5 Disclosure of Nutrition Information by Demonstration SFAs Disseminate information about nutrient content of meals Yes, on printed menus sent home Yes, in newspapers/on local TV Yes, post on cafeteria line No Reasons for not disclosing nutrition information Don't have time Don't want to do so until NSMP is implemented in all schools Software only gives monthly (rather than weekly) average; monthly summary is not useful Not completely comfortable with accuracy of information yet Believe it will confuse parents more than help them No money to print/send home menus Note: Respondents could report more than one form of disclosure »nd more thin one reason for non-disclosure Source: SFA director interviews. Spring 1997. AM AttoclatM Inc. ■ttpwentnurtion of NSMP In D#monsti Alton SFAs 3-12 ¥&- SFA directors who did not disclose nutrition information had several reasons for not doing so. Four directors said they simply did not have the time to prepare and disseminate this information. Two directors did not want to disclose nutrition information until NSMP was implemented in all schools. Two directors reported that they could only get monthly printed summaries from their software analysis system and did not believe that a monthly summary was useful. One director was not yet comfortable enough with the results of her NSMP analyses to disseminate the results. Another believed that disclosure of information would be more confusing than helpful. Finally, one director indicated that the SFA did not have enough money to support printing and distribution of menus for parents, the vehicle she saw as the most viable one for disseminating information on the nutrient content of planned menus. Promotional Activities in School Cafeterias Another activity that was encouraged, but not required, was use of cafeteria promotions to increase awareness of NSMP and to encourage children to accept new/modified menu items. Three-quarters of the fully or partially implemented SFAs conducted at least one special cafeteria promotion before Spring 1997 (Exhibit 3.6). Most, but not all, promotions took place in elementary schools and most involved taste testing activities. SFAs also reportedly assisted with classroom nutrition/health education activities and held special 5 A Day promotions to encourage fruit and vegetable consumption. Exhibit 3.6 Cafeteria Promotions Conducted by Demonstration SFAs Conducted special promotions in school cafeterias Yes 15 No 5 Activities conducted Taste testing new products/recipes 10 Coordinated activity with classroom nutrition/health 8 education Held S A Day or other promotion to increase fruit and 5 vegetable consumption Special nutrition awareness week (including National 2 School Lunch Week) Note: Respondents could report more than one activity. Source: SFA director interviews. Spring 1997. Abt Associates Inc. Imptem«ntetton of NSMP in Demonstration SFAs 3-13 «3 UM of Fortified Foods While formulating plans for the demonstration, FNS staff were concerned that some SFAs might satisfy nutrient standards by offering highly-fortified foods rather than planning an appropriately balanced menu using foods from the basic food groups. Program planners considered establishing restrictions en the use of fortified foods. Although such restrictions were never implemented, with the exception of limitations on the use of vegetable protein products (VPP), SFA staff were discouraged from using such foods. Staff in two-thirds of the SFAs that remained in the demonstration through Spring 1997 were approached by vendors offering fortified foods (Exhibit 3.7). The most common items offered Exhibit 3.7 Use of Fortified Foods by Demonstration SFAs Approached by vendor offering fortified foods Yes No Food offered by vendors "Super" donuts/s veet rolls Other fortified bread products Milk-based drinks Dessert items Vegetarian/soy entrees "Energy" bars Fruit drinks Fruit rollups Yogurt Other Incorporated fortified foods not used prior to NSMP Yes No 23 Note: Fortified foods have been specially fortified with one or more Source: SFA director interviews. Spring 1997. vitamins or minerals Inc. ImpfOffi+otsffOfi of NSMP wt rvfTiontlfMlon SFAs 3-14 &L were "super" donuts and sweet rolls similar to those already used in many SBP programs across the country. Less commonly encountered fortified foods included other fortified bread products such as bagels and muffins, milk-based drinks, desserts, vegetarian entrees., "energy" bars, fruit drinks, fruit rollups, and yogurt. Although two-thirds of the SFAs that did not withdraw from the demonstration were solicited by vendors offering specially fortified foods, none or these SFAs incorporated a fortified food that was not already being offered before NSMP (e.g., several SFAs used "super" donuts before joining the demonstration). Meeting Nutrient Standards Directors of the 16 fully implemented SFAs were asked about their problems and successes in meeting nutrient standards, based on the menus being offered in Spring 1997. Two SFAs reported that all of their planned menus satisfied all of the relevant NSMP standards. The remaining 14 SFAs reported that one or more of their planned menus failed to meet at least one of the nutrient standards. Exhibit 3.8 summarizes SFA directors' reports for breakfast and lunch menus. With the exception of calories and the percent of calories from saturated fat, more than three-quarters of the SFAs that offered breakfasts reportedly satisfied individual nutrient standards for breakfasts. The calorie standard was the most problematic. Seven of the 15 fully implemented SFAs that offered breakfast in Spring 1997 reported that one or more of their planned menus did not meet this standard. Four SFAs were unable to consistently meet the standard for saturated fat. Other standards that were problematic for one or more SFAs included the percent of calories from fat (3 SFAs); vitamin A (3 SFAs); iron (1 SFA); and vitamin C (1 SFA). Many of the same nutrients were problematic for lunch menus. Only eight of the 16 fully implemented SFAs reported meeting the standards for calories from fat and saturated fat on a consistent basis. Four SFAs reported difficulty meeting the standard for calories, specifically in secondary school menus. A comparable number reported similar difficulty with the standard for iron. Finally, two SFAs reported difficulty with the standard for vitamin A and one SFA had difficulty with the standard for vitamin C. SFAs were encouraged to monitor levels of carbohydrate, cholesterol, sodium, and dietary fiber in planned menus, although specific standards were not established for these nutrients. Only two of the 16 fully implemented SFAs reported monitoring these nutrients. SFA directors who were not monitoring these nutrients consistently reported that they (and their staff) did not have time to do anything that was not required. Calendar Time Required for Full Imptomentatlon Most SFAs required substantially more time to implement NSMP than originally anticipated. Among the 16 SFAs that fully implemented NSMP, an average of 19 months elapsed between the time SFA staff started actively working on NSMP implementation (which in many cases was several months after receiving training) and the time NSMP menus were served in all schools (Exhibit 3.9). The range was very broad, with a minimum of three months and a maximum of 33 months. In the great majority of the SFAs that achieved full implementation, active work on NSMP spanned more than one calendar year. Only three SFAs achieved full implementation in less than 12 months, while six SFAs required more than two years. Inc. of NSMP in Demonstration SFAs 3-1S if Exhibit 3.8 Reported Successes and Probkms in Meeting Nutrient Standards Breakfast Number of Fuify Implemented SFAs (■-IS) Calories 8 7 Protein 15 0 Calcium IS 0 Iron 14 1 Vitamin A 12 3 Vitamin C 14 1 Percent of calories from fat 12 3 Percent of calories from saturated fat 11 - 4 Lunch Calories 12 Protein 16 Calcium 16 Iron 12 Vitamin A 14 Vitamin C IS Percent of calories from fat 8 Percent of calories from saturated fat 8 Number of Fully Implemented SFAs (■-16) 4 0 0 4 2 1 8 8 Note: One fully implemented SFA does not offer breakfut. Source: SFA director interviews, Spring 1997. Inc. of NSMP In Demonstration SFA* -# 3-16 MMII Time Required by Demonstration SFAs to Achieve Full Implementation Month! elapsed between initiation of active work on NSMP and full implementation Less than 12 months 3 13 -24 months 7 More than 24 months 6 Mean (months) 19 Minimum (months) 3 Maximum (months) 33 SFA directors' estimates of time required under more favorable circumstances Less than 12 months 9 13-24 months 6 More than 24 months 1 Mean (months) 16 Minimum (months) 3 Maximum (months) 30 Source: SFA director interviews. Spring 1997 Although SFAs were able to work on implementation tasks during the period when approved software was not available (e.g., gathering menu production data, standardizing and/or modifying recipes, evaluating potential new commercial products, etc), delays in software approval may have introduced some inefficiencies and led to a more protracted implementation period than would have been experienced under more optimal conditions. Likewise, the fact that SFA staff had to acquire and enter nutrition information for many commercially-prepared foods (because the version of the nutrient database included in the NSMP software did not include this information) may have contributed to an extended implementation period. To get some idea of the time period required for NSMP implementation in the absence of these complications, directors of the 16 fully implemented SFAs were asked to estimate the amount of time AM AsaociatM Inc. knptomontation of NSMP hi Demonstration SFAs 3-17 XL they would need to implement NSMP if they were to do it again under more optimal circumstances (i.e., with readily available and more complete software) and with the knowledge gained during their initial experience. Responses are summarized in Exhibit 3.9. As shown, the time line for NSMP implementation projected by most SFA directors was shorter than their actual experience in the demonstration, but not dramatically so. On average, projected implementation time lines were about three months shorter than actual demonstration experience. Nine of the 16 SFA directors estimated that they could implement NSMP u. 12 months or less. Six directors estimated that they would need one to two years to implement Only one SFA director indicated that he would need more than two years to implement NSMP. Characteristics That May Predict Successful NSMP Implementation One of the research objectives defined for the evaluation was to determine whether particular SFA characteristics influence the likelihood of successful NSMP implementation. In view of the implementation rates described in preceding sections, this question seems especially important However, the study's ability to draw conclusions about relationships between SFA characteristics and successful implementation is limited by small sample sizes. Group sizes for "implementers" and "non-implementers" are so small that only extremely large differences between the two groups can be deemed statistically significant It is therefore more informative to examine the data and identify patterns that ire suggestive of meaningful associations. Exhibit 3.10 shows tabulations of final implementation status by selected SFA characteristics. Included are baseline characteristics considered in selecting demonstration SFAs as well as several other variables thought to be potentially important to NSMP implementation. The latter group includes whether the SFA director assumed direct responsibility for NSMP implementation and whether there was a change in SFA directors during the course of the demonstration. All 34 of the SFAs selected for the demonstration are included in these tabulations. It is important to note that several of the characteristics that could potentially be influential are highly correlated with each oth s. For example, urban SFAs tend to include more schools, have larger total enrollments, serve more reimbursable meals on an average day, and include a greater proportion of children in poverty than suburban and rural SFAs. The small available sample size does not support multivanate estimates of the separate effects of each characteristic. In comparing SFAs that achieved full implementation, with and without modifications, to SFAs that did not achieve full implementation, several interesting patterns emerge, as summarized below. Urban SFAs were ranch less Mkefy to impfcmeat NSMP than either suburban or rural SFAs. Only two of nine urban SFAs achieved full implementation, and one of these SFAs used a modified approach to weighted analysis. In contrast, eight of 16 suburban SFAs achieved full implementation (four SFAs used a modified approach), as did six of nine rural sites (one SFA used a modified approach). ?r Exhibit 3.10 Characteristics of Demoastratioa SFA* by Ffaul ImpknieaULoa Statin Coaaaaity Type Urban 1 Suburban Rural Enrollment Under 2,500 2^00-9,999 10,000-24,999 25,000 or more Perceat of Laacaes Served Free 25 percent or less 26-50 percent 51 percent or more Characteristics of Food Service Program Prior to NSMP Some centralized food production1 Yes No Fully centralized menu planning Yes No Cycle mcnu(s) used Yes No NatrMaa Experience of Staff Registered dietitian or nutritionist on staff Yes No 7 9 1 16 3 9 4 6 4 9 2 1 t 11 5 10 4 10 9 14 1 17 10 17 11 21 7 13 5 10 13 24 13 5 23 II AM Inc. of KSMP in Demonstration SFAs W 3-19 Exhibit 3.10 (c Natritioa Experience of Staff (continued) SFA director or NSMP coordinator is registered dietitian Yes No Prior with Computerized Nutrient District menus previously analyzed by computer Yes No SFA director or NSMP coordinator had prior experience with computerized nutrient analysis Other Factors that May hai SFA director personally managed NSMP iMaaaaaaaaal Yes Chaaajs in SFA directors during demonstration Yes No 10 I 7 11 10 t 11 7 4 14 18 16 13 21 17 17 24 10 7 27 1 50% or n.ore of schools in district ire served by base or central kitchens. Note: Some data may differ from application forms or from previous reports (Fox. MK. 1997) because information for SFAs was clarified/updated in subsequent interviews Source: Application forms for NSMP demoiisuanon (Fall 1993) and Spring 1997 implementation status. AM Inc. of NSMP in SFAa Jt> Although most of the urban SFAs were also either large or very large, exploratory analyses suggest that being urban had an influence on implementation that was independent of size. Potential reasons that urban SFAs may have more trouble with NSMP implementation than their suburban or rural counterparts include financial constraints (several urban SFA directors were in the midst of or facing a hiring freeze) and a more transient or less reliable work force (several urban SFA directors mentioned long-standing vacant positions and frequent staff turnover). It is also possible that urban SFAs are dealing with food preparation facilities that are aging or poorly equipped. In short, urban SFAs may be less able to devote the time and energy required for NSMP implementation because of numerous competing priorities. Very large SFAs and, to a lesser extent, small SFAs were less Hkery to implement NSMP than medium or large SFAs. Only one of 11 very large SFAs (enrollment of 25,000 or more) implemented NSMP without modifications. An additional two very large SFAs implemented NSMP using a modified approach to weighted nutrient analysis. Most very large SFAs that did not implement NSMP (7 of 8 SFAs) withdrew from the demonstration. As discussed earlier in this chapter, the primary reason for termination was concern about the use of weighted analysis, specifically the level of effort involved in collecting and organizing district-wide menu production data. The burden associated with complying with this requirement clearly escalates as district size increases, unless a district is fully computerized or has highly centralized food production. In fact, the data suggest that SFAs with enrollments between 10,000 and 25.000 may also have trouble implementing the weighted analysis requirement Only three of eight SFAs in this group were able to fully implement NSMP without use of a modified approach to weighted analysis. At the opposite end of the spectrum, small SFAs (less than 2.500 students) also appeared to have some difficulty with implementation. Four of the six small SFAs selected for the demonstration never implemented NSMP. Three of these SFAs dropped out. Feedback from SFA directors indicates that while the issue of weighted analysis played a role in the decision to drop out, a larger issue was the fact that these directors were personally responsible for multiple aspects of day-to-day operations. Those who dropped out found the amount of work involved in NSMP implementation, particularly within the constrained time frame in which they were ultimately asked to work, to be more than they could handle. SFAs that had aoaae ceatratatd food production prior to NSMP were aaore andy to achtrre tnl napli mi nUHna, wanont modaVanon, than SFAs that had nttte or no centrahaed food production. Seven of the 17 SFAs that had a moderate degree of centralized food production prior to NSMP, defined in this analysis as 50 percent or more of schools serviced by a base or central kitchen, achieved full implementation without using a modified approach to weighted analysis. The same was true for only three of the 17 SFAs that had less centralized food production. i SFA* Ml 61 The presence of centralized food production may benefit NSMP implementation by reducing the amount of work required to document district-wide food usage. In addition, SFAs with some centralized food production may face fewer problems in identifying standardized recipes and ensuring that those recipes are used in all food production locations. SFAs that had fully centralized menu planning prior to NSMP were more likely to achieve foil implementation, without modification, than SFAs that had fully or partially decentralized menu planning. Most SFAs (21 of 34) had fully centralized menu planning systems in place prior to the demonstration. Among the 13 SFAs that entered the demonstration with some degree of decentralized menu planning, only two achieved full implementation without using a modified approach to weighted analysis. To implement NSMP in situations where individual schools have limited or total autonomy in menu planning, SFAs must either reduce the level of local control—i.e., move to a fully centralized system—or analyze menus for each individual school. The latter approach significantly increases the time and labor required to implement NSMP. fete. fmvtmmMaben o* NSMP In OwiHXtliaOon SFA« 3-22 SI Chapter 4 Nutrient Content of Lunches Before and After NSMP This chapter presents information on the nutrient content of lunches served in demonstration SFAs before and after implementation of NSMP. The following research questions are addressed: • How do lunches served after implementation of NSMP compare to established NSMP standards and NRC recommendations? • Does the nutrient content of lunches served after implementation of NSMP differ significantly from lunches served prior to the use of NSMP? • Does the nutrient profile of NSMP lunches differ significantly if the nutrient analysis is unweighted? Do conclusions about the impact of NSMP differ if the analysis is unweighted? The chapter is divided into four sections. The first section describes the analysis that was conducted and how the data are presented in this report. The second section provides a context for the analysis of nutrient content by summarizing differences in the types of foods included in lunch menus before and after NSMP. The third section describes and compares the nutrient content of lunches served before and after the implementation of NSMP. Results are based on weighted analyses, the analytic approach that is the foundation of the current NSMP system (The difference between weighted and unweighted analyses is described later in this chapter.) The fourth and final section of the chapter compares results of weighted and unweighted analyses of Spring 1997 lunch menus. The objective of this comparison is to determine whether the two analytic techniques lead to similar conclusions about the extent to which NSMP lunches satisfy nutrient standards and recommendations. Overview of the Analysis The analysis includes 19 SFAs that implemented NSMP, either district-wide (16 SFAs) or for elementary school lunches only (3 SFAs), prior to Spring 1997. Data are presented for elementary school lunches and secondary school lunches. In cases where SFAs planned two elementary school menus (n=6) or separate middle and high school menus (n=10), data were combined in a single, appropriately weighted, analysis. All analyses were completed using NUTRIKIDS software.1 A detailed description of the methodology' used in conducting all nutrient analyses is provided in Appendix D. Appendix A includes supplementary exhibits that summarize data for SFAs that implemented NSMP by Spring 19%, referred to as "early-implementing SFAs," as well as data for middle school lunches and high school lunches for the ten SFAs that analyzed separate menus. LunchByte Systems. Inc Version 7.0.1997 Abt Assocl ites Inc. Nutrient Content of Lunches Before and After NSMP 4-1 /J Reference RDA Standards The RDAs specify allowances for four different groups of school-age children (4-6 year olds, 7-10 year olds, 11-14 year olds, and 15-18 year olds), as well as for 1-3 year olds, the oldest of whom may attend school-district-operated preschools. For purposes of the evaluation, RDA standards were assigned to each SFA based on the grade spans included in the districts' schools. In some cases, a defined RDA standard was used, for example, RDAs for 7-10 year olds for an elementary school that included grades two through five, or RDAs for 11-14 year olds for a middle school that included grades six through eight. When the grade span for a particular type of school encompassed more than one established RDA age group, for example, an elementary school with kindergarten through grade six (ages 5-11), the NSMP software was used to develop a customized RDA standard (see Appendix D). Abt Associates' coding staff ensured consistency in RDA standards across SFAs for schools with comparable grade spans as well as a consistent standard for Spring 1994 and Spring 1997 analyses. In addition to an analysis tailored to reflect the actual makeup of SFA schools, a separate analysis was conducted to compare lunches served before and after NSMP to the minimum nutrient (RDA) standards defined in current program regulations. Results of this analysis are summarized in Appendix A. Weighted and Unweighted Analy-ta The principal analysis presented in this chapter describes the average nutrient content of lunches served in participating SFAs before and after NSMP, based on a weighted analysis of lunch menus. Weighted analysis is the standard defined for the NSMP demonstration and in current program regulations. A weighted analysis takes into account the number and types of foods actually taken by students, giving greater weight to foods selected more frequently. As such, results of a weighted nutrient analysis provide a picture of the average lunch served to students. An unweighted analysis, in contrast, does not incorporate information on student selection patterns and constitutes a simple average of all foods made available to students. An unweighted analysis includes an average serving of every type of food offered (i.e., milk, entree, fruits and vegetables, and desserts), and thereby reflects the average lunch offered to students. To assess similarities and differences between the two analytic approaches, lunch menus were analyzed both ways. Results for Spring 1997 menus are discussed in this chapter. Results for Spring 1994 menus and a comparison of pre-NSMP and post-NSMP menus using an unweighted analysis are summarized in Appendix A. Frequency of Foods Offered in Lunch Menus Because nutrient content is driven by the mix of foods included in a menu, it is helpful to understand changes SFAs' may have made over the course of the demonstration in the mix of foods offered. Foods included in lunch menus offered before and after NSMP were assigned one of 52 different food/food group codes. Data were tabulated to produce frequency distributions showing the percentage of daily lunch menus in which each food/food group was included. For example, a frequency of 67 percent for pizza indicated that 67 percent of the daily lunch menus reported by demonstration SFAs included pizza. Atrt AaaoclatM Inc. Nutrient Content of LuncheeMMand After NSMP 4-2 J-f Data for pre-NSMP and NSMP menus were compared, and foods or food groups that increased or decreased over time by fiv; or more percentage points were identified. Results are summarized in Exhibit 4.1. Foods listed in the Offered More Often section of the exhibit appeared more often in NSMP menus than in menus planned before NSMP. Likewise, foods listed in the Offered Less Often section of the exhibit appeared less often in NSMP menus than in pre-NSMP menus. Changes noted in planned menus were generally consistent with recommendations made during NSMP training and with changes reported by SFA directors. Elementary school lunch menus planned after NSMP included: • more skim milk and flavored lowfat milk and less whole milk; • more unbreaded meat, poultry, and fish, and fewer breaded versions of these items; • more raw vegetables (other than salad); and • more "extra" bread/grain choices. The number of changes noted in middle and/or high school menus was greater than elementary school menus, but the pattern of change was similar. Compared to pre-NSMP lunch menus for middle and/or high schools, NSMP lunch menus included: • more skim milk and less whole milk; • more fresh fruit; • more raw vegetables and salad and fewer french fries and other types of potato products; • more pasta-based entrees, salad bars, and Chefs salads, and fewer entrees that tend to be high in fat, including breaded meat, poultry, and fish, burgers, nachos, hot dogs, and specialty food bars (e.g., taco ban, potato bars); • more rice; • more desserts; and • fewer snack chips. In addition, NSMP menus for all types of schools included fewer cooked vegetables (other than potatoes); elementary and middle school menus included less flavored skim milk; and middle and high school menus included less fruit juice. These findings are somewhat counter-intuitive but explainable. Comments from SFA directors indicate that use of cooked vegetables was decreased and counterbalanced by an increase in raw vegetables and both side-dish and entree salads. According to SFA staff, student acceptance of raw vegetables and salads is substantially higher than cooked vegetables (other than french fries and other types of potatoes). The decrease in the frequency of flavored skim milk can be traced to three SFAs that were using this type of milk prior to NSMP but discontinued it because of poor student acceptance. Finally, the change in the frequency of juice is attributable to changes made in four SFAs 'hat offered juice every day or almost every day prior to NSMP. After NSMP, menu planners in three of these SFAs offered fresh fruit rather than juice. In the fourth SFA, the frequency of juice dropped because of the elimination of a sack lunch, previously offered every day, in which the beverage was juice. AM Associate* Inc. Nutrient Content of Lunches Before and After NSMP 4-3 ft Exhibit 4.1 Foods Offered More and Less Often in NSMP Lunch Menus Compared to Pre-NSMP Loach Menus Offered More Often' Skim milk / / / Flavored lowfat milk / Fresh fruit / / Raw vegetables (other than salad) / / Salad / Unbreaded meat, poultry, or fish / Pasta-based entrees • Salad bar (entree) / Chefs salad / Rice / Extra bread/grain3 / Desserts / / Offered Less Often2 Whole milk / / / Flavored skim milk / / Fruit juice / / crench fries / / Potatoes other than french fries / • Cooked vegetables other than potatoes / / / Breaded meat, poultry or fish / / Hamburgers/cheeseburgers / / Nachos / Hot dogs / Specialty food bars / Chips / 1 Foods were offered in mor# daily menus after NSMP than before NSMP. Difference of more than S percentage points 2 Foods were offered in fewer dairy menus after NSMP than before NSMP. Difference of more than 5 percentage point.. 1 Bread or grain item available to all students, regardless of entree selected Nutrient Content of LunchM Before and After NSMP # Nutrient Profile of Lunches Served Before and After NSMP This section presents data on the average nutrient content of lunches served before and after NSMP, relative to NSMP standards and NRC recommendations. All results are based on weighted analyses. The section includes three separate discussions as outlined below. • Nutrient Content Relative to RDAs. Mean contribution to RDAs for calories, protein, vitamin A, vitamin C, calcium, and iron is evaluated in light of the NSMP standard for lunch (33% of the RDA). • Percentage of Calories from Total Fat, Saturated Fat, Carbohydrate, and Protein. The mean percentage of calories provided by each macronutrient is compared to NSMP standards (total fat and saturated fat) or NRC recommendations (carbohydrate and protein). • Cholesterol, Sodium, and Fiber Content Mean cholesterol and sodium content are compared to NRC recommendations. Information on mean fiber content is presented but not compared to a standard because no scientific consensus exists about a recommended daily intake of fiber. Discussions are organized around two research questions identified at the beginning of the chapter (1) the extent to which lunches served after implementation of NSMP (NSMP lunches) met defined nutrient standards and NRC recommendations, and (2) significant differences in the nutrient content of lunches offered before and after implementation of NSMP. Comments about non-significant differences between pre-NSMP and NSMP lunches are limited to situations where conclusions about meeting NSMP standards or NRC recommendations differ, (e.g., when the mean for NSMP lunches satisfies an NSMP standard or NRC recommendation but the mean for pre-NSMP lunches does not, or vice versa). Unless otherwise noted, findings for the supplementary analyses summarized in Appendix A (early implementers; middle school lunches and high school lunches; and minimum nutrient standards defined in the regulations) a*e comparable to those reported here. Mean Nutrient Content Relative to RDAs In general, lunches served both before and after imp' ^mentation of NSMP met NSMP standards for calories and all key RDA nutrients. Indeed, as shown in Exhibit 4.2, lunches served at both points in time provided more than one-third of the RDA, with the exception of calories in secondary school lunches.2 Elementary school lunches served after NSMP provided significantly fewer calories and less protein than lunches served prior to NSMP. These differences have little substantive importance, however, because lunches served at both points in time exceeded the NSMP standard of one-third of the RDA. Secondary school lunches served after NSMP were somewhat lower in calories than lunches served before NSMP. Although this difference was not statistically significant, the mean for NSMP lunches fell 2 DMI on actual mnacacrtyi^mitrwnicaaicrt of hiKJMKrvedafcpfCKMedmExMM A 1 A more detailed summary of the ( !«Exla*att42ll»out^4 5upfovtdedinExhibttsA2-A4 Abt Associates Inc. Nutrient Content of Lunches Before and After NSMP ^7 Exhibit 4.2 With the Exception of Calorie* in Secondary Schools. Lunches Served Before and After NSMP Provided More than On. -third of the RDA Elementary School Lunches 110% 100% Calories Protein Vitamin A Vitamin C Calcium Ion Secondary School Lunches 100% Calories Protein Vitamin A Vitamin C Calcium Iron Before NSMP After NSMP 'Difference is statistically significant at the 5 percent level. •Difference is statistically significant at the 1 percent level. AM Associate* Inc. Nutrient Content of Lunches Before and After NSMP 4-6 JT just below the one-third RDA standard. (Pre-NSMP lunches provided, on average, exactly one-third of the RDA.) The only nutrient which differed significantly for secondary school lunches was vitamin A. Lunches served after NSMP provided significantly more vitamin A than lunches served before NSMP. Again, however, the relative importance of this difference is minimal in terms of this evaluation because the vitamin A content of lunches served at both points in time more than satisfied the one-third RDA standard. In summary, the data indicate that the contribution of NSMP lunches to students' daily nutrient needs was comparable to the contribution of lunches planned before NSMP. In other words, improvements in NSMP lunches, as described in subsequent sections, were achieved without a negative impact on the relative contribution of lunches to students' daily needs for calories and nutrients. Number ofIndividual SFAs Meeting RDA Standards In addition to examining mean nutrient content across all SFAs, the analysis assessed whether implementation of NSMP affected the number of individual SFAs that met NSMP standards for calories and RDA nutrients. With the exception of calories in secondary schools, a majority of SFAs provided one-third of the RDA for each target nutrient both before and after NSMP (Exhibit A.3). Calories in secondary school lunches were problematic both before and after implementation of NSMP. As noted in the preceding section, the pre-NSMP mean, across all SFAs, just met the one-third RDA standard. On an individual basis, only eight of the 16 SFAs met the standard before NSMP. After NSMP, the number of SFAs meeting the standard dropped to six and thr mean calorie level was slightly lower, falling just below tlte one-third benchmark. This pattern is consistent with SFA directors' reports that some SFAs found it difficult to meet the standard for calories. Percentage of Calories from Total Fat, Saturated Fat, Carbohydrate, and Protein On average, neither NSMP lunches nor lunches served prior to NSMP met the NSMP standards for percent of calories from total fat and saturated fat or the NRC recommendations for percent of calories from carbohydrate and protein (Exhibit 4.3). Failure to meet NSMP standards for calories from fat and saturated fat is consistent with SFA director reports that these standards were the most difficult to meet at lunch, particularly for secondary schools. Nonetheless, the nutrient profile of NSMP lunches was significantly better than pre-NSMP lunches. Specifically, NSMP lunches derivedfewer caloriesfrom fat and saturatedfat and more calories from carbohydrate than lunches served before NSMP. This is true for both elementary and secondary school lunches and, in all cases, differences between pre-NSMP and NSMP lunches were statistically significant. The mean percentage of calories from fat in elementary school NSMP lunches was five percentage points lower than the mean for pre-NSMP lunches. This is a substantial difference, representing an overall decrease in calories from fat of 19 percent.3 As a result, elementary school NSMP lunches came close to meeting the NSMP standard of no more than 30 percent of calories from fat. For secondary school 3 SMEINMA.1 Ota!* Iran fci.pw of hi* 9 oOonei Abt Associate* Inc. Nutrtent Content of Lunch** B*for* and After NSMP 4-7 4\I Exhibit 4.3 Lunches Served After NSMP Provided Fewer Calories from Fat and Saturated Fat than Lunches Served Before NSMP Elementary School Lunches Total Fat Saturated Fat Carbohydrate Protein Secondary School Lunches 70% 60% ^50% I "§ 40% Zo 30% | 20% £ 10% 0% 52%* NSMP Standard orNRC Recommendation Before NSMP After NSMP 'Difterence is statistically significant «; the 5 percent level. "Difference is statistically significant at the 1 percent level. Nutrient Content of Lunch*. Bofor* and After NSMP 44 ± lunches, the mean percentage of calories from fat in NSMP lunches was three percentage points lower than pre-NSMP lunches, equivalent to a 12 percent decrease in actual fat calories. In addition, the percentage of calories from saturated fat in elementary school lunches dropped three percentage points, representing an overall decrease in calories from saturated fat of 25 percent.4 The equivalent for secondary sdnol lunches was a decline of one percentage point and a 16 percent drop in calories from saturated faL Finally, NSMP lunches provided significantly more calories from carbohydrate than pre-NSMP lunches and, particularly in the case of elementary school lunches, came close to meeting the NRC recommendation of rrore than S3 percent of calories from carbohydrate. Thus, although, on average, NSMP lunches continued to exceed the standards for calories from fat and saturated fat and to fall short of the recommendation for calories from carbohydrate, the data suggest that implementation of NSMP had a significant impact in moving school lunches closer to these goals. Overall, the magnitude of the change was larger for elementary school lunches than for secondary school lunches. Number of Individual SFAs Meeting Standardsfor Fat and Saturated Fat Before NSMP, only one of the 19 SFAs included in this analysis met the NSMP standard of no more than 30 percent of calories from fat in elementary school lunches (Exhibit 4.4). Further, elementary school lunches served in almost three-quarters of SFAs (14 of 19) exceeded this standard by a substantial margin, providing more than 35 percent of calories from fat. After NSMP, the number of SFAs meeting the standard for percent of calories from fat increased to more than one-third (7 of 19), a statistically significant difference, and only three SFAs provided more than 35 percent of calories from fat. The number of SFAs meeting the NSMP standard for percent of calories from saturated fat in elementary school lunches also shifted dramatically over the course of the demonstration (Exhibit 4.4). Before NSMP, none of the 19 SFAs met die NSMP standard of less than 10 percent of calories from saturated faL After NSMP, about one-quarter of the SFAs (5 of 19) met the standard (a statistically significant difference). Similar shifts were noted in the number of SFAs meeting fat and saturated fat standards in secondary school lunches, however, neither change was statistically significant. Finally, the number of SFAs meeting the NRC recommendation for percent of calories from carbohydrate was significantly higher, for both elementary school lunches and secondary school lunches, a
Object Description
Page/Item Description
Title | Part 1 |
Full-text | Ab« Associates Inc. Evaluation of the Nutrient Standard Menu Planning Demonstration 55 Wheeler Street Cambridge, Massachusetts 02138-1168 617 492-7100 telephone 617 492-5219 ftaimiU Final Report August 1998 Contract No. 53-3198-4-001 Hampden Square, Suite 500 4800 Montgomery Lane Bethesda, Maryland 20814-5341 301 913-0500 telephone 301 652-1618fusmik 640 North LaSalle Street Suite 400 Chicago, Illinois 60610-3781 312 867-4000 telephone 312 867 -4200 ftatmtU Prepared/or: JohnEndahl U.S. Dept. of Agriculture Food and Nutrition Service Office of Analysis and Evaluation 3101 Park Center Drive Alexandria, VA 22302 Prepared by. Mary Kay Fox Mary Kay Crepinsek Nancy Buratein Patty Connor Frederic Glantz Abt Associates Inc. 55 Wheeler Street Cambridge. MA 02138 Table of Contents Page List of Exhibits iii Acknowledgments xiv Chapter 1 Introduction 1-1 The National School Lunch and School Breakfast Programs 1-1 The NSMP Demonstration 1-5 The School Meals Initiative for Healthy Children 1-5 Organization of this Report 1-7 Chapter 2 Evaluation Design , 2-1 Research Objectives 2-1 Evaluation Design 2-2 Standards Used in Evaluating Nutrient Content 2-6 Estimation of Program Impact 2-8 Chapter 3 Implementation ofNSMP in Demonstration SFAs 3-1 Selection of Demonstration SFAs 3-1 Early Implementation Challenges 3-4 Status of Implementation in Spring 1997 3-4 Implementation ofNSMP in Demonstration SFAs 3-8 Characteristics That May Predict Successful NSMP Implementation 3-18 Chapter 4 Nutrient Content of Lunches Before and After NSMP 4-1 Overview of the Analysis 4-1 Frequency of Foods Offered in Lunch Menus 4-2 Nutrient Profile of Lunches Served Before and After NSMP 4-5 Differences Between Weighted and Unweighted Analyses 4-11 Chapter 5 Nutrient Content of Breakfasts Before and After NSMP 5-1 Frequency of Foods Offered in Breakfast Menus 5-1 Nutrient Profile of Breakfasts Served Before and After NSMP 5-3 Differences Between Weighted and Unweighted Analyses 5-10 Chapter 6 Impact ofNSMP on Program Operations and Costs 6-1 Program Participation 6-1 Meal Acceptability 6-3 Meal Costs 6-4 A la Carte Revenues 6-12 Use of USDA-Donated Commodities 6-12 Menu Planning Practices 6-16 Abt Associates Inc. U Table of Contents i Table of Contents (Continued) Page Chapter 7 Burden and Acceptability of the NSMP System 7-1 Start-up Costs 7-1 Maintenance Costs 7-6 SFA Directors' Perceptions About Burden of NSMP Tasks 7-9 Perceptions of School Food Service Staff 7-11 SFA Directors' Attitudes Toward NSMP 7-13 State Directors' Attitudes Toward NSMP 7-16 Chapter 8 State Monitoring of SFAs Using the NSMP Option 8-1 State Monitoring Requirements 8-1 State Agency Experiences with Demonstration SFAs 8-3 Changes in Monitoring Procedures to Accommodate SMI Reviews 8-6 Perceptions about SMI Monitoring Requirements 8-11 Chapter9 Accuracy of SFAs'Nutrient Analyses 9-1 Problems Detected in Nutrient Analyses Conducted by SFA Staff 9-1 Findings from On-Site Meal Observations 9-12 References Appendix A Supplementary Exhibits: Nutrient Content of Lunches A-1 Appendix B Supplementary Exhibits: Nutrient Content of Breakfasts B-1 Appendix C Study Implementation C-l Appendix D Methodology Used in Nutrient Analysis D-l Appendix E Methodology Used in Study of Meal Costs E-l Appendix F Methodology Used in Sampling and Weighting Surveys of School Food Service Staff F-l AM Associates Inc. Table of Contents ii 3 List of Exhibits Page Exhibit 1.1 Traditional Meal Pattern Requirements for the National School Lunch Program . 1 -3 Exhibit 1.2 Traditional Meal Pattern Requirements for the School Breakfast Program 1-4 Exhibit 2.1 Evaluation Design 2-3 Exhibit 2.2 Data Collected at Each Measurement Point 2-4 Exhibit 2.3 Nutrient Standards Used in the Evaluation 2-7 Exhibit 3.1 Characteristics of Demonstration SFAs 3-2 Exhibit 3.2 Final Implementation Status of Demonstration SFAs 3-5 Exhibit 3.3 Reasons for Withdrawal from the Demonstration 3-7 Exhibit 3.4 Software Systems Used by Demonstration SFAs 3-9 Exhibit 3.S Disclosure of Nutrition Information by Demonstration SFAs 3-12 Exhibit 3.6 Cafeteria Promotions Conducted by Demonstration SFAs 3-13 Exhibit 3.7 Use of Fortified Foods by Demonstration SFAs 3-14 Exhibit 3.8 Reported Successes and Problems in Meeting Nutrient Standards 3-16 Exhibit 3.9 Time Required by Demonstration SFAs to Achieve Full Implementation 3-17 Exhibit 3.10 Characteristics of Demonstration SFAs by Final Implementation Status 3-19 Exhibit 4.1 Foods Offered More and Less Often in NSMP Lunch Menus Compared to Pre-NSMP Lunch Menus 4-4 Exhibit 4.2 With the Exception of Calories in Secondary School, Lunches Served Before and After NSMP Provided More than One-third of the RDA 4-6 Exhibit 4.3 Lunches Served After NSMP Provided Fewer Calories from Fat and Saturated Fat than Lunches Served Before NSMP 4-8 Exhibit 4.4 More SFAs Met Standards for Calories from Fat and Saturated Fat in Lunches After Implementing NSMP 4-10 AM Associates Inc. List of Exhibits Si f List of Exhibits (Continued) Page Exhibit 4.5 Lunches Served Before and After NSMP Met the NRC Recommendation for Cholesterol but Not for Sodium 4-12 Exhibit 4.6 Weighted and Unweighted Analyses of 1997 Lunches Yield Slightly Different Mean Values for Calories and RDA Nutrients, but Conclusions About Whether Lunches Met NSMP Standards are Similar 4-14 Exhibit 4.7 Weighted and Unweighted Analyses of Lunches Yiel \ Similar Results for Calories from Fat, Saturated Fat, Protein, and Carbohydrate 4-16 Exhibit 4.8 Weighted and Unweighted Analyses of 1997 Lunches Yield Slightly Different Mean Values for Cholesterol, Sodium, and Fiber, but Conclusions About Whether Lunches Met NRC Recommendations are Similar 4-17 Exhibit S. 1 Foods Offered More and Less Often in NSMP Breakfast Menus Compared to Pre-NSMP Breakfast Menus 5-2 Exhibit 5.2 With the Exception of Calories, Breakfasts Served Before and After NSMP Provided More than One-Fourth of the RDA 5-4 Exhibit 5.3 Breakfasts Served After NSMP Provided Significantly Fewer Calories from Fat and Saturated Fat than Breakfasts Offered Before NSMP 5-7 Exhibit 5.4 More SFAs Met Standards for Calories from Fat and Saturated Fat in Breakfasts After Implementing NSMP 5-8 Exhibit 5.5 With the Exception of Sodium in Secondary Schools, Breakfasts Served Before and After NSMP Met the NRC Recommendations for Cholesterol and Sodium 5-9 Exhibit 5.6 Weighted and Unweighted Analyses of Breakfasts Yield Similar Mean Values for Calories and RDA Nutrients 5-11 Exhibit 5.7 Weighted and Unweighted Analyses of Breakfasts Yield Similar Results for Calories from Fat, Saturated Fat, Protein, and Carbohydrate 5-13 Exhibit 5.8 Weighted and Unweighted Analyses of 1997 Breakfast Menus Yield Comparable Results for Cholesterol. Sodium and Fiber Content 5-15 Exhibit 6.1 Average Daily NSLP Participation Was Essentially Constant Over the Course of the Demonstration 6-2 Abt Associates Inc. „ List of ExnlWts Iv r List of Exhibits (Continued) Page Exhibit 6.2 Mean SBP Participation Rates Before and After NSMP 6-3 Exhibit 6.3 Kitchen Managers' Perceptions About Changes in Lunch Plate Waste Since Implementation of NSMP 6-5 Exhibit 6.4 Kitchen Managers' Perceptions About Changes in Breakfast Plate Waste Since Implementation of NSMP 6-7 Exhibit 6.5 School Food Service Staff Perceptions About Students' Acceptance of School Meals Since NSMP 6-8 Exhibit 6.6 On Average, the Cost of Producing a Reimbursable Lunch Under NSMP Was Comparable to or Lower than the Cost Before NSMP 6-10 Exhibit 6.7 Costs of Producing a Reimbursable Lunch Before and After Implementation of NSMP 6-11 Exhibit 6.8 On Average, the Cost of Producing a Reimbursable Breakfast Under NSMP Was Comparable to or Lower than the Cost Before NSMP 6-13 Exhibit 6.9 Costs of Producing a Reimbursable Breakfast Before and After Implementation of NSMP 6-14 Exhibit 6.10 A la Carte Revenues Before and After Implementation of NSMP 6-15 Exhibit 6.11 Value of USDA-Donated Commodities Used Before and After Implementation of NSMP 6-16 Exhibit 6.12 On Average, Lunches Offered After NSMP Included as Many Options as Lunches Offered Before NSMP 6-18 Exhibit 6.13 The Amount of Variety in Weekly Lunch Menus Planned After NSMP Was Comparable to Menus Planned Before NSMP 6-19 Exhibit 6.14 On Average, Breakfasts Offered After NSMP Included as Many Options as Breakfasts Offered Before NSMP 6-21 Exhibit 6.15 The Amount of Variety in Weekly Breakfast Menus Planned After NSMP Was Comparable to Menus Planned Before NSMP 6-22 AM Associates Inc. \ List of Exhibits v List of Exhibits (Continued) Page Exhibit 7.1 Non-Labor Start-up Costs Reported by Demonstration SFAs 7-2 Exhibit 7.2 Labor Hours Devoted to NSMP Implementation, by Task 7-5 Exhibit 7.3 Most of the Time Devoted to NSMP Implementation Was Spent on Data Entry and Staff Training 7-7 Exhibit 7.4 Time Required for NSMP Maintenance Compared to Time Spent on Previous Menu Planning Practices 7-8 Exhibit 7.5 SFA Directors Found Obtaining Menu Production Data to be the Most Burdensome NSMP Task 7-10 Exhibit 7.6 School Food Service Staff Perceptions About Impact of NSMP on Overall Paperwork Burden 7-12 Exhibit 7.7 School Food Service Staff Perceptions About Effect of NSMP on Job 7-13 Exhibit 7.8 SFA Directors' Opinions About NSMP 7-14 Exhibit 7.9 SFA Directors' Perceptions About Advantages and Disadvantages of NSMP ... 7-15 Exhibit 7.10 State Directors'Opinions About NSMP 7-17 Exhibit 7.11 State Directors' Estimates of Non-Demonstration SFAs' Menu Planning Selections 7-18 Exhibit 7.12 State Directors' Perceptions About Why SFAs WiU Not Select NSMP 7-19 Exhibit 7.13 Recommendations of SFA and State Directors to Increase Likelihood of Successful NSMP Implementation 7-21 Exhibit 8.1 Reasons for State Visits to Demonstration SFAs 8-3 Exhibit 8.2 NSMP Topics Covered During State Visits to Demonstration SFAs 8-4 Exhibit 8.3 Methods Used by State Agency Staff to Assess Accuracy of SFAs' Nutrient Analyses 8-5 Exhibit 8.4 Summary of Problems Identified by State Monitoring Staff 8-7 Inc. U«t of Exhibits vi n List of Exhibits (Continued) Page Exhibit 8.5 Changes in State Schedules and Procedures to Accommodate SMI Monitoring .. 8-9 Exhibit 8.6 State Directors' Opinions About SMI Monitoring Requirements 8-12 Exhibit 9.1 Errors Found in Weekly Menu Analyses Completed by SFA Staff 9-3 Exhibit 9.2 Errors Found in Recipes Entered and Analyzed by SFA Staff 9-6 Exhibit 9.3 Data Entry Errors in Foods Added to the Nutrient Database by SFA Staff 9-8 Exhibit 9.4 Prevalence of Missing Data in Information Provided by Manufacturers and Data Entry Decisions Made by SFAs 9-9 Exhibit 9.5 Age Groups Analyzed by Predominant Grade Configuration 9-11 Exhibit 9.6 Adherence to Planned NSMP Menus 9-13 Exhibit 9.7 Items Omitted and Not Substituted in Observed Lunches 9-M Exhibit 9.8 Unanalyzed Items Offered During Observed Lunches 9-15 Exhibit 9.9 Adherence to Planned Portion Sizes 9-16 Exhibit A.1 Mean Calorie and Nutrient Content of Lunches Served Before and After NSMP A-1 ExhibitA.2 Mean Nutrient Profile of Lunches Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations A-2 Exhibit A.3 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Lunch Before and After NSMP A-3 Exhibit A.4 Distribution of Percentage of Calories from Fat and Saturated Fat in Lunches Served Before and After NSMP A-4 Exhibit A.5 Mean Calorie and Nutrient Content of Lunches Served Before and After NSMP: Unweighted Analysis A-6 Exhibit A.6 Mean Nutrient Profile of Spring 1997 Lunches: Comparison of Weighted and Unweighted Analyses A-7 Inc. I1 LMofExhMts vN List of Exhibit* (Continued) Pay* Exhibit A.7 Mean Nutrient Profile of Spring 1994 Lunches: Comparison of Weighted and Unweighted Analyses A-8 Exhibit A.8 Mean Nutrient Profile of Lunches Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Unweighted Nutrient Analysis A-9 Exhibit A.9 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Lunch Before and After NSMP: Unweighted Nutrient Analysis A-10 Exhibit A. 10 Distribution of Percentage of Calories from Fat and Saturated Fat in Lunches Served Before and After NSMP: Unweighted Nutrient Analysis A-l 1 Exhibit A. 11 Mean Calorie and Nutrient Content of Lunches Served Before and After NSMP: Middle Schools and High Schools A-12 Exhibit A.12 Mean Nutrient Profile of Lunches Served Before and After NSMP Compared to NSMP Standards arid NRC Recommendations: Middle Schools and High Schools A-13 Exhibit A. 13 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Lunches Before and After NSMP: Middle Schools and High Schools A-14 Exhibit A. 14 Mean Calorie and Nutrient Content of Lunches Served Before and After NSMP: Middle Schools and High Schools Unweighted Analysis A-13 Exhibit A.15 Mean Nutrient Profile of Lunches Served Before a.id After NSMP Compared to NSMP Standards and NRC Recomrnendaticns: Middle Schools and High Schools Unweighted Nutrient Analysis A-16 Exhibit A. 16 Number of SFAs That Satisfied Nutrient Standards and Recommendatior for Lunches Before and After NSMP: Middle Cchools and High Schooi- Unweighted Nutrient Analysis A-17 Exhibit A.17 Mean Calorie and Nutrient Content of Lunches Served Before and After NSMP Elementary Schools in Early-Implementing SFAs A-19 Exhibit A. 18 Mean Nutrient Profile of Lunches Served Before and After NSMP Compart 1 to NSMP Standards and NRC Recommendations: Elementary Schools in Early-Implementing SFAs A-20 Inc. ^ List of Exhibits vW 7 List of Exhibits (Continued) Exhibit A. 19 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Lunches Before and After NSMP: Elementary Schools in Eariy-Implementing SFAs A-21 Exhibit A.20 Mean Calorie and Nutrient Content of Lunches Served Before and After NSMP: Elementary Schools in Early-Implementing SFAs Unweighted Nutrient Analysis A-22 Exhibit A.21 Mean Nutrient Profile of Lunches Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Elementary Schools in Eariy-Implementing SFAs Unweighted Nutrient Analysis A-23 Exhibit A.22 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Lunches Before and After NSMP. Elementary Schools in Early-Implementing SFAs Unweighted Nutrient Analysis A-24 Exhibit A.23 Mean Calorie and Nutrient Content of Lunches Served Before and After NSMP: Secondary Schools in Early-Implementing SFAs A-25 Exhibit A. 24 Mean Nutrient Profile of Lunches Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Secondary Schools in Eariy-Implementing SFAs . A-26 Exhibit A.25 Number of SFAs That Satis*rd Nutrient Standards and Recommendations for Lunches Before and After NSMP: Secondary Schools in Early Implementing SFAs A-27 Exhibit A.26 Mean Calorie and Nutrient Content of Lunci.es Served Before and After NSMP: Secondary Schools in Early-Implementing SFAs Unweighted Nutrient Analysis A-28 Exhibit A.27 Mean Nutrient Profile of Lunches Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Secondary Schools in Eariy-Implementing SFAs Unweighted Nutrient Analysis A-29 Exhibit A.28 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Lunches Before and After NSMP Secondary Schools in Eariy-Implementing SFAs Unweighted Nutrient Analysis A-30 Inc. List of Exhibits ix /0 List of Exhibits (Continued) Page Exhibit A.29 Grade Groups Used by Demonstration SFAs and Associated Nutrient Standards for Lunches A-32 Exhibit A.30 Mean Percent FDA Provided by Lunches Served Before and After NSMP Compared to Minimum Nutrient Standards Defined in Regulations A-33 Exhibit A.31 Mean Percent RDA Provided by Lunches Served Before and After NSMP Compared to Minimum Nutrient Standards Defined in Program Regulations: Unweighted Nutrient Analysis A-34 Exhibit B. 1 Mean Calorie and Nutrient Content of Breakfasts Served Before and After NSMP B-l Exhibit B.2 Mean Nutrient Profile of Breakfasts Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations B-2 Exhibit B.3 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Breakfast Before and After NSMP B-3 Exhibit B.4 Distribution of Percent of Calcries from Fat and Saturated Fat in Breakfasts Served Before and After NSMP B-4 Exhibit B.5 Mean Calorie and Nutrient Content of Breakfasts Served Before and After NSMP: Unweighted Analysis B-5 Exhibit B.6 Mean Nutrient Profile of Spring 1997 Breakfasts: Comparison of Weighted and Unweighted Analyses B-6 Exhibit B.7 Mean Nutrient Profile of Spring 1994 Breakfasts: Comparison of Weighted and Unweighted Analyses B-7 Exhibit B.8 Mean Nutrient Profile of Breakfasts Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Unweighted Nutrient Analysis B-8 Exhibit B.9 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Breakfast Before and After NSMP: Unweighted Nutrient Analysis B-9 Exhibit B.10 Distribution of Percent of Calories from Fat and Saturated Fat in Breakfasts Served Before and After NSMP: Unweighted Nutrient Analysis B -10 tac UatofExMM* a // List of Exhibits (Continued) Page Exhibit B. 11 Mean Calorie and Nutrient Content of Breakfasts Served Before and After NSMP: Middle Schools and High Schools B-12 Exhibit B. 12 Mean Nutrient Profile of Breakfasts Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Middle Schools and High Schools B-13 Exhibit B. 13 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Breakfast Before and After NSMP: Middle Schools and High Schools B-14 Exhibit B. 14 Mean Calorie and Nutrient Content of Breakfasts Served Before and After NSMP: Middle Schools and High Schools Unweighted Analysis B-15 Exhibit B. 15 Mean Nutrient Profile of Middle and High School Breakfasts Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Middle Schocls and High Schools Unweighted Nutrient Analysis B-16 Exhibit B. 16 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Breakfast Before and After NSMP Middle Schools and High Schools: Unweighted Nutrient Analysis B-17 Exhibit B. 17 Mean Calorie and Nutrient Content of Breakfasts Served Before and After NSMP: Elementary Schocls in Early-Implementing SFAs B-19 Exhibit B. 18 Mean Nutrient Profile of Breakfasts Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Elementary Schools in Early-Implementing SFAs B-20 Exhibit B. 19 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Breakfast Before and After NSMP: Elementary Schools in Early- Implementing SFAs , B-21 Exhibit B.20 Mean Calorie and Nutrient Content of Breakfasts Served Before and After NSMP: Elementary Schools in Early-Implementing SFAs Unweighted Nutrient Analysis B-22 Exhibit B.21 Mean Nutrient Profile of Breakfasts Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Elementary Schools in Early- Implementing SFAs Unweighted Nutrient Analysis B-23 Abt Associates Inc. List of Exhibits xi ^ List of Exhibits (Continued) Page Exhibit B.22 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Breakfast Before and After NSMP: Elementary Schools in Early-Implementing SFAs Unweighted Nutrient Analysis B-24 Exhibit B.23 Mean Calorie and Nutrient Content of Breakfasts Served Before and After NSMP: Secondary Schools in Early-Implementing SFAs B-2S Exhibit B.24 Mean Nutrient Profile of Breakfasts Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Secondary Schools in Early- Implementing SFAs B-26 Exhibit B.25 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Breakfast Before and After NSMP: Secondary Schools in Early-Implementing SFAs B-27 Exhibit B.26 Mean Calorie and Nutrient Content of Breakfasts Served Before and After NSMP: Secondary Schools in Early-Implementing SFAs Unweighted Nutrient Analysis B-28 Exhibit B.27 Mean Nutrient Profile of Breakfasts Served Before and After NSMP Compared to NSMP Standards and NRC Recommendations: Secondary Schools in Early- Implementing SFAs Unweighted Nutrient Analysis B-29 Exhibit B.28 Number of SFAs That Satisfied Nutrient Standards and Recommendations for Breakfast Before and After NSMP: Secondary Schools in Early-Implementing SFAs Unweighted Nutrient Analysis B-30 Exhibit B.29 Grade Groups Used by Demonstration SFAs and Associated Nutrient Standards for Breakfasts B-32 Exhibit B.30 Mean Percent RDA Provided by Breakfasts Served Before and After NSMP Compared to Minimum Nutrient Standards Defined in Regulations B-33 Exhibit B.31 Mean Percent RDA Provided by Breakfasts Served Before and After NSMP Compared to Minimum Nutrient Standards Defined in Program Regulations: Unweighted Nutrient Analysis B-34 Exhibit D.l Weighted and Unweighted Coding for Sample Menu D-4 ExnibitEl Overview of Meal Coit Methodology Framework E-2 AW Associate* Inc. .1st of Exhibits xll /J List of Exhibits (Continued) Pag* Exhibit E.2 Calculation of the Total Food Costs Attributable to Breakfast and Lunch E-4 Exhibit E.3 Calculation of the Labor Costs Attributable to Breakfast and Lunch E-6 Exhibit E.4 Calculation of the Total Reported Cost of Producing Breakfast and Lunch E-7 Exhibit E.5 Estimation of the Percentage of Food Costs That Arc Reimbursable E-8 Exhibit F. 1 Number of Schools in Universe and Number of Schools Sampled by Stratum and Across All School Types: Spring 1996 and Spring 1997 F-l Exhibit F.2 Response Rates for Surveys of School Food Service Staff F-3 Abt Associates Inc. List of Exhibits xiii H Acknowledgments The NSMP Demonstration and its attendant evaluation are indebted to the staff in the 34 school districts who participated in the demonstration and supported the evaluation through its three-year time line. Special thanks are due to staff in the 23 districts that remained in the demonstration all three years. Their unfailing cooperation made this evaluation possible. Thanks are also due to staff in the 17 State offices who provided information on experiences with demonstration SFAs and on reactions to new regulations governing meal planning practices in Child Nutrition programs. Staff of the Office of Analysis and Evaluation, Food and Nutrition Service, of the U.S. Department of Agriculture had responsibility for overseeing the project. The Project Officer, John Endahl, guided all stages of project planning and implementation as well as development of study reports. Leslie Cristovich also provided valuable insights and direction throughout the project and in preparation of the final report. Finally, several staff members at Ab* Associates played important roles on the project including Patty Connor (Manager of On-site Data Collection and Nutrition Coding); Diane Greene (Survey Director); Gail Langeloh and Mary Kay Crepinsek (Senior Nutritionists); Nancy Burstein (Director of Analysis); Frederic Glantz (Task Manager for the Cost Analysis); K.P. Srinath (Sampling Statistician); Ellie Lee and Don Laliberty (Programmers); and Susan Altman (Editor). Eileen Fahey served as the contract secretary, responsible for production of all study reports, and was ably assisted by Eileen McEnaney and Judi Groves. AMAssociates Inc. ti Acknowledgments xiv Chapter 1 Introduction The National School Lunch Program (NSLP), operated by the Food and Nutrition Service (FNS) of the U.S. Department of Agriculture (USDA), has been providing meals to the Nation's school children since 1946. The School Breakfast Program (SBP) has been in operation since the early 1970s. Over the years, research has shown that meals offered in both the NSLP and SBP have provided children with the calories, vitamins, and minerals needed to sustain health and promote normal growth. More recent research has shown, however, that in comparison to current public health recommendations, school lunches are high in fat, saturated fat, and sodium, and low in carbohydrate. Since 1993, when this situation was first identified, FNS has been working on many fronts to enhance the nutritional quality of meals offered in the NSLP and SBP. As part of this ongoing initiative, FNS sponsored a demonstration project to evaluate the acceptability and impact of an alternative system for planning lunch and breakfast menus. This system, known as Nutrient Standard Menu Planning (NSMP), utilizes computerized nutrient analysis to monitor the nutrient content of meals served to students. Abt Associates Inc. of Cambridge, Massachusetts was awarded a contract to conduct an independent evaluation of the three-year demonstration project. This report presents results of that evaluation. The National School Lunch and School Breakfast Programs All public and private nonprofit schools are eligible to participate in both the NSLP and the SBP. The NSLP was established in 1946 "to safeguard the health and well-being of the Nation's children and to encourage the domestic consumption of nutritious agricultural commodities and other foods."1 Today the NSLP operates in over 94,000 schools and institutions. More than 26 million children receive meals through the program on any given day; about half of these meals are provided free of charge. The SBP began as a pilot project in the mid-1960s to support the provision of breakfast to children living in "poor areas and areas where children [had] to travel a great distance to school."2 The SBP was officially authorized as a permanent program in 1975, and the target population was expanded to include "all schools where [the program] is needed to provide adequate nutrition for all children in attendance."1 Currently, the SBP operates in approximately two-thirds of the schools and institutions that offer the NSLP, most commonly in schools that serve large numbers of economically disad vantaged children. On an average day, roughly seven million children receive breakfast through the SBP. The vast majority of these meals are provided free of charge. School Food Authorities (SFAs) participating in the NSLP and SBP receive two types of federal assistance: donated surplus commodities and cash reimbursements. Entitlement to surplus commodities 1 N«uon«l School Lunch Act oflM6. PL 79-396. 2 Child Nutrition Act of 1996. PL 19-642. 3 1975 Amendments to theChikJ Nutrition Act, PL 94-105 Abt Associates Inc. tf Introduction 1-1 is based on an established per-meal flat rate ($0,145 per meal in 1996-97) applied to the number of reimbursable lunches served the previous year. In addition, SFAs may receive bonus commodities which do not count against entitlements. Subject to availability, SFAs may request bonus commodities in amounts that can be used without waste. The type and amount of bonus commodities available varies from year to year depending on purchasing decisions made by USDA. Cash reimbursements for NSLP and SBP meals are based on the number of meals served, established per-meal reimbursement rates, and the poverty level of participating students. SFAs receive a base payment for each meal served, with substantially higher rates paid for meals served free or at a reduced price to income-eligible students. Childrens' household size and income determine their eligibility for free and reduced-price meal benefits. Nutrition Standards for School Meals Program regulations require that meals served in the NSLP and SBP meet defined nutrition standards in order to be eligible for Federal subsidies. Program regulations for the NSLP stipulate that "the nutrients of the lunch, averaged over a period of time, approximate one-third of the National Academy of Science's Recommended Dietary Allowances (RDAs) for children in specific age/grade groups."4 To ensure that these standards are met, program regulations have historically included food-based menu planning guidelines. These guidelines, originally known as the Type A meal pattern," define specific types of food iO be offered as well as minimum acceptable portion sizes (Exhibit 1.1). Although Lie authorizing legislation for the SBP did not include a specific RDA goal for breakfast meals, a meal pattern, modeled after the NSLP meal pattern, was developed to ensure that breakfasts would provide approximately one-fourth of the RDA (Exhibit 1.2). Over the years, research has shown that, with few exceptions, the NSLP has provided students the opportunity to satisfy one-third of their daily nutritional requirements (Wellisch, J., et al., 1983; St. Pierre, R., et al., 1992; and Burghardt, J., et al., 1993). Similarly, meals offered in the SBP have provided, on average, 20-25 percent of the RDAs (St. Pierre, R., et al., 1992; and Burghardt, J., et al., 1993). The most recent nationally representative study of school meals, the School Nutrition Dietary Assessment Study (SNDA), published in 1993, focused attention on another aspect of nutritional quality (Burghardt, J., et al., 1993). The SNDA study found that NSLP meals were high in fat, saturated fat, and sodium, and low in carbohydrate, in comparison to recommendations included in the Dietary Guidelines for Americans (U.S. Departments of Health and Human Services and Agriculture, 1995) and in the National Research Council's (NRC) Diet and Health report (National Research Council, 1989). At the time, programs were not required to meet these guidelines. In response to the SNDA findings, and as part of the national Healthy People 2000 initiative, USDA made a commitment to implement the Dietary Guidelines in the NSLP and SBP. The Department pledged to "increase to at least 90 percent [by the year 2000] the proportion of school lunch and breakfast services ... that are consistent with the nutrition principles in the Dietary Guidelinesfor Americans'' (U.S. Department of Health and Human Services, 1991). 4 7 CTR 245. Pwt 210 Abt Associate* Inc. Introduction 1-2 n Exhibit 1.1 Traditional Meal Pattern Requirements for the National School Lunch Program1 Meat or meat alternate Lean meat, poultry, or fish Cheese Large egg(s) Cooked dry beans or peas Peanut butter Peanuts, soy nuts, tree nuts, or seeds 1 serving per meal 2 OK. 2oz. 1 serving 1/2 cup 4 tbsp. 1 oz. s 1/2 the requirement Vegetables, fruits and/or full-strength jukes2 2 or more servings per meal, 3/4 cup total portion Bread or bread alternate Enriched or whole-grain bread Enriched or whole-grain biscuit, muffin, roll or equivalent Cooked enriched or whole grain rice, macaroni, noodles, or other cereal grains such as bulgur or corn grits 1 or more servings per meal/ 8 servings per week 1 slice 1 serving 1/2 cup Milk Fluid milk (whole milk and low fat milk must be offered daily) 1 serving per meal 1/2 pint (8 fluid oz.) 1 Requirement* were changed in June 1995 when the School Meals Initiative for Healthy Children wai implemented (tee text). 2 No more than one-half of the total requirement may be met with full-strength fruit or vegetable juice. a Introduction 1-3 Exhibit 12 Traditional Meal Pattern Requirements for the School Breakfast Program1 HMHRHHHHHBHBI 4 components must be offered: 1 serving of fluid milk 1 serving of vegetables, fruit, or juice 2 servings of bread/bread alternate, meat/meat alternate, or one serving of each Meat or meat alternate Lean meat, poultry, or fish Cheese Large egg Cooked dry beans or peas Peanut butter Peanuts, soy nuts, tree nuts, or seeds loz. loz. 1/2 4 tbsp. ■ 2 tbsp. loz. Vegetables, fruits, and/or full-strength juices 1/2 cup Bread or bread alternate Enriched or whole-grain bread Enriched or whole-grain biscuit, muffin, roll or equivalent Enriched, whole-grain, or fortified cereal 1 slice 1 serving 3/4 cup or 1 oz. Milk Fluid milk (whole milk and low fat milk must be offered daily) 1/2 pint (8 fluid oz.) Requirements were changed in June 1995 when the School Meals Initiative for Healthy Children was implemented (see text). AM Associate* Inc. n Introduction 1-4 The NSMP Demonstration As part of an ongoing initiative to enhance the nutritional quality of meals offered in the NSLP and SBP, FNS sponsored a demonstration project to evaluate the acceptability and impact of an alternative system for planning lunch and breakfast menus. This system, known as Nutrient Standard Menu Planning (NSMP), uses computerized nutrient analysis to plan menus rather than food-based menu planning guidelines. Under NSMP, SFAs are free to use whatever portions and combinations of food they wish in order to meet specific nutrient-based standards. Thus, in theory, NSMP provides more flexibility in menu planning while, at the same time, providing a greater degree of assurance that meals served to students meet nutrition standards. FNS selected 35 SFAs to participate in the demonstration, from a pool of 127 applicants. SFAs were purposefully selected to provide diversity in geographic location, district size, student participation rates, food service program characteristics, and staff experience with computerized nutrient analysis. One SFA dropped out shortly after selection because school board approval for the project was not obtained, and the district was not replaced. Thus, the demonstration began with 34 SFAs. Participating SFAs were expected to implement NSMP for both lunch and breakfast in all schools. Menus were to be planned using newly developed nutrient analysis software systems approved by FNS. Approval was granted only to systems that met detailed specifications set forth by the Agency. A key requirement was that the nutrient analysis be based on a weighted average of all foods served in reimbursable meals. A weighted analysis incorporates information about students' food selection patterns and gives more weight to foods that are selected more often. Approved NSMP software systems also incorporated the Child Nutrition Database (CN Database), which was developed by USDA specifically for use in the Child Nutrition programs. The CN Database includes nutrient information from the USDA standard nutrient database for foods routinely used in school food service programs. Initially, USDA planned to augment the main database with nutrient information for an extensive number of commercially prepared food items. USDA staff requested nutrition information from major food manufacturers that serve the school food service industry. As discussed later in this report, however, food manufacturers were slow to respond to this request. Menus planned under NSMP were expected to meet a defined set of nutrient standards. Lunches were expected to provide one-third of the RDA and breakfasts were expected to provide one-fourth of the RDA. Both meals were expected to be consistent with Dietary Guidelines goals for the percentage of calories from total fat and saturated fat. Chapter 2 includes detailed descriptions of the demonstration time line and implementation protocol as well as the research design used in the associated evaluation. The School Meals Initiative for Healthy Children USDA's commitment to implementing the Dietary Guidelines in school meals was formalized in the School Meals Initiative for Healthy Children (SMI),' which promoted expansive changes in nutrition S Federal RefM»er 60 113. 31188 31222. Jwt 13. 1995 to Introduction 1-6 standards for school meals. The original SMI proposal, issued in June 1994, required that all SFAs use NSMP (referred to as NuMenus) or an alternative known as Assisted NuMenus (ANSMP) to plan NSLP and SBP menus. (ANSMP allows SFAs to arrange or contract for menu development and nutrient analysis by an outside source such as a State agency, a consortium of SFAs, or a consultant.) Further, the initial SMI proposal required that school meals be consistent with the Dietary Guidelines by the start of SY 1998-99. The extended time line was built into the proposed regulation because comments received during public hearings and in response to an initial Federal Register notice indicated that some SFAs would need a considerable amount of time to implement NSMP or ANSMP and to develop menus consistent with the Dietary Guidelines. In November 1994, as part of the reauthorization of the Child Nutrition programs. Congress enacted The Healthy Meals for Healthy Americans Act.6 This law precipitated two important changes in the Department's initial SMI proposal. Fust, it required the Department to develop a modified food-based menu planning system, similar to those shown in Exhibits 1.1 and 1.2, that schools could use in lieu of NSMP or ANSMP. Second, the law shortened the time line for implcmc nation of the Dietary Guidelines, requiring that all SFAs be in compliance by tl * first day of the 1996-97 schooi year (two years earlier than the Department had suggested), unless a waiver was granted by the cognizant State agency. The final SMI rule, issued in June 1995, provided three options for menu planning: NSMP, ANSMP, and a modified food-based meal pattern (one for breakfasts and one for lunches).7 Menu planning options were further expanded in May 1996, when The Healthy Meals for Children Act allowed use of the traditional NSLP and SBP meal patterns, i.e., those in effect before the modified pattern defined in the SMI rule, or "any reasonable approach'' to satisfy the nutrient standards defined under SMI. SFAs that elect to use either of the food-based systems (the modified meal patterns defined in the SMI rule or the traditional meal patterns) or another approach approved by a cognizant State agency do not have to analyze the nutrient content of planned menus. However, every five years State tgency staff must analyze a representative weekly menu as part of a mandated SMI re view and compare results of the analysis to the same nutrient standards used in NSMP and ANSMP. The regulatory changes described above, which were implemented after the inception of the NSMP demonstration, have already incorporated NSMP as a menu planting option fcr the NSLP and SBP. Nonetheless, findings from the demonstration are important in understanding the challenges and benefits associated with use of NSMP. The collective experience of SFAs that participated in the demonstration will contribute to meaningful dialogue about the current NSMP system and may inform future decisions about program requirements, training, technical assistance, and monitoring. 6 PL I03-44S. 7 M. 104-149. Abt Aaaoclrt—Inc. Introduction- 1-6 *l Organization of this Report Eight additional chapters in this report present the following information: • Chapter 2 provides an overview of the design and objectives of the evaluation. • Chapter 3 describes the demonstration SFAs and the process of NSMP implementation. • Chapters 4 and 5 present information about the impact of NSMP on the nutrient content of lunches and breakfasts. These chapters also include comparisons of weighted and unweighted analyses of the same menus. • Chapter 6 reports on the impacts cf NSMP on food service program costs and operations. • Chapter 7 describes start-up costs associated with implementing NSMP It also summarizes the opinions and recommendations of key stakeholders, including SFA directors, school food service staff, and State agency staff. • Chapter 8 discusses monitoring and accountability under the new SMI regulations. Problems encountered by State agency staff who visited demonstration SFAs or completed SMI reviews in other SFAs are described. • Chapter 9 reports on problems encountered in the nutrient analyses completed by SCA staff. Findings from on-site observations of meal service are also included. Appendices provide supplementary exhibits as well as additional information about data collection methodologies, the development of sample weights for surveys of school food service staff, and methodologies used in assessing nutrient content and meal costs. Inc. u Introduction 1-7 Chapter 2 Evaluation Design This chapter outlines the research objectives defined for the evaluation and describes the research design. The sequence and timing of data collection are also discussed and resoondents and data sources are identified. Finally, the chapter identifies the nutrients that were examined in the analysis and the standards 'hat were usrd to evaluate the average nutrient content of school meals. Research Objectives The study included both process and impact evaluations, with the following research objectives: Process Evaluation To identify difficulties and successes experienced during NSMP implementation. To assess start-up and maintenance costs associated with NSMP. To determine which SFA characteristics are associated with successful implementation ofNoMP. To assess monitoring and accountability under NSMP, including the accuracy of nutrient analyses completed by SFAs. To assess the user-friendliness and efficiency of NSMP software systems. "o assess the opinions and recommendations of key stakeholders (SFA directors, school food service staff, and State agency staff) regarding the NSMP system. hacMCt Evahttfioa To compare the average nutrient content of meals served under NSMP to defined nutrient standards. To compare the average nutrient content of meals served before and after implementation of NSMP. To compare results of weighted and unweightr) nutrient analyses of menus planned under NSMP. To compare costs (food, labor, and other) associated with producing school meals before and after implementation of NSMP. To compare school food service operations, including participation rates and commod. ty usage, before and after implementation of NSMP. to a-1 Evaluation Design The demonstration spanned three academic years, from SY 1994-95 through SY 1996-97. The sequence and timing of major milestones in both the demonstration and its associated evaluation are summarized in Exhibit 2.1. FNS selected SFAs in January 1994. Shortly after that, while traditional meal patterns were still in use and before SFA staff received any training on NSMP, baseline data for the impact -valuation were collected. Key staff from demonstration SFAs attended a three-day training session in June 1994. Staff were expected to begin serving meals planned using NSMP the following fall, with the goal of achieving full implementation (i.e., NSMP-planned menus served for both breakfast and lunch in all schools in the district) by Fall 1995. The original design called for collection of initial process evaluation data in Spring 199S, when SFAs were expected to be approximately six months into NSMP implementation. As discussed in Chapter 3, however, work on NSMP implementation was substantially delayed in most SFAs for reasons that were beyond SFA control. Consequently, initial process evaluation data were collected about six months later, in the Fall of 1995 Follow-up data were collected in Spring 1996 and Spring 1997. Comparable data were collected each year, with the exception of data on meal costs, which were collected only in the Spring of 1997 and only from SFAs that h~d fully implemented NSMP. Data Collection This section provides an overview of the data collection methodologies used. Exhibit 2.2 provides a summary of the types of data collected at each measurement point. An expanded discussion of study implementation is provided in Appendix C. Interviews with SFA Directors SFA directors were interviewed at three points during the demonstration. In Fall 1995. directors were interviewed via telephone for purposes of the process evaluation. These interviews focused on experience with NSMP implementation to date, including progress made, difficulties encountered, and solutions employed. SFA directors were asked to assess advantages and disadvantages of the NSMP system and to offer recommendations for improving the demonstration as well as the NSMP system in general. SFA directors were also interviewed in person in Spring 19% and 1997. in conjunction with follow-up data collections.1 Interviews included questions about resources devoted to NSMP implementation. experiences with NSMP software, and perceptions about the burden and acceptability of NSMP. In districts where someone other than the SFA director had primary responsibility for actually working with NSMP software, this staff member (referred to as the data manager) was also interviewed. toe. Evaluation OMign 2-2 tn BLANK PAGE Exhibit 2.1 Evaluation Design Jan 1994 Jure 1994 Fall 1994 Fall 1995 NSMP ImplctTtcntotion FNS selects SFAs SFAs receive NSMP training SFAs expected to begin NSMP implementation SFAs expected to be fully implemented I i I I ■ 1 i SY 93-94 SY 94-95 SY 95-96 SY 96-97 Spring 1994 FaU 1995 Fo oring 1996 Soring 1997 Data Collection Baseline data collected Initial process evaluation data collected How-up data collected Follow-up data collected Notes: Process evaluation data were originally scheduled to be collected in Spring 1995. See Exhibit 2.2 for a summary of data collected at each measurement point. Inc. Evaluation Design Jf' Exhibit 2.2 Data Collected at Each Measurement Point 1 ! , —1 Measurement Point Data Codec Interviews wfthSFA Directors,: SFA Menus and Recipes Data on Meal Costs . -vcyscf On-S d Service Observation* | S|aff Interviews [ fdthStat Directors j Baseline Data: Spring 1994 / / / /3 Initial Process Evaluation Data: Fall I99S / / /2 Follow-up Data: Spring 1996 / / / / /2 Follow- «p Data: Spring 1997 / / / / / /' /2 /3 1 Interviews actually completed in Summer 1997 after the school year had ended. 2 SFAs were asked to submit start-up logs every six months beginning in January 1995. ' Administrative records were collected by mail in Fall 1994 (for SY 1993-94) and Fall 1997 (for SY 1996-97) after SFA staff had prepared year-end summaries for the preceding school year. Notes: Interviews were also conducted with FNS Regional office liaisons in Fall I99S. These interviews were not repeatcl in subsequent years, however, because it was determined that Regional office staff were not very involved with demonstration sites and were therefore unable to contribute new information. SFA directors who dropped out of the demonstration were interviewed by phone shortly after withdrawal. The interview focused on factors contributing to the decision to terminate participation in the demonstration. AM Associate* Inc. PZ£ Evaluation Design 2-4 SFA Menus and Recipes Detailed information on menus and recipes used in demonstration SFAs was collected in Spring 1994, 1996, and 1997. Each time, data were collected on the menus offered during a specified five-day period (referred to as the target week).2 Target weeks were spread between mid-April and mid-May. Data were collected for lunches and breakfasts served in elementary schools, middle schools, and high schools. The following information was collected for all reimbursable menu items: full name; type of food (e.g., white vs. whole wheat bread; fat content of milk or ground meat); form of food (e.g., fresh, frozen, canned; raw or cooked; convenience entree or prepared from scratch); brand name, if appropriate; preparation method; and planned portion sizc(s). Recipes were collected for all items prepared by combining two or more foods or ingredients. In addition, SFA staff were asked to provide, for each menu item, the total number ofservings anticipated for the district as a whole, i.e., the number of individual servings of each menu item expected to be served, exclusive of a la carte servings and servings sold to teachers or other adults. This information was needed to support a weighted nutrient analysis, as required under NSMP (see Chapter 3). In Spring 1997, when SFAs were using NSMP, additional data were collected to permit an assessment of the accuracy and completeness of nutrient analyses conducted by SFA staff. This included printouts of recipe and menu analyses for the target week (to assess the accuracy of data entered and analyzed); vendor-supplied nutrient information sheets (to assess the accuracy of data entry for foods added to the nutrient data base at the local level); and back-up menu production records (to assess the accuracy of serving projections used in weighted nutrient analyses). Data on Meat Costs To assess the costs of producing reimbursable meals before and after NSMP, a substudy of meal costs was conducted in three randomly selected schools (one elementary school, one middle school, and one high school) in each district. In Spring 1994 and again in Spring 1997, school food service staff in these schools were trained by Abt Associates staff to collect data for the cost substudy. Data were collected during the same target week covered in the menu analysis (see above) and included detailed menu production records and time studies. Menu production records included, for each food served during the target week (including a la carte foods and foods sold to adults), information on the number of servings prepared, the number of servings left over, and the number of servings actually served or sold. Daily time records were completed by all employees in the sampled schools for the full five-day period. The time records were used to allocate time spent on breakfast and time spent on lunch. A time allocation grid was also obtained for central office staff to identify time devoted to breakfast and lunch production. Data from administrative records (bid/inventory lists and salary rosters) were used to translate food production information and time re.ords into food and labor costs, respectively. These data were used to compute the average cost of producing a reimbursable lunch and breakfast, if offered, in each school. A detailed description of the methodology used in the study of meal costs is provided in Appendix E. 2 To obtain a reasonable aiataamrnr of nutrient content, it is necessary to examine meals offered over a period of time rather than a single meal. The National Research Council (SRC) recommends that group feeding programs plan menus so that nutrient standards are met over a five- lo ten-day period. A sample five-day period, equivalent to a full week in most school districts, is routinely used in USDA-sponsored evaluations of Child Nutrition programs. NSMP requirements specify that analyses be based on a typical school week, ranginc from three to seven days. AM Associate* Inc. h Evaluation Design 2-5 On-Site Observations On-site observations of meal preparation and service assessed compliance with planned menus, recipes, and portion sizes. One-day observations were completed during the target week in each of the three schools selected for the cost substudy. Observers documented whether items on the planned menu were actually offered and collected descriptive information on any additional items offered. They also observed food preparation and documented the nature and extent of deviations from planned recipes. In addition, observers weighed five random portions of all foods offered to children to determine average portion sizes.3 Finally, for purposes of the substudy of meal costs, staff observed students as they passed through servings lines and recorded which items students included in their reimbursable meals. Surveys ofSchool Food Service Staff In Spring 1996 and 1997, school food service staff in a random sample of schools in each SFA completed brief mail surveys. Surveys assessed staff perceptions about NSMP, particularly perceptions about burden and about the impact of NSMP on the acceptability of school meals. The methodology used in selecting schools and in developing weights for use in the analysis is described in Appendix F. Interviews with State Agency Staff Brief telephone interviews were completed with directors of State agencies and other relevant staff in Fall 1995 and Summer 1997. Interviews included questions about problems encountered in demonstration SFAs or other SFAs using NSMP as well as State agency directors' perceptions about monitoring under the new SMI guidelines (see Chapter 1). Start-Lip Logs SFA directors were asked to maintain detailed logs during the implementation period, documenting the number of staff hours devoted to a variety of different NSMP tasks. Directors were instructed to record only hours associated with work that would not have been done in the absence of NSMP. Respondents also provided information on the hourly wages of individual staff members so that costs associated with the reported labor could be calculated. SFAs were asked to submit start-up logs every six months, beginning in January 1995 and continuing until full implementation was achieved. Administrative Records Data on program participation, food service income and expenses, a la carte revenues, and commodity utilization (total dollar value of commodities received) were collected by mail in Fall 1994 (for SY 1993- 94) and Fall 1997 (for SY 1996-97), after year-end summaries had been prepared by SFA staff. Standards Used in Evaluating Nutrient Content Two sets of nutrient standards weie used in evaluating the nutrient content of NSLP and SBP meals (Exhibit 2.3). The first set is comprised of standards specified in the NSMP demonstration protocol (and, under SMI, carried over into current NSLP and SBP regulations). These include standards for calories and target nutrients for which RDAs have been established (protein, vitamin A, vitamin C, calcium, and iron) as well as for the percentage of calories from fat and saturated fat. 3 Commercially pre-park«|rrl item* such as cartons of mult and |wtas^ of cookie*, crackers, or chip* were excluded became previous experience has thown minor, if any. deviations front expectations due to quality control in commercial pack aftng processes AM Associate* Inc. Evaluation Design 24 & Exhibit 23 Nutrient Standards Used in the Evaluation Nutrient Standard Standards Defined for NSMP Nutrients with established RDAs:' Breakfast: One-fourth of the RDA Calories, protein, vitamin A, vitamin C, * calcium, and iron Lunch: One-third of the RDA Nutrients included in the Dietary Guidelines for Americans'} Total fat S 30% of total calories Saturated fat < 10% of total calories NRC Recommendations3 Carbohydrate > 55% of total calories Protein < 15 % of total calories4 Cholesterol Breakfast: £ 75 mg' Lunch: slOOmg6 Sodium Breakfast: soOOmg5 Lunch: sSOOmg6 1 National Research Council (1989). Recommended Dietary Allowances, 10th edition. Washington. DC: National Academy Press. 1 U.S. Departments of Health and Human Services and Agriculture (1995). Nutrition and Your Health: Dietary Guidelines for Americans, 4th edition. Washington, DC: U.S. Government Printing Office. 1 National Research Council (1989). Diet and Health: Implicationsfor Reducing Chronic Disease Risk. Washington DC: National Academy Press. 4 Diet and Health recommends a maximum protein intake equivalent to leu than twice the RDA. To achieve recommended levels of calories from fat and carbohydrate, the percentage of total calories from protein needs to be in this range. ' One-fourth of suggested maximum intake of cholesterol (300 mg) and sodium (2,400 mg). 6 One-third of suggested maximum intake of cholesterol (300 mg) and sodium (2,400 mg). Abt Associates Inc. * Evaluation Design 2-7 A second set of standards, based on the National Research Council (NRC) recommendations, was defined for nutrients and food components that are analyzed by NSMP software but not included in standards defined for NSMP. These include the percentage of energy from carbohydrate and protein as well as total cholesterol and sodium content. Neither demonstration SFAs, nor current NSMP users outside the demonstration, are required to meet these additional standards. They are used in this report solely for the purpose of facilitating interpretation of the data. Data on fiber content are also presented in this report. However, these data are not compared to a standard because there is no scientific consensus about an appropriate benchmark for daily consumption of fiber. Estimation of Program Impact The impact evaluation examined the effect of NSMP on nutrient content, meal costs, and key aspects of food service operations (e.g., participation rates, a la carte revenues, use of donated commodities, number of daily choices offered, and variety of foods offered over the course of a week). Because relatively few SFAs had implemented NSMP by Spring 1996, impact analyses focus on comparisons of baseline (Spring 1994) and final follow-up (Spring 1997) data. Throughout the text the terms 'pre- NSMP' or 'before NSMP* are used to refer to Spring 1994 data and 'after NSMP' or 'NSMP' are used to refer to Spring 1997 data. The basic measure of program effect is the difference between estimates made prior to NSMP implementation and those made afterward. Simple /-tests are used to test the statistical significance of differences in means or proportions. It is important to realize that impacts that are detected cannot necessarily be ascribed to NSMP. In the absence ot a control group, it is impossible to rule out the possibility that changes over time were influenced by factors external to the demonstration. Thus, information on program impacts should be considered suggestive rather than conclusive, and is not necessarily representative of what might happen in programs nationwide. Impact analyses are further limited by small sample sizes. Some of the SFAs selected to participate in the demonstration dropped out or never implemented NSMP. Others implemented NSMP only in elementary schools. Sample sizes vary from one analysis to the next depending on the number of SFAs included. The maximum sample size for any SFA-lcvel analysis is 19 (analysis of elementary lunch menus). These small samples provide adequate statistical power to detect only very large effects. Consequently, the absence of statistical significance cannot be interpreted with confidence as evidence that a meaningful difference does not exist. Rather, the conclusion must be that very large differences do not exist. AM AMoeif■ Inc. Evaluation Daaign 2-f Jd Chapter 3 Implementation of NSMP in Demonstration SFAs This chapter describes the SFAs that participated in the demonstration, the extent to which SFAs were successful in implementing NSMP, and difficulties and successes experienced by SFA staff. The first section of the chapter describes the SFAs selected to participate in the demonstration. The next two sections summarize challenges faced by SFAs during early stages of implementation and the final implementation status of all SFAs at the end of the demonstration (Spring 1997). The next section describes expectations of the original NSMP protocol and the extent to which SFAs were able to meet these expectations. The final section of the chapter identifies characteristics that may predict successful NSMP implementation. Selection of Demonstration SFAs SFAs were invited to volunteer for the NSMP demonstration in the Summer of 1993. Thirty-five SFAs were ultimately selected by FNS staff from a pool of 127 applicants. One SFA dropped out shortly after selection because school board approval for the project was not obtained. Thus, the demonstration began with a total of 34 SFAs. SFA selection was structured to ensure that the demonstration would provide an opportunity to assess the feasibility and impacts of NSMP in a variety of different school districts. As summarized in Exhibit 3.1, FNS staff purposefully structured the sample of SFAs to ensure diversity in geographic location, community type, school district size, and program participation rates. The nutrition and food service experience of the SFA director and the availability of nutrition expertise among other SFA staff were also considered in selecting sites. About two-thirds of the selected SFAs had a registered dietitian on staff. In about half of die SFAs, the director and/or NSMP coordinator was a registered dietitian. SFAs selected for the demonstration were managed both by veteran school food service professionals (up to 38 years of experience) and by relative newcomers (1 -2 years of experience). Characteristics of existing food service programs were also taken into account. Sites were selected to ensure variation in centralization of menu planning, use of cycle menus, the extent of centralized food production, use of standardized recipes, and prevalence of scratch food preparation (as opposed to use of pre-prepared convenience items). A final factor that influenced site selection was prior experience with computerized nutrient analysis. Because ready access to a computer is a prerequisite for NSMP implementation, all of the districts selected had some pre-existing computer capability. However, only 13 of the 34 selected districts had actually analyzed the nutrient content of their menus by computer prior to the demonstration. In about half of the selected districts, the SFA director and/or NSMP coordinator had prior experience with computerized nutrient analysis. Abt Associate* Inc. lmp4«m»nUtk>n of NSMP in Demonstration SFA» 3-1 Exhibit 3.1 Characteristics of Demonstration SFAs FNS Region Mid-Atlantic Mountain Plains Midwest Northeast Southeast Southwest Western Community Type Urban Suburban Rural Enrollment Under 2400 2400-9,999 10,000-24,999 25.000 or more Mean enrollment Average Daly NSLP Participation Rate 40 percent or less 41-55 percent 56-70 percent 71 percent or more Mean participation rate Percent of Lunches Served Free 25 percent or less 26-50 percent 51-74 percent 75 percent or more Mean percentage 4 4 5 5 5 6 5 9 16 9 6 9 t 11 33.984 8 9 13 4 53% 10 10 8 6 44% AM A«wocia«M Inc. ■pMMMM of NSMP In Demonstration SFAs 3-2 $SL Exhibit 3.1 (continued) Nutrition/Food Service Experience of Staff Registered dietitian or nutritionist on stiff SFA director or NSMP coordinator is registered dietitian SFA directors' total food service experience (years) Minimum Maximum Mean SFA directors' experience as director (years) Minimum Maximum Mean Characteristics of Food Service Program Prior to NSMP Fully centralized menu planning Cycle meniKs) used Some centralized food production1 Percentage of entree items prepared from scratch 25% or less 26-30% More than 50% Mean percentage Prior Experience with Computerized Nutrient Analysis District menus previously analyzed by computer SFA director or NSMP coordinator had prior experience with computerized nutrient analysis 23 It 1 38 15 1 19 8 21 10 17 7 15 12 51% 13 17 1 50% or rnore of schools in distrkt are served by base or central kitchens. Note: Some data may differ from application forms or from previous reports (Few. MK, 1997) because mformation for SFAs was clarified/updated in subsequent interviews. Source: Application forms for NSMP demonstration (Fall 1993). Abt Associates Inc. Imptemontetion of NSMP In Domooolintioo SI As 3-3 33 Early Implementation Challenges From the outset, NSMP implementation in all SFAs was complicated by a number of factors over which SFAs had no control. Most important was a significant delay in the approval of NSMP software systems. Software vendors were slow to submit products for USDA review and approval, and many of the programs submitted did not satisfy the functional criteria defined by USDA. The delay in the availability of approved software systems effectively postponed concerted implementation efforts in most SFAs. While most SFAs began working on NSMP at some level shortly after receiving training in June 1994, the majority did not begin to work in earnest until sometime afterward. By Jaruary 199S, six months after SFA staff were trained and the point at which process study data were originally scheduled to be collected, only two NSMP software packages had been approved. At that tune, FNS contacted all demonstration SFAs and asked that they select one of the two available systems, move forward with NSMP implementation during the remainder of SY 1994-95, and continue to work toward full implementation by Fall 1995. Several SFAs lobbied for approval of other softwait systems, and a third system was ultimately approved.1 While all but two of the SFAs that completed the demonstration eventually selected one of the three available software systems, many SFAs did not really begin working with NSMP software until well into the latter half of SY 1994-95 or the subsequent summer months. This delayed start-up had a substantial impact on the progress SFAs were able to make during the first year of the demonstration. Another problem that complicated implementation was the poor response USDA received from food manufacturers contacted to supply nutrient information for the CN Database. Very few manufacturers responded to the request, and much of the data submitted was found to be incomplete, inaccurate, or otherwise questionable. As a result, the database included in the approved software systems did not include many of the commercially purchased products used in school food service. This situation placed an unanticipated burden on demonstration SFAs. SFA staff had to assume responsibility for obtaining nutrient information tor nationally-available commercial products and for entering these data into the their own databases. Status of Implementation in Spring 1997 By Spring 1997, when final follow-up data collection began, less than half (16) of the original 34 SFAs were considered fully implemented (Exhibit 3.2). Ten of these SFAs had implemented NSMP in accordance with expectations—NSMP menus were being served in all schools and menus had been developed using approved NSMP software and district-wide menu production information. The other six SFAs were serving NSMP menus in all schools but used a modified approach to weighted nutrient analysis. Modifications included basing saving projections used in weighted analyses on data from a subset of schools rather than all schools in the district or on staff predictions rather than actual menu production data. One of these SFAs also used an unapproved software system one that had been in use prior to the demonstration. I Sawe January 1995. 1! other software sysetns have been approved AM Associates lr>c. ■nptewiHtJun or NSMP In DwwonsUaHon SFAs 3-4 Final Implementation Status of Demonstration SFAs Fully implemented Implemented without modifications Weighted analysis completed using approved software and district wide menu production data Implemented with modifications Weighted analysis completed using approved software but used menu production data from a subsample of schools1 Weighted analysis completed using approved software but used staff predictions rather than district-wide menu production data Weighted analysis completed using unapproved software and staff predictions rather than district-wide menu production data Partially implemented Implemented for elementary lunch; no analysis done for lunch at other levels or for breakfast at any level Implemented for elementary breakfast and lunch; no analysis done for other levels Implemented for breakfast at all levels; no analysis done for lunch Not implemented No menus analyzed fully Lunch menus analyzed, but with unapproved software and without weighting; breakfast menus not analyzed Breakfast and lunch menus analyzed but without weighting; analysis based on one "set" menu each day, without consideration of numerous other available choices Dropped out 10 i 2 4 2 1 1 3 1 1 11 l One SFA used district-wide menu production data for 'inch menus but based breakfast analysis on data from one elementary school one middle school and one high school Inc. hnphimentation of NSMP in Demonstration SFAs J-5 3^ Because the unapproved software did not have the ability to conduct a weighted analysis, SFA staff developed a companion spreadsheet program to compute weighted weekly averages (based on staff projections). Four SFAs achieved only partial implementation and three SFAs never implemented NSMP. Three of the partially implemented SFAs implemented NSMP only in elementary schools, and the other SFA implemented NSMP only for breakfast In all four of these SFAs, directors reported that they had not achieved full implementation because they were unable to dedicate the required number of staff labor hours. Among the three SFAs that had not implemented NSMP by Spring 1997, one was still in the process of developing and analyzing menus. The director was doing all the work herself and had experienced several false starts and multiple software problems. The other two non-implemented SFAs had analyzed breakfast and lunch menus, but the analyses deviated substantially from the NSMP protocol. One of these SFAs used unapproved software and used unweighted nutrient analysis. The other SFA based analyses on a single defined meal each day rather than incorporating all of the menu options that were actually available. For example, daily lunch analyses included only one entree, ignoring a second entree offered in elementary schools and numerous other entrees (more than ten on some dayv) offered in middle and high schools. Finally, a total of 11 SFAs withdrew from the demonstration sometime before Spring 1997. In most cases, several different factors contributed to the decision to terminate participation. Concern about the use of weighted nutrient analysis was the primary reason for withdrawal, cited by eight of 11 SFA directors (Exhibit 3.3). Most often, the concern centered around the amount of staff labor required to obtain district-wide menu production data. Some SFA directors were also concerned that weighted nutrient analysis would limit flexibility in menu planning and decrease the ability of individual schools to cater to students' preferences. A few directors were worried that a weighted analysis could only meet nutrient standards if popular high-fat food items were eliminated or offered much less frequently. These SFA directors believed that such changes would have a negative impact on lunch participation in middle and high schools. Problems with NSMP software contributed to the withdrawal of five SFAs. Specific problems included incompatibility of available NSMP software with other software currently in use for other aspects of program operations; the need for upgraded hardware in order to run NSMP software; and numerous problems with the prototype NSMP software received at the June 1994 training and/or with early releases) of approved systems. Five of the SFA directors who dropped out of the demonstration were concerned about the time line for implementation. Specifically, they were unhappy that they were expected to implement NSMP by the start of the 1993-% school year despite the lengthy delay in the software approval process. Directors reported that they could not dedicate the amount of labor required to implement NSMP within the condensed time frame. X Exhibit 33 Reason for Withdrawal from the Demonstration Concerns about weighted nutrient analysis Labor requirements Potential negative impact on flexibility in menu planning Potential negative impact on secondary school participation NSMP software problems Approved NSMP software systems not compatible with existing system Could not afforJ necessary hardware upgrade Numerous problems with NSMP software Demonstration time line too constrained Change in SFA director Concerns about NSLP SMI regulations issued prematurely NSLP may be block granted Note: Respondents could report more than one reason for withdrawal. Source: Interviews with directors of drop-out SFAs. November 1995-January 1997. Two SFAs terminated participation because there had been a change in SFA directors. The new directors were not prepared to assume responsibility for continuing with NSMP implementation. Finally, two SFA directors were influenced by situations that were not directly related to NSMP or to the demonstration. One director believed that issuance of the SMI regulations (see Chapter 1) prior to the conclusion of the evaluation undermined the intent of the demonstration. He was therefore unwilling to devote the time and resources necessary to continue with the project. The second SFA director was concerned that the NSLP was going to be block granted (an idea that was being debated by the Congress at the time), and that, as a consequence, her district would cease to participate in the program. AM of NSMP in Demonstration SFAs 3-7 il Implementation of NSMP in Demonstration SFAs The sections that follow describe key components of NSMP and summarize the difficulties and successes experienced by SFA staff in attempting to follow implementation guidelines in each area. All 23 of the SFAs that remained in the demonstration through Spring 1997 are included in these analyses. Additional information related to implementation is provided in Chapter 7 which reports on start-up and maintenance costs incurred by demonstration SFAs as well as SFA staff perceptions about the burden and acceptability of the NSMP system. Staff Responsibility To increase the probability of success, SFAs were required to identify one individual who would assume responsibility for overseeing NSMP implementation. In 17 of the 23 SFAs that remained in the demonstration through Spring 1997, SFA directors assumed direct responsibility for NSMP implementation. Ten of these directors did virtually everything themselves, including data preparation and entry. The other seven directors delegated data preparation and entry tasks or received substantial assistance from other staff or volunteer assistants, including dietetic interns and State office staff. For one SFA, the State office essentially took over these tasks because neither the SFA director nor his limited staff had time to do it. All of the SFAs in which responsibility for NSMP implementation was delegated were either very large (enrollments in excess of 25,000) or large (enrollments between 10,000 and 24,999). NSMP Software Systems As noted above, demonstration SFAs had three approved NSMP software systems from which to choose. These were School Nutrition Accountability Program (SNAP); Computer-Assisted Food Service (CAP'S); and NUTRIKIDS (developed and marketed by Lundifiyte Systems, Inc.). By the time the demonstration was over, many other software systems had been approved. Aoout half of the 23 SFAs that remained in the demonstration through Spring 1997 used NUTRIKIDS (Exhibit 3.4). Five of these SFAs originally started with SNAP, the prototype software distributed at training, but switched to NUTRIKIDS during the course of the demonstration. SNAP was the next most common software package, used by five SFAs. CAFS was used by three SFAs. Two of these SFAs already owned the CAFS food service management software system and simply purchased the NSMP software as an enhancement. The other SFA that elected to purchase CAFS did so because of the system's other features, i.e., modules that support a variety of food service administration and operations tasks. One SFA elected to use one of the NSMP systems that received approval later in the demonstration— PCS, a multi-module school food service management system offered hy PCS Revenue Control Systems, Two SFAs continued to use a software system that was in place prior to Lie demonstration (Practorcare), even though the software never received USDA approval. One of diese SFAs was in the fully-implemented- with-modifications group and the other SFA never implemented NSMP. Some SFAs experienced problems with NSMP software during the initial stages of implementation. Most of these problems were related to glitches in initial versions of the software, inexperience on the AM *IIOCIHM Inc. Impiomerrtaoon of NSMP in Demonstration SFAs 3-f 3? Exhibit 3.4 Software Systems Used by Demonstration SFAs Approved NSMP Systems NUTRIKIDS SNAP CAFS PCS Unspproved Software Systems Practorcarc 12 5 3 1 Note: Excludes 11 SFA» that dropped out of the demonstration. Source: SFA director interviews, Spring 1997. part of SFA staff, or problems with hardware. The frequency of software problems decreased as the demonstration continued and both software manufacturers and SFA staff gained more experience. Overall, SFAs using approved NSMP software systems were pleased with their selections. The majority found the software to be both easy to use and time efficient. For most SFAs, the only major problem with NSMP software was the fact that the nutrient database did not include information for brand name food products. This shortcoming placed the burden of acquiring and entering these data on the shoulders of SFA staff. In many SFAs the task of collecting, reviewing and entering data for literally hundreds of food items consumed a sizeable amount of SFA resources. Data for Weighted Nutrient Analyses To support development of serving projections for weighted nutrient analyses, SFAs were required to maintain menu production records which documented the number of individual portions actually served in reimbursable meals. A la carte and adult servings were not to be included in these counts. Unless an SFA planned to conduct separate analyses for individual schools within the district, weighted analyses were to be based on district-wide data. A sample menu production record was provided in the NSMP training manual, but SFAs were free to adapt the form for their own use. As noted in the preceding discussion of reasons for SFA withdrawal, many SFAs found it difficult to comply with this requirement. The amount of labor involved in collecting and synthesizing data from all schools in the district was the major problem Although virtually all SFAs maintained some form of menu production records, these records were maintained only at individual schools. Few SFAs dealt with Abt Associates Inc. Imptomsntatlon of NSMP in Demonstration SFAs 3-9 * these records at the district level. Most SFAs used purchasing records, ordering requisitions and inventory reports, rathet than detailed menu production records to track district-wide food usage. Consequently, the need to gather and synthesize menu production data required most SFAs to implement and manage an extensive stream of paperwork. SFA staff also identified more specific difficulties involved in obtaining these data. These included the need to convert food production information traditionally reported in volume or weight (e.g., ten gallons of gravy) to servings, the unit of measure needed for NSMP; separating foods sold a la carte from those sold in reimbursable meals; maintaining separate counts for similar items that were traditionally counted together, e.g., cookies, sandwiches, chips; and obtaining production information for self-serve foods, including, but not limited to, salad bars and other specialty food bars. While the sentiment regarding the requirement for district-wide menu production data was largely negative, seven SFA directors found this requirement, in combination with the requirement that all recipes be standardized, to be beneficial. These directors reported that the increase in accountability and standardization allowed them to identify and rectify inefficiencies and weaknesses in their overall operations. Standardized Recipes SFAs were expected to have a standardized recipe for every menu item that included more than one ingredient or involved preparation other than simple heating. A standardized recipe was defined as "one that has been tested to provide an established yield and quality through the use of ingredients that remain constant in both measurement and preparation methods." Districts were free to use standardized USDA recipes or to use/develop their own recipes. Approved NSMP software included modules to develop, analyze, and modify standardised recipes. SFAs that used salad bars or other theme bars were encouraged to develop recipes based on menu production information, i.e., the number of reimbursable meals served from the theme bar and the total quantity of food used. Several versions of a recipe might be needed if different types of schools offered different items or, for any pre-portioned foods, different portions. SFA staff were expected to reanalyze/update theme bar recipes if there were changes in the food items offered. This requirement proved to be more challenging than some SFA directors expected. When initial process evaluation data were collected, few SFA directors expected standardized recipes to be a problem. As implementation progressed, however, and SFAs gained experience in this area, difficulties increased. The level of effort required to modify and test existing recipes was greater than many SFA directors anticipated. Recipes often needed to be tested several times before a product was deemed acceptable. SFA directors complained that there were not enough "healthy" quantity recipes available. Although USDA issued an expanded set of recipes during the course of the demonstration, many SFAs incorporated only a few of these items, reporting that many of them were "inappropriate" for their populations (i.e., "my students will not eat that"). Resistance among SFA staff, particularly cooks, was another problem faced by SFA directors striving to standardize food production. Many cooks were unhappy with the increased emphasis on standardization, which required that they abandon some of their favorite recipes and refrain from making individual decisions about how a recipe might be improved. The need for a standardized approach to recipe preparation and food portioning was a recurrent theme in training sessions nek) for SFA staff. Abt Aaeoclatee Inc. " Implementation of NSMP In Demonstration SFA* 3-10 Y0 Age/Grade Groups Used In Analyzing Menus The demonstration protocol defined four different age/grade groupings for use in menu analysis: preschool and early elementary grades K-l (ages 3-6); upper elementary grades 2-5 (ages 7-10); middle school grades 6-8 (ages 11-13); and high school grades 9-12 (ages 14-17). SFAs were expected to plan and analyze separate menus for these four age groups. Because all school districts were not expected to have exactly identical grade configurations, the protocol allowed menu planners to incorporate one additional age group at either end of the defined span (e.g., 6-10 or 10-14). If the disparity in age/grade groupings included two or more years at either end of the spectrum, menu planners were required to develop customized standards using a specialized function available in approved software. Technical guidance issued after the training session stipulated that the calorie requirement in any customized standard be within ten percent of the calorie requirement for the oldest and youngest children in the group. This requirement effectively limited the number of ages that could be included in developing a customized standard and required that two separate menus be analyzed for most schools with wide grade spans, e.g., elementary schools covering grades K-6 or K-8 or combination schools covering grades K-l2. From the beginning, many SFAs had difficulty de.ermining which age groups to use in their nutrient analyses. This was particularly true for SFAs that had a number of different grade configurations for one type of school (e.g., elementary schools that covered grades K-6,1 -3, and K-12). SFAs' initial confusion about this issue was exacerbated by the issuance of the SMI regulations because the requirements specified in the regulations differed from those defined for the demonstration. First, although the SMI regulations encourage use of the four basic age/grade groups defined in the NSMP protocol, these are presented as an option rather than a requirement An alternative scheme presents minimum, less stringent, age/grade groupings. Second, the SMI regulations permit calculation of a combined middle/high school menu. Finally, although the -egulations allow for development of customized standards to meet individual district needs, no restriction is placed on the number of age groups that can be included in a combined, customized standard. In the end, the age/grade groups used by many SFAs were not consistent with expectations of the initial protocol. Only two of the 20 SFAs that planned elementary school menus (16 fully implemented SFAs and four partially implemented SFAs) planned two separate menus for lower and upper elementary grades. Six of the 16 SFAs that planned middle and high school menus (all of the fully implemented SFAs) used a combined middle and high school analysis rather than separate analyses for each school level. Finally, as discussed in Chapter 9, the age spans chosen for analysis, particularly for elementary menus, were not always appropriate. Alternative Offer-versus-Serve Rule NSMP menus, whether for breakfast or lunch, must include a minimum of three menu items, and one of the items must be milk. In addition, one of the three items offered in lunch menus must be an entree. Menu items are defined as any single food or combination of foods served together, e.g., a sandwich is considered one menu item. Demonstration SFAs were required to implement offer-versus-serve (OVS), a regulatory provision that allows students in secondary schools and, at the discretion of SFAs, elementary schools, to refuse some of the food items offered in reimbursable lunches. The OVS rule defined for the demonstration required that students take a minimum of two menu items. No restriction was placed on the items that could be refused. AbtAeaoclaOalnc. Implementation of BBS In Pemonetretlon SFAe i£TT HI Confusion about the OVS rule increased sharply when final SMI regulations were published in June 1995. The SNA regulations require that students select a minimum of two menu items and one of the selected items must be an entree. Most SFA directors incorrectly believed that the SMI regulations superseded the guidelines provided during the June 1994 NSMP training. Consequently, only five of the 20 fully or partially implemented SFAs followed the OVS rule defined for the demonstration. The remaining 15 SFAs followed the SMI version of the OVS rule or maintained the OVS rule used under the traditional meal pattern (students must take 3 of 5 meal components). This situation made it impossible for the evaluation to provide FNS with information about the acceptability or impact of the "any two items" OVS rule. Disclosure of Nutrition Information SFAs were not required to disclose nutrient information, but they were encouraged to do so as a means of increasing awareness of both NSMP and the SFA's efforts to improve the nutritional quality of school meals. Half of the SFAs that fully or partially implemented NSMP disseminated information about the nutrient content of school meals (Exhibit 3.5); some SFAs used more than one method. Eight SFAs used printed menus that were sent home to parents, three used posted information in cafeterias, and two published information in newspapers or local cable television bulletin boards. ExMbtt 3.5 Disclosure of Nutrition Information by Demonstration SFAs Disseminate information about nutrient content of meals Yes, on printed menus sent home Yes, in newspapers/on local TV Yes, post on cafeteria line No Reasons for not disclosing nutrition information Don't have time Don't want to do so until NSMP is implemented in all schools Software only gives monthly (rather than weekly) average; monthly summary is not useful Not completely comfortable with accuracy of information yet Believe it will confuse parents more than help them No money to print/send home menus Note: Respondents could report more than one form of disclosure »nd more thin one reason for non-disclosure Source: SFA director interviews. Spring 1997. AM AttoclatM Inc. ■ttpwentnurtion of NSMP In D#monsti Alton SFAs 3-12 ¥&- SFA directors who did not disclose nutrition information had several reasons for not doing so. Four directors said they simply did not have the time to prepare and disseminate this information. Two directors did not want to disclose nutrition information until NSMP was implemented in all schools. Two directors reported that they could only get monthly printed summaries from their software analysis system and did not believe that a monthly summary was useful. One director was not yet comfortable enough with the results of her NSMP analyses to disseminate the results. Another believed that disclosure of information would be more confusing than helpful. Finally, one director indicated that the SFA did not have enough money to support printing and distribution of menus for parents, the vehicle she saw as the most viable one for disseminating information on the nutrient content of planned menus. Promotional Activities in School Cafeterias Another activity that was encouraged, but not required, was use of cafeteria promotions to increase awareness of NSMP and to encourage children to accept new/modified menu items. Three-quarters of the fully or partially implemented SFAs conducted at least one special cafeteria promotion before Spring 1997 (Exhibit 3.6). Most, but not all, promotions took place in elementary schools and most involved taste testing activities. SFAs also reportedly assisted with classroom nutrition/health education activities and held special 5 A Day promotions to encourage fruit and vegetable consumption. Exhibit 3.6 Cafeteria Promotions Conducted by Demonstration SFAs Conducted special promotions in school cafeterias Yes 15 No 5 Activities conducted Taste testing new products/recipes 10 Coordinated activity with classroom nutrition/health 8 education Held S A Day or other promotion to increase fruit and 5 vegetable consumption Special nutrition awareness week (including National 2 School Lunch Week) Note: Respondents could report more than one activity. Source: SFA director interviews. Spring 1997. Abt Associates Inc. Imptem«ntetton of NSMP in Demonstration SFAs 3-13 «3 UM of Fortified Foods While formulating plans for the demonstration, FNS staff were concerned that some SFAs might satisfy nutrient standards by offering highly-fortified foods rather than planning an appropriately balanced menu using foods from the basic food groups. Program planners considered establishing restrictions en the use of fortified foods. Although such restrictions were never implemented, with the exception of limitations on the use of vegetable protein products (VPP), SFA staff were discouraged from using such foods. Staff in two-thirds of the SFAs that remained in the demonstration through Spring 1997 were approached by vendors offering fortified foods (Exhibit 3.7). The most common items offered Exhibit 3.7 Use of Fortified Foods by Demonstration SFAs Approached by vendor offering fortified foods Yes No Food offered by vendors "Super" donuts/s veet rolls Other fortified bread products Milk-based drinks Dessert items Vegetarian/soy entrees "Energy" bars Fruit drinks Fruit rollups Yogurt Other Incorporated fortified foods not used prior to NSMP Yes No 23 Note: Fortified foods have been specially fortified with one or more Source: SFA director interviews. Spring 1997. vitamins or minerals Inc. ImpfOffi+otsffOfi of NSMP wt rvfTiontlfMlon SFAs 3-14 &L were "super" donuts and sweet rolls similar to those already used in many SBP programs across the country. Less commonly encountered fortified foods included other fortified bread products such as bagels and muffins, milk-based drinks, desserts, vegetarian entrees., "energy" bars, fruit drinks, fruit rollups, and yogurt. Although two-thirds of the SFAs that did not withdraw from the demonstration were solicited by vendors offering specially fortified foods, none or these SFAs incorporated a fortified food that was not already being offered before NSMP (e.g., several SFAs used "super" donuts before joining the demonstration). Meeting Nutrient Standards Directors of the 16 fully implemented SFAs were asked about their problems and successes in meeting nutrient standards, based on the menus being offered in Spring 1997. Two SFAs reported that all of their planned menus satisfied all of the relevant NSMP standards. The remaining 14 SFAs reported that one or more of their planned menus failed to meet at least one of the nutrient standards. Exhibit 3.8 summarizes SFA directors' reports for breakfast and lunch menus. With the exception of calories and the percent of calories from saturated fat, more than three-quarters of the SFAs that offered breakfasts reportedly satisfied individual nutrient standards for breakfasts. The calorie standard was the most problematic. Seven of the 15 fully implemented SFAs that offered breakfast in Spring 1997 reported that one or more of their planned menus did not meet this standard. Four SFAs were unable to consistently meet the standard for saturated fat. Other standards that were problematic for one or more SFAs included the percent of calories from fat (3 SFAs); vitamin A (3 SFAs); iron (1 SFA); and vitamin C (1 SFA). Many of the same nutrients were problematic for lunch menus. Only eight of the 16 fully implemented SFAs reported meeting the standards for calories from fat and saturated fat on a consistent basis. Four SFAs reported difficulty meeting the standard for calories, specifically in secondary school menus. A comparable number reported similar difficulty with the standard for iron. Finally, two SFAs reported difficulty with the standard for vitamin A and one SFA had difficulty with the standard for vitamin C. SFAs were encouraged to monitor levels of carbohydrate, cholesterol, sodium, and dietary fiber in planned menus, although specific standards were not established for these nutrients. Only two of the 16 fully implemented SFAs reported monitoring these nutrients. SFA directors who were not monitoring these nutrients consistently reported that they (and their staff) did not have time to do anything that was not required. Calendar Time Required for Full Imptomentatlon Most SFAs required substantially more time to implement NSMP than originally anticipated. Among the 16 SFAs that fully implemented NSMP, an average of 19 months elapsed between the time SFA staff started actively working on NSMP implementation (which in many cases was several months after receiving training) and the time NSMP menus were served in all schools (Exhibit 3.9). The range was very broad, with a minimum of three months and a maximum of 33 months. In the great majority of the SFAs that achieved full implementation, active work on NSMP spanned more than one calendar year. Only three SFAs achieved full implementation in less than 12 months, while six SFAs required more than two years. Inc. of NSMP in Demonstration SFAs 3-1S if Exhibit 3.8 Reported Successes and Probkms in Meeting Nutrient Standards Breakfast Number of Fuify Implemented SFAs (■-IS) Calories 8 7 Protein 15 0 Calcium IS 0 Iron 14 1 Vitamin A 12 3 Vitamin C 14 1 Percent of calories from fat 12 3 Percent of calories from saturated fat 11 - 4 Lunch Calories 12 Protein 16 Calcium 16 Iron 12 Vitamin A 14 Vitamin C IS Percent of calories from fat 8 Percent of calories from saturated fat 8 Number of Fully Implemented SFAs (■-16) 4 0 0 4 2 1 8 8 Note: One fully implemented SFA does not offer breakfut. Source: SFA director interviews, Spring 1997. Inc. of NSMP In Demonstration SFA* -# 3-16 MMII Time Required by Demonstration SFAs to Achieve Full Implementation Month! elapsed between initiation of active work on NSMP and full implementation Less than 12 months 3 13 -24 months 7 More than 24 months 6 Mean (months) 19 Minimum (months) 3 Maximum (months) 33 SFA directors' estimates of time required under more favorable circumstances Less than 12 months 9 13-24 months 6 More than 24 months 1 Mean (months) 16 Minimum (months) 3 Maximum (months) 30 Source: SFA director interviews. Spring 1997 Although SFAs were able to work on implementation tasks during the period when approved software was not available (e.g., gathering menu production data, standardizing and/or modifying recipes, evaluating potential new commercial products, etc), delays in software approval may have introduced some inefficiencies and led to a more protracted implementation period than would have been experienced under more optimal conditions. Likewise, the fact that SFA staff had to acquire and enter nutrition information for many commercially-prepared foods (because the version of the nutrient database included in the NSMP software did not include this information) may have contributed to an extended implementation period. To get some idea of the time period required for NSMP implementation in the absence of these complications, directors of the 16 fully implemented SFAs were asked to estimate the amount of time AM AsaociatM Inc. knptomontation of NSMP hi Demonstration SFAs 3-17 XL they would need to implement NSMP if they were to do it again under more optimal circumstances (i.e., with readily available and more complete software) and with the knowledge gained during their initial experience. Responses are summarized in Exhibit 3.9. As shown, the time line for NSMP implementation projected by most SFA directors was shorter than their actual experience in the demonstration, but not dramatically so. On average, projected implementation time lines were about three months shorter than actual demonstration experience. Nine of the 16 SFA directors estimated that they could implement NSMP u. 12 months or less. Six directors estimated that they would need one to two years to implement Only one SFA director indicated that he would need more than two years to implement NSMP. Characteristics That May Predict Successful NSMP Implementation One of the research objectives defined for the evaluation was to determine whether particular SFA characteristics influence the likelihood of successful NSMP implementation. In view of the implementation rates described in preceding sections, this question seems especially important However, the study's ability to draw conclusions about relationships between SFA characteristics and successful implementation is limited by small sample sizes. Group sizes for "implementers" and "non-implementers" are so small that only extremely large differences between the two groups can be deemed statistically significant It is therefore more informative to examine the data and identify patterns that ire suggestive of meaningful associations. Exhibit 3.10 shows tabulations of final implementation status by selected SFA characteristics. Included are baseline characteristics considered in selecting demonstration SFAs as well as several other variables thought to be potentially important to NSMP implementation. The latter group includes whether the SFA director assumed direct responsibility for NSMP implementation and whether there was a change in SFA directors during the course of the demonstration. All 34 of the SFAs selected for the demonstration are included in these tabulations. It is important to note that several of the characteristics that could potentially be influential are highly correlated with each oth s. For example, urban SFAs tend to include more schools, have larger total enrollments, serve more reimbursable meals on an average day, and include a greater proportion of children in poverty than suburban and rural SFAs. The small available sample size does not support multivanate estimates of the separate effects of each characteristic. In comparing SFAs that achieved full implementation, with and without modifications, to SFAs that did not achieve full implementation, several interesting patterns emerge, as summarized below. Urban SFAs were ranch less Mkefy to impfcmeat NSMP than either suburban or rural SFAs. Only two of nine urban SFAs achieved full implementation, and one of these SFAs used a modified approach to weighted analysis. In contrast, eight of 16 suburban SFAs achieved full implementation (four SFAs used a modified approach), as did six of nine rural sites (one SFA used a modified approach). ?r Exhibit 3.10 Characteristics of Demoastratioa SFA* by Ffaul ImpknieaULoa Statin Coaaaaity Type Urban 1 Suburban Rural Enrollment Under 2,500 2^00-9,999 10,000-24,999 25,000 or more Perceat of Laacaes Served Free 25 percent or less 26-50 percent 51 percent or more Characteristics of Food Service Program Prior to NSMP Some centralized food production1 Yes No Fully centralized menu planning Yes No Cycle mcnu(s) used Yes No NatrMaa Experience of Staff Registered dietitian or nutritionist on staff Yes No 7 9 1 16 3 9 4 6 4 9 2 1 t 11 5 10 4 10 9 14 1 17 10 17 11 21 7 13 5 10 13 24 13 5 23 II AM Inc. of KSMP in Demonstration SFAs W 3-19 Exhibit 3.10 (c Natritioa Experience of Staff (continued) SFA director or NSMP coordinator is registered dietitian Yes No Prior with Computerized Nutrient District menus previously analyzed by computer Yes No SFA director or NSMP coordinator had prior experience with computerized nutrient analysis Other Factors that May hai SFA director personally managed NSMP iMaaaaaaaaal Yes Chaaajs in SFA directors during demonstration Yes No 10 I 7 11 10 t 11 7 4 14 18 16 13 21 17 17 24 10 7 27 1 50% or n.ore of schools in district ire served by base or central kitchens. Note: Some data may differ from application forms or from previous reports (Fox. MK. 1997) because information for SFAs was clarified/updated in subsequent interviews Source: Application forms for NSMP demoiisuanon (Fall 1993) and Spring 1997 implementation status. AM Inc. of NSMP in SFAa Jt> Although most of the urban SFAs were also either large or very large, exploratory analyses suggest that being urban had an influence on implementation that was independent of size. Potential reasons that urban SFAs may have more trouble with NSMP implementation than their suburban or rural counterparts include financial constraints (several urban SFA directors were in the midst of or facing a hiring freeze) and a more transient or less reliable work force (several urban SFA directors mentioned long-standing vacant positions and frequent staff turnover). It is also possible that urban SFAs are dealing with food preparation facilities that are aging or poorly equipped. In short, urban SFAs may be less able to devote the time and energy required for NSMP implementation because of numerous competing priorities. Very large SFAs and, to a lesser extent, small SFAs were less Hkery to implement NSMP than medium or large SFAs. Only one of 11 very large SFAs (enrollment of 25,000 or more) implemented NSMP without modifications. An additional two very large SFAs implemented NSMP using a modified approach to weighted nutrient analysis. Most very large SFAs that did not implement NSMP (7 of 8 SFAs) withdrew from the demonstration. As discussed earlier in this chapter, the primary reason for termination was concern about the use of weighted analysis, specifically the level of effort involved in collecting and organizing district-wide menu production data. The burden associated with complying with this requirement clearly escalates as district size increases, unless a district is fully computerized or has highly centralized food production. In fact, the data suggest that SFAs with enrollments between 10,000 and 25.000 may also have trouble implementing the weighted analysis requirement Only three of eight SFAs in this group were able to fully implement NSMP without use of a modified approach to weighted analysis. At the opposite end of the spectrum, small SFAs (less than 2.500 students) also appeared to have some difficulty with implementation. Four of the six small SFAs selected for the demonstration never implemented NSMP. Three of these SFAs dropped out. Feedback from SFA directors indicates that while the issue of weighted analysis played a role in the decision to drop out, a larger issue was the fact that these directors were personally responsible for multiple aspects of day-to-day operations. Those who dropped out found the amount of work involved in NSMP implementation, particularly within the constrained time frame in which they were ultimately asked to work, to be more than they could handle. SFAs that had aoaae ceatratatd food production prior to NSMP were aaore andy to achtrre tnl napli mi nUHna, wanont modaVanon, than SFAs that had nttte or no centrahaed food production. Seven of the 17 SFAs that had a moderate degree of centralized food production prior to NSMP, defined in this analysis as 50 percent or more of schools serviced by a base or central kitchen, achieved full implementation without using a modified approach to weighted analysis. The same was true for only three of the 17 SFAs that had less centralized food production. i SFA* Ml 61 The presence of centralized food production may benefit NSMP implementation by reducing the amount of work required to document district-wide food usage. In addition, SFAs with some centralized food production may face fewer problems in identifying standardized recipes and ensuring that those recipes are used in all food production locations. SFAs that had fully centralized menu planning prior to NSMP were more likely to achieve foil implementation, without modification, than SFAs that had fully or partially decentralized menu planning. Most SFAs (21 of 34) had fully centralized menu planning systems in place prior to the demonstration. Among the 13 SFAs that entered the demonstration with some degree of decentralized menu planning, only two achieved full implementation without using a modified approach to weighted analysis. To implement NSMP in situations where individual schools have limited or total autonomy in menu planning, SFAs must either reduce the level of local control—i.e., move to a fully centralized system—or analyze menus for each individual school. The latter approach significantly increases the time and labor required to implement NSMP. fete. fmvtmmMaben o* NSMP In OwiHXtliaOon SFA« 3-22 SI Chapter 4 Nutrient Content of Lunches Before and After NSMP This chapter presents information on the nutrient content of lunches served in demonstration SFAs before and after implementation of NSMP. The following research questions are addressed: • How do lunches served after implementation of NSMP compare to established NSMP standards and NRC recommendations? • Does the nutrient content of lunches served after implementation of NSMP differ significantly from lunches served prior to the use of NSMP? • Does the nutrient profile of NSMP lunches differ significantly if the nutrient analysis is unweighted? Do conclusions about the impact of NSMP differ if the analysis is unweighted? The chapter is divided into four sections. The first section describes the analysis that was conducted and how the data are presented in this report. The second section provides a context for the analysis of nutrient content by summarizing differences in the types of foods included in lunch menus before and after NSMP. The third section describes and compares the nutrient content of lunches served before and after the implementation of NSMP. Results are based on weighted analyses, the analytic approach that is the foundation of the current NSMP system (The difference between weighted and unweighted analyses is described later in this chapter.) The fourth and final section of the chapter compares results of weighted and unweighted analyses of Spring 1997 lunch menus. The objective of this comparison is to determine whether the two analytic techniques lead to similar conclusions about the extent to which NSMP lunches satisfy nutrient standards and recommendations. Overview of the Analysis The analysis includes 19 SFAs that implemented NSMP, either district-wide (16 SFAs) or for elementary school lunches only (3 SFAs), prior to Spring 1997. Data are presented for elementary school lunches and secondary school lunches. In cases where SFAs planned two elementary school menus (n=6) or separate middle and high school menus (n=10), data were combined in a single, appropriately weighted, analysis. All analyses were completed using NUTRIKIDS software.1 A detailed description of the methodology' used in conducting all nutrient analyses is provided in Appendix D. Appendix A includes supplementary exhibits that summarize data for SFAs that implemented NSMP by Spring 19%, referred to as "early-implementing SFAs," as well as data for middle school lunches and high school lunches for the ten SFAs that analyzed separate menus. LunchByte Systems. Inc Version 7.0.1997 Abt Assocl ites Inc. Nutrient Content of Lunches Before and After NSMP 4-1 /J Reference RDA Standards The RDAs specify allowances for four different groups of school-age children (4-6 year olds, 7-10 year olds, 11-14 year olds, and 15-18 year olds), as well as for 1-3 year olds, the oldest of whom may attend school-district-operated preschools. For purposes of the evaluation, RDA standards were assigned to each SFA based on the grade spans included in the districts' schools. In some cases, a defined RDA standard was used, for example, RDAs for 7-10 year olds for an elementary school that included grades two through five, or RDAs for 11-14 year olds for a middle school that included grades six through eight. When the grade span for a particular type of school encompassed more than one established RDA age group, for example, an elementary school with kindergarten through grade six (ages 5-11), the NSMP software was used to develop a customized RDA standard (see Appendix D). Abt Associates' coding staff ensured consistency in RDA standards across SFAs for schools with comparable grade spans as well as a consistent standard for Spring 1994 and Spring 1997 analyses. In addition to an analysis tailored to reflect the actual makeup of SFA schools, a separate analysis was conducted to compare lunches served before and after NSMP to the minimum nutrient (RDA) standards defined in current program regulations. Results of this analysis are summarized in Appendix A. Weighted and Unweighted Analy-ta The principal analysis presented in this chapter describes the average nutrient content of lunches served in participating SFAs before and after NSMP, based on a weighted analysis of lunch menus. Weighted analysis is the standard defined for the NSMP demonstration and in current program regulations. A weighted analysis takes into account the number and types of foods actually taken by students, giving greater weight to foods selected more frequently. As such, results of a weighted nutrient analysis provide a picture of the average lunch served to students. An unweighted analysis, in contrast, does not incorporate information on student selection patterns and constitutes a simple average of all foods made available to students. An unweighted analysis includes an average serving of every type of food offered (i.e., milk, entree, fruits and vegetables, and desserts), and thereby reflects the average lunch offered to students. To assess similarities and differences between the two analytic approaches, lunch menus were analyzed both ways. Results for Spring 1997 menus are discussed in this chapter. Results for Spring 1994 menus and a comparison of pre-NSMP and post-NSMP menus using an unweighted analysis are summarized in Appendix A. Frequency of Foods Offered in Lunch Menus Because nutrient content is driven by the mix of foods included in a menu, it is helpful to understand changes SFAs' may have made over the course of the demonstration in the mix of foods offered. Foods included in lunch menus offered before and after NSMP were assigned one of 52 different food/food group codes. Data were tabulated to produce frequency distributions showing the percentage of daily lunch menus in which each food/food group was included. For example, a frequency of 67 percent for pizza indicated that 67 percent of the daily lunch menus reported by demonstration SFAs included pizza. Atrt AaaoclatM Inc. Nutrient Content of LuncheeMMand After NSMP 4-2 J-f Data for pre-NSMP and NSMP menus were compared, and foods or food groups that increased or decreased over time by fiv; or more percentage points were identified. Results are summarized in Exhibit 4.1. Foods listed in the Offered More Often section of the exhibit appeared more often in NSMP menus than in menus planned before NSMP. Likewise, foods listed in the Offered Less Often section of the exhibit appeared less often in NSMP menus than in pre-NSMP menus. Changes noted in planned menus were generally consistent with recommendations made during NSMP training and with changes reported by SFA directors. Elementary school lunch menus planned after NSMP included: • more skim milk and flavored lowfat milk and less whole milk; • more unbreaded meat, poultry, and fish, and fewer breaded versions of these items; • more raw vegetables (other than salad); and • more "extra" bread/grain choices. The number of changes noted in middle and/or high school menus was greater than elementary school menus, but the pattern of change was similar. Compared to pre-NSMP lunch menus for middle and/or high schools, NSMP lunch menus included: • more skim milk and less whole milk; • more fresh fruit; • more raw vegetables and salad and fewer french fries and other types of potato products; • more pasta-based entrees, salad bars, and Chefs salads, and fewer entrees that tend to be high in fat, including breaded meat, poultry, and fish, burgers, nachos, hot dogs, and specialty food bars (e.g., taco ban, potato bars); • more rice; • more desserts; and • fewer snack chips. In addition, NSMP menus for all types of schools included fewer cooked vegetables (other than potatoes); elementary and middle school menus included less flavored skim milk; and middle and high school menus included less fruit juice. These findings are somewhat counter-intuitive but explainable. Comments from SFA directors indicate that use of cooked vegetables was decreased and counterbalanced by an increase in raw vegetables and both side-dish and entree salads. According to SFA staff, student acceptance of raw vegetables and salads is substantially higher than cooked vegetables (other than french fries and other types of potatoes). The decrease in the frequency of flavored skim milk can be traced to three SFAs that were using this type of milk prior to NSMP but discontinued it because of poor student acceptance. Finally, the change in the frequency of juice is attributable to changes made in four SFAs 'hat offered juice every day or almost every day prior to NSMP. After NSMP, menu planners in three of these SFAs offered fresh fruit rather than juice. In the fourth SFA, the frequency of juice dropped because of the elimination of a sack lunch, previously offered every day, in which the beverage was juice. AM Associate* Inc. Nutrient Content of Lunches Before and After NSMP 4-3 ft Exhibit 4.1 Foods Offered More and Less Often in NSMP Lunch Menus Compared to Pre-NSMP Loach Menus Offered More Often' Skim milk / / / Flavored lowfat milk / Fresh fruit / / Raw vegetables (other than salad) / / Salad / Unbreaded meat, poultry, or fish / Pasta-based entrees • Salad bar (entree) / Chefs salad / Rice / Extra bread/grain3 / Desserts / / Offered Less Often2 Whole milk / / / Flavored skim milk / / Fruit juice / / crench fries / / Potatoes other than french fries / • Cooked vegetables other than potatoes / / / Breaded meat, poultry or fish / / Hamburgers/cheeseburgers / / Nachos / Hot dogs / Specialty food bars / Chips / 1 Foods were offered in mor# daily menus after NSMP than before NSMP. Difference of more than S percentage points 2 Foods were offered in fewer dairy menus after NSMP than before NSMP. Difference of more than 5 percentage point.. 1 Bread or grain item available to all students, regardless of entree selected Nutrient Content of LunchM Before and After NSMP # Nutrient Profile of Lunches Served Before and After NSMP This section presents data on the average nutrient content of lunches served before and after NSMP, relative to NSMP standards and NRC recommendations. All results are based on weighted analyses. The section includes three separate discussions as outlined below. • Nutrient Content Relative to RDAs. Mean contribution to RDAs for calories, protein, vitamin A, vitamin C, calcium, and iron is evaluated in light of the NSMP standard for lunch (33% of the RDA). • Percentage of Calories from Total Fat, Saturated Fat, Carbohydrate, and Protein. The mean percentage of calories provided by each macronutrient is compared to NSMP standards (total fat and saturated fat) or NRC recommendations (carbohydrate and protein). • Cholesterol, Sodium, and Fiber Content Mean cholesterol and sodium content are compared to NRC recommendations. Information on mean fiber content is presented but not compared to a standard because no scientific consensus exists about a recommended daily intake of fiber. Discussions are organized around two research questions identified at the beginning of the chapter (1) the extent to which lunches served after implementation of NSMP (NSMP lunches) met defined nutrient standards and NRC recommendations, and (2) significant differences in the nutrient content of lunches offered before and after implementation of NSMP. Comments about non-significant differences between pre-NSMP and NSMP lunches are limited to situations where conclusions about meeting NSMP standards or NRC recommendations differ, (e.g., when the mean for NSMP lunches satisfies an NSMP standard or NRC recommendation but the mean for pre-NSMP lunches does not, or vice versa). Unless otherwise noted, findings for the supplementary analyses summarized in Appendix A (early implementers; middle school lunches and high school lunches; and minimum nutrient standards defined in the regulations) a*e comparable to those reported here. Mean Nutrient Content Relative to RDAs In general, lunches served both before and after imp' ^mentation of NSMP met NSMP standards for calories and all key RDA nutrients. Indeed, as shown in Exhibit 4.2, lunches served at both points in time provided more than one-third of the RDA, with the exception of calories in secondary school lunches.2 Elementary school lunches served after NSMP provided significantly fewer calories and less protein than lunches served prior to NSMP. These differences have little substantive importance, however, because lunches served at both points in time exceeded the NSMP standard of one-third of the RDA. Secondary school lunches served after NSMP were somewhat lower in calories than lunches served before NSMP. Although this difference was not statistically significant, the mean for NSMP lunches fell 2 DMI on actual mnacacrtyi^mitrwnicaaicrt of hiKJMKrvedafcpfCKMedmExMM A 1 A more detailed summary of the ( !«Exla*att42ll»out^4 5upfovtdedinExhibttsA2-A4 Abt Associates Inc. Nutrient Content of Lunches Before and After NSMP ^7 Exhibit 4.2 With the Exception of Calorie* in Secondary Schools. Lunches Served Before and After NSMP Provided More than On. -third of the RDA Elementary School Lunches 110% 100% Calories Protein Vitamin A Vitamin C Calcium Ion Secondary School Lunches 100% Calories Protein Vitamin A Vitamin C Calcium Iron Before NSMP After NSMP 'Difference is statistically significant at the 5 percent level. •Difference is statistically significant at the 1 percent level. AM Associate* Inc. Nutrient Content of Lunches Before and After NSMP 4-6 JT just below the one-third RDA standard. (Pre-NSMP lunches provided, on average, exactly one-third of the RDA.) The only nutrient which differed significantly for secondary school lunches was vitamin A. Lunches served after NSMP provided significantly more vitamin A than lunches served before NSMP. Again, however, the relative importance of this difference is minimal in terms of this evaluation because the vitamin A content of lunches served at both points in time more than satisfied the one-third RDA standard. In summary, the data indicate that the contribution of NSMP lunches to students' daily nutrient needs was comparable to the contribution of lunches planned before NSMP. In other words, improvements in NSMP lunches, as described in subsequent sections, were achieved without a negative impact on the relative contribution of lunches to students' daily needs for calories and nutrients. Number ofIndividual SFAs Meeting RDA Standards In addition to examining mean nutrient content across all SFAs, the analysis assessed whether implementation of NSMP affected the number of individual SFAs that met NSMP standards for calories and RDA nutrients. With the exception of calories in secondary schools, a majority of SFAs provided one-third of the RDA for each target nutrient both before and after NSMP (Exhibit A.3). Calories in secondary school lunches were problematic both before and after implementation of NSMP. As noted in the preceding section, the pre-NSMP mean, across all SFAs, just met the one-third RDA standard. On an individual basis, only eight of the 16 SFAs met the standard before NSMP. After NSMP, the number of SFAs meeting the standard dropped to six and thr mean calorie level was slightly lower, falling just below tlte one-third benchmark. This pattern is consistent with SFA directors' reports that some SFAs found it difficult to meet the standard for calories. Percentage of Calories from Total Fat, Saturated Fat, Carbohydrate, and Protein On average, neither NSMP lunches nor lunches served prior to NSMP met the NSMP standards for percent of calories from total fat and saturated fat or the NRC recommendations for percent of calories from carbohydrate and protein (Exhibit 4.3). Failure to meet NSMP standards for calories from fat and saturated fat is consistent with SFA director reports that these standards were the most difficult to meet at lunch, particularly for secondary schools. Nonetheless, the nutrient profile of NSMP lunches was significantly better than pre-NSMP lunches. Specifically, NSMP lunches derivedfewer caloriesfrom fat and saturatedfat and more calories from carbohydrate than lunches served before NSMP. This is true for both elementary and secondary school lunches and, in all cases, differences between pre-NSMP and NSMP lunches were statistically significant. The mean percentage of calories from fat in elementary school NSMP lunches was five percentage points lower than the mean for pre-NSMP lunches. This is a substantial difference, representing an overall decrease in calories from fat of 19 percent.3 As a result, elementary school NSMP lunches came close to meeting the NSMP standard of no more than 30 percent of calories from fat. For secondary school 3 SMEINMA.1 Ota!* Iran fci.pw of hi* 9 oOonei Abt Associate* Inc. Nutrtent Content of Lunch** B*for* and After NSMP 4-7 4\I Exhibit 4.3 Lunches Served After NSMP Provided Fewer Calories from Fat and Saturated Fat than Lunches Served Before NSMP Elementary School Lunches Total Fat Saturated Fat Carbohydrate Protein Secondary School Lunches 70% 60% ^50% I "§ 40% Zo 30% | 20% £ 10% 0% 52%* NSMP Standard orNRC Recommendation Before NSMP After NSMP 'Difterence is statistically significant «; the 5 percent level. "Difference is statistically significant at the 1 percent level. Nutrient Content of Lunch*. Bofor* and After NSMP 44 ± lunches, the mean percentage of calories from fat in NSMP lunches was three percentage points lower than pre-NSMP lunches, equivalent to a 12 percent decrease in actual fat calories. In addition, the percentage of calories from saturated fat in elementary school lunches dropped three percentage points, representing an overall decrease in calories from saturated fat of 25 percent.4 The equivalent for secondary sdnol lunches was a decline of one percentage point and a 16 percent drop in calories from saturated faL Finally, NSMP lunches provided significantly more calories from carbohydrate than pre-NSMP lunches and, particularly in the case of elementary school lunches, came close to meeting the NRC recommendation of rrore than S3 percent of calories from carbohydrate. Thus, although, on average, NSMP lunches continued to exceed the standards for calories from fat and saturated fat and to fall short of the recommendation for calories from carbohydrate, the data suggest that implementation of NSMP had a significant impact in moving school lunches closer to these goals. Overall, the magnitude of the change was larger for elementary school lunches than for secondary school lunches. Number of Individual SFAs Meeting Standardsfor Fat and Saturated Fat Before NSMP, only one of the 19 SFAs included in this analysis met the NSMP standard of no more than 30 percent of calories from fat in elementary school lunches (Exhibit 4.4). Further, elementary school lunches served in almost three-quarters of SFAs (14 of 19) exceeded this standard by a substantial margin, providing more than 35 percent of calories from fat. After NSMP, the number of SFAs meeting the standard for percent of calories from fat increased to more than one-third (7 of 19), a statistically significant difference, and only three SFAs provided more than 35 percent of calories from fat. The number of SFAs meeting the NSMP standard for percent of calories from saturated fat in elementary school lunches also shifted dramatically over the course of the demonstration (Exhibit 4.4). Before NSMP, none of the 19 SFAs met die NSMP standard of less than 10 percent of calories from saturated faL After NSMP, about one-quarter of the SFAs (5 of 19) met the standard (a statistically significant difference). Similar shifts were noted in the number of SFAs meeting fat and saturated fat standards in secondary school lunches, however, neither change was statistically significant. Finally, the number of SFAs meeting the NRC recommendation for percent of calories from carbohydrate was significantly higher, for both elementary school lunches and secondary school lunches, a |