Evaluation of grants to states for the reduction of payment error in the food stamp program final report

              ' I Oolt-A-ot ki?.2-e*i/F"j*L
Corporation
Evaluation of Grants to States for the Reduction of
Payment Error in the Food Stamp Program
Final Report
Submitted to:
Food and Consumer Service
Department of Agriculture
Contract Number 53-3198-3-C 21
September 20,1996
A Diversified Small Business Professional and
Technical Services Organization
1010 Wayne Avenue • Suite 850 • Silver Spring, Maryland 20910
(301) 495-1591 • Fax (301) 495-2919
E-Mail: infosys@kra.com • Internet http://www.kra.com
<\\',&0*<>
q4\
£-
Evaluation of Grants to States for the Reduction of
Payment Error in the Food Stamp Program
Final Report
Submitted to:
Food and Consumer Service
U.S. Department of Agriculture
Contract Number 53-3198-3-021
Authors:
Cynthia Lynn Holmes
Mary Kay Sistik
Robert F. Cook
KRA
Corporation
1010 Wayne Aveswe
Suite 850 September 20,199*
Silver Spring, Maryland 20910
This document was prepared under Contract Number 53-3198-3-021 between the U.S. Department of Agriculture and KRA Corporation of Silver
Spring, Maryland. Trie contents of this document do not necessarily reflect the ews or policies ofthe Department, nor does mention of trade
names, commercial products, or organizations imply endorsement by the U.S. Government.
a-1
Table of Contents
Page Number
Executive Summary Executive Summary-i
Chapter 1. Background and Introduction 1
Background 1
Grants to States for the Reduction of
Payment Error 2
Organization of the Report 3
Chapter 2. Study Objectives and Methodology 4
Study Objectives and Research Questions 4
Study Design 5
Study Limitations 6
Chapter 3. Description of Maryland's Food Stamp Error
Reduction Demonstration 8
Description of the Error Reduction Demonstration 8
Implementation of the Payment Accuracy Program 9
Operation of Demonstration 10
Results 12
Impact of the Demonstration 19
Chapter 4. Description of Illinois' Food Stamp Error
Reduction Demonstration 22
Description ofthe Error Reduction Demonstration 22
Implementation of the Illinois Food Stamp
Error Reduction Demonstration 24
Operation of Demonstration 24
Results 27
Impact of the Demonstration 32
Appendix
Appendix A: Food Stamp Program Data by Fiscal Year
Appendix B: Statistical Approach to Integrated Quality Control Sample Error Rate Data
Table of Contents-i
Tables
Table 1
Table 2
Table 3
Table 4
Table 5
Table 6
Table 7
Average Number of Cases Investigated Each Month
During the Payment Accuracy Program 11
Maryland—Case Overissuance Error Rates 16
Maryland—Dollar Overpayment Error Rates 17
Maryland—Estimated Annual Dollar Savings as a Result
of the Demonstration 18
Illinois—Case Overissuance Error Rates 30
Illinois—Dollar Overpayment Error Rates 31
Illinois—Estimated Annual Dollar Savings as a Result
of the Demonstration 32
Table of Contonts-ii
Executive Summary
BACKGROUND
The Food Stamp Program (FSP) is available to low-income individuals and families to increase their
purchasing power for food, ensuring access to a more nutritious diet. Directly administered by State
agencies, the administrative costs incurred in running the FSP are jointly funded by States and the
Federal Government. Actual FSP benefits are federally funded. At the Federal level, the FSP is
funded and administered by the Food and Consumer Service (FCS), U.S. Department ofAgriculture.
The FSP is the largest food assistance program administered by FCS, providing 22.7 billion in
benefits to 26.6 million people in Fiscal Year (FY) 1995.
Issuing FSP benefits accurately is increasingly important as more households participate in the FSP
and benefit levels increase. Although most benefits are issued accurately, overpayment of benefits
to eligible and ineligible households represented approximately 8 percent of all benefits issued
during a 10-year period.1
In 1993, FCS offered States an opportunity to apply for grant funds to demonstrate methods of
reducing overpayment errors (case and dollar) and improving payment accuracy. Maryland and
Illinois were awarded 18-month grants under this program. Maryland received $188,687 in grant
funds while Illinois received $184,985 in grant funds. States used these funds to hire new staff,
purchase computers, and pay for any travel required to support error reduction activities.
STUDY OBJECTIVES AND METHODOLOGY
FCS contracted with KRA Corporation (KRA) to document and evaluate the error rate reduction
interventions undertaken in the two demonstrations. This evaluation was designed to meet the
following study objectives:
• Determine which error rate reduction initiatives were effective in reducing
overpayment error rates
• Determine whether these error rate initiatives were cost-effective
• Determine whether the demonstration initiatives are easily transferable to other States
In order to address the study's research questions, KRA staff performed a process evaluation, an
analysis of error rate data, and a synthesis of the demonstration reports as part of the evaluation.
1 Food and Consumer Service Grant Number 59-3198-3-003, Purpose and Goals of Grants.
KRA Corporation —-7 Executive Summary-i
MARYLAND'S ERROR REDUCTION DEMONSTRATION
Description of the Demonstration
Maryland's Payment Accuracy Program (PAP) began operation in October 1993 and operated until
February 1995. PAP focused on payment errors in nonpublic assistance food stamp cases resulting
from either applicants providing inaccurate information at application or recipients failing to report
changes in resources subsequent to certification and prior to recertification. The PAP project
combined front-end investigations2 of application information with followup or recall activities
(updating and verifying information) before recertification. The project was designed to operate as
a demonstration project; it was implemented and operated as such. The overall project design,
however, changed from that originally proposed in the grant application.
Although the Income Maintenance Administration3 within Maryland's Department of Human
Resources originally conceived and developed the error reduction demonstration grant, a
departmental reorganization placed the PAP in a newly developed division, the Office of the
Inspector General (OIG). The OIG had no direct authority over the FSP or local district offices and
OIG staff have different skills than FSP staff. OIG hired an investigative supervisor and four
investigators without FSP experience to administer and operate the PAP. The departmental
reorganization delayed implementation of the error reduction demonstration.
The demonstration operated in three local district offices located in Baltimore City—Govans-
Collington, Clifton, and Liberty-Garrison. Office procedures and processes differed somewhat
across offices.
PAP staff investigated a total of 1,330 cases prior to applicants being certified for FSP benefits, and
investigated 893 cases as part of the recall process during the demonstration period. Over the course
of the demonstration, each investigator handled an average of 33 cases per month. The number of
cases investigated each month varied greatly and the process used for referrals to PAP investigators
differed across offices. This variance in cases investigated each month occurred because district
office staff believed referral of cases was optional. State staff viewed PAP as a mandatory program.
Impact of the Demonstration
Based on our analysis using Integrated Quality Control System (IQCS) error rate data, Maryland's
demonstration was effective. Maryland showed statistically significant decreases in the case and
dollar error rates. The three demonstration offices, when compared to other areas in Maryland,
experienced a larger decrease in case overissuance and dollar overpayment errors during the course
of the demonstration (for the data available). For example, case error rates at Govans-Collington,
Liberty-Garrison, and the weighted average of all demonstrations' case error rates were significantly
different from the error rates for the balance of the City of Baltimore (minus the three demonstration
2 Investigations occur prior to certifying households eligible for FSP benefits. The investigative process
includes verifying the information provided at application, such as making collateral contacts, verifying residency,
and the like.
3 The Income Maintenance Administration administers the FSP.
KRA Corporation . Executive Summary-ii
)/
offices) at the 1 percent level of significance. In addition, case error rates at Govans-Collington,
Liberty-Garrison, and the weighted average of all demonstrations were significantly different from
those for the balance of the State of Maryland (less the demonstration offices, City of Baltimore, and
Baltimore County) at the 1 percent level of significance.
We found a statistically significant change in the demonstration offices' overpayment dollar error
rates when compared to other areas within Maryland. That is, for Govans-Collington, Liberty-
Garrison, and the weighted average of all demonstrations' dollar error rates, changes in overpayment
dollar error rates were significantly different from the balance of the City of Baltimore at the 1
percent level of significance. In addition, all demonstration offices individually and the weighted
average of the three demonstrations were significantly different from overpayment dollar error rates
for the balance of the State of Maryland at the 1 percent level of significance.
Ofthe FSP applications reviewed by PAP investigators, 28 percent were denied benefits. For recall
investigations, approximately 19 percent were denied recertification as a result of adverse
information obtained through the PAP investigative process. However, we view these findings with
caution as we could not fully ascertain the role PAP staff had in these reported denials.
We conducted a limited cost analysis for the Maryland demonstration. The estimated annual benefit
savings for the three demonstration local district offices measured against the error rate change in
the State is 3.8 times the estimated 18-month total cost of the demonstration and 5.7 times the 12-
month cost. When we estimated the annual dollar savings, we determined that a 1 percent reduction
in the overpayment error rate for the weighted average of the demonstrations produced
approximately a $100,000 annual benefit savings.
ILLINOIS' ERROR REDUCTION DEMONSTRATION
Description of the Demonstration
The Illinois Food Stamp Error Rate Reduction Demonstration began in April 1993 and operated until
October 1994. Two staffexperienced in food stamp policy and procedures were hired to review FSP
cases and the processes related to determining eligibility for the FSP. Under the supervision of a
steering committee, the Error Reduction Analysts (ERAs) had total responsibility for the design and
implementation of procedures used to collect information regarding the type of errors being made
and why these errors occurred. Specifically, they focused their activities on earned income and
household composition errors. Three contiguous offices in Chicago were selected for the project,
two with high error rates in earned income and household composition (Lower North and Humboldt
Park), and a third with considerably lower error rates (Northwest).
The Illinois Department of Public Aid (DPA), the agency that administers the FSP, operated the
demonstration. A steering committee, composed of five individuals from different divisions within
DPA, provided overall management for the project. The State Food Stamp Director served as the
demonstration project director and chaired the steering committee. Two administrative staff
members from the DPA's Chicago zone office, who had immediate and direct oversight as well as
line authority over the targeted local offices, were also recruited for the project. They provided
direct supervision and guidance to the ERAs, and provided the steering committee members with
feedback regarding the progress of the project.
KRA Corporation / Executive Summary-iii V
The ERAs reviewed case records, observed interviews, interviewed caseworkers, and made
recommendations regarding problem areas and solutions to those problems. The demonstration
consisted oftwo phases with a period for analysis between phases. The first phase was introductory
and exploratory in nature. It consisted of intake interview observations and case record reviews.
Demonstration staff were in each of the 3 local offices for 6 weeks during this phase. The second
phase operated for 2 months in each office and primarily involved recertification observations and
case record reviews. The demonstration staff also held entrance and exit conferences with local
office staff, solicited local office staff input, and developed and administered a supervisory quiz as
part of their activities.
Impact of the Demonstration
The Illinois project operated more like a program improvement effort than a demonstration project.
Changes to improve the FSP were made statewide during the course of the demonstration and thus
changes that may have occurred as a result of the demonstration could not be isolated to the
demonstration offices.
Several changes were made to FSP policies and procedures as a result of the demonstration. These
changes included developing special "Quality Tips" memos that focused on specific topics
associated with errors and statewide interview training for intake staff. In addition, revisions to the
household composition module of the Automated Intake System (AIS) that were already in progress
were accelerated and completed during the demonstration.
Other changes that have occurred (or are under way) as a result of the demonstration include
updating the computerized client database, developing an FSP conference/training for senior local
office staff, providing translation techniques training for bilingual intake staff, and analyzing
additional quality control data. In addition, changes have been made (or will be made) to several
of the forms used during the FSP (re)certification process.
Illinois showed a decrease in accuracy (through their own Quality Assessment and Quality Control
analysis) and the IQCS data do not show consistent significant error rate effects at the individual
office level or compared to other areas within the State. It should be noted, however, that it would
be difficult to show any significant changes in Illinois' error rates, because the Illinois demonstration
did not limit the effects of the demonstration to the demonstration offices. In addition, the State
error rate (case and dollar) decreased during the demonstration period.
Of the demonstration sites, the only decrease in case error rates occurred in Humboldt Park. This
change was significant at the S percent level of significance relative to the balance of the State
(minus Cook County and the demonstration offices). For dollar error rates, only the change between
the Lower North demonstration office and the balance of Cook County (not including the three
demonstration offices) was significant and negative at the 1 percent level of significance. We
concluded that the demonstrations produced no demonstrable difference in the dollar overpayment
rates between the demonstration areas and either of the comparison areas. In fact, there was a
relative rise in error rates in the demonstration offices over the period relative to the comparison
areas. Because the changes in error rates relative to the comparison areas were positive, the Illinois
demonstration areas did not show any savings that could be measured against cost.
KRA Corporation VI Executive Summary-iv
FUTURE IMPACT OF THE DEMONSTRATIONS
Staff from both States plan to share the results of their demonstration for the purposes of replication.
Demonstration staff believe other States will find their approach useful in decreasing case and
payment error rates. Maryland staff modified PAP and expanded it to other designated areas of the
State.
Within the demonstration States, some changes may continue to occur as result ofthe demonstration.
For example, in Illinois, State staff recognized that communication between the local offices and the
State administrative offices improved during the course of the demonstration. Staff plan to keep this
communication open so that State staff can continue to learn about what tools are needed to
effectively implement and improve FSP policies and procedures. Illinois staff also plan to continue
to make improvements to policies and procedures based on the information learned from the
demonstration.
KRA Corporation
*'l
Executive Summary-v
Chapter 1
Background and Introduction
BACKGROUND
The Food Stamp Program
The Food Stamp Program (FSP) is available to low-income individuals and families to increase their
purchasing power for food, ensuring access to a more nutritious diet. The FSP is the largest food
assistance program administered by the U.S. Department of Agriculture's Food and Consumer
Service (FCS)1, providing $22.7 billion in benefits to 26.6 million people in Fiscal Year (FY) 1995.
The FSP is directly administered by State agencies. Benefits are federally funded; however,
administrative costs incurred in running the FSP are jointly funded by States and the Federal
Government.
Food stamps are available to all persons who meet specified income and resource standards.
Benefits are determined on a household basis; households are determined as eligible based on gross
and net income limits, assets, and other nonfinancial eligibility standards, such as student or alien
status, and work registration requirements.
Household monthly food stamp benefits are computed based on the household's net countable
monthly income, the benefit reduction rate, and the maximum food stamp benefit for household size
and location. The maximum benefit to which a household is entitled is based on the June cost ofthe
Thrifty Fo Plan (TFP) for a family offour, adjusted for different household sizes and geographic
areas outside the contiguous United States. The cost of the TFP is based on an economical and
nutritious diet, adjusted for household size and composition. The maximum allotment in any given
FY (from 1991 until the present) is based on 103 percent of the cost of the TFP for the previous
June. Only households with zero net income receive the maximum allotment.
The maximum benefit increased from 100 percent of the Congressional ly specified cost ofthe TFP
in FY 1988, to 103 percent in FY 1991. This increase in the maximum benefit, coupled with the
caseload increase, resulted in an increase in FSP benefits.
Several activities are completed as part of the FSP application process. These activities include
obtaining and completing an application form, completing a certification interview, and providing
verification of information. Once these activities are completed, FSP eligibility workers in local
offices determine whether the household is eligible to receive food stamps; the eligibility worker also
determines the amount ofmonthly benefits each household receives.
Prior to 1995, the agency was known as the Food and Nutrition Service.
KRA Corporation
Trends in the Food Stamp Program
FSP participation fell slowly from FY 1983 to 1989, then rose sharply and peaked at 27.5 million
persons in FY 1994, representing approximately 11.1 million households. By FY 1995, participation
dropped slightly below the FY 1993 level. The increase in participation is attributed to the economic
recession that began in 1990. Benefit levels remained relatively constant until FY 1990. From FY
1989 to 1993 benefit levels increased over 10 million dollars. Participant and benefit data, by year,
are presented in Appendix A.
FSP payment error rates errors fell slowly during the 1981-1991 period, then spiked up in 1992,
stayed high in 1993, fell somewhat in 1994, and continued falling in 1995.2 The overpayment error
rate trends lor FY 1981 through 1994 mirror the overall payment error rates, with a decrease over
the 10 year period (1981 -1991) and an increase in 1992 and again in 1993. Overpayment error rates
fell in 1994 and fell again in 1995. Payment error rates, by year, are presented in Appendix A.
Issuing FSP benefits accurately is increasingly important as more households participate in the FSP
and benefit levels increase. Although most benefits are issued accurately, overissuance of benefits
to eligible and ineligible households represented approximately 8 percent of all benefits issued
during a 10-year period.3
GRANTS TO STATES FOR THE REDUCTION OF PAYMENT ERROR
Purpose of the Grants to States
Over the years, FCS has funded special projects that enable States to test new approaches that allow
them to issue benefits accurately and thereby reduce State error rates and resulting dollar losses. For
example, a two-phase study, using process analysis as a means to reduce errors in the FSP, was
conducted in 1991 in several States. The project identified effective error reduction management
initiatives, determined whether these initiatives could be transferred to other States, and quantified,
through development ofa model, successful management initiatives that improved the accuracy of
State FSP case processing. Currently, FCS funds a payment accuracy initiative that seeks to impact
the national error rate. Activities funded under this initiative include funding a National Payment
Accuracy Coordinator at the Federal level, publishing a national payment accuracy report and
catalogue of successful State strategies, and sponsoring State Exchange Program activities and a
national conference.
In 1993, FCS offered States an opportunity to apply for grant funds to demonstrate methods of
reducing payment errors. The primary objective ofthe grant program was to determine successful
methods that improved payment accuracy and subsequently reduced dollar losses.
2 Official Error Rates, U.S. Department of Agriculture, November 1995 and July 1996.
3 Food and Consumer Service Grant Number 59-3198-3-003, Purpose and Goals of Grants.
KRA Corporation
Grantees Funded Through the Demonstration
In the spring of 1993, FCS awarded grants to two States, Maryland and Illinois, to reduce FSP
payment errors. Grants were for an 18-month period. Maryland received $188,687 in grant funds
while Illinois received a grant for $184,985. Grant awards were used to fund new staff, purchase
computers, and pay for any travel required for error reduction activities.
In January 1994, Maryland represented 1.5 percent oftotal FSP households while Illinois represented
4.S percent of all FSP households nationally. Maryland had 381,953 persons receiving benefits
through the FSP, representing 162,058 households (January 1994). Illinois had 1,195,209 persons
on food stamps during this month. This represented 501,003 households.
On a national basis, the FSP monthly benefit averaged $168 in 19944. In Maryland it was slightly
higher at $ 186 while Illinois' monthly benefit during this same time period was less than the national
average at $161. Nationally, FSP households averaged 2.5 persons. Illinois was somewhat less at
2.3 persons per food stamp household; Maryland's FSP household size matched the national
average. The average certification period for food stamp recipients in 1994 was 9.8 months.
Maryland's average certification period was 9.5 months while Illinois' was 11.3 months. The
average certification periods did not change from the previous year.
Grant Requirements
Grantees were required to focus their payment accuracy activities in large, urban, high-error-rate
areas. Efforts to reduce the overpayment error rate could not adversely impact other FSP operations,
and the demonstration activities were to supplement basic FSP administrative functions already
being performed by the State. FCS wanted States to develop realistic and straightforward methods
in their approach to improving payment accuracy.
As a condition offunding, FCS required States to submit quarterly progress reports and a final report
oftheir demonstrations. Grantee final reports were to include self-assessments of project outcomes
and impacts, based on quantifiable measures. States were also required to cooperate with the outside
evaluation planned by FCS.
ORGANIZATION OF THE REPORT
This report summarizes the results of the two demonstration projects and presents findings from the
analysis of Integrated Quality Control System (IQCS) data. In the following chapter (Chapter 2),
we provide an overview of the study objectives and methodology. Chapters 3 and 4 present a
description of Maryland's and Illinois' error reduction demonstration, respectively.
4 Data reported in this paragraph is based on the 1994 Food Stamp Quality Control sample, as reported in the
Characteristics ofFood Stamp Households, Summer 1994, U.S. Department of Agriculture.
KRA Corporation
Chapter 2
Study Objectives and Methodology
STUDY OBJECTIVES AND RESEARCH QUESTIONS
As part of the reduction in payment error demonstration grant program, FCS set aside funds to
conduct an overall evaluation of the demonstration efforts. FCS retained KRA Corporation (KRA)
to document and evaluate the error rate reduction interventions undertaken in the two demonstration
States.
Three study objectives and eight major research questions framed the study. The study design
required the conduct of a process evaluation, analysis of error rate data, and synthesis of
demonstration reports. The study design is described in more detail in the following sections.
Study Objectives
The evaluation of the FSP error rate reduction demonstrations was designed to meet the following
study objectives:
• Determine which error rate reduction initiatives are effective in reducing
overpayment error rates
• Determine whether these error rate initiatives are cost-effective
• Determine whether the demonstration initiatives are easily transferable to other States
Research Questions
Several research questions also provided a framework for the study. These questions included the
following:
• What were the problems experienced by the demonstration sites?
• What interventions were used?
What specific interventions were planned? How did they address the problems
experienced at the sites? How did they differ from standard procedure?
• How were the interventions implemented?
Did the sites fully implement the planned interventions?
KRA Corporation
• Did the process of operating the interventions vary between demonstration
sites?
If the implementation varied, did the interventions implemented address the original
problems and likely sources of error?
• Did the interventions affect payment and case error rates?
Did the changes implemented have an observable effect on overpayment error rates?
Were certain types of payment errors and cases affected? Did the changes affect
program operations in a positive way (e.g., program effectiveness), regardless of the
error rate effects?
• Did these impacts vary across different subgroups?
How did the impacts vary between demonstration sites? Were the payment error
rates reduced for the types of cases envisioned, or did the improved procedures result
in a reduction in error across all classes of cases?
• Did the interventions save FSP costs?
• Did the interventions allow FSP offices to maintain or improve normal
operations?
Were the interventions' impacts on client service and timeliness neutral or positive?
STUDY DESIGN
The methodology used in this study included a process evaluation, an analysis of error rate data, and
a synthesis ofthe demonstration reports. Each element is briefly described below.
• Process Evaluation—The process evaluation addressed the implementation of the
demonstrations. We examined whether the planned interventions were actually
implemented and whether they had the desired effect on the anticipated outcome
variables (case and dollar error rates). A series of site visits to the demonstration
projects, interim telephone calls, and a review ofthe demonstration grant applications
and reports informed the process evaluation for this study.
• Analysis of Error Rate Data—To determine whether the demonstrations had a
significant impact on case and dollar error rates, we analyzed historical IQCS data.
States collect these data and provide them on a monthly basis to FCS. States
randomly select a certain number of FSP cases at each local office monthly. The
local office subsequently reviews these cases for errors. Any errors found in the
sample cases determine the "error rate" for each office. States participating in the
demonstration did not supplement the official IQCS sample with additional cases for
the purposes of this evaluation. We used the IQCS data to determine whether the
KRA Corporation
demonstration had the desired effect of reducing food stamp benefit errors and the
magnitude ofthe resultant savings. The objective of this analysis was to compare the
error rate, on both a case and dollar basis, prior to the implementation of the
demonstration to that after the implementation ofthe demonstration.
• Synthesis of Demonstration Reports—A synthesis of reports produced by the
demonstrations and developed as part of the overall evaluation comprised the final
component of this study. The information collected by the KRA project team during
the site visits and through the IQCS analysis augmented the information provided by
the demonstration projects. Quarterly and final reports were the primary sources of
information from the demonstration programs.
STUDY LIMITATIONS
Demonstration Design Limitations
The intent ofthis demonstration program was to learn whether certain types of interventions could
reduce error rates, in particular, reduce overpayment dollar error rates. The design of Maryland's
demonstration was such that one could isolate the effects of the demonstration to the local offices
participating in the demonstration because comparison areas, not affected by the demonstration, were
available.
In contrast, Illinois approached the demonstration as a program improvement exercise; as staff
learned why certain errors occurred, they changed procedures and implemented training statewide.
The demonstration effects were not isolated to the three demonstration offices and comparison areas
did not exist. Any changes in error rates that may occur in the future cannot be directly attributed
to the demonstration effort because we do not know if these changes would have occurred without
the demonstration.
Data Limitations
States participating in the error reduction demonstration were required to conduct self-assessments.
The two demonstrations did not complete comprehensive self-assessments based on quantifiable
measures. The demonstrations' resulting final reports lacked descriptive information or quantifiable
results and did not compare findings to nondemonstration offices. The demonstrations' final reports,
because of their general nature, were not as useful as anticipated in conducting the final analysis for
the overall evaluation.
Use ofthe IQCS data for error rate analysis limited Lie conclusions that could be drawn about the
effectiveness of the demonstrations. For example, IQCS data is reported by office. Only three
offices in each State participated in the demonstration. IQCS sample sizes for each office were
small, particularly given the short postimplementation period available (only four to six quarters of
data). Therefore, more emphasis was put on the demonstrations taken as a whole than on individual
offices. Also, the use of IQCS data precluded an analysis of the effects of the demonstrations on
specific classes ofcases.
KRA Corporation
To some extent, the time frame for the study limits the conclusions that can be drawn. For example,
we were only able to obtain four to six quarters' worth of postimplementation IQCS data for the
error rate analysis. If additional data were available, we might be able to learn whether the
demonstrations had a lasting impact on error rates.
KRA Corporation
Chapter 3
Description of Maryland's Food Stamp
Error Reduction Demonstration
DESCRIPTION OF THE ERROR REDUCTION DEMONSTRATION
The Payment Accuracy Program (PAP) was funded in April 1993 but, because of a reorganization
in the Maryland State government and resulting staff changes, began operating in October 1993.
PAP focused on nonpublic assistance food stamp (NPA/FS) cases because approximately 40 percent
of the statewide error rate was caused by recipients receiving NPA/FS who provided inaccurate
information at application or failed to report changes in resources subsequent to certification and
prior to recertification.' The State saw the project as one that would reduce FSP errors and fraud.
The PAP project combined investigations prior to certification with follow-up or recall investigative
activities prior to recertification.
Planned Intervention2
The Maryland Department of Human Resources (DHR), Income Maintenance Administration
(IMA), proposed to hire four investigators and one investigative supervisor to perform certain tasks
geared toward improving payment accuracy and reducing the Food Stamp Program (FSP) error rate.
Focusing on NPA/FS cases exclusively, the State envisioned the demonstration as a mandatory
referral process for all NPA/FS cases in three demonstration district offices. Once each NPA/FS
case was referred, front-end investigations3 were planned. Follow-up or recall activity before
recertification was also planned. Front-end investigations and recall activities included verification
ofapplication information. In Baltimore City, NPA/FS cases represented between 33 and 35 percent
of all food stamp cases in 1994.4
Project staff were to conduct investigations ofNPA/FS cases or cases in which recipients received
no other benefits. Investigations were to occur prior to issuance of benefits with intake staff
referring all NPA/FS cases for investigation prior to certification. Investigators then were to validate
the information submitted by the applicant. This validation included conducting home visits;
contacting landlords, neighbors, and other individuals; and reviewing certain information in
databases, such as wage data. As part ofthe recall process, clients were to be contacted by telephone
or a home visit to identify discrepancies in client reporting. After contacting the recipient, the
i Maryland Department of Human Resources grant application, February 1993.
2 Information in this section is from the Maryland Department of Human Resources grant application,
February 1993.
3 Investigations occur prior to certifying households eligible for FSP benefits. The investigative process
includes verifying the information provided at application, including making collateral contacts, verifying residency,
and the like.
4 Food Stamp Program Integrated Information System, January and July 1994, Food and Consumer Service.
KRA Corporation 8
investigator was to call collaterals listed in the record. This process allowed the investigator to
reestablish eligibility on all items that could have changed. Recall was slated for 3 months after
original certification and 3 months prior to recertification.
Mandatory referrals of certain cases for investigation were new to IMA. While the agency had
previously conducted front-end investigations in an earlier effort to identify fraud, the combination
of front-end investigations with recall reviews was considered a "novel approach."
Operationally, the PAP project was not viewed as a mandatory process by eligibility workers;
therefore, NPA/FS cases were inconsistently referred to PAP staff. During the course of the
demonstration, PAP staff broadened the type of cases investigated; otherwise, the project operated
as planned.
Research design
The grant application detaileu a multi-group design with random assignment of FSP applicants to
groups. The research design used four FSP district offices for the study and within each office
established four applicant groups. Hypothesizing that the use of recall activities coupled with
investigations during the application process was more effective than investigations alone, IMA
established a design that tested each intervention alone in an office (front-end investigations or recall
activities), tested the two interventions in one office, and used one office (no interventions) as a
control office. Within each demonstration office, applicants were to be randomly assigned to one
of four groups. These groups were:
• Intervention with notification
• Intervention without notification
• No intervention (control) with notification
• No intervention (control) without notification
In other words, across the 4 offices (3 demonstration and 1 control), 13 applicant groups existed.
The design allowed IMA to isolate the most effective combination of interventions for reducing
error rates. Some notification ofapplicants occurred during the first 6 months ofthe demonstration,
however, staff dropped all other aspects of the research design early in the project.
IMPLEMENTATION OF THE PAYMENT ACCURACY PROGRAM
IMA, the State organizational entity responsible for administering the FSP, originally conceived and
developed the error reduction demonstration grant. A departmental reorganization placed the PAP
in a newly developed division, the Office of Special investigations (OSI) within the Office of the
Inspector General (OIG). The OIG had no direct authority over the FSP or local district offices and
the staff who conceived the project had no role in PAP once the reorganization took effect.
The State hired new staff to operate the PAP. The investigative supervisor and two investigators
were hired in August 1993; two additional investigators were hired in October 1993. None ofthe
KRA Corporation
demonstration staff had prior experience with the FSP. Investigators received training for 2 weeks
prior to beginning activities in local district offices. This training included use of the PAP forms,
using the database, and other activities. The training did not include a review of FSP policies and
procedures.
The demonstration operated in three local district offices located in Baltimore City. The district
offices were Govans-Collington, Clifton, and Liberty-Garrison. They were preselected prior to
beginning the demonstration. Govans-Collington was the largest of these offices, with roughly
twice the number ofcases as each ofthe other offices. The offices operated somewhat independently
from each other, using different processes and operational procedures.
Prior to beginning work in the three local district offices, PAP staff developed a database to track
the information collected as part of the demonstration, project forms, and an operations manual. The
supervisor also conducted staff training of the investigators.
After prerequisite forms and procedures were completed, PAP staff visited the local district offices.
On October 1, 1993, PAP staff met with district office managers, assistant managers, and the
Baltimore City Liaison from IMA. These individuals were provided background information about
the project and given a project handbook. The handbook contained information about the project,
assignment procedures, and intervention schedules (e.g., informing applicants about the project).
At this meeting, district office staff were instructed as to how they were to "group" applicants, such
as intervention with or without notification.
District office staffvaried in their accounts ofhow the PAP project was implemented. Some recalled
a large meeting held with IMA staff and OSI staff. Others referred to an IMA transmittal letter
informing the district offices about the project. All local district office staff reported some type of
meeting where they learned about the PAP project. For example, one respondent indicated that the
PAP supervising investigator visited the district office to explain the role of the investigators and to
supply staff with written materials about the project. Staff from one district office specifically stated
that they did not receive anything "in writing" about the project. PAP staff also held follow-up
meetings.
Within district offices, the process ofintroducing the demonstration to workers varied. In one office,
the assistant administrator explained the project to first-line supervisors. In this office, the
supervisors were the point ofcontrol for PAP referrals. In other words, they made the case referrals
to PAP investigators. In another office, the project was introduced at an all-staff meeting. Written
materials also were distributed.
OPERATION OF DEMONSTRATION
The project operated from October 1993 through February 1995, a 17-month period. Staff
investigated a total of 1,330 cases as part ofthe front-end investigations effort, and investigated 893
cases as part of the recall process during the demonstration period. Over the course of the
demonstration, each investigator handled an average of 33 cases per month.
KRA Corporation 10
The number of cases investigated each month varied greatly. For example, in December 1993, PAP
investigators only received three cases from Liberty-Garrison staff. During the same month,
investigators received 82 cases from staff at Govans-Collington. The average number of cases
investigated each month at each district office is presented in Table 1.
The number of recall investigations also varied, ranging from a low of 99 cases during a 3-month
period across offices to a high of 231 cases during a 3-month period across offices.
Table 1—Average Number of Cases Investigated Each Month
During the Payment Accuracy Program
Office Liberty-
Garrison
Govans-
Collington
Clifton
Average Number of
Cases Investigated
Each Month
18 43 17
The process used for referrals to PAP investigators differed among offices; in particular, who
referred cases to investigators varied. Workers gave cases directly to PAP staff in one office;
supervisors performed this function in another office. One investigator indicated a clerk referred
cases to PAP staff. Across supervisors, efforts to encourage workers to refer cases for investigation
varied.
The project originally developed as a mandatory program. In other words, all NPA/FS cases were
to be referred for investigation. District office staff did not view the project as mandatory. They
believed referral of cases was optional. The data also confirm this—PAP staff only received
between 3 percent and 17 percent of all NPA/FS cases handled by the three offices. This difference
in viewpoint between district office staff and investigative staff created tension during the course of
the demonstration.
Local district office staff were not familiar with the research aspects of the PAP project. Some fully
understood that they were part of a research project in which they would be investigating certain
types of cases, including that they were giving information to some clients about what was
happening, but were not giving information to other participants. Some district office staff were
unfamiliar with the control aspect of the investigation, while some were confused about its duration.
Investigative staff sometimes viewed the small number of referrals of food stamp cases as an issue
of malicious intent while local district office staff, because of awkward referral systems, viewed the
project as one more step in the work process for an already underpaid and overworked staff. In one
office, staff did not fill out the PAP form because the office was short staffed and had a backlog of
cases. District office staff also believed that the requirement of giving the investigator the case
record hindered their work productivity. Workers often needed to see the case record when it was
with the investigator.
Changes during the course of the project The PAP project experienced a few changes during
the course of the demonstration. Changes occurred in four areas—the types of cases reviewed, the
KRA Corporation 11
number of PAP forms used, reporting the findings of the investigators to eligibility workers, and
notifying food stamp recipients about the project.
In the initial stages ofthe demonstration, staff received referrals for and investigated NPA/FS cases
exclusively. Later, the types of cases expanded to include all FSP cases, including Expedited
Service5 cases. District office staff indicated they were not clear on the types of cases investigators
handled. District office staff also indicated they were unsure about the criteria for referral and cited
the example of income sources. Early in the project, investigators only handled cases in which
applicants were employed or had no income. This criterion later changed, and investigators handled
cases that received unemployment insurance or public assistance.
PAP staff used 12 forms during the first few months of the project. During the course of the
demonstration, they found they could streamline efforts and reduce forms and paperwork.
Consequently, project staff reduced the number of forms to four.
One change in project procedures had a positive impact on relationships between demonstration staff
and district office staff. This change was in the way investigators reported the results of their
investigations. Originally, investigators made case disposition recommendations based on their
findings. District office staff believed the investigators were setting policy, and the reports became
a point ofcontention. Later, this was changed so that investigators simply documented their findings
without interpreting them. District office staff workers then determined the disposition ofthe case.
The notification process for FSP applicants changed during the course of the project. During the first
6 months of the project, recipients were either notified or not notified about the project procedures
based on the district office and whether they had an odd or even Social Security number. After 6
months, demonstration staff analyzed the impact of this selective notification by comparing groups.
PAP staff found there was no difference in the amount of false information reported between groups,
so all notification was discontinued.
RESULTS
Changes to Local District Office Operations and FSP Policies and Procedures
There were basically no changes in office operations at the district offices, nor were changes made
to policies and procedures during the life of the demonstration. District offices provided the
investigators with a desk or place to do their work, provided investigators access to a computer, and
developed systems that allowed investigators access to the case files. This process, however, varied
at each office, particularly for the process used access the case files. Otherwise, operational
procedures remained the same.
5 Expedited Service is an administrative mechanism used in the FSP to provide immediate assistance to
households with few resources to purchase food in the month they apply for benefits. Households with under $ 150 in
gross monthly income and under SI00 in countable assets, destitute migrant and seasonal farm workers, homeless
individuals and families, and households at risk of becoming homeless because their gross monthly income and liquid
resources are less than their monthly housing costs all qualify for Expedited Service benefits.
KRA Corporation 12
PAP staff thought the project improved the work of the eligibility workers, because they knew
someone else was reviewing their cases. PAP staff also thought the demonstration had a positive
impact on FSP participants, because the project deterred them from misrepresenting their situations.
The participants learned very quickly that the investigators were around. Some district office staff
thought the participants did not like the program because they were concerned they would not
receive their benefits right away. Demonstration staff believed the demonstration helped with the
timeliness of benefits in that all the verification needed was provided by the investigators. They
believed PAP improved the quality of services because it excluded persons who should not have
received benefits. In addition, PAP staff felt quality was improved because the eligibility workers
knew the case would be reviewed by others.
Denial Rates
Instead of the original evaluation design, PAP project staff received approval from FCS to conduct
an analysis of FSP benefits denied as a result of the project. The PAP supervisor checked the status
of all cases reviewed by PAP staff against the automated system indicating the outcome of the case.
If the case was denied, the reason was recorded. If the case was denied but PAP did not find a
negative result, the case was not counted as a PAP denial. If the case was certified, the case was
marked so that it would be reviewed in 3 months.
Of the NPA/FS applications reviewed by PAP investigators, 28 percent were denied benefits as a
result of the investigative process. For recall investigations, approximately 19 percent were denied
recertification as a result of adverse information obtained through PAP investigations. However,
we caution that these findings do not fully explain the role PAP staff had in the denials. For
example, we do not know the number of cases in which adverse information was obtained from
eligibility workers following routine procedures that resulted in the denial of benefits. Other studies,
such as the Food Stamp Program Application Process: Office Operations and Client Experiences
(1992), suggest that only 71 percent of those who file an application for FSP benefits are approved
for benefits. Thirteen percent are found ineligible and 16 percent fail to either complete the
certification interview or submit all required documentation. The data collected for the PAP project
do not allow us to determine the portion of ineligible applicants and number of applicants that do
not complete the process. We also do not know the extent to which the PAP project many have
influenced the behavior of eligibility workers with their cases.
PAP staff analyzed the characteristics of the population denied FSP benefits. Of those cases
subsequently denied benefits after PAP investigation, 62 percent were issued Expedited Service
benefits before investigation.6 PAP staff also found the following:
• Single-member households comprised 80 percent of all NPA/FS applicants
investigated by PAP.
• Males comprised 72 percent of the single-member households investigated.
6 A 1995 study (Evaluation ofExpedited Services in the Food Stamp Program) found that 45 percent of
Expedited Service applications involved postponed verification. These cases were more likely to be later found
ineligible than regular cases, suggesting that Maryland's 62 percent figure is a reasonable finding.
KRA Corporation 13
• Applicants often used nonexistent or false addresses.
• Applicants frequently failed to provide accurate information about household size
and composition.
Changes in Error Rates
We used a comparative analysis of pre- and postimplementation error rates in the demonstration
areas; however, this type ofanalysis could be misleading. For example, suppose the IQCS FSP case
error rate declined from 8 percent in the period before the demonstration to 6 percent after the
implementation of the demonstration procedures. This might appear to indicate that the
demonstration was successful. However, if the error rate for the State as a whole (excluding the
demonstration areas) also declined by 2 percent over the same period, one might be inclined to
question the results ofthe demonstration. Therefore, the procedure followed for this evaluation was
to compare the change in the error rate pre- and postimplementation to the change in the comparable
error rate in several comparison areas to test the results of the demonstration for statistical
significance.
The case error rate is the number of cases in the quality control (QC) sample that had overissuance
errors in a given office in a given period (quarter) over the total number ofsample cases in that office
in that time period. The dollar overpayment error rate represents the dollars paid in error in the QC
sample in the quarter divided by the average dollar payment times the number in the sample in that
period. For the overall demonstration period, the error rates are the rates in each demonstration
office for each quarter weighted by the QC sample size in the local office divided by the sum of the
quarterly samples (please see the Appendix B).
Originally, we planned to collect demonstration cost data to use for a cost-benefit calculation. This
calculation was to compare the marginal dollar savings resulting from the reduction in dollar error
rates in the demonstration area to the additional cost of implementing and operating the
demonstration. However, demonstration staff did not collect actual cost data on a consistent basis.
Therefore, for each office site and for the weighted average of all sites in the demonstration area, we
calculated the savings in benefits on an annual basis that resulted from the implementation of the
demonstration. In each case, this is the change in the estimated dollar overpayment error rate
between the predemonstration and postimplementation periods multiplied by the sum of the average
dollar payment in each quarter times the quarterly caseload. We compared this to the cost data from
the budget submitted with the project application.
The Maryland demonstration began in October 1993 (FY1994-1). Therefore, we compared quarterly
IQCS data from FY1991-I through FY1993-IV (the predemonstration period) to FY1994-I through
FY1994-1V (the postimplementation period). We had 12 quarters ofpreimplementation data and
4 quarters of data following the implementation of the demonstration.
To assess the significance of error rate changes, we compared the error rate experience in the
demonstration offices to that of the rest of the City of Baltimore. We also compared the
demonstration areas to Baltimore County, which surrounds the city to the west but does not include
KRA Corporation 14
the City ofBaltimore. Finally, we compared the experience of the demonstration areas to that of the
State less the city, county, and demonstration areas. Quarterly IQCS data were provided to us in
electronic format by the Office of Quality Control, in FCS.
Case Error Rate Changes. Table 2 presents the percent of cases that were paid in error.7 The first
three columns in Table 2 indicate the case overissuance error rate in the predemonstration period,
in the postimplementation period, and the change between the two periods. The rates for each period
are the averages ofthe quarterly rates weighted by the IQCS sample size in each quarter. Since the
objective of the demonsti .-ti, >ns was to reduce error rates, we do not consider the significance of
increases in error rates and use one-tailed tests to judge the statistical significance of error rate
declines.
The change in the case error rates in Govans-Coringlon*, Liberty-Garrison, and the weighted
average of the error rates in the three demonstration areas are all negative (see column 3 of Table
2)—a decline of 3.6 percent in Govans-Collington, a whopping 14-percent reduction in Liberty-
Garrison, and a decline of 4.1 percent in the combined rate for all three demonstration areas. Largely
driven by the decline in Liberty-Garrison, the combined three-office demonstration area changes are
significant at the 1-percent level against the balance of the City of Baltimore and the balance of the
State. Only Liberty-Garrison is significant relative to the change in error rates in Baltimore County
(minus 7.9 percent).
The last three rows ofTable 2 show the change in error rates in the three comparison areas and the
results for the balance ofthe City of Baltimore and Baltimore County relative to the change in error
rates for the balance of the State less Baltimore City and County. Note that the change in the balance
of the City of Baltimore is not significantly different from the change in the balance of the State,
while the relative change in Baltimore County (minus 8.8 percent) is significantly different from that
in the State at the 1-percent level of significance.
7 The Appendix contains the statistical considerations behind the results presented here. Those interested in
how the significance ofthe results was determined should consult the appendix. It also contains charts showing the pre-and
postimplementation sample sizes for each of the demonstration and comparison areas.
'Govans and Collington were combined during the life ofthe demonstration. Therefore, we merged the IQCS
data for these offices for the purpose of this evaluation.
KRA Corporation 15
Table 2 - Case Overissuance Error Rates
Local
Office
Pre-Demo
Error Rate
(Percent)
Demo
Error Rate
(Percent)
Change in
Error Rate
(Percent)
Compared
to City of
Baltimore
(Percent)
Compared
to
Baltimore
County
(Percent)
Compared
to State
(Percent)
Clifton 15.0 18.4 +3.4 +1.4 +9.5 +0.7
Govans-
Collinglon
17.0 13.4 -3.6 -5.6** +2.5 -6 2**
Liberty-
Garrison
24.0 10.0 -14.0 -16.0" -7.9* -16.6**
All Demos 17.9 13.8 -4.1 -6.1** +2.0 -6.8**
Balance of
State
8.0 10.7 +2.7 0.0
City of
Baltimore
Less Demos
14.7 16.7 +2.0 -0.7
Baltimore
County
16.9 10.8 -6.1 -8.8**
Significant at the I percent level
Dollar Error Rate Changes. Table 3 shows the changes in the dollar overpayment error rates in
each of the demonstration areas as well as the weighted average of the three demonstration areas
taken together. They are shown relative to the balance of the City of Baltimore, Baltimore County,
and the State (less Baltimore City, County, and demonstration sites). The combined error rate for
the three demonstration areas (minus 3.6 percent) is significantly different from the balance ofthe
City ofBaltimore and the balance of the State of Maryland at the 1-percent level. The change in the
dollar overpayment error rate for Govans-Coliington (minus 2.S percent) and Liberty-Garrison
(minus 10.3 percent) are significantly different from the balance of the City of Baltimore and the
balance ofthe State at the 1 percent level. Although the dollar error rate in Clifton increased (0.4
percent), compared to the balance of the City of Baltimore (an increase of 1.2 percent) it decreased
by 0.9 percent, which is statistically significant at the 5-percent level. That is, although the error rate
increased in the Clifton demonstration area, the error rate in the comparison area increased more.
KRA Corporation 16
Table 3 - Dollar Overpayment Error Rates
Local
Office
Pre-Demo
Error Rate
(Percent)
!>*mo
Error Rate
(Percent)
Change in
Error Rate
(Percent)
Compared
to City of
Baltimore
(Percent)
Compared
to
Baltimore
County
(Percent)
Compared
to State
(Percent)
Clifton 5.4 5.8 +0.4 -0.9* +1.2 -3.2**
Govans-
Collington
6.0 3.5 -2.5 -3.7** -1.7* -6.1**
Liberty-
Garrison
14.8 4.5 -10.3 -11.6** -9.5** -13.9**
All Demos 7.9 4.3 -3.6 -4.8** -2 8** -7.2**
Balance of
State
7.1 10.7 +3.6 0.0
City of
Baltimore
Less Demos
5.4 6.6 +1.2 -2.4**
Baltimore
County
8.1 7.3 -0.8 -4.4**
* Significant at the 5 percent level
*• Significant at the 1 percent level
Although the declines in the dollar overpayment error rates for Govans-Collington and Liberty-
Garrison and for all three demonstration areas taken together would probably be significant
measured individually, measured relative to the change in the balance ofthe City ofBaltimore and
the balance of the State, they are even more significant. The change in the error rates in both the
balance of the City of Baltimore and the State are positive. Indeed, the balance of the State
experienced a rise of 3.6 percent in its dollar overpayment rate over the period in which the
demonstrations were implemented. Note also that the balance ofthe State was significantly below
the changes in the error rates in the balance ofthe City ofBaltimore and in Baltimore County. This
suggests that the major rise in the error rate in the State that occurred in this period was outside the
City and County of Baltimore.
Govans-Collington is significantly different from Baltimore County at the 5-percent level. Liberty-
Garrison and the weighted average ofthe three demonstration sites are significantly different from
Baltimore County at the 1-percent level. The change in the dollar error rate in 'he Clifton
demonstration area is not significantly different from that in Baltimore County, but it is significantly
different from that ofthe balance ofthe State at the 1-percent level (minus 3.2 percent).
KRA Corporation 17
Resulting Dollar Savings Estimates
Table 4 shows the dollar savings associated with the changes in the error rates in the
demonstration areas. The dollar savings are calculated using the demonstration related change in
the dollar error rate estimated for the QC sample cases. This error rate is converted to dollars
by multiplying the dollar error rate for the quarter by the average benefit amount in the office for
the quarter, multiplied by the quarterly QC caseload for that period. As indicated in the note to
the table, the error rates associated with the estimated savings that are statistically significant are
indicated in bold. Statistical significance is measured by the significance of the change in the
dollar error rate from the predemonstration period to the demonstration period relative to the
change in the dollar error rate in the balance of the City of Baltimore, Baltimore County, or the
balance of the State of Maryland. As in the case of the changes in the estimated dollar error rate,
the change in the error rate in each of the demonstration areas and all of the demonstration areas
combined is significantly different from the change in the error rate in the balance of the State of
Maryland during the same period. The estimated annual dollar savings in the demonstration areas
on an annual basis is $717,424 relative to the balance of the State. Relative to the balance of the
City of Baltimore, the error rate change in all of the demonstration areas and the demonstration
areas combined is significant; and the estimated annual dollar saving as a result of the change in
the error rate in the demonstration areas is $479,727 on an annual basis, ranging from $26,286
in Clifton to $237,943 in Liberty-Garrison. The estimated saving in each of the demonstration
areas will not add to the savings in all demonstration areas combined because we estimated the
saving in each area and for all areas combined separately.
Table 4 - Estimated Annual Dollar Savings as a Result of the Demonstration
Local
Ofnce
Annual
Allotment
Pre-Demo vs Demo Demo vs State Demo vs Baltimore
County
De.no vs City of
Baltimore
Change in
Error
Rate*
Amount
Saved**
Change in
Error
Rate*
Amount
Saved**
Change in
Error
Rate*
Amount
Saved**
Change in
Error
Rate*
Amount
Saved"
Clifton $2,920,672 r.4°/. ($10,363) -3.2% $94,634 1.2% ($35,048) -0.9% $26,286
Govans-
Collington
J5.022.398 -2.5% $125,560 -6.1% $306,366 -1.7% $82,870 -3.7% $185,829
Liberty-
Garrison
B2.0S 1,234 -10.3% $211,752 -13.9% $285,122 -9.5% $194,867 -11.6% $237,943
All Demos R9.994.304 -3.6% $358,132 -7.2% $717,424 -2.8% $279,841 -4.8% $479,727
* Bolded error rate changes are significant
•• Dollar savings in demonstration sites may not add up to all-demonstration totals. Savings for each were estimated separately.
Although the demonstration sites did not collect consistent data on costs, it is likely that the
savings resulting from the reduction in dollars paid in error in the demonstration sites, as
estimated, is greater than the cost of the operation of the demonstrations.
KRA Corporation 18
We compared the estimated savings for the year in the three demonstration local district offices
to the budgeted cost for the demonstration submitted as a part of the original grant application for
demonstration funding. The estimated annual benefit savings for the three demonstration local
district offices measured against the error rate change in the State ($717,424) is 3.8 times the
estimated 18-month total cost of the demonstration ($188,687)9 and 5.7 times the 12-month cost
($125,791). The annual benefit saving measured against the error rate change in the balance of
the City of Baltimore ($479,727) is 2.5 times the 18-month estimated cost of the demonstration
and 3.8 times the estimated 12-month cost.
IMPACT OF THE DEMONSTRATION
Demonstration project staffthought the PAP demonstration was very effective. The statistics suggest
that the project kept ineligible people from participating in the FSP and that the error rate (case and
dollar) was reduced. Beca. ;e of gaps in information about the applicant notification process, small
IQCS sample sizes, and other factors, it is not possible to conclusively attribute the reduction in error
rate to the demonstration.
Impact on Local Office Operations
The demonstration did not have an impact on local office operations nor did it impact State FSP
policies and procedures. Client services and timeliness of benefits did not change as a result of the
demonstration.
The eifects of this demonstration are limited to the time the investigators are in a local office; once
investigators leave the office, the effect of the demonstration ends. The project did not appear to
provide any learning at the worker level and the demonstration was not designed to change local
district office policies and procedures.
Future Impact of the Demonstration
IMA is continuing this project by using reinvestment dollars to expand the program statewide. The
project has expanded to other offices in Baltimore City and two urban counties near Washington,
D.C. There continue to be tension and turf issues with the expanded program. OSI/OIG is
responsible for the front-end investigations, while IMA is responsible for the follow-up
investigations. IMA has responsibility for the funds, which gives OSI/OIG little control over staff
and equipment expenses. OSI/OIG and IMA continue to try to work together on this effort, as both
divisions believe the effort has merit and is effective.
In terms of Maryland's replication effort, it would be interesting to know if Maryland's expanded
program is as effective as the demonstration. The expanded statewide project has some activities
and characteristics that are different than the original demonstration. For example, the characteristics
ofthe new areas in which the effort is operating are quite different demographically than the three
FSP district offices in which the demonstration originally operated. The PAP demonstration was
initially limited to NPA/FS applicants. This pool of applicants was primarily composed of single
9 Budget information was obtained from the Maryland Department ofHuman Resources grant application,
February 1993.
KRA Corporation 19
household males residing in an inner city area. How well the program works when applied to other
FSP populations is not known. It would be interesting to learn which activities of the original PAP
demonstration were replicated across the State, ifthey were effective with the FSP population groups
in these areas, and if error rates declined in these new areas.
Transferability to Other States
The ease with which this demonstration can be transferred to other States relies on the availability
of detailcu documentation about the demonstration, self-assessment information, and the nature of
the demonstration; and the type of system or structure (staff and organization) required to replicate
the demonstration. A brief discussion of these factors follows below.
FCS required demonstration staff to document their efforts in quarterly progress reports and a final
report. Maryland staff emphasized the collection of program statistics and their final report included
many numerical tables. However, the data elements they collected were inconsistent across time
periods, the elements themselves changed during the course ofthe demonstration, and the numbers
collected were not comparable or related to the universe of cases within an office. For example,
project staff did not report how many cases they reviewed each month relative to all NPA/FS cases
in each office. Staff only included a brief description of the operational aspects of the
demonstration. For example, we do not know with any certainty when local district office staff
stopped notifying applicants about the investigations as this information was not detailed in reports.
FCS also required grantees to perform self-assessments. The resulting final report and self-assessment
was not as rigorous as Maryland staff originally planned nor did staff meet the
requirements specified by FCS.
Evaluation technical assistance provided early in the implementation phase of the demonstration
would have helped the demonstration staff develop appropriate and realistic evaluation measures.
These measures would have enabled them to better document the successes of and barriers to their
interventions. Evaluation technical assistance also would have provided a better balance of
qualitative and quantitative measures for the demonstrations.
In addition, development ofan evaluation/self-assessment design would have provided a framework
for the quarterly reports. For example, if the demonstration staff had developed research questions
or demonstration objectives, quarterly reports could have detailed progress made in achieving
objectives or in answering the research questions.
The PAP demonstration could be replicated in States where "investigative" or follow-up procedures
are in place or can easily be implemented. Maryland's investigative approach requires an
organizational structure that supports investigative activities. This could be a separate organizational
entity or one that is part of the FSP. The types of staff needed to conduct these investigations are
relatively easy to recruit—Maryland used staff who did not have FSP experience. However, this
lack ofexperience may have compromised the overall program effectiveness. Other States may find
replicating Maryland's program difficult because demonstration staff did not develop detailed
handbooks or materials to guide replication efforts.
KRA Corporation 20
Lessons Learned
PAP staff indicated that if they were to implement this type of project again, they would hire more
investigators, provide more training, conduct more follow-up reviews, require more cooperation at
the administrative/supervisory district office level, and review Aid to Families with Dependent
Children cases as well as NPA/FS cases. They would also fully explain the p.ogram to the eligibility
workers so that the workers would not be so resistant to the program. They would work with all
involved parties so that procedures could be set up ahead of time. Finally, they would use fewer
forms.
Local district office staff suggested that more training be provided to the investigators. Investigators
needed training in the areas of client sensitivity and FSP policies and procedures, particularly in the
area of FSP Expedited Services. District office staff shared an underlying belief that they could not
deny food stamps to needy people. Investigators did not share this value, and this often caused
conflict.
PAP demonstration staff learned many lessons from this demonstration project. These include the
following:
• There were people who actively tried to defraud the food stamp system.
• FSP procedures should be applied uniformly, particularly in the area of Expedited
Services.
• The demonstration has good deterrent value—if clients know their cases may be
reviewed, they may be less inclined to be dishonest.
• Most clients are honest and in need of their benefits.
KRA Corporation 21
Chapter 4
Description of Illinois' Food Stamp
Error Reduction Demonstration
DESCRIPTION OF THE ERROR REDUCTION DEMONSTRATION
The Illinois Food Stamp Error Rate Reduction Demonstration began in April 1993 and ended in
October 1994. Two staff experienced in FSP policies and procedures were hired to review food
stamp cases and the processes related to determining eligibility for the FSP. These individuals were
giver, total responsibility, under the supervision ofa steering committee, to design and implement
the procedures used to collect information regarding the type of errors being made and why these
errors occurred. Specifically, the error reduction activities focused on earned income and household
composition errors that result in overpayment of FSP benefits.
Planned Intervention
The goal ofthe project was "to implement a reasonable and easy-to-use payment accuracy initiative
which would target errors, identify causes and develop specific recommendations for on-going error
reduction activities". The project concentrated on the two most frequently occurring errors (e.g.,
earned income and household composition) that together caused $18 million in misspent Federal
funds annually. The Illinois' Department of Public Aid (DPA) initiative was specifically designed
to meet departmental needs in an urban, high volume area. However, the effort was sufficiently
generic so that it could be applied to any office or public assistance agency.
DPA staff selected three contiguous offices in Chicago for the project, two with high error rates in
earned income and household composition (Lower North and Humboldt Park), and a third with
considerably lower error rates (Northwest). All were on the north and northwest side of Chicago.
Each of the three offices issued considerably more than $10 million per year in FSP benefits. The
three offices had similar characteristics. All offices had a large caseload (10,000 to 16,000 cases).
All had a large Hispanic caseload and had bilingual staff to work with Spanish-speaking clients. In
each of the offices, the process was automated (for 10 years), and workers used computer screens
to guide the interview process. Office characteristics differed only slightly among the offices.
Demonstration staff reported that worker use ofcomputers varied somewhat among the offices and
that one office1 was able to process and interview clients more quickly than the other two offices,
thus reducing client time in the waiting room. All offices handled a large number ofaliens, requiring
staff to grapple with the complicated policies related to serving them.
State staff hired two individuals with FSP experience to review cases to determine why the two most
common types of errors were being made in the two offices with the high error rates, and why the
same errors were not being made in the office with the fewer errors. The staff were responsible for
developing specific recommendations for corrective actions that could be taken to address the
problems, and ensure that appropriate action would be taken on the specific errors identified. Within
This was a high error rate office.
KRA Corporation 22
this overall goal, the demonstration staff had the flexibility, with appropriate supervision, to use the
approaches that they thought were best and most productive. These staff, known as Error Reduction
Analysts (ERAs) reviewed food stamp cases, interviewed staff, and observed local office operations.
DPA staff designed the ERA review process around three basic principles:
• The review process be markedly different from a quality control review or mandated
audit
• The ERAs be given a great deal offlexibility to develop and implement new methods
and tools, as necessary, to aid in the review process and to achieve project goals
• The ERAs be able to solicit local office cooperation and input for problem solving
throughout the review process
DPA developed and operated the demonstration. A steering committee, composed of five
individuals from different divisions within DPA, provided overall management for the project.
Divisions represented on the steering committee included the Division of Policy, Bureau of Food
Stamps; Division of Field Operations, Bureau of Program and Field Management; Division of
Program Integrity, Bureau of Quality Control; and the Division of Planning and Community
Services, Bureau of Planning. Steering committee members were to use their expertise and
experience to assist in the design, implementation, and on-going management ofthe project. The
steering committee was chaired by the State Food Stamp Director who was also designated as the
project director.
With steering committee members headquartered in Springfield, staff recognized that additional on-site
management assistance was beneficial and would enhance the project Two administrative staff
members from the DPA's Chicago zone office, who had immediate and direct oversight as well as
line authority over the targeted local offices, were recruited for the project This "zone support staff'
provided direct supervision and guidance to the ERAs, and provided the steering committee
members with feedback regarding the progress of the project. Zone support staff also provided
assistance in expressing the importance ofthe project to local office administrative staffand helped
coordinate the project implementation schedule with the selected local offices.
Background
The Illinois FSP is administered through DPA's full service offices where applications are taken,
eligibility is determined, and case records are maintained. Cook County, which includes the city of
Chicago and the surrounding suburbs, has 26 full service offices and several other specific-service
offices. The FSP is administered directly from the State's central offices in Springfield. The
Division of Field Operations has direct line authority over the administrators and staff in each ofthe
local offices. The Division of Policy contains both the Bureau of Policy and Training and the
Bureau of Food Stamps. The Bureau of Policy and Training is responsible for drafting the DPA's
policy instructions for all programs and providing training. The Bureau of Food Stamps is the
primary point of State administrative responsibility to USDA for FSP operations.
KRA Corporation 23
During FY 1994, Illinois' total FSP caseload varied from approximately 494,000 to 507,000
households per month, and its total issuance ranged from $88.4 million to $91 million in benefits
per month. Cook County cases comprised approximately 62 percent of FSP households, making it
the ideal site from which to choose the localized urban area FCS was seeking for its payment
accuracy improvement project.
IMPLEMENTATION OF THE ILLINOIS FOOD STAMP ERROR REDUCTION
DEMONSTRATION
In April 1993, the FCS authorized funding for Illinois' Food Stamp Error Reduction Demonstration.
The steering committee began the project by meeting and clarifying the roles and responsibilities for
project implementation, securing cooperation from the Chicago zone staff and local office
administrators, and initiating the hiring process for the two ERAs. The steering committee
recognized that the selection ofthe proper staffwas critical to the success ofthe project. The desired
skills for the ERAs included extensive FSP knowledge and experience, good analytical skills, the
ability to communicate both in writing and orally, and excellent "people" skills. The original grant
request also included clerical support staff. However, it was later decided that the purchase oflaptop
computers and software would be a better use ofthe limited resources available.
Zone and State staff hired the analysts on August 17, 1993; the analysts began working on the
project on September 1,1993. During the first week, the analysts met with the steering committee
and Chicago zone staff, obtained and began familiarizing themselves with their laptop computers,
and scheduled visits to each of the local offices involved in the demonstration. They selected
Humboldt Park district office as the office where the ERAs would begin their work.
During the first month of the project, in addition to case reviews and interview observations, the
ERAs designed the forms needed to collect the desired information. These included a data collection
form to gather the necessary project data from the case files, a worksheet to collect data obtained
during the observation ofintake eligibility interviews, and a supervisor and caseworker questionnaire
that allowed staff to express their ideas about the project. ERAs created management reporting
forms and daily logs with an emphasis on the need for open two-way lines of communication
between the steering committee and ERAs. By the second month of the project, most working tools
were developed and implemented, and the ERAs moved to the second targeted office.
Experiences in the first office helped the ERAs plan and conduct their reviews in the second and
third offices more effectively. By January 18, 1994, the ERAs completed the first phase of the
project. After a brief period to evaluate phase one and plan phase two, phase two reviews began in
the Humboldt office in the middle of February. ERAs completed the phase two review on August
17,1994.
OPERATION OF DEMONSTRATION
From September 1993 through August 1994, ERAs reviewed case records, observed interviews,
interviewed caseworkers, and made recommendations regarding problem areas and solutions to those
problems. The two-phase demonstration allowed a period of analysis between phases. The first
phase was more introductory and exploratory in nature. It consisted of intake interview observations
and case record reviews. Demonstration staff were in each of the three local offices for 6 weeks
KRA Corporation 24
during this phase. The second phase operated for 2 months in each office and involved
recertification observations, case record reviews, and other activities.
In each office, ERAs initiated each phase with an entrance conference. The local office
administrator, office supervisors, zone staff, and steering committee members attended the entrance
conferences. The conferences introduced the project to local office staff and introduced the
demonstration staff to the offices. The entrance conference served to kick off the first phase of
activities and officially marked the beginning of work in thaf particular office.
The ERAs developed a questionnaire to initiate worker input and circulated it to all staff at each of
the local offices. This questionnaire asked for suggestions for improvement to anything related to
FSP operations, in general, and any problems with earned income and household composition,
specifically. Demonstration staffused the responses to help define the future direction ofthe project.
Each Local Office Administrator designated an individual to serve as a demonstration project liaison.
The duties of the liaison included recommending procedures for selection of case records to be
reviewed, deciding which units in the local office should be analyzed, establishing a non-threatening
system to implement recommended case actions on documented errors, handling specific requests
by the ERAs, helping the ERAs resolve problems as a result of project activities, and providing the
local office administration with project updates.
ERAs analyzed a total of six casework units (two in each office) in phase one. The analysis
consisted of two major components case record review and follow-up, and caseworker/supervisor
interviews. Cases were selected for review based on four criteria, including case record data
indicated that there were people living in the house who were not receiving food stamps, persons
were added to or deleted from the food stamp household, a recertification of eligibility for FSP
benefits had recently been completed, and the case was subject to the Illinois monthly reporting
system.
Using the Case Record Review document to record case data, the ERAs reviewed all the case
information to identify evidence of any procedural or policy-related errors during the period since
the last recertification SRAs provided feedback regarding specific case errors and recommended
corrective actions to the local office through the local office liaison. Using a standard local office
memo, the ERAs documented errors and discrepancies found on specific cases and attached this
information to a copy of the narrative portion of the Case Record Review document. During the
caseworker/supervisor interviews, the ERAs concentrated on problems related to the FSP as
perceived by the staff. The interviews highlighted some problem areas about forms (the monthly
reporting form and FSP recertification forms in particular) as well as policy. Certain aspects of FSP
policy, for example the policy affecting aliens, seemed to be widely misunderstood. Many workers
indicated that they relied on their supervisors for policy interpretations, yet interpretations seemed
to vary between offices and also between supervisors within offices. This information became a
source of ideas for additional approaches to further pinpoint causes for errors.
In addition to case record reviews and caseworker/supervisor interviews, the ERAs observed the
intake eligibility interviews. They interviewed a total of 60 local office staff, and completed 63
observations in the three local offices. When the ERAs began the interview observations, they only
observed the Intake Eligibility Specialists. However, after observing several intake eligibility
KRA Corporation 25
interviews, it soon became apparent that screeners were collecting information and making decisions
that affected the determination of household composition, and that screener activity could be a
substantial source for household composition errors. As a result, the ERAs scheduled observation
of screening interviews.
Phase two activities included a case record review for each of the cases observed during the intake
interviews in phase one in addition to the standard unit analysis (case record reviews and
caseworker/supervisory interviews). It also included observation of the interviews required as part
of the FSP recertification process, and a supervisory policy comprehension quiz. ERAs completed
a total of 137 recertification interview observations. The recertification interview observation was
essentially the same as the intake eligibility interview observation. The ERAs did not actively
participate in the interview process; they answered caseworker questions and provided assistance
when asked.
The E?As created a supervisory quiz because inconsistent policy interpretation and a lack of basic
policy knowledge by supervisory staff was often mentioned by caseworkers as a problem. Staff
designed the quiz as a method of testing the supervisors' comprehension skills, not their
memorization skills. That is, the supervisors could use their FSP manual when completing the quiz.
The quizzes were distributed and completed at a supervisory meeting. Upon completion of the quiz,
the ERAs collected them and then discussed the questions and answers in an open session. A total
of 42 supervisory quizzes were completed. Supervisors averaged 11.22 correct responses out of 23
questions.
The ERAs held exit conferences at each of the local offices at the conclusion of the activities. The
purpose of these conferences was to provide feedback to the local office administration on the
project's findings during the just-completed phase, and to express appreciation for their staffs
cooperation and input.
Procedures and Staffing Changes Impacting the Demonstration. Immediately prior to the
demonstration, the method of assigning caseworkers to caseloads changed. Previously, if a client
received both medical assistance and FSP benefits, two separate caseworkers were assigned. Under
the new organization, one caseworker handled both cases. DPA trained local staff in the new
programs they were assigned, but one office reported that some caseworkers still were not proficient
in the policies for both programs.
Staff shortages were common during the demonstration period. One office experienced a lack of
bilingual caseworkers. Caseworker shortages became particularly problematic as caseloads
increased in local offices. One office reported 13 uncovered caseloads2 during the course of the
demonstration. A caseload could contain as many as 350 cases at various stages of activity. A
second office also reported uncovered cases until the last 2 months of the demonstration.
Caseworkers and supervisors in local offices are unionized, which has a major impact on the changes
the State can make in worker responsibilities and similar matters. In July 1994, supervisors, who
2 Cases not assigned to workers because of staff shortages.
KRA Corporation 26
D
were previously not part of the bargaining unit, became part of the union. This change altered staff
roles and responsibilities and created a new administrative level within local offices.
Policies related to the FSP frequently changed. One local office staff member reported that
approximately 60 policy or procedural changes were made each year. Typically, memos announcing
the change were distributed to local office staff. Staff reported that the memos were not always as
explicit as they would like.
Revisions to the FSP calculation sheet and FSP eligibility worksheet occurred during the course of
the demonstration. Several respondents credited demonstration staff with the new forms; however,
these forms were developed independently of the demonstration and demonstration staff.
RESULTS
Changes to Local Office Operations and FSP Policies and Procedures
The ERAs found no difference in procedures between the two high error rate offices (Lower North
and Humboldt Park) and the lower error rate office (Northwest). They were unable to determine
why Northwest had a lower error rate at the beginning of the demonstration than the other two
offices.
Although on the surface we can state that changes to local office operations did not occur as a result
of the demonstration, DPA initiated several efforts and modified FSP procedures as a direct result
of information collected by the ERAs. In other words, DPA staff used the information generated by
the ERAs in three urban, local offices to make immediate statewide changes that they believed
would ultimately reduce the error rate. The areas in which the ERAs and demonstration project
influenced DPA FSP policy and procedural changes are described below.
State staff made several changes to FSP policies and procedures as a result of the project. These
changes included the Quality Tip memos and statewide interview training for intake staff. In
addition, revisions to the household composition module ofthe Automated Intake System (AIS) that
were already in progress were accelerated and completed during the demonstration. The steering
committee was influential in making these revisions a high priority because oftheir impact on the
error rate.
In December 1993, State DPA staff distributed the first Quality Tips memorandum to local office
staff. It addressed the subject of incorrect determination ofthe applicant's household composition
at screening/intake. Six additional Quality Tips memoranda prepared during the course of the
demonstration covered subjects such as correctly updating the forms to identify additional persons
added to the FSP household, counting income in the month it was received (not the month it was
earned), important questions to ask about household composition, and reviewing alien status
documentation at recertification.
The DPA's Bureau ofPolicy and Training developed interview training in a very short time frame.
This training focused on appropriate interviewing techniques and policy application within the AIS
context. The ERAs recommended this training because they found early in phase one that the
caseworkers did not have the comprehensive policy knowledge needed to make accurate eligibility
KRA Corporation 27
decisions and relied too heavily on the automated system for this purpose. Supervisors and
caseworkers tested the training in the three demonstration offices and upon their overwhelmingly
favorable response, DPA expanded the training statewide.
The ERAs also influenced accelerating revisions to the State's AIS household composition module.
Efforts to enhance the AIS to include a household composition module began before the
demonstration as part ofthe Corrective Action process. During phase one, the ERAs determined that
household composition was being addressed incorrectly in AIS (the automated screen instructed
workers to ask who "eats together" rather than who "purchases and prepares" food together). Once
ERAs identified and documented this, finishing a new household composition module became a top
priority. The State piloted a full module in January 1995 and modified it after the pilot test.
Training on the module was then completed and implemented statewide.
Other changes that have occurred since the demonstration ended (or that are under way) include
updating the computerized client database, developing a FSP conference/training for senior local
office staff, providing Translation Techniques training for bilingual intake staff, and analyzing
additional quality control data. In addition, changes have been made (or will be made) to several
of the forms used during the FSP (re)certification process.
The ERAs found that excluding ineligible household members from the client database led to
budgeting errors in some FSP cases. As a result, they recommended that procedures be changed so
that ineligible household members are included in the data base. This improvement remains a
priority for State staff; however, its progress is sic • Staff believe that the implementation of the
new household composition module will make it easier to update the client database.
The ERAs found that knowledge about FSP policies among supervisors was inadequate. As a result
ofthis finding, a "forum" for discussion of policies on a regular basis was recommended. However,
the realignment of supervisory staff into the collective bargaining unit precluded such a forum.
Under the new structure, supervisors became lead workers supervised by Assistant Local Office
Administrators (ALOAs). Related to this effort, the State held two 2-day conferences in September
199S for ALOAs. One conference was held in Springfield, the other in Chicago. The conference
highlighted FSP issues as they related to ALOA responsibilities. It also included information on the
FSP in general and quality control and quality assessment issues.
The ERAs discovered that even workers with a conversational command of both Spanish and
English had trouble translating the specialized FSP terminology. The Metro Chicago Training
Center created and began conducting a one-day Translation Techniques training session for bilingual
intake staff. The training concentrates on translation of agency-specific terms and phrases, and on
communicating in the variety ofdialects used by Spanish-speaking people. DPA provided training
in all offices with significant numbers of Spanish-speaking clientele by August 1995.
The creation of general recertification guidelines is another recommendation that was still under
consideration at the conclusion of the demonstration. These guidelines would include forms
applicable to each case type, acceptable types of verification, an explanation of items that require
verification, and additional questions that should be asked when applicants present new information.
KRA Corporation 28
In general, office operations remained the same throughout and after the demonstration. However,
local office staff felt their ability to correctly implement policy improved. Staff generally indicated
they had a better understanding of policy and were able to provide a better quality of services to the
clients.
Changes in Error Rates
DPA demonstration staffand KRA evaluation staff conducted an analysis to determine the impact
of the Food Stamp Error Rate Reduction Demonstration on the error rate. DPA demonstration staff
augmented their IQCS sample with Quality Assessment (QA) reviews of case records in the three
local offices. They also examined Quality Control (QC) cases for these offices. KRA obtained
quarterly IQCS data from FCS for analytic purposes.
Both the DPA QA and QC analysis and the KRA analysis were inconclusive in demonstrating any
change in error rate.
DPA Quality Assessment Reviews and Augmented Quality Control Sample Analysis. QA
reviews use case record material to determine whether the required forms, documentation, or sources
of verification exist within the case record. If such documentation or verification does not exist, an
error exists. DPA staff conducted 292 QA reviews for the three offices prior to the implementation
ofthe demonstration in 1993; in August 1994, staff conducted post-implementation reviews on 696
cases. The QA review conducted prior to the demonstration included only recertification actions
while the post-implementation review included intake and recertification actions. The reviews only
examined the household composition and earned income areas of the records.
For the three offices, the pre-demonstration QA review showed a relatively high accuracy rate for
household composition (96 and 100 percent accurate). Post-implementation household composition
accuracy averaged 95 percent across offices. This was a decease from the pre-implementation
accuracy rate.
When compared to household composition accuracy, earned income accuracy was lower across
offices. For the pre-demonstration QA review of earned income, offices ranged from 100 percent
accurate to only 33 percent in one office. The post-implementation review showed that offices
averaged 86 percent accuracy.
Although the offices were performing at an acceptable level of accuracy (with the exception ofthe
33 percent level of accuracy for earned income), accuracy levels generally declined in the three
offices during the demonstration period. Several factors can be attributed to this, including a change
in forms, and a change in procedures that included certain aliens as eligible FSP recipients.
DPA staff also performed a QC review. Staff reviewed a total of 103 QC cases for the three offices
in mid-1993 (pre-demonstration). There were five agency-caused errors (earned income and
household composition) in the three offices. Later, staff reviewed a second QC sample of 72 cases
during the time the demonstration was in its early stages (late 1993 through early 1994). In this
sample, only two agency-caused errors were found across offices, one household composition error
and one earned income error.
KRA Corporation 29
Integrated Quality Control System Analysis. KRA analysts used IQCS error rate data to evaluate
the efficacy of the error reduction demonstration. In addition, we used a dollars-paid-in-error
measure to estimate the annual savings from the implementation of the demonstration.
The demonstration started in April 1993 (FY1993-III). Therefore, the quarterly IQCS data for
FY1991-I through FY1993-II (the pre-demonstration period) were compared to FY1993-III through
FY1994-IV (the post implementation period). Thus, we used ten quarters of pre-demonstration data
and six quarters of data following the implementation of the demonstration in the analysis. The
comparison areas used in the analysis include the demonstration offices (Lower North, Humboldt
Park and Northwest), the balance of Cook County less the demonstration offices, and the State of
Illinois less Cook County. The results of this analysis are discussed below.
Case Error Rate Changes. Table 5 shows the case overissuancc error rates for the demonstration
local offices. The table shows the pre- to post-implementation change in the case error rate (column
three) and the change in the error rate in the demonstration area relative to the comparison area
(columns four and five).
Table 5 - Case Overissuance Error Rates
Local Office Pre-Demo
Error Rate
(Percent)
Demo Error
Rate
(Percent)
Change in
Error Rate
(Percent)
Compared to
Cook County
(Percent)
Compared to
State
(Percent)
Lower North 19.3 19.4 +0.1 +0.8 +0.4
Humboldt
Park
20.5 18.6 -1.9 -1.2 -1.6*
Northwest 14.8 16.3 +1.5 +2.3 +1.8
All Demos 17.8 17.9 +0.1 +0.9 +0.4
Cook County
Less Demos
17.9 17.1 -0.8 -0.4**
State Less
Cook County
and Demos
16.1 15.8 -0.3 0.0
* Significant at the 5 percent level
** Significant at the 1 percent level
As can been seen from Table 5, only the change in the case error rate in Humboldt Park relative to
the balance of the State (-1.6 percent) is significant at the 5 percent level, using a one-tailed test of
significance. The fact that the change in the error rate in Humboldt Park is not significantly different
KRA Corporation 30
from that in the balance of Cook County is not surprising given that the error rate change in the
balance ofCook County is significantly different, at the 1 percent level, from that in the balance of
the State. None ofthe other demonstration area offices or sum of demonstration area offices changes
in error rates are significant. It is interesting to note that the office with the lowest case overissuance
error rate at the beginning of the demonstration showed the greatest increase in error rates relative
to the other demonstration offices during the course of the demonstration. Despite this increase, the
Northwest office continued to have the lowest case overissuance error rates than the other two
demonstration offices.
Dollar Error Rate Changes. As seen in Table 6, only the change in the dollar error rate between
the Lower North demonstration area and the balance ofCook County is significant and negative (at
the 1 percent level). However, the change in the error rate in Lower North is not significantly
different from the change in the error rate for the balance of the State of Illinois and the change in
Cook County is not statistically different from that in the balance of the State. Thus, we conclude
that the demonstrations produced no demonstrable difference in the dollar overpayment rates
between the demonstration areas and either of the comparison areas. In fact, the difference in the
dollar error rates for all the demonstration areas combined and the balance of Cook County and the
balance of the State are positive, indicating a relative rise in error rates in the demonstration offices
over the period relative to the comparison areas. As with the case overissuance error rates, the
Northwest office showed the greatest increase in dollar overpayment error rate changes, despite
having the lowest dollar erroi rate at the beginning of the demonstration.
Table 6 - Dollar Overpayment Error Rates
Local Office
Pre-Demo
Error Rate
(Percent)
Demo Error
Rate
(Percent)
Change in
Error Rate
(Percent)
Compared to
Cook County
(Percent)
Compared to
State
(Percent)
Lower North 7.3 5.1 -2.2 ~3.<r -1.4
Humboldt
Park
9.8 10.9 +1.1 +0.2 +1.9
Northwest 5.0 8.3 +3.3 +2.4 +4.1
All Demos 7.1 8.5 +1.4 +0.5 +2.2
Cook County
Less Demos
7.3 8.2 +0.9 +1.7
State Less
Cook County
and Demos
68 6.0 -0.8 0.0
Significant at the 1 percent level
KRA Corporation 31
Resulting Dollar Savings Estimates
As indicated in Table 7, only the change in the error rate for Lower North relative to the balance of
Cook County is statistically significant (bolded). None of the other changes in error rates relative
to the comparison areas are significant. In fact, because the point estimates ofthe change in the error
rates in the demonstration areas are positive, although not significant, the estimates of the dollar
"savings" relative to the two comparison areas are negative—$297,549 relative to the balance of the
State and $66,060 relative to the balance of Cook County for the weighted sum of the three
demonstration areas.
Table 7 - Estimated Annual Dollar Savings as a Result of the Demonstration
Local
Office
Annual
Allotment
Pre-Demo vs Demo Demo vs State Demo vs Cook
County
Change in
Error Rate*
Amount
Saved**
Change in
Error Rate*
Amount
Saved**
Change in
Error Rate*
Amount
Saved**
Lower North $4,606,750 -2.2% $101,349 -1.4% $62,836 -3.0% $139,999
Humboldt
Park
$4,606,750 1.1% ($50,674) 1.9% ($88,205) 0.2% ($11,042)
Northwest $4,606,750 3.3% ($152,023) 4.1% ($188,200) 2.4% ($111,037)
Ml Demos $13,820,251 1.4% ($193,484) 2.2% ($297,549) 0.5% ($66,060)
* Bolded error rate changes are significant
** Dollar savings in demonstration sites may not add up to all-demonstration totals. Savings for each were estimated
separately.
In Illinois, KRA analysts could discern no appreciable reduction in the change in error rates between
the demonstration areas and the balance ofCook County or the rest of the State of Illinois on either
a case or dollar basis. In fact, because the changes in error rates relative to the comparison areas
were positive, the Illinois demonstration areas exhibited a loss of benefit payments rather than a
savings. In Illinois, a 1 percent change in the dollar error rate in the demonstration areas would
produce almost a $135,000 change in benefit payments on an annualized basis.
We note that it would be difficult to show any significant changes in Illinois' IQCS error rates
because the demonstration did not limit the demonstration effects to the demonstration offices.
Policy and procedural changes implemented as a result of the demonstration were made statewide.
Thus, the demonstration offices could not be compared to other locales to determine the
effectiveness oftheir interventions.
IMPACT OF THE DEMONSTRATION
The Illinois project operated more like a program improvement effort than a demonstration project,
as staff obtained information and subsequently used that information to clarify policy and make
KRA Corporation 32
changes to procedures. As State staff were interested in why errors occurred, they designed their
demonstration effort to use what they learned from the demonstration to improve the system and
institutionalize certain changes that would ultimately reduce errors. However, changes to "improve"
the program were made statewide during the course ofthe demonstration and thus changes that nay
have occurred as a result of the demonstration could not be isolated to the demonstration offices.
Impact on Local Office Operations
The demonstration operated within the existing system of policies and procedures. Although
changes in local office procedures and policies did not occur while the demonstration was under
way, changes are more likely to occur in the future as a result ofthe demonstration. These changes
have the potential ofdecreasing the error rate on a sustained basis. For example, modifications to
computerized intake modules, training to improve interviewing techniques, and other similar local
office activities could ultimately reduce case and dollar error rates over several years. Further, any
future changes to FSP operations will be implemented statewide, generalizing the "lessons learned"
from the demonstration to all FSP offices.
Future Impact of the Demonstration
The demonstration provided DPA staff valuable insight into how the FSP is operationalized in local
offices. Illinois continues to make changes to FSP practices as a resuli of the demonstration. The
demonstration findings have prompted several procedural changes that occurred early in the
demonstration and continue through the present. Additional changes are planned but may take time
to implement, given changing priorities and multiple programs within DPA.
Transferability to Other States
The general consensus ofthe Illinois staffwas that the basic procedures and concepts of this project
are replicable in other agencies. Adjustments would need to be made for individual circumstances.
However, the attitude and background knowledge of the demonstration team were constantly
mentioned as crucial to this demonstration. The successful transfer ofthis project to another location
will rely heavily on the staff available to conduct the reviews and work with local agency staff.
The handbook created at the conclusion ofthis project provides a guide for implementing a similar
study in another local office or State. At the local office level, a supervisor could use the handbook
developed by demonstration staff to develop and implement procedures to examine why error rates
occurred. Designed primarily for use in other Illinois offices, it could also be used in other States
as background material for designing an error reduction initiative appropriate for that State or local
agency. The authors intentionally tried to avoid writing the document in the typical bureaucratic
style. Instead, they used humor and an informal style to present the facts. The handbook is
basically a report, written in a direct easy-to-read manner, describing what was done and how it was
done. It is not a "how to" handbook. However, it emphasizes all the major points that must be
considered to make a similar demonstration effective.
Illinois' self-assessment and final report also provide information that would be useful to States
planning to replicate the demonstration. The Illinois demonstration staff created a rich description
from which one can learn what activities they performed as part ofthe demonstration and when they
KRA Corporation 33
occurred. Staff spelled out the demonstration activities and detailed results of their activities. The
final report, however, included few statistics and quantitative aspects of the demonstration.
Hiring the right staff to implement the project, and obtaining the full support of the State and local
level administrators are the critical components of this demonstration. The complete demonstration
can only be replicated effectively in a State willing to meet these criteria. However, major portions
of this demonstration could be replicated by local agencies that have the authority to implement
changes such as creating or changing forms, or providing training on identified problem areas.
A replication challenge is recruiting the right staff to conduct the case and procedural reviews. The
Illinois demonstration staff had a combination of strong interpersonal skills and FSP substantive
knowledge. For those localities that plan to replicate this model, this combination of skills may be
difficult to assemble and may not readily be available in all locations.
Lessons Learned
DPA staff do not expect to continue this project. However, Illinois staff realize that the progress
they made while conducting this demonstration may disappear when staff attention is focused on
other priorities and programs. They expect to continue to implement the recommendations made
to reduce errors. More importantly, they plan to keep the communication lines open so staff can
keep administration informed about what tools are needed to effectively implement and improve FSP
policies and procedures.
KRA Corporation 34
APPENDIX
u
Appendix A
Food Stamp Program Data
by Fiscal Year
Table 1—Payment Error Rates
Combined Paymtnt Error Rate* Overpayment Error Rates
mJlamnyiiwannrat Wrote U.S. Avaraga Maryland IHrnote U.S. Avaraga
1961 16.68 11.47 12.40 14.22 8.50 9.90
1982 11.33 10.95 11.99 9.70 8.93 9.54
1963 9.31 9.63 10.77 7.12 7.23 8.32
1984 8.32 11.23 10.91 6.99 8.31 8.59
1965 8.75 10.58 10.50 7.32 8.16 8.27
1966 10.10 11.10 10.39 6.06 9.17 8.09
1987 9 38 11.03 10.25 7.27 8.74 758
1968 8.62 10.52 993 6.45 8.37 7.41
1989 10.07 10.28 981 7.99 8.17 7.27
1990 10.64 10.77 9.61 8 34 8.65 7.34
1991 9.00 9.85 9.30 6.96 7.23 6.96
1992 8 99 9.95 10.68 6.99 7.55 8.19
1993 10.71 10.20 10.81 7.80 7.90 8.27
1994 11.24 9.47 10.32 848 6.96 7.65
1995 12.09 11.70 9.72 9.66 9.49 7.30
Source: Official Error Rates, U.S. Department of Agriculture, November 1995 and Fiscal Year 1995 Error
Rate Data, U.S. Department of Agriculture, Jury 1996.
KRA Corporation
%
Appendix A-1
Table 2—Maryland Food Stamp Program Data
Year Average Number
of Participants
Average Number
of Households
Total
Benefits
1981 346.283 145,356 170,654,259
1982 321,470 130,933 163,407,300
1983 318,497 126.011 177,509,962
1984 301,952 119,987 168,778,418
1985 286,816 114.950 171,382,240
1986 271,731 110.571 170,021.447
1987 253,674 104.136 159.187,825
1988 243,257 101.859 166,342,843
1989 248,814 106,209 175,873.999
1990 254,661 108.903 202.720,166
1991 303,684 129.813 258,979,379
1992 342,232 146,155 315,935,510
1993 374,522 159.059 336.182.307
1994 390.204 164,793 350.118.442
1995 398,727 169,440 365,169.777
Source: Food Stamp Program Information for FY1980 Through 1995, Data Base Monitoring
Branch, Program Information Division, Food and Consumer Service, U.S. Department of
Agriculture, April 1996.
KRA Corporation
2>1
Appendix A-2
Table 3—Illinois Food Stamp Program Data
Year Average Number
of Participants
Average Number
of Households
Total
Benefits
1981 984,399 380,057 505,963.521
1982 1,028,830 396,338 547.333.793
1983 1,117,872 424,897 662.825.687
1984 1,133,466 431,539 696,400,189
1985 1,110,242 430,246 713.131,942
1986 1.098,113 431,173 707.459,735
1987 1,079,356 429,714 702.546.528
1988 1,031,571 415,242 727.841.385
1989 989,500 402,126 729.311.679
1990 1,013,087 417,775 835.062,481
1991 1,096,441 460,331 961.130.472
1992 1,156,380 488,383 1.069.843.950
1993 1.179,488 493,429 1.080.091.166
1994 1,188,760 499,445 1.069.487.941
1995 1.151,035 487,586 1,056,482,588
Source: Food Stamp Program Information fbrFY 1980 Through 1995, Data Base Monitoring
Branch, Program Information Division, Food and Consumer Service, U.S. Department of
Agriculture, April 1996.
KRA Corporation i Appendix A-3
Table 4—United States Food Stamp Program Data
Year Average Number
of Participants
Average Number
of Households
Total
Benefits
1981 22,429.574 8,215,411 10,629,866,494
1982 21,717,398 7,906.747 10,208,297,750
1983 21,624,639 7.851.017 11,152.312.661
1984 20,853,631 7,593,926 10.696.130,140
1985 19,899,052 7,331,142 10,743,554,537
1986 19,429,101 7,213,147 10,605.196,145
1987 19,113,128 7,132.443 10.500,343,713
1988 18,644,192 7,055,303 11,149,051,131
1989 18,806,463 7,231,080 11,700,524,584
1990 20,066,750 7,795,796 14,186,724,695
1991 22,624.627 8.877,435 17,338,697,344
1992 25,405.615 10,059,659 20,905,665,401
1993 26.982,399 10,788.012 22,005,889,570
1994 27.468.779 11,088,962 22,746,355,319
1995 26.608,496 10.880,868 22,766,109,338
Source: Food Stamp Program Information tbrFY 1980 Through 1995, Data Base Monitoring
Branch, Program Information Division, Food and Consumer Service, U.S. Department of
Agriculture, April 1996.
KRA Corporation A Appendix A-4
Appendix B
Statistical Approach to Integrated Quality Control System (IQCS)
Error Rate Data
Dttmonftntjon Qfflctt ind Companion JMM
• Maryland
Demonstration Period: FY1994-I to FY19W-IV
• Demonstration offices (individually and collectively) compared to City ofBaltimore
minus demonstration offices
• Demonstration offices (individually and collectively) compared to Baltimore County,
minus City of Baltimore and demonstration offices
• Demonstration offices (individually and collectively) compared to State minus City
of Baltimore City, Baltimore County, and demonstration offices
• Illinois
Demonstration Period: FY1993-III to FY1994-IV
• Demonstration offices (individually and collectively) compared to Cook County
minus demonstration offices
• Demonstration offices (individually and collectively) compared to State minus
Cook County and demonstration offices
Meaaur— For Error Rata Data
(Formulas are for Case Error Rates; Dollar Error Rates are similar)
• Below
i» quarter
j = d, c (where d denotes demonstration office, and c denotes comparison area)
k = a, b (where a denotes institution of demonstration, and d denotes before
institution of demonstration)
• Overissuance Case Error Rate:
Number of QC Cues with Overpayment Error..
CE^= = w Number of QC Sample Ceiei^
KRA Corporation < Appendix B-1
Overpayment Dollar Error Rate:
DE
QC Sample Dollar Error
* Average Dollar Payment,.
Weighted Case Error Rate:
?(CV8*)
Estimated Change in Error Rate:
Pj = WCE^ - WCEj,
Standard Deviation of Change in Error Rate.
V> V1
Difference in Error Rate:
D = P,P.
Standard Deviation of the Difference in Err' r Rate:
•jtt.
Test to see whether error difference is significant at the 5 percent level:
. _D_ significant at the 5 percent level, where z <. -1.65
o ' significant at the 1 percent level, where z 4 -1.96
KRA Corporation H\ AppondixB-2
Where:
CE., = weighted case error rate
DEp = weighted dollar error rate
WCE^ = weighted case error rate
Sp ■ sample case load
P, = change in error rate pre to post
o. = standard deviation of P,
n, = Number of quarters
D = difference between Pd and Pt
oD = standard deviation of D
Z = Number of standard deviations of D
Tables 1 and 2 indicate the IQCS sample sizes for pre- and postdemonstration periods for demonstration and
comparison areas.
KRA Corporation
Hi-
Appendix B-3
Table 1—Maryland Quality Control Case and Payment Error Sample Sizes
Local Office
Case and Payment Error Sample Sizes
Pre-Demo1 Demo Period2
Clifton 140 38
Govans-Collington 265 82
Liberty-Garrison 121 30
All Demos 526 150
City of Baltimore
Less Demos
1.153 334
Baltimore County 289 83
State Less Baltimore City,
Baltimore County, and
Demos
1,485 447
Table 2—Illinois Quality Control Caae and Payment Error Sample Sizes
Local Office
Caae and Payment Error Sample Shea
Pre-Demo' Demo Period4
Lower North 145 62
Humbott Park 185 94
Northwest 243 103
All Demos 573 259
Cook County Less Demos 3,949 1,539
State Less Cook County and
Demos
2.560 1.093
1 Consists of 12 quarters of IOCS data.
2 Consists of4 quarters of IQCS data.
5 Consists of 10 quarters of IQCS data.
4 Consists of 6 quarters of IQCS data.
KRA Corporation a Appendix B-4