Retailer pre-authorization visit demonstration final summary report

              s»«r 'tew'*0** Aft-xfjt&ffr'
Cwwir
Officsof
Retailer Pre-Authorization
Visit Demonstration
Final Summary Report
March 1998
■KM
IISnA United States
Departmwtof
Agriculture
Food and
Nutrition
Service
Retailer Pre-Authorization Visit
Demonstration
Office of
Analysis and
Evaluation
FINAL SUMMARY REPORT
March 1998
Authors:
Christopher Logan
Julio Masker
John Blomquist
Submitted by: Submitted to:
Abt Associates Inc.
66 Wheeler Street
Cambridge. MA 02138
Project Director: Christopher Logan
Office of Analysis and Evaluation
USDA Food and Nutrition Service
3101 Park Cantor Drive, Rm. 214
Alexandria. VA 22302
Project Officer: Kan Offsrman
Thta «tudy was conducted under Contract No. 63 3198-8-016 with tfw Food end Nutrition Servtoo. UnMed
Deportment of AericuRure. under ttw authority of fne Food Stamp Act of 1977, es emended. Point* of
•urtod in tfra report do not nocoMorwy roprooont itw offfCMri position of tno Food end NutrrSon Sonflci.
\
RETAILER PRE-AUTHORIZATION VISIT DEMONSTRATION
FINAL SUMMARY REPORT
INTRODUCTION
The Food and Nutrition Service (FNS) of the U.S. Department of Agriculture
administers the Food Stamp Program (FSP), the nation's largest nutritional assistance program.
The purpose of the FSP is to improve the food-purchasing power of financially needy
households. The program provides low-income households with benefits in the form of food
stamp coupons or electronically-enroded cards that enable recipients to purchase eligible food
items at authorized retail food stores.
Background. In order to be eligible to accept food stamp benefits, retailers must meet
the eligibility criteria established by law, regulations, and policy, which are administered by
FNS. According to the current legislation governing the FSP, one of the requirements for a
store to be eligible is that it:
"sells food for home preparation and consumption and (A) offers for sale, on
a continuous basis, a variety of foods in each of the four categories of staple
foods, as specified in subsection (u)(l), including perishable foods in at least
two of the categories; or (B) has over SO percent of the total sales of the
establishment or route in staple foods."1
Retailers interested in participating in the FSP for the first time must apply to their
FNS Field Office and be approved before participating in the FSP. FNS periodically
reevaluates the eligibility of participating FSP stores, typically every three to five years. Field
Office staff review the application information, follow up with a phone call if necessary and, in
some cases, pay an in-person visit to the store. Due to limited resources, however, it has
become increasingly difficult for Field Office staff to visit stores.
In order to prevent ineligible retailers from participating in the FSP, the USDA budget
for FY 1997 includes funds to support FSP retailer authorization site visits. This money will
1 Public Law 102-225, section 201. Subsection 00(1) defines staple foods u including meat, poultry, or
fish; bread or cereal; fruit or vegetables; and dairy products.
Final Summary Report
be used primarily to fund store visits by contractors, who will provide the information gathered
to the FNS Field Office staff making the eligibility decisions.2
In the spring of 1996, with the expectation of the FY 1997 funding to support pre-authorization
store visits, FNS set out to determine the best way to manage these newly-acquired
resources. FNS also wanted to determine how to ensure that contractors provide
sufficient data to enable Field Offices to make sound, sustainable determinations of program
eligibility. Therefore, FNS conducted a large-scale, four-month-long demonstration, the Food
Retailer Pre-Authorization Visit Demonstration, in which contractor representatives visited over
7,000 stores on behalf of FNS.
Evaluation Objectives and Sources. This report summarizes the findings of an FNS-sponsored
evaluation of the demonstration. In a competitive procurement, FNS selected Abt
Associates Inc. as the evaluator for the demonstration.3 The primary goal of the evaluation
was to determine how best to manage and allocate the additional resources that FNS expected
in FY 1997 for contractor-conducted pre-authorization store visits.
In order to answer the research questions posed by FNS, Abt Associates Inc. conducted
two rounds of on-site data collection activities between July 15 and September 30, 1996.
Evaluation staff interviewed FNS staff at the Field and Regional Offices, and contractor
personnel at the corporate and field levels. In addition, evaluation staff observed store visits
conducted by FNS Field Office staff and by contractor personnel.
Special tracking forms created for the demonstration and other FNS administrative
records were collected from the 16 Field Offices that participated in the demonstration. The
database from these sources was used to analyze the following: numbers of new applications
and reauthorization applications for which the subject stores were visited by contractors or FNS;
timeliness and usefulness of contractor reports; FNS actions to gather additional data from
contractor-visited stores; and outcomes of the application/reauthorization process (i.e., approval,
denial, or withdrawal).
2 FNS issued a Request for Proposals (RFP FNS 97-003LMW) to procure these services on February 14,
1997.
1 USDA Food and Nutrition Service Contract No. 53-3198-6-016.
Final Summary Report
Organization of the Report. The remainder of this report is organized in three parts.
First, the report describes how FNS organized and operated the demonstration. Second, the
results of the demonstration are presented, including findings based on the interviews with FNS
and contractors, the tracking forms, contractor cost information, and the documentation of
appeals of decisions to the FNS Administrative Review Branch. The final section presents the
conclusions from the evaluation, combining the insights of FNS and contractor representatives
with those of the evaluation team. Appendix A presents supplemental information on the
sampling and analysis of the tracking forms.
DESCRIPTION OF THE DEMONSTRATION
The demonstration involved the efforts of various staff at FNS headquarters, Regional
Offices, and Field Offices. Headquarters staff provided contract support and guidance. Each
Regional Office designated a contracting officer and a contracting officer's representative
(COR), with backup for the COR. Sixteen Field Offices, spread over all seven FNS regions
across the nation, participated in the demonstration. Taken together, these Field Offices were
responsible for nearly one-third of all authorized retailers in the nation. The sites included three
large Field Offices, with over 5,000 retailers each; eight medium-sized offices, with 3,000 to
5,000 retailers each; and five small offices, with fewer than 3,000 retailers each. Through
competitive procurements, the Regional Offices contracted with private firms to work in the
territories covered by these Field Offices.
Demonstration planning activities began in late 1995, culminating in the release of a
Statement of Work (SOW) by each Regional Office in April 1996. After the contractors were
selected, the Regional and Field Offices trained contractor representatives in the demonstration
procedures. The contractors began visiting stores in June 1996.
Nine firms were awarded contracts and participated in the demonstration. Six of the
contractors served two or more sites. The contractors' fees, based on their competitive bids in
response to the regional SOWs, ranged from $25 to $125 per visit. In most sites multiple
contracts were awarded, but visits were assigned to the contractor with the lowest bid for the
site unless that firm failed to perform as required.
Each of the 16 Field Offices ordered contractor store visits for both new authorizations
and reauthorizations on an as-needed basis. Field Offices had the flexibility to establish the
Final Summary Report
criteria for which stores were assigned to the contractors, based on type of store, geographic
location, and other considerations. The variety of visits ordered ranged from one site's special
focus on stores with patterns of excessive FSP redemptions to sites where most stores applying
for authorization or reauthorization were visited, occasionally including supermarkets. Most
Field Offices issued separate orders for new authorizations, which had deadlines for contractor
reports of 7 to 10 days, and for reauthorizations, which had contractor deadlines of 10 to 30
days. (The contractor deadlines varied from region to region, depending on the terms of the
region's SOW.) Most Field Offices geographically grouped the stores in their orders to some
degree, in order to reduce the contractors' travel time, but low numbers and geographic
dispersion of the available stores to be visited limited or precluded geographic clustering in
several sites. During the demonstration all but four Field Offices continued to send their own
staff to visit some stores, most often when stores required special attention (e.g., because of
known concerns about the store's eligibility). On rare occasions FNS staff made followup visits
when the information from contractor visits was inconsistent or inconclusive.
Contractors made in person store visits in which, after receiving the consent of a store
representative, they completed a checklist of the food inventory and took photographs (including
the staple food stock, the check-out area, and the store exterior). The procedures for the
inventory checklist and photographs were modeled in part on approaches used by FNS staff, but
the demonstration contractors took substantially more photographs (at least 12 per store) than
has been the practice of FNS. (FNS staff typically took photographs only to document stores'
deficiencies, but contractors were expected to photograph foods from all four groups of staple
foods regardless of how well-stocked the store was in each food group.) In the Western
Region, contractors were also required to sketch the store's layout. The contractors then
submitted reports, comprising a cover page, the checklist, the photographs, and (in some
regions) supplemental information, to the FNS Field Offices. Contractors spent an average of
95 minutes per store on preparations, conducting the visit, and producing the report; the typical
in-store time was 1S-2S minutes.
The FNS Field Offices reviewed the information provided by the contractors, both to
determine the acceptability of the contractor reports and to assess what the reports indicated
about the subject stores' eligibility. Most Field Offices conducted separate quality control
checks on the contractors' reports before using them to determine the stores' eligibility. The
Final Summary Report
most common problems detected by these reviews were missing or unclear photographs and
inconsistencies between the checklist and the photographs. Most Field Offices provided prompt
feedback to the contractors regarding deficiencies in the reports, either in writing, by telephone,
or both.
The Field Offices were almost always able to decide retailer eligibility using the
contractor reports. FNS made very few follow-up visits to stores visited by the contractors.
About half of the Field Offices viewed the photographs and the checklists as equally important
to their eligibility decisions, but the other half rated the photographs as more important. A
major challenge was the need to review each store on its own merits, because the FSP's
Criterion A for retailer eligibility (ample variety of staple foods) does not provide quantitative
standards for staple food inventory. An additional challenge was the need to assess the store's
potential eligibility under Criterion B (over 50 percent of total sales in staple foods).
DEMONSTRATION RESULTS
Basic Statistics. When the authority to commit demonstration funds expired on
September 30, 19%, the Field Offices had ordered a total of 7,232 contractor visits, as shown
in Exhibit 1. The number of visits ordered ranged from 173 in the smallest site (Wichita) to
1,111 in the largest (Los Angeles). The average cost across all sites and visits was $70 per
visit; the average cost in individual sites (reflecting combinations of two or more contractors in
the sites that had turnover) ranged from $30 in Boston to $118 in New York City.4
Field Office Experiences. All but three of the Field Offices experienced significant
problems with contractor performance at some point in the demonstration, but most of the
difficulties were in the early months. The principal problems were late, incomplete,
inconsistent, or poorly-presented reports; refusal of work assignments was a less common
problem that was important in a few sites. Nearly all of the early performance problems were
resolved by the fourth (and last) month of the demonstration, either through improved effort by
the original contractors or by replacing them with better-performing (albeit more expensive)
contractors. A substantial minority of the Field Offices (38 percent) had to change contractors
4 The range of die final avenge cost differs from the $25 to $125 range of bids by participating
contractors, because each of the sites with die lowest and highest bids (Detroit and New York, respectively)
used two or more contractors with different fees over the course of the demonstration.
Final Summary Report
Exhibit 1
ACTUAL CONTRACTOR VISITS ORDERED BY SITE
1 Region/Field Office
Total Number of
Retailers'
Store Visits
Ordered Total Spending* Cost per Visit*
Mid-Atlantic Region
Trenton 4,010 568 $40,179 $71
Harrisburg 3,138 243 20,431 84
Northeast Region
New York City 9,788 974 115,406 118
1 Wallingford 1,748 236 8,140 34
Boston 3,782 297 8,901 30
Southeast Region
Tampa 4,200 349 11,639 33
Raleigh 6,442 383 35,079 92
Midwest Region
Detroit 3,618 407 35,827 88
Grand Rapids 3,361 525 31,284 60
Southwest Region
Austin 4,887 727 39,985 55
Little Rock 2.461 522 29,993 57
Mountain Plains Region
Denver 1,788 236 16,237 69
Wichita 1,393 173 11,929 69
Western Region
Los Angeles 8,497 1,111 65,175 59
Sacramento 3,418 291 23,037 79
Phoenix 2,185 190 13,343 70
TOTALS 64,716 7,232 $506,585 $70
* Retailer counts as reported in the SOW for the demonstration.
" Cost per visit represents the average foe for all visits as ordered. Final cost per visit may have been lower to tLc extent
that penalties for late deliverables were assessed.
Final Summary Report
at least once. Sixty-one percent of the Field Offices gave ratings of "moderately high" or
"high" satisfaction to the contractors they were using near the end of the demonstration.
Seventeen percent rated their level of satisfaction as "moderate," and 22 percent said that then-level
of satisfaction was "low" to "moderately low."
Most Field Offices felt that the additional steps in the retailer authorization and
reauthorization process during the demonstration affected how they allocated staff time. Some
offices spent more time overall on retailer management activities, relative to other Field Office
duties.1 Others reported reductions in the proportion of their retailer management time devoted
to monitoring retailer participation (such as reviewing redemption reports). Some offices
reported born of these effects.
Fieid Office staff shared several widespread views of the demonstration experience:
• Field Office staff saw the demonstration as reducing FNS time devoted to store
visits, but increasing the overall number of visits conducted.
• Sites that experienced significant problems with contractor performance stressed the
added workload associated with correcting this situation.
• The compressed schedule of the demonstration and the record-keeping for the
evaluation also contributed to the perceived workload effects.
• Contractors were seen by Field Office staff as an important resource, particularly
for covering territories with sparse, far-flung retailer populations.
• Field Office staff believed, however, that contractor visits are, in some ways, less
useful than FNS staff visits. FNS staff can answer retailer questions, spot signs of
fraudulent behavior, verify ownership, and make preliminary authorization
decisions on the spot.
Nearly two-thirds of the Field Offices substantially underspent their allocations for the
demonstration (i.e., ordered two-thirds or fewer of the number of visits projected at the outset,
based on the available funds and contractor fees). The reasons for the shortfall included Field
Office staff shortages, competing priorities (such as the FNS "sweep" of a large number of
5 The general category of FSP retailer management includes the following activities: authorization and
reauthorization of retailers to participate; providing training and technical assistance to retailers regarding FSP
rales and procedures; monitoring retailers' patterns of redemption activity to detect store closures and possible
program violations; and sanctioning retailers for program violations.
Final Summary Report
stores in one site), and lost time due to problems with contractor performance. The shortfall
was partially offset by the four Field Offices that ordered more than the originally-projected
number of visits, because of lower-than-expccted contractor costs or availability of reallocated
funds from the sites that underspent their allocations.
Outcomes of Contractor and FNS Store Visits. As part of the evaluation, Field
Office staff completed Initial Tracking Sheets for all new authorizations and reauthorizations
processed during the demonstration, and Follow-Up Action Reports on all contractor-visited
stores. The evaluators analyzed the store visit tracking data contained in a sample of these
forms completed during the demonstration period. The sample is composed of 2,194 store
visits, 1,934 of which were conducted by contractors, with the remaining 260 visits conducted
by FNS. The sample includes data from all 16 demonstration Field Offices and nine
contractors.6
Overall, the tracking data indicate that contractors successfully conducted retailer
visits on behalf of FNS. Nearly 99 percent of visits were completed. Exhibit 2 summarizes
the outcomes of the completed contractor visits, overall and by whether the visit was for an
initial authorization or reauthorization. Work was delivered to FNS on a timely basis in 90
percent of the visits. Information from contractor reports contributed to or supported
authorization or reauthorization decisions in 96 percent of cases (i.e., without a follow-up
contact visit by FNS).
There is considerable variability in outcomes, however, by type of visit. The second
column of Exhibit 2 shows that contractor work was delivered on a timely basis more frequently
for reauthorization visits than for new authorization visits—92 percent vs. 81 percent. This
statistically significant difference7 is not altogether surprising, because the time allotted by FNS
for delivery of new authorization visit reports was almost always less than for reauthorizations,
and new authorization visits tended to be more geographically dispersed than reauthorization
visits because they could not be grouped in batches to ease scheduling. Both characteristics
* For further information on the sample and weighting methodology, tee Appendix A.
7 Throughout mis report we say (hat a difference between two means estimated from sample data is
"statistically significant" if there is a 10 percent or smaller chance mat the true difference in the population
is actually zero.
Final Summary Report
Exhibit 2
OUTCOMES OF CONTRACTOR STORE VISITS, BY TYPE OF VISIT
Visit Type
Work
Delivered
on Timely
Basis
Requested
More
Information
from
Contractor
Authorization
Decision Based
on Visit
Store
Approved or
Reauthorized
Store Denied
or
Withdrawn
All contractor visits 89.9% 3.7% 95.7% 92.2% 7.8%
New authorizations 81.0" 6.8* 95.3 87.2" 12.8"
Reauthorizations 92.3' 2.8' 95.8 93.7" 6.3'
SOURCE: Initial Tracking Sheets and Follow-up Action Reports for visits in the analysis sample.
' Difference between new authorizations and reauthorizations significant at the 0.10 level.
made it more difficult for contractors to turn in new authorization deliverables quickly and
efficiently.
The fifth column in Exhibit 2 indicates that store reauthorization rates were over six
percentage points higher than approval rates for new stores, a statistically significant difference.
Also, as displayed in the third column, FNS was more than twice as likely to request that
contractors provide additional information or repeat visits to stores following new authorization
visits than following reauthorization visits (in 7 percent vs. 3 percent of cases, again a
statistically significant difference). FNS made such requests, as authorized under the SOW,
when the contractors failed to provide complete and consistent information. The results are
consistent with evidence from the process interviews that Field Offices subjected new
authorizations to a higher level of scrutiny than reauthorizations, rosing a greater need for
complete and reliable data on the store in question. Stores that are applying for reauthorization
have already passed through the initial authorization process and require less verification. FNS
was no less likely, however, to be able to make an authorization decision using information
from the contractor visit in new authorization cases than in reauthorization cases.
The outcomes of FNS store visits, by type of visit, are compared to those of contractor
visits in Exhibit 3. It should be noted that the sample of FNS visits is much smaller than the
sample of contractor visits (260 FNS visits versus 1,934 contractor visits). With this in mind,
the sample indicates that FNS visits also led to high rates of authorization decisions, both for
Final Summary Report
new store applications and for reauthorizations, and high rates of approval or reauthorization.
Over 97 percent of FNS visits provided sufficient basis for an authorization decision, and just
over 95 percent of FNS visits resulted in approval or reauthorization. The rate of authorization
decisions based on FNS visits was slightly higher among new authorizations than
reauthorizations, whereas the approval/reauthorization rate was higher among reauthorizations.
Neither difference, however, is statistically significant.
Exhibit 3
OUTCOMES OF CONTRACTOR AND FNS STORE VISITS BY TYPE OF VISIT
Visit Type
Percent of Total
Visits
Authorization Decision
Based on Visit
Store Approved or
Reauthorized
Contractor FNS Contractor FNS Contractor FNS
All 93.2% 6.8% 95.7% 97.2% 92.2% 95.2%
New authorization 84.0 16.0 95.3 97.5 87.2* 94.8»
Reauthorization 96.0 4.0 95.8 96.5 93.7 96.1
SOURCE: Initial Tracking Sheets and Follow-up Action Reports for visits in the analysis sample.
" Difference between contractor and FNS visits significant at the 0.10 level.
Exhibit 3 allows direct comparison of the outcomes of visits conducted by contractors
with those conducted by FNS. As shown in the last two columns of the exhibit, overall
approval/reauthorization rates were slightly higher among FNS-visited stores (95 percent) than
among contractor-visited stores (92 percent). In addition, FNS had a slightly higher proportion
of its visits that directly resulted in authorization decisions (97 percent for FNS visits versus 96
percent for contractor visits). Although the differences between contractor and FNS visits on
these overall outcomes are not statistically significant, a different pattern emerges when new
authorizations and reauthorizations are examined separately.
Among all new authorizations, approval rates were significantly higher among visits
conducted by FNS (95 percent) compared with contractor visits (87 percent)—a more than 7
percentage point difference. Among all reauthorizations, there were no significant differences
in approval rates or in the percentage of authorization decisions based on store visits without
followup between contractor and FNS visits. Thus, approval rates for new contractor-visited
stores were significantly lower than for either new FNS-visited stores or contractor-visited
10
Final Summary Report
stores applying for reamhoriwtion. All other approval rates were not significantly different
from each other.
One possible explanation for the lower approval rate for new stores when visited by
contractors (compared with stores visited by FNS) is that the contractor visits provided a basis
for more systematic and objective comparisons among stores, thereby making clearer the
differences between eligible and ineligible stores. As the preceding description shows, the
process of deciding whether to authorize a new applicant retailer when a contractor visited the
store was different in several important ways from the process when FNS visited the store. For
the contractor-visited stores, FNS had access to a comprehensive set of photographs, in addition
to the food inventory checklists (which also were completed by FNS when its staff made visits).
The SOW provided a structure to ensure consistency in the format and scope of the information.
Because the information on the stores was delivered to the Field Offices, multiple FNS staff
(particularly for both program specialists and the Officers-in-Charge) could review and discuss
the stores. The fact that contractors visited stores that were too distant for FNS staff to visit
may also have contributed to the lower approval rate, if more distant stores are for some reason
less likely to pass FNS' tests of eligibility.
An alternative explanation is that the observed difference in approval rates reflects
underlying differences in approval rates between Field Offices regardless of who conducts the
visit. The samples of contractor-visited and FNS-visited new stores were distributed differently
across Field Offices. The Field Offices that conducted the most visits with their own staff were
more likely to approve new stores whether the visit was conducted by FNS or a contractor. On
the other hand, two of the three Field Offices with the most contractor visits had approval rates
of 72 percent or less for new contractor-visited stores, driving down the average for all new
contractor-visited stores. The differences in approval rates across Field Offices may reflect
differences in the retailer applicant pool, or in the relative willingness to risk having decisions
reversed in the administrative review process, or a combination of these and other factors.
The limitations of the demonstration design (especially the non-random way that visits
were assigned to contractor versus FNS visits) m-'-e it impossible to determine conclusively
which of the preceding reasons contributed most to the difference in approval rates. It is
therefore uncertain to what extent this result is likely to be replicated when contractor visits are
used on a larger scale. It is important to note, however, that both contractor and FNS visits
11
Final Summary Report
presumably reduced the number of ineligible retailers entering the FSP, because the applicants
that could be denied without a visit were generally screened out before stores were selected to
be visited. The demonstration results suggest that contractor visits can be at least as effective
as FNS visits at detecting stores that should be denied authorization to accept food stamp
benefits.
Other findings from lie analysis of store visit tracking data include:
• Field Offices differed greatly in the mix of type of visit (new vs. reauthorization);
• all Field Offices had the majority of visits conducted by contractors;
• most FNS visits were conducted at stores applying for new authorization, although
some variation is evident across Field Offices;8 and
• among Field Offices, the approval rates for contractor-visited stores ranged from
two sites below 85 percent to five sites with approval rates of 100 percent, and the
approval rates for new contractor-visited stores ranged from 81 to 100 percent.
Administrative Review. The denials of new applications and withdrawals of
previously authorized stores can, like all adverse actions against retailers, be appealed to the
Administrative Review Branch of FNS. Upon request by the retailer, the case is assigned to an
Administrative Review Officer (ARO), who considers the appropriateness of the decision in
light of the applicable laws, regulations, and policies, the retailer file supplied by the Field
Office, and any additional information provided by the retailer. The decision of the ARO to
sustain or reverse the Field Office's action represents FNS' final decision on the case;
involuntary withdrawals can, however, be appealed to the US District Court for judicial review.
Among the decisions based on contractor reports, about one-fifth (21 percent) of the
denials of new applications and more than one-quarter '28 percent) of the decisions to withdraw
retailers from the FSP were appealed.9 Of those denials that were appealed, 63 percent were
* The Field Offices with relatively high proportions of FNS visits conducted at stores applying for
reauthorization also tended to have more FNS visits of all kinds (relative to the number of contractor visits).
These offices were more likely than others to assign stores to FNS visits based on location or marginal
eligibility.
' These estimates are based on data supplied by the Field Offices and on exception reports of reversed
decisions prepared by the Administrative Review Branch.
12
Final Summary Report
sustained by the Administrative Review Branch, and 37 percent were reversed.10 Thus,
approximately 8 percent of all denials were reversed. For withdrawals, the story is much the
same: 60 percent of the withdrawals were sustained by administrative review, whereas 40
percent were reversed. When the higher appeal and reversal rates for withdrawals are
combined, however, about 11 percent of all withdrawals were reversed—a modest rate but
almost half again as high as the 8 percent rate for denials. These overall averages reflect the
substantial influence of one large Field Office with especially high rates of denials, withdrawals,
appeals, and reversals. If this Field Office is excluded, the reversal rate for all denials drops
from 8 to 3 percent, and the reversal rate for all withdrawals drops from 11 to 4 percent.
Reversal rates for denials and withdrawals based on FNS visits in the demonstration
sites were much lower, but this difference should not be overemphasized, for two reasons. The
FNS visits were to a quite select group of stores, and relatively few negative actions arose from
these visits (leading to a substantial probability that the difference in reversal rates was due to
chance). FNS tracks the incidence of denials and withdrawals but does not track the rates of
appeals and reversals for these actions, so there is no baseline against which the demonstration
experience can be compared.
There were three common reasons for reversals of Field Office decisions based on
store visits by contractors and FNS staff:
1. The ARO found that the store, although wonsidered ineligible by the Field Office,
met the minimum standards set by the most recent FNS policy memorandum (dated
January 31, 1996). This policy memorandum indicates that "prudent judgment"
should be used in making determinations on marginal stores. This reason was cited
in 58 percent of all reversed cases examined for this study. Among cases involving
contractor visits this reason was cited 'n 81 percent of reversals."
10 Appeals withdrawn by the retailer are included in the count of sustained decisions; actions rescinded
by the Field Office are included in the reversals.
11 Exception reports explaining reversals were examined for 112 cases reviewed during the demonstration
period, of which 21 were identified as invmving contractor visits. More than one reason could be cited, so
percentages in this analysis sum to more than 100.
13
Final Summary Report
2. The Field Office did not provide sufficient documentation for the basis of its
determination, either in the store's file or in communications with die retailer. A
Field Office must document the specific grounds for a decision made on the basis
of a contractor store visit. This reason was cited in 21 percent of all reversals, but
in only 10 percent of cases involving contractor visits.
3. The Field Office did not adequately review the retailer for qualification under both
Criterion A (ample variety of staple foods) and Criterion B (over 50 percent of total
sales in staple foods), or the ARO disagreed with the Field Office's determination
regarding eligibility under Criterion B. This reason was cited in 20 percent of all
reversals but only 10 percent of cases involving contractor visits.
Administrative Review Branch representatives indicated that well-prepared, consistent
contractor reports contributed to the sustaining of Field Office decisions. The lack of
photographs and checklists from some FNS visits outside the demonstration contributed to the
reversal of some of the decisions based on those FNS visits. The AROs relied heavily on the
photographs provided by the contractors, in some cases citing inconsistencies between the
inventory checklist and the photographs as grounds for reversal. The positive response of the
AROs to the use of contractor-provided photographs helped encourage the Field Offices to see
photography as a very effective way to substantiate eligibility decisions.
Contractor Costs. Using data from interviews with contractors and other available
sources, the evaluation produced independent estimates of contractors' costs to conduct store
visits. The estimated average total cost per visit incurred by the contractors was about $32,
substantially below the weighted average fee of $70 per visit paid to the contractors.12 The
range of the evaluation's estimated contractor costs was from $30 to $93. Field labor costs
(including preparation for visits, travel time, in-store time, and report preparation) accounted
for about 55 percent of total contractor costs; field non-labor costs (mainly travel expenses)
accounted for 20 percent; and management costs (mainly labor) represented about 25 percent.
Across the sites, variations in average travel time and store visit volume were the most
important influences on the estimated contractor cost. Travel time was the most variable
component of the field labor cost per visit. Much of the management cost for a given site was
a fixed cost associated with contracting and other startup activities, so the management cost per
12 The contractor cost estimate includes wages, fringes, direct expenses and overhead, but not profit The
weighted average fee paid to the contractors, as cited earlier in this report, does not account for penalties for
late or unacceptable reports, for which data were not available.
14
Final Summary Report
visit generally declined with visit volume. As a result, contractor costs were lowest in largely
urbanized, high-volume sites and highest in sparsely-populated, low-volume sites. Analysis of
the effect of different assumptions regarding pay rates for field staff confirmed that the pay rate
is an influential variable, posing a clear trade-off between staff qualifications and costs.
CONCLUSIONS
The demonstration showed that using contractors to visit stores is viable and can work
very well. This success is highlighted by the following key results:
• Field Offices found this resource useful enough to order 85 percent of the
originally-projected number of visits.
• By the end of the demonstration, after unsatisfactory contractors had been replaced
in 38 percent of the sites, over three-quarters of the Field Offices were moderately
to highly satisfied with contractor performance.
• Despite the unfamiliar challenges of using contractors' reports, over 60 percent of
the sites rated the quality of contractor deliverables as good or better; a similar
percentage of sites rated contractors as generally or consistently on time.
• In the tracking analysis sample, 90 percent of contractor deliverables were timely
and 96 percent did not require correction by the contractor.13
• Contractor visits provide sufficient information in the vast majority of cases: 96
percent of contractor visits enabled the Field Office to make an authorization or
reauthorization decision without further information-gathering (other than routine
documentation).
• Contractor visits can and do lead to the exclusion of ineligible stores from the FSP.
The evaluation data indicate that Field Offices took negative actions (denial or
withdrawal) against 8 to 9 percent of contractor-visited stores; the estimated total
was 649 stores.
13 The fact thai about 60 percent of the Field Offices rated die overall quality and timdinrss of contractor
deliverables as good or better may seem inconsistent with the tracking data indicating that over 90 percent of
individual contractor deliverables were timely and did not require correction. These two sets of data,
however, are not comparable. The ratings given by the Field Offices reflect their complex and lometimes
subjective expectations regarding what contractors should do to meet their needs, above and beyond the
minimum standards of compliance with the SOW. The tracking data reflect performance only relative to
those more objective but lower standards for which data were collected.
15
Final Summary Report
• Negative Field Office actions based on contractor visits were, in about 90 percent
of cases, accepted by the retailer or upheld upon appeal.
On several of these indicators, there was considerable variability across sites. Some
contractors substantially exceeded contractual requirements, whereas others were unable to meet
those requirements and had to be replaced. A minority of Field Offices exceeded their
projected number of visits ordered, whereas the majority substantially underutilized this
resource. Rates of approval or reauthorization for individual Field Offices (which are subject
to some sampling error) varied considerably as a result of the differences in contractor report
quality combined with variations in approaches to authorization decisions.
The contractors in the demonstration proved that they could effectively serve as FNS'
"eyes" and credibly represent the FSP. Contractors can be a valuable extension of FNS' reach,
especially in more distant areas.
Lessons Learned. Exhibit 4 presents a list of the best practices employed by
contractors and Field Offices. This list reflects the observations of FNS staff, contractors, and
the researchers. In each section of the exhibit the best practices begin with startup activities,
progress through the issuance and completion of call orders, and end with the submission and
review of deliverables. These practices include approaches, techniques, and tools that
contributed to smooth, efficient, and effective operations. They are worthy of consideration as
FNS proceeds to implement store visit contracting on a larger scale, although they may require
adaptation to other contractors' or Field Offices' circumstances.
FNS has incorporated these and other lessons from the demonstration into the Request
for Proposals (RFP) for the FY 1997 store visit contracts and other preparations for the future
use of contractors to conduct store visits. Both FNS and the future contractors will benefit from
the following advances:
• Better information about FNS'plans and expectationsfor contractor store visits.
The FY 1997 RFP contained much more information on the timing and location of
planned store visits, and on their likely distribution between new authorizations and
reauthorizations, than was available to the demonstration contractors. This
information should help ensure that contractors are better prepared to meet FNS'
needs.
16
Final Summary Report
Exhibit 4
BEST PRACTICES
Best Practice* by Contractor*
• Thorough training by contracting organization, which includes a field component
• Good working relationship with the Field Office
• Automated tracking system
• Mapping software that plots store locations and assists in planning travel routes
• Photographs that are sufficient in number and content to permit an adequate assessment of the
staple food stock, as well as overall sense of the store
• Methods to correctly match photos with stores, such as using a store identification placard in
photos or one roll of film per store
• Computer software that generates labels for photographs and mailings
• In-house quality control process dial identifies problems prior to submission of reports to FNS
• Neat, professional, flat deliverables that are organized in a consistent fashion
Best Practices by Field Offices
• Identification of a store visit coordinator
• Adequate training and tools (such as checklist) for staff to ensure consistency in die approach to
assessing the adequacy of deliverables and making eligibility decisions
• Detailed and thorough training of contractors by Field Offices
• Communicating clearly the expectations for Field Office needs and contractor performance
• Automated process for issuing and tracking call orders
• Regular issuance of call orders; steady and adequate flow of work, negotiated by Field Office
and contractor
• Geographic grouping of stores in call orders
• Prompt review of contractor deliverables
• Ongoing feedback to contractors on the quality of deliverables and issues/problems as they arise
17
Final Summary Report
• More emphasis on trainingfor reviewers employed by contractors. As part of the
demonstration evaluation, a guidebook for contractors has been produced,
explaining how to conduct visits and providing samples of photographs, survey
forms, checklists, and store sketches. Training for the FY 1997 contractors will be
more formalized, including sessions at the national and regional or field level. To
remove disincentives to training, contractors will be paid separately for training
costs, rather than having to recover those costs from fees for conducting store
visits.
Changes in rules and procedures for ordering visits to increase efficiency. Field
Offices will be able to issue larger orders for store visits, both to reduce ordering
time and to give contractors a better flow of work. The plans for FY 1997 put
more emphasis on reauthorization visits, which can be conducted more efficiently
through geographic clustering and advance scheduling.
Changes in rules and procedures for conducting visits to elicit more consistent
contractor performance. The FY 1997 RFP spells out more clearly the
expectations for contractor deliverables, including a specific requirement for at least
17 photographs. Contractors are required to employ field coordinators to facilitate
communications with the FNS Regional and Field Offices, and to propose and
follow clear plans for training reviewers. Contractors will be allowed an initial
trial period of 30 visits per region, after which they will be subject to liquidated
damages of up to $200 per deliverable.
Changes in the format of deliverables to facilitate use by Field Offices. The
staple food checklist will be modified so that Field Office staff can more easily
determine the extent of variety in the store's inventory in each staple food group.
A store layout sketch, as used in the Western Region during the demonstration, will
be used to provide better information on the nature of the business and the stock of
non-staple foods and non-food products.
Realigning responsibilities for administering store visit contracts. To ensure
consistent contract terms and to better target agency resources, FNS has issued a
single RFP for the FY 1997 procurement, and the contracts will be administered at
the national level, with technical support from the Regional Offices. The Field
Offices will have less responsibility for contract administration and will be able to
focus more on using contractor visits to accomplish their ultimate objective:
screening out ineligible stores from the FSP.
18
APPENDIX A
TRACKING DATA SAMPLE AND WEIGHTING METHODOLOGY
During the four months of the demonstration, FNS Field Office staff completed up to
two evaluation forms for each new applicant store and each store subject to reauthorization.
The Initial Tracking Sheet (ITS) was completed for all stores, regardless of assignment status.
For each store, the ITS records tracking information, assignment status (no visit, contractor
visit, or FNS visit), and (for contractor-visit stores) contractor name and assignment date. For
stores visited by FNS, it records whether additional information was requested from the retailer,
the final outcome (approval or denial/withdrawal), and the final action date. For no-visit stores,
the final outcome is recorded, along with the date of the action.1
In addition to the ITS, a Follow-up Action Report (FAR) was completed following
administrative action for all contractor-visited stores. The FAR records, from the viewpoint of
FNS staff, the outcomes associated with contractor visits, including timeliness of contractor
deliverables, completeness of contractor information and additional information requested,
action taken on contractor reports, whether a follow-up FNS visit was conducted, the final
administrative action taken (approval or denial/withdrawal), and the date of the action.
Analysis Sample and Weights
Because the sample generally represents the entire period of demonstration operation,
there should be few inherent biases in the data other than inevitable sampling error stemming
from the random process used to select store visits for analysis. We use weighted frequencies
in our analysis to make the sample representative of the known universe of visits. Contractor
visits are weighted so that die weighted sample represents the total contractor visits ordered by
each Field Office over the period of the demonstration, except in three sites where die weighted
sample reflects visits ordered through August 16.2 Individual weights for contractor visits were
determined separately for new and reauthorization visits in each Field Office; the weights were
computed as die total number of (new or reauthorization) visits ordered, divided by the number
1 This analysts does not include data on no-visit stores.
1 Delays in receipt of data from these sites limited the analysis to this time period.
A-l
fl
Appendix A: Tracking Data Sample and Weighting Methodology
of visits in the sample.3 The weighted sample is then equal to the total number of new and
reauthorization visits ordered. FNS visits are weighted to represent the total number of visits
in each Field Office for which Abt Associates received and batched ITS forms.4
The effect of these weights on relative sample sizes is shown in Exhibit A-1, where the
weighted and unweighted samples are compared by Field Office. References to numbers of
visits or percentages in the report apply to the weighted sample, unless otherwise indicated. In
interpreting the results, readers are cautioned to recall the size of the unweighted sample, which
determines the precision of the estimates. The entire sample of 2,194 visits allows fairly
precise estimates, but the much smaller samples for individual Field Offices yield much less
precise estimates.
1 The numbers of new and reauthorization visits ordered by a Field Office were obtained from semi-monthly
Regional Tracking Reports produced by FNS.
' The actual universe of FNS visits at any exact point during the demonstration is unknown. All 495 ITS
forms for FNS visits were batched, but only the 260 forms in die sample were processed and used in the
analysis.
A-2 dd
Appendix A: Tracking Data Sample and Weighting Methodology
Exhibit A-l
UNWEIGHTED AND WEIGHTED SAMPLES, BY FIELD OFFICE
Fldd Office
Total
Visits in
Sample
Percent of All
Visits
Weighted Total
Visits
Percent of
Weighted Visits
Trenton 287 13.1% 568 7.8%
Harrisburg 79 3.6 282 3.9
New York City 303 13.8 1112 15.3
Wallingford 60 2.7 246 3.4
Boston 133 6.1 358 4.9
Tampa 105 4.8 349 4.8
Raleigh 36 1.6 125 1.7
Detroit 112 5.1 422 5.8
Grand Rapids 129 5.9 571 7.9
Austin 114 5.2 619 8.5
Little Rock 127 5.8 646 8.9
Denver 119 5.4 236 3.2
Wichita 62 2.8 179 2.5
Los Angeles 438 20.0 1120 15.4
Sacramento 57 2.6 245 3.4
Phoenix 33 1.5 190 2.6
Total 2194 100% 7268 100%
A-3 <M