**ATTACHED ARE SOME RESOURCES
A PROGRAM I WOULD LIKE OF FOCUS IS Federal housing programs THAT HELP HOMELESS. THANKS.
To ensure the success of a program evaluation, a social worker must generate a specific detailed plan. That plan should describe the goal of the evaluation, the information needed, and the methods and analysis to be used. In addition, the plan should identify and address the concerns of stakeholders. A social worker should present information about the plan in a manner that the stakeholders can understand. This will help the social worker receive the support necessary for a successful evaluation.
To prepare for this Assignment, identify a program evaluation you would like to conduct for a program with which you are familiar. Consider the details of the evaluation, including the purpose, specific questions to address, and type of information to collect. Then, consider the stakeholders that would be involved in approving that evaluation. Review the resources for samples of program evaluations.
By Day 7
Submit the following:
- A 1-page stakeholder analysis that identifies the stakeholders, their role in the agency and any concerns that they might have about the proposed program evaluation
- A 2- to 3-page draft of the program evaluation plan to submit to the stakeholders that:
Identifies the purpose of the evaluation
Describes the questions that will be addressed and the type of information that will be collected
Addresses the concerns of the stakeholders that you identified in your Stakeholder Analysis
Program Evaluation
Studies
TK Logan and David Royse
A
variety of programs have been developed to address social problems such
as drug addiction, homelessness, child abuse, domestic violence, illiteracy,
and poverty. The goals of these programs may include directly addressing
the problem origin or moderating the effects of these problems on indi-
viduals, families, and communities. Sometimes programs are developed
to prevent something from happening such as drug use, sexual assault, or crime.
These kinds of problems and programs to help people are often what allracts many
social workers to the profession; we want to be part of the mechanism through which
society provides assistance to those most in need. Despite low wages, bureaucratic red
tape, and routinely uncooperative clients, we tirelessly provide services tha t are invaluable
but also at various Limes may be or become insufficient or inappropriate. But without
conducting eva luation, we do not know whether our programs are helping or hurting,
that is, whether they only postpone the hunt for real solutions or truly construct new
futures for our clients. This chapter provides an overview of program evaluation in gen –
eral and outlines the primary considerations in designing program evaluations.
Evaluation can be done informally or formally. We are constantly, as consumers, infor-
mally evaluating products, services, and in formation. For example, we may choose not to
return to a store or an agency again if we did not evaluate the experience as pleasant.
Similarl y, we may mentally take note of unsolicited comments or anecdotes from clients and
draw conclusions about a program. Anecdotal and informal approaches such as these gen-
erally are not regarded as carrying scientific credibility. One reason is that decision biases
play a role in our “informal” evaluation. Specifically, vivid memories or strongly negative or
positive anecdotes will be overrepresented in our summaries of how things are evaluated.
This is why objective data are necessary to truly understand what is or is not working.
By contrast, formal evaluations systematically examine data from and about programs
and their outcomes so that better decisions can be made about the interventions designed
to address the related social problem. Thus, program evaluation involves the usc of social
research meLhodologies to appraise and improve the ways in which human services, poli-
ci~s, and programs are co nducted. Formal eva l.uation, by its very nature, is applied research.
Formal program evaluations attempt to answer the following general ques tion: Does
the p rogram work? Program evaluation may also address questions such as the following:
Do our clients get better? How does our success rate compare to those of other programs
or agencies? Can the same level of success be obtained through less expensive means?
221
222 PART II • QUANTITATIVE A PPROACHES: TYPES OF STUD IES
What is the experience o f the typical client? Sho uld this prog ram be terminated and its
funds applied elsewhere?
Ideally, a tho rough program eval uation would address more complex questions in
three main areas: (1) Does the program produce the intended outcomes and avoid unin-
tended negative o u tcomes? (2) For whom does the program work best and un der what
conditions? and (3) Ilow well was a p rogram model developed in one setting adapted to
another setti ng?
Evaluation has taken an especially p rominent role in practi.ce today because o f the focu~
on evidence-based practice in social programs. Social work, as a profession, has been asked
to use evidence-based practice as an ethical obligation (Kessler, Gira, & Poertner, 2005).
Evidence-based practice is defined diLTerently, but most definit ions include using program
evaluation data to help determine best practices in whatever area of social programming is
being considered. In other words, evidence-based practice incl udes using objective indica-
tors of success in addition to p ractice or more subjective indicators of success.
Formal program evaluations can be found on just about every topic. For instance,
Fraser, Nelson, and Rivn rd ( 1997) h ave examined th e effectiveness of family preservation
services; Kirby, Korpi, Adivi, and Weissman ( 1997) have evalu ated an AIDS and preg-
nancy prevention middle school program. Mo rrow- Howell, Beeker-Kemppainen, and
Judy ( 1998) evaluated an interven tion designed to reduce the risk of suicide in elderl y
adult clients of a crisis hotline. Richter, Snider, and Gorey ( 1997) used a quasi-experimental
design to study the effects of a g roup work interven tio n on female sur vivors of childho od
sexual abuse. Leukefeld and colleagues ( 1998) examined the effects of an I IlV prevention
intervention with injecting drug and crack users. Logan and colleagues (2004) examin ed
the effects of a drug co urt in terven tion as well as the costs of drug co urt compared with
t he economic benefits of the drug court progra m.
Basic Evaluation Considerations
Before beginning a program eva luntion, several issues must be initially considered. These
issues are decisions 1 hat are critical in determining the evaluation methodology and goals.
Although you may not have complete answers to th ese qu estions when beginning to plan
a n evaluation, these ques tion s help in developing th e plan and must be answered before
a n evaluation ca n be carried out. We can 1.um up these considerations with the following
questions: who, what, where, when, and why.
First, who will do the evaluation? This seems like a simple question at first glance.
llowever, this particular consideration has major implications for the evaluation results.
P rogram evaluators ca n be categorized as being either in ternal or external. An internal
evaluator is someone who is a program staff member or regular agency employee, whereas
an external evaluator is a professional, on contract, hired for the specific purpose of evalu-
a tion. Th ere are adva ntages nnd disa dvan tages to using either type of evaluato r. For
example, the internal evaluator probably will be very familia r with the staff and the
program . This may save a lot of planning time. The d isadvnn tage is that eva luatio ns com-
pleted by an internal eva luator may be considered less valid by outside agencies, including
the funding source. The external evaluator gene rally is thought to be less biased in terms of
evaluation outcomes beca use he or she has no persona l investment in the program. One
disadvantage is that an externa l evaluator frequently is viewed as an “o utsider” by the staff
w ithin an agency. This may affect the amount of time necessar)’ to conduct the eva lua tion
or cause problems in the overall evaluation if agency staff are reluctant to cooperate.
CHAPTER 13 • P ROGRAM E VALUATION S1 UD I ES 223
Second, what resources are available to conduct the evaluation? Hiring an outside eval-
uator ca n be expensive, whi le having a staff person conduct the evaluation m ay be less
expensive. So, in a sense, you may be trading credibility for less cost. In fact, each method-
ological decision will have a trade-off in credibility, level of information, and resources
(including time and mo ney). Also, t he amount and level of infor mation as well as the
research design \ .. ciU be determined, to some e11.”1ent, by what resources are available. A
comprehensive and rigorous eval uation does take significant resources.
Third, where will the information come from? If an eval uation can be done using exist-
ing data, the cost will be lower than if data must be collected from numerous people such
as clien ts and/or staff across m ultiple sites. So having some sense of where the data will
come from is important.
Fou rth, when is the evaluation information needed? In o ther wo rds, what is the time-
fra me for the evaluation? The timeframe will affect costs and design of research methods.
Fifth, why is the evaluation being conducted? Is the evaluation being conducted at the
request of th e fun ding so urce? Is it being cond ucted to improve services? Is it being con-
ducted to document the cost-benefit trade-off of the program? If future program funding
decisions will depend on the results of the evaluation, then a lot more importance will be
attache d to it than 1f a new manager simply wants to know whether clients were satisfied
with services. The more that is riding on an evaluation, the more attention will be given
to the methodology and the more threa tened staff ca n be, especially if they think that th e
purp ose of the evaluation is to down size and trim excess employees. In other words, there
arc many reasons an evaluation is being considered, and these reasons may have implica-
tions for the evaluati on methodology and implemen tation.
Once the issues described above have been considered, more complex questions and
trade-offs will be needed in planning the evaluation. Specifically, six ma in issues guide
and shape the design of any program evaluation effort and m ust be given thoughtful and
delib erate consideration.
L Defining the goal of the program evaluation
2. Un dersta ndi ng the level of infor mation needed for the program evaluation
3. Determining the methods and analysis that need to be used for the program evaluation
4. Consider in g issues that might a ri se and strategies to keep the eval uation on course
5. Developing results into a useful fo rm at for the program stakeholders
6. Providing practical and useful feedback about the program strengths and weak-
nesses as well as providing infor matio n about next steps
Defining the Goal of the Program Evaluation
It is essen tial that the evaluator has a firm understanding of the short- and long-term
objectives of the evaluation. Imagine being hired for a position but not being given a job
descrip tio n or informed aboul how the job fits into the overall organization. Without
knowing why an evaluation is called for or needed, the evaluator might attempt to answer
a d ifferent set of c.1uestio ns from those of interest to the age ncy director or advisory board.
The management might want Lo know why the majo rity of clients do not return after one
or two visits, whereas the evaluator might think that his or her task is to determ ine
224 PART II • QUANTITATIVF APPROACHES: TYPlS Or SIUDIES
whether clien ts who received group therapy sessions were better off than cl ien ts who
received ind ividua l counseling.
In defini ng the goals of t he prog ram evaluation, severa l steps should be taken. First, the
program goals should be examined. These can be lea rned through examining official
program docum ents as well as through talking to key program stakeholders. In clarifying
the overall purpose of the evaluation, it is critical to talk with different program “stake-
holders.” Scriven ( 199 1) defines a program stakeholder as “one who has a substantial ego,
credibility, power, futures, or other capital invested in the program . . .. This includes
program staff and many who arc no t ac tively invo lved in the day-to-day operations”
(p. 334) . Stakeholders incl ude both supporters and opponents of the program as well as
program clients or consumers or even potential consumers or clients. lt is essential that
the evaluator obtain a variety of different views about the program. By listening and con-
sidering stakeholder perspectives, the evaluator can ascertain the most important aspects
of the program to target for the evaluation by looking for overlapping concerns, ques-
tions, and comments from the various stakeholders. However, it is important th at the
stakehol ders have so me agreement on what program success means. Otherw ise, it may be
d ifficult to conduct a satisfactory evalua tio n.
It is also important to consult the extant literature to understand what similar
programs have used to evaluate their outcomes as well as to understand the theoretical
basis of the program in defining the program evaluation goals. Furthermore, it is critical
that the evaluator works closely with whoever initia ted the evaluation to set priorities for
the evaluation. This process should identify the intended o utcomes of th e program an d
which of those outco mes, if not all of them, will be evaluated. Takin g the eval uation a step
further, it may be important to include the exam ination of un intended negative outcomes
that may result from the program. Stakeholders and the literature will also help to deter-
mine those kinds of outcomes.
Once the overall purpose and priorities of the evaluation a re established, it is a good
idea to develop a written agreement, especially if the eva I uator is an external one.
Misunderstandings can and will occu r m onths later if things are no t wr itten in black
and white.
Understanding the Level of Information
Needed for the Program Evaluation
The success of the program evaluation revolves around the evaluator’s ability to develop
practical, researchable questions. A good rule to follow is to focus the evaluation on one
or two key questions. Too many questions can lengthen the process and overwhelm the
evaluator with too much data that, instead of facilitating a decision, might produce
inconsistent findings. Sometimes, funding sources require only that some vague unde-
fined type of evaluation is conducted. The funding sources m ight nei ther expect nor
desire disserta tio n-quality researc h; they simply migh L expect “good fa ith” efforts when
beginning eva luation processes. Other agencies may be quite demand ing in the types and
forms of data to be provided. Obviously, the choice of methodology, data collection
procedures, and reporting formats will be strongly affected by the purpose, objectives,
and questions exam ined in the study.
It is important to note the difference between general research and evaluation. In
resea rch, th e investigator often· focuses on q uestions based on theoretical considerations
o r hypotheses gene rated to hu ilcl o n research in a specific area of study. Altho ugh
CHAPTER 13 • PROGRAM EVALUATION $ TUU I ES 22 5
prog ram evaluatio ns m ay foc us on an intervention derived from a theory, the evalua-
tio n questions should, first and foremost, be driven by the program’s objectives. The eval-
uator is less con ce rned with buildi ng o n prior litera ture o r cont ributing to the
development of practice theory than with determinin g whether a program worked in a
specific community or location.
T here are actually two main types of evalu ation questi ons. There are quc~>tions that
focus on client outcomes, such as, “What impact did the program have?” Th ese kinds of
questions are addressed by using outcome evaluation methods. Then there are questions
that ask, “Did the program achieve its goals?” “Did the program ad here to the spec ified
procedures or standards?” o r “vVh at was learned in operating this program?” These kinds
of questions are addressed by using process evaluation methods. We will examine both of
these two types o f evaluation approaches in the following sec tions.
Process Evaluation
Process evaluations offer a “snapshot” of the program at any given time. Process evalua-
tions typically describe the day-to- day program effo rts; program modifica tions and
changes; outs ide even ts that infl uenced the program; people and institutions involved;
culture, customs, and traditions that evolved; and sociodemographic makeup of the clien-
tele (Scarpitti, In ciardi, & Pottieger, 1993). P rocess evaluation is conce rned with identify-
ing p rogra m st rengths and weaknesses. T his level of p rogram cvalua rion can be usefuhn
several ways, including providing a contex-t within wh ich to interpret program outcomes
and so that other agen ci es o r localities wishin g to sta rr sim ilar programs ca n benefit with-
out havin g to make the same mistakes.
As an example, Bentelspacher, DeSilva, Goh, and La Rowe ( 1996) conducted a process
eva luation o f the cultural co mpatibility of psychoed ucational fam ily grou p treatment
with eth n ic Asian cl ients. As another example, Logan, Williams, Leukefeld, an d Minton
(2000) conducted a detailed process evaluation of the drug court programs before under-
taking an outcome evalual ion of the same programs. T he Loga n et al. sl udy used multiple
m ethods to condu ct the process evaluati o n, including .in-depth i nterviews with the
program administra tive personnel, inten,iews with each of five judges involved in the
progr am, surveys a nd face- to -face interviews with 22 randomly selected current clients,
and surveys of all program staff, 19 community treatment provider representatives, 6 ran –
domly selected d efense attorney representatives, 4 prosecu tin g attorney representatives, l
representative 6:om the probation and parole offi ce, 1 representa tive from the local
co unty jail, an d 2 police depa rtmen l representatives. In all, 69 different individuals repre-
senting I 0 different agency perspectives provided information about the drug court
program. Also, all agency documents were ex amined and analyzed , observations of vari-
ous aspects of the program process were conducted, and client intake data were analyzed
as pa rt of the process evaluation. The results were all integrated an d compiled into one
co mprehensive repo r t.
What makes a process evaluation so important is that resea rchers often have relied only
on selected program outcome indicators such as termination and grad uation rates or
number of rearrests to determine effectiveness. However, to better understand how an d
why a program such as drug court is effective, an analysis of how the p rogram was con cep-
tualized, implemented, and revised is needed. Consider this exan1ple-say one outcome
eva luation of a drug cou rt p rogram showed a gra duat ion rate of 80% of those who began
the program, while another outcome evaluation found that only 40o/o of those who began
the program graduated. Then, the graduates of the second program were more likely to be
free from substance usc an d crimin al behaviors at the l2- month foUow-up than the graduates
226 PART II • QuANTITATIVE APPROACHES: TYPES OJ SJUDIES
from the first program. A process evaluation could help to explain the specific differences
in facto rs such as selection (how clients get into the programs), treatment plans, monitor-
ing, program length, and other program features that may influence how many people
graduate and slay free from drugs and criminal behavior at follow-up. Tn other words, a
process evaluation, in contrast to an examina tion of program outcome only, can provide a
clearer and more com prehensive pictm e of how drug cou rt affects those involved in the
program. More specifically, a process evaluation can provide information about program
aspects that need to be improved and those that work well (Scarpilli, Inciard i, & Pottieger,
1993). Finally, a process evaluation m ay help to facilita te replicatio n of the drug cou rt
program in other areas. This often is referred to as technology transfer.
A different but related process evaluation goal might be a description of the failures
and depa r tures from the way in which the interventio n o riginally was designed. How were
the staff trained and hired? Did the intervention depart from the treatment manual rec-
ommendations? Influences that shape and affect the intervention that clients receive need
to be identified because they affect the fidelity of the treatment p rogram (e.g., delayed
funding or staff hires, ch anges in policies or procedu res). \”/hen program implementation
deviates significantly from what was intended, this might be the logical explanation as to
why a program is not working.
Outcome or Impact Evaluation
Outcome or impact evaluation focuses on the targeted objectives of the program, often
looking at variables such as behavior change. For example, many drug t reatment programs
may measure outcomes or “success” by the number of clients who abstain from drug use.
Questions always arise, though. For instance, an evaluation might reveal that 90% of those
who graduate from the program abstai n from drug use 30 days after the prog ram was com-
pleted. However, only 50% report abstai ning from drug use 12 months after the program
was completed. Would key stakeholders involved all consider that a success or failure of the
progr am? This exam ple brings up three critical issues in outcome evaluations.
One of the critical issues in outcome evaluations is related to understanding for whom
docs the program work best and under what conditions. In other words, a more interest-
ing and important question , rather than just asking whether a program works, would be
to ask, “Who are those 50% of people who remained abstinent from drug use 12 mo nths
after completing the program, and how do they differ from the 50% who relapsed?” It is
not unusual for some evaluation questions to need a combination of both process and
im pact evaluation m ethodol ogies. For example, if it turned o ut that r esults of a particular
evaluatio n showed that the program was not effective (impact), then it might be useful to
know why it was not effective (process ). Tn such cases, it would be important to know how
the program was im plemented, what changes were made in the pro gram during the
im plementation, what problems were experienced dur ing the implem entation, and what
was done to overcome those problems.
Another important issue in outcome evaluation has to do with the timing of meas ur-
ing the o utcomes. Ou tcome effects are usually measured after treatmen t or postin terven-
tion. These effects may be either short term or long term. immediate outcomes, or those
generally measured at the end of the treatment or intervention, might or might not pro-
vi de the same resu lts as one would get later in a 6- or 12-m onth follow- up, as highlighted
in the exa mple above.
The third important issue in outcome evaluation has to do with what specific measures
were used. Is abstinence, for example, the only measure of interest, or is reduction in use
something that might be of inte rest? Refra inin g from cri minal activity or holding a steady
CHAPTER l3 • PROGRAM EVALUATION STUOIES 2 27
job may also be an important goal of a subslance abuse program. If we only m easure
abstinence, we would never know about other kinds of outcomes the program may affect .
These last two issues in outcome evaluations have to d o with the evaluation methodol-
ogy and analysis and are add ressed in more detail below.
Determining the Methods and
Analysis That Need to Be Used
for the Program Evaluation
The next step in the evaluation process is to determine the evaluation design. There
are several interrelated steps in this process, including determining the (a) sources of data,
(h) research design, (c) measures, (d ) analysis of change, and (e) cost- benefit assessment
of the program.
Sources of Data
Several main so urces of data can be used for evaluat ions, includ ing quali tative informa-
tion and quantitative information.
Qualita t ive Data Sources
Qualitative data sources are often used in p rocess evaluations and might include o bsen a-
tions, analysis of existing program documents such as policy and procedure manuals, in –
depth interview data, or focus group data. There are, however, trade-offs when using
qualitative data so urces. On the positive side, q ua litative evaluation data provide an “in-
depth” snapshot of var ious topics such as how the program functions, what staff think are
the positive or negative aspects of the programs, or what clients really think of the O\’erall
program exp eriences. Reporting cl ients’ experiences in their own words is a characteristic
of qualitative evaluations.
Interviews arc good for collecting qualitative or sensitive data such as values and atti –
tud es. This method requires an interview prolocol or questionnaire. These usual!) are
structured so that respondents are asked questions in a specific order, but they can be
semistructured so t.hat there are fewe r topics, and the interviewer has the ability to cha nge
the order based on a “reading” of the client’s responses. Surveys can request information
of clients by mail, by telephone, or in person. They may or may not be 1>clf-administered.
So, besides considering what data are desi red, evaluators must be concerned with prag-
matic considerations regarding the best way in which to collect the desired data.
Pocus groups also offer insight in to cer tain aspects of the program or program func-
tioning; participants add their input, and input is interpreted and discussed by other
group members. This discussion component ml!y provide an opportunity to uncover
information that might otherw ise remain undiscovered such as the m eaning of certain
things to different people. Focus gro ups typically are small inform al groups of persons
asked a series of questions that start out very general and then become more specific.
Focus groups are increasingly being used to provide evaluative info rmation about human
services. They work pa rt icula rly well in identifying t he questio ns that might be important
to ask in a survey, in testing planned procedures or the phrasing of items for the spec ific
target population, and in exploring possible reactions to an intervention or a service.
228 P!IRT II • QuANTITATIVE APP ROACHF.S: TYPES OF SruOI[S
On the other hand, qualitative studies Lend to use small samp les, and care mus t be used
in analyzing and interpreting the information. FurLhermore, although both qualitative
and quantitative data are su bject to m ethod bias and threats to validity, qualitative data
may be more sensitive to bias depending on how participants are selected to be inter-
viewed, the nu mber of observations or focus groups, and even subtleties in the questions
asked. With qualitative approaches, the evaluator often has less abil ity to account for alter-
n ative expla nation s because th e data are more limited. Making strong conclusions about
representativeness, validity, and reliability is more difficult with quali tative d ata corn-
pared to something like an average rating of satisfaction across respondents (a quantita –
tive measu re). Yet, an average rating do es not tell us much about why parti cipants a re
satis fi ed with the program or why they may be dissatisfied with other aspects of the
p rogram. Thus, it is often imperative to use a mixture of q ualitative and quantitative
information to evaluate a program.
Quantitative Data Sources
Two main types of quantitative data sources ca n be used for program evalu ations: sec-
ondary data and original data.
Secondary Data. One option for ob taining needed data is to use existi ng data. Collecting
new data often is more expensive than using existing data. Examining the data on hand
an d already available always is a good llrst step. H owever, the evaluator migh t want to
rearrange or reassemble the data, for example, dividing it by quarters or combining it into
12 -m onth periods that help to reveal patterns and trends over t ime. Existing data can
come from a variety of places, including the following:
Client records maintained by the program: These may include a host of demographic
and service-related data items about the population served.
Program expense and financial data: T hese can help the evaluator to determ ine whether
one intervention is much more expensive than another.
Agenc.y annual reports: These can be used to identify trends in service delivery and
program costs . The evaluator can compare an n uil l reports from year to year and can
develop graphs to easily identify trends wilh clientele and programs.
Databases maintained by the state health department and other state agencies. Public
data such as births, d eaths, and divorces are available from each state. Furthermore,
mos t state agencies produce annual reports that may reveal the number of clients
served by program, geographic region, and on occasion, selcc t·ed sociodemographic
variables (e.g., race or age).
Local and regional agencies. Planning boards for mental health services, child protec-
tion, school boards, and so forth may be able to furnish statistics on outpatient and
in patien t services, special school populations, or child abuse cases.
The federa l government. The fed era l governmen t collects and maintains a large amount
of data on many different issues and topics. State and national data provide bench-
marks fo r comparing local demographic or social indicators to national-level demo-
graphic or social indicators. For instance, if you were worki ng as a cancer educator
whose objective is to red uce the incidence of b reast cancer, you might want to consult
cancercontrolplanct.ca ncer.gov. That Web site w ill furnis h natio nal -, state-, and
CHAPTER 13 • PROGRAM EVAlUA II ON S TUD ICS 229
county-level data on the nwnber of new cancer cases and deaths. By compariso n, it
will be possible to determine if the rate in one county is higher than the state or
national average. Demographic information about communities can be found at
www.census.gov.
Foundations. Certain well-established foundations provide a wealth of information
about problems. For example, the Annie E. Casey Foundation provides an incredible
Kids Count Data Book that provides an abundance of child welfare-related data at the
state, national, and county level. By using their data, you could determine if infant
mortality rates were rising, teen births were increasing, or high school dropouts were
decreasing. You can find the Web site at www.aecf.org .
lf existing data cannot be used or cannot answer all of the eva luation q uestions, then
o riginal data rnust be coll ec lcd.
Original Data Sources. There a re rnan y typ es or evalua ti o n designs (rom wh ich to choose,
and no single one will be ideal for every project. Th e specific approach chosen for the
eva luation will depend on the purpose of the evaluation, the research questions to be
explored, the h oped-to r or in ten d ed res ults, the quali ty and vo lume of data available or
needed, and staff, time, and financial reso urces.
The evaluation design is a critical decision for a number of reasons. Without the
appropriate evaluation design, confidence in the resuiL<> of the evaluation might be lack~
ing. A strong evaluation design minimizes alternative explanations and assists the evalua-
tor in gauging the true effects attributable to the intervention. In other words, the
evaluation design directly affects tl1e interpretation that can be made regarding whether
an intervention should be viewed as the reason for change in clients’ behavior. Howewr,
there are trade offs with each design in the credibility of information, causality of
an)’ observed changes, and resources. These trade- off.~ must be carefully considered and
discussed with the program staff.
Quantitative designs include surveys, pretest-posttest studies, quasi-experiments with
noncquivalcnt control groups, lo ngit u dinal designs, and randomized experimental
designs. Quantitative approaches transform answers to specific questions into numerical
data. Outcome and impact evaluations nearly always are based on quantitative evaluation
desig ns. Also, sa mpli ng strategies must be co nsidered as a n in regr<1l p<1rt of th e research
design. Below is a brief overview of the major types of quantitative evaluation designs. For
a n expanded disc ussio11 o r these topics, refe r Lo Royse, Thyer, Padgell, and Logan (2005 ).
Research Design
Cross -Sectional Surveys
A survey is limited to a description of a sa mple at o ne point in time and provides us with
a “snapshot” of a group of respondents and what they were like or what knowledge or atti-
tudes they held at a particular point in time. If the survey is to generate good generalizable
data, then the sampling procedures must be carefully planned and implemented. A cross-
sectional survey requires rigorous random sampling procedures to ensure that the sample
closely represents the population of interest. A repeated survey is similar to a cross-
sectional study but collects information at two or more points in time from the same
respondents. A repeated (longitudinal) survey is effective at measuring changes in facts,
attitudes, or opinions over a co urse of Lime.
230 PART II • QuANTITATIVE APPROACH ES: TYPES Of S TUOIES
Pretest-Po sttest Design s (Nonexperimental)
Perhaps the mosl common quantitative evaluation design used in social and human
service agencies is the pretest-posttest. In this design, a group of clients with some specific
problem or diagnosis (e.g., depression) is admin istered a pretest pr ior to the start of inter-
vention. At some point toward the end or after the inter vention, the same inst rument is
admi nistered to the group a second time (the pos ttest) . The one-group pretest-posttest
design can measure change, but the evaluator has no basis for attributing change solely to
the program. Confidence about change increases and the design strengthens when control
groups are added and when participan ts are randomly assigned to either a control or
experimental condition.
Quasi-Experimental De signs
Also known as nonequivalent control group designs, quasi-experiments generally use
comparison groups whereby two similar groups are selected and foll owed for a period of
time. One group typically receives some program or benefit, v,rhereas the other gro up (th e
control) does nol. Both groups are m easured and compared for any differences at the end
of some time period. Participants used as controls may be clien ts who are on a waiting list,
those who are enrolled in another treatment program, or those who live in a different city
or county. The problem with this design is that the control or compa rison group might
no t, in fact, be equivalent to the group receiving the intervention. Comparing Ocean View
School to Inner City School might not be a fair comparison. Even two differen L sch ools
within the same rural county might be more different than similar in terms of the learn-
ing milieu, the proportion of students receiving free lunches, the number of computers
and books in the school librar y, the principal’s hiring pract~ces, and the like. With this
design, there always is the possibility that whatever the results, they might have been
obtained because the intervention group really was different from the contro l group.
However, many of these issues can be considered and either controlled for by collecting
the information and performing statistical analysis with these considerations or at least
can be considered within the contex1: of interpreting the resu lts. Even so, this type of study
does not provide proof of cause and effect, and the evalu ator always must cons ider o th er
facto rs (both known and measured a nd unknown or unmeasured) that co uld h ave
affected the study’s outcomes.
Longitudinal Designs
Longitudinal designs are a type of quasi- experimental design t hat involves tracking a par-
ticula r group of individu als over a substantial period of time to discover potential
changes due to the influence of a program. It is not uncommon for evaluators to want to
know about the effects of a program after an extended period of Lime has passed. The
questio n of interest is whether treatment effects last. These ~tudies typically are compli-
cated and expensive in time and reso urces. In add ition, the longer a study runs, the higher the
expected rate of attrition from cl ients who drop out or move away. High rates of allrition can
bias the sample.
Randomized Experimental Designs
l.n a true experimental desig n, participants are ran domly assigned to either the control
or treatment group. This design provides a persuasive argument about causal effects of a
program on participants. The random assignment of respondents lo treatm ent and con-
trol groups helps to ensure both groups are equivalent across key variables such as age,
race, area of residency, and treatment history. This design provides the best evidence Lhat
CIIAPT ER 13 • P ROC RA M EVALUATIO N STUDI ES 231
any observed differences between the tl’IO groups after the intervention can be attributed
to the intervention, assuming the two groups were equal before the interven tion. E\·en
w ith random assignmen t, group differences preinLervention co uld exist, and the eval uato r
should carefu lly look for them and use statistical controls when necessary.
One word of warning about random assignment is that key program stakeholders
often view random ass ignment as unethical, especially if they view the treatment p rogram
as benefici al. One o utcome of this diffk ulty of accepting random assignment is that staff
mi gh t have problems not giving the intervention they believe is effective to specific needy
clients or to all of their clients instead of just to those who were randomly assigned. If they
do succumb to this temptation, then the eval uation effo r t can be unintentional ly sabo-
ttlged . The evalua tor must tra in an d prepa re all of those individ uals involved in the e\’al-
uation to help them und erstand the purpose and importance of the random assignment.
That, more than any other procedure, prov ides the evidence that the treatmen t really does
benefit the clients.
Sampling Strategies and Consideration s
vVhen the client population of interest is too large to obtain information from each
individual member, a sample is drawn. Sampling allows the eva luator to make predictions
abou t a population based on study findings from a set of cases. Sampling st rategi es can be
very complex. lf the evaluator needs th e type of precision afforded b y a p robability sam-
ple in which there is a known level of confidence and margin of error (e.g., 95% confi-
dence, pl us or m inus 3 pe rcentage points), th en he o r she m igh t need to hire a sampling
consultant . A co nsultant is particularly recommended wh en the decisions about the
program or intervention are critical such as in drug research or when treatments could
have potentially harmful side effects. However, there is a need to recognize the trade-offs
that a re made when deter mining sampling strategy and samp le si ze. Large samples can be
more accurate than sma ller ones, yet they usually are much more expensive. Small
samples can be acceptable if a big change or effect is ell.lJected. As a rule, the m ore critical
the decision, the larger (and more precise) the sample should he.
T here are two main c Wit h probabi lity sampling, the primary idea is tha t every in d ivi du al, object, or institu- The evaluator must balance the need for scientific rigor against convenience and often Measures 232 PART I I • QUANTITATIVE APPROACHES: T YPES OF S TUOICS
evaluation, depending on the questions being asked. In one project, the focus migh L be on Once there is agreement on the outcome variables, objective meas ures for those The second complicating factor in measurement is determining what other constructs ls one instrument or scale better than another fo r measuring depression? ·what are the When no suitable instrument o r available data exist for the evaluation, the evaluator A note mu st be made here about a special case of program evaluatio n: evaluating pre- Evaluators use a multitude of methods and instruments to collect data for their stud- CHAPrtR 13 • PROCRAM EVALUATION STUD I ES 233
when random assignment is not possible. That way, one can possibly look for convergence Analysis of Change The unit of analysis refers to the person or things being stud ied or measured in the eval- The analysis will also be determined by the research design such as the number of Cost-Benefit Analysis A comprehensive economic cost analysis would include c.slimates of the value of all 234 PART II • QuANIITATIVE APPROACHES: T YPES OF STUDIES
Including costs fo r which the program may not explicitly pay presents an interesting A comprehensive analysis of econom ic benefits also presents challenges. The goal of So, although the direct cost of the program usually is easily computed, the full costs When looking <~l the cost -benefit analys is of programs o r compa r ing these costs and
ben efi ts across programs, it is important to keep in mind that cost- benefit analysis may be
done very differently, and a careful assessment of the methods must be under taken to
ensure comparabilily across programs. The Step-by-Step Guide to The W.K. Kellogg Foundation, founded in 1930 by breakfast cereal pioneer Will Keith Kellogg, This publication may be shared without charge for educational, charitable, nonprofit or other non-commer- The Step-by-Step Guide to Evaluation | page i Page ii | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page iii Page iv | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page v Page vi | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page vii Page viii | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page ix Page x | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 1 Page 2 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 3 Page 4 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 5 Page 6 | The Step-by-Step Guide to Evaluation http://betterevaluation.org/resource/guide/ http://betterevaluation.org/resource/guide/ The Step-by-Step Guide to Evaluation | page 7 Page 8 | The Step-by-Step Guide to Evaluation http://www.communityscience.com/pdfs/CrossCulturalGuide.r3 http://www.communityscience.com/pdfs/CrossCulturalGuide.r3 http://www.nsf.gov/pubs/2002/nsf02057/nsf02057_5 The Step-by-Step Guide to Evaluation | page 9 http://www.eval.org Page 10 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 11 Page 12 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 13 Page 14 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 15 Page 16 | The Step-by-Step Guide to Evaluation Evaluative thinking is about understanding — using a systematic process of The Step-by-Step Guide to Evaluation | page 17 http://www.evaluativethinking.org/docs/EvaluativeThinking. http://www.evaluativethinking.org/docs/EvaluativeThinking. Page 18 | The Step-by-Step Guide to Evaluation 2.2 http://tamarackcci.ca/files/developmental_evaluation_primer The Step-by-Step Guide to Evaluation | page 19 Culturally responsive evaluation recognizes that cultural values, beliefs and context http://www.cdc.gov/dhdsp/docs/cultural_ http://www.cdc.gov/dhdsp/docs/cultural_ http://education.illinois.edu/crea http://education.illinois.edu/crea Page 20 | The Step-by-Step Guide to Evaluation Experimental designs assess the causal effects of a program by comparing two groups The Step-by-Step Guide to Evaluation | page 21 Page 22 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 23 Page 24 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 25 Page 26 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 27 Ev Page 28 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 29 Allocation of time and Page 30 | The Step-by-Step Guide to Evaluation Frequent inclusion of Ev The Step-by-Step Guide to Evaluation | page 31 Useful resources about these evaluation approaches include: http://tamarackcci.ca/files/developmental_ http://tamarackcci.ca/files/developmental_ http://www.mrc.ac.za/crime/evaluation http://education.illinois.edu/crea Page 32 | The Step-by-Step Guide to Evaluation Ex Q The Step-by-Step Guide to Evaluation | page 33 Let’s consider this example. You have been implementing a district-wide academic enrichment Page 34 | The Step-by-Step Guide to Evaluation http://onlinelibrary.wiley.com/doi/10.1002/casp.1131/full http://onlinelibrary.wiley.com/doi/10.1002/casp.1131/full http://www.outcomemapping.ca/ http://www.outcomemapping.ca/ http://www.outcomemapping.ca/ http://www.pacenterofexcellence.pitt.edu/documents/study_designs_for_ http://www.pacenterofexcellence.pitt.edu/documents/study_designs_for_ The Step-by-Step Guide to Evaluation | page 35 Page 36 | The Step-by-Step Guide to Evaluation The community foundation was interested in the following evaluation questions: The Step-by-Step Guide to Evaluation | page 37 Page 38 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 39 Page 40 | The Step-by-Step Guide to Evaluation A logic model is a graphic representation of the theory of change that illustrates the The Step-by-Step Guide to Evaluation | page 41 http://www.cdc.gov/eval/guide/cdcevalmanual http://www.cdc.gov/eval/guide/cdcevalmanual http://betterevaluation.org/sites/default/files/Rainbow%20Framework%20-%20compact%20version http://betterevaluation.org/sites/default/files/Rainbow%20Framework%20-%20compact%20version http://www.ifrc.org/global/publications/monitoring/ifrc-me-guide- http://www.ifrc.org/global/publications/monitoring/ifrc-me-guide- http://ctb.ku.edu/en/tablecontents/index.aspx Page 42 | The Step-by-Step Guide to Evaluation http://infed.org/mobi/peter-senge-and-the- http://infed.org/mobi/peter-senge-and-the- The Step-by-Step Guide to Evaluation | page 43 Page 44 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 45 Identify stakeholder role(s) in evaluation planning, implementation, interpretation of Page 46 | The Step-by-Step Guide to Evaluation http://www.wkkf.org/knowledge-center/resources/2006/02/wk-kellogg-foundation- http://www.wkkf.org/knowledge-center/resources/2006/02/wk-kellogg-foundation- http://www.theoryofchange.org/pdf/Superwomen_Example http://www.aecf.org/upload/publicationfiles/cc2977k440 The Step-by-Step Guide to Evaluation | page 47 Once the theory of change and logic model have been developed, evaluation questions can be Page 48 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 49 Page 50 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 51 Page 52 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 53 Page 54 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 55 example above, this could include testing a new curriculum for training staff to become more Page 56 | The Step-by-Step Guide to Evaluation http://www.evaluationinnovation.org/sites/default/files/Pathways%20for%20Change http://www.evaluationinnovation.org/sites/default/files/Pathways%20for%20Change http://www.hfrp.org/var/hfrp/ http://www.hfrp.org/var/hfrp/ The Step-by-Step Guide to Evaluation | page 57 Page 58 | The Step-by-Step Guide to Evaluation http://www.evaluationinnovation.org/publications/practical-guide- http://www.evaluationinnovation.org/publications/practical-guide- https://www.mathematica-mpr.com/our-publications-and-findings/publications/evaluating-system-change-a-planning-guide https://www.mathematica-mpr.com/our-publications-and-findings/publications/evaluating-system-change-a-planning-guide http://docs.geofunders.org/?filename=geo2014_indicators_framework http://tamarackcci.ca/resource-library/evaluation/approaches-measuring- http://tamarackcci.ca/resource-library/evaluation/approaches-measuring- The Step-by-Step Guide to Evaluation | page 59 http://www.jrsa.org/pubs/juv-justice/evaluator mailto:info%40communityscience.com?subject=Selecting%20an%20Evaluator%20resource Page 60 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 61 Page 62 | The Step-by-Step Guide to Evaluation http://www.eval.org The Step-by-Step Guide to Evaluation | page 63 The evaluation team proposed a methodology to answer the primary evaluation questions. The Page 64 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 65 Page 66 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 67 Page 68 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 69 Page 70 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 71 Page 72 | The Step-by-Step Guide to Evaluation http://www.rwjf.org/en/library/research/2009/12/a-practical-guide-for-engaging- http://www.rwjf.org/en/library/research/2009/12/a-practical-guide-for-engaging- http://docs.geofunders.org/?filename=do_nothing_about_me_without_me http://docs.geofunders.org/?filename=do_nothing_about_me_without_me http://www.cpc.unc.edu/measure/resources/publications/ms-11-46-e http://www.cpc.unc.edu/measure/resources/publications/ms-11-46-e The Step-by-Step Guide to Evaluation | page 73 Page 74 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 75 Page 76 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 77 http://www.eval.org http://tei.gwu.edu Page 78 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 79 Page 80 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 81 Page 82 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 83 Determine Stakeholders and Page 84 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 85 Page 86 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 87 Page 88 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 89 Page 90 | The Step-by-Step Guide to Evaluation http://www.jrsa.org/pubs/juv-justice/evaluator http://www.cdc.gov/dhdsp/pubs/docs/ http://www.cdc.gov/dhdsp/pubs/docs/ mailto:info%40communityscience.com?subject=Selecting%20an%20Evaluator%20resource The Step-by-Step Guide to Evaluation | page 91 Page 92 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 93 http://www.communityscience.com/pdfs/CrossCulturalGuide.r3 https://www.nsf.gov/pubs/2002/nsf02057/nsf02057_5 https://www.nsf.gov/pubs/2002/nsf02057/nsf02057_5 Page 94 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 95 Page 96 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 97 Page 98 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 99 Page 100 | The Step-by-Step Guide to Evaluation http://docs.geofunders.org/?filename=do_nothing_about_me_without_me http://docs.geofunders.org/?filename=do_nothing_about_me_without_me http://docs.geofunders.org/?filename=do_nothing_about_me_without_me http://docs.geofunders.org/?filename=do_nothing_about_me_without_me http://www.cpc.unc.edu/measure/resources/publications/ms-11-46-e http://www.cpc.unc.edu/measure/resources/publications/ms-11-46-e The Step-by-Step Guide to Evaluation | page 101 Page 102 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 103 Page 104 | The Step-by-Step Guide to Evaluation introduction www.wkkf.org/resource- www.wkkf.org/resource- www.wkkf.org/resource- The Step-by-Step Guide to Evaluation | page 105 Page 106 | The Step-by-Step Guide to Evaluation 7.1 The Step-by-Step Guide to Evaluation | page 107 http://www.wkkf.org/knowledge-center/resources/2006/02/wk-kellogg-foundation- http://www.wkkf.org/knowledge-center/resources/2006/02/wk-kellogg-foundation- http://www.theoryofchange.org/pdf/Superwomen_Example http://www.aecf.org/upload/publicationfiles/cc2977k440 Page 108 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 109 Page 110 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 111 Page 112 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 113 Page 114 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 115 Page 116 | The Step-by-Step Guide to Evaluation Keep in Mind: If you do it this way, explain to everyone why some comments might not have The Step-by-Step Guide to Evaluation | page 117 Short-term (sometimes called immediate and intermediate outcomes) and long-term https://www.bttop.org/sites/default/files/public/W.K.%20Kellogg%20LogicModel Page 118 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 119 Page 120 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 121 http://www.rwjf.org/en/library/research/2009/12/a-practical-guide-for-engaging- http://www.rwjf.org/en/library/research/2009/12/a-practical-guide-for-engaging- Page 122 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 123 Page 124 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 125 Page 126 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 127 Page 128 | The Step-by-Step Guide to Evaluation A SMART approach means: The Step-by-Step Guide to Evaluation | page 129 Page 130 | The Step-by-Step Guide to Evaluation http://www.urban.org/uploadedpdf/411404_nonprofit_performance The Step-by-Step Guide to Evaluation | page 131 Page 132 | The Step-by-Step Guide to Evaluation Evaluation Plan Evaluation Report The Step-by-Step Guide to Evaluation | page 133 Page 134 | The Step-by-Step Guide to Evaluation http://www.opportunitystudies.org/ http://www.opportunitystudies.org/ https://www.wilder.org/Wilder-Research/Publications/Studies/Program%20Evaluation https://www.wilder.org/Wilder-Research/Publications/Studies/Program%20Evaluation https://www.wilder.org/Wilder-Research/Publications/Studies/Program%20Evaluation https://www.wilder.org/Wilder-Research/Publications/Studies/Program%20Evaluation The Step-by-Step Guide to Evaluation | page 135 http://cphs.berkeley.edu/compensation http://cphs.berkeley.edu/compensation http://www.affordablelanguageservices.com/translation-costs http://www.affordablelanguageservices.com/translation-costs http://www.affordablelanguageservices.com/translation-costs Page 136 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 137 Page 138 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 139 Page 140 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 141 Page 142 | The Step-by-Step Guide to Evaluation A PREPARE for conducting The Step-by-Step Guide to Evaluation | page 143 Page 144 | The Step-by-Step Guide to Evaluation http://toolkit.pellinstitute.org/evaluation-guide/collect-data/ http://www.nsf.gov/pubs/2002/nsf02057/nsf02057_5 http://www.nsf.gov/pubs/2002/nsf02057/nsf02057_5 The Step-by-Step Guide to Evaluation | page 145 Page 146 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 147 Different types of quantitative data collection methods. Open-ended questions that ask respondents to resply through written text. Page 148 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 149 Page 150 | The Step-by-Step Guide to Evaluation http://learningstore.uwex.edu/assets/pdfs/g3658-10 http://learningstore.uwex.edu/assets/pdfs/g3658-10 The Step-by-Step Guide to Evaluation | page 151 http://learningstore.uwex.edu/assets/pdfs/G3658-6 http://learningstore.uwex.edu/assets/pdfs/G3658-6 http://www.evaluatod.org/assets/resources/evaluation-guides/analyzing-interpretingdata-8-09 http://www.evaluatod.org/assets/resources/evaluation-guides/analyzing-interpretingdata-8-09 Page 152 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 153 Page 154 | The Step-by-Step Guide to Evaluation A useful approach when no readily available, The Step-by-Step Guide to Evaluation | page 155 Page 156 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 157 Page 158 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 159 http://www.evaluatod.org/assets/resources/evaluation-guides/analyzing-interpretingdata-8-09 http://www.evaluatod.org/assets/resources/evaluation-guides/analyzing-interpretingdata-8-09 Page 160 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 161 Page 162 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 163 Page 164 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 165 Page 166 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 167 Page 168 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 169 Page 170 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 171 Page 172 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 173 Page 174 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 175 Page 176 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 177 Page 178 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 179 Page 180 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 181 Page 182 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 183 Page 184 | The Step-by-Step Guide to Evaluation It’s not necessary to wait until the end of the program to share The Step-by-Step Guide to Evaluation | page 185 Page 186 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 187 aw kn de im Page 188 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 189 Page 190 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 191 Page 192 | The Step-by-Step Guide to Evaluation http://annkemery.com/introducing-the-essentials/ The Step-by-Step Guide to Evaluation | page 193 Page 194 | The Step-by-Step Guide to Evaluation The Step-by-Step Guide to Evaluation | page 195 Page 196 | The Step-by-Step Guide to Evaluation Turnitin Report Formatting Title Page Citation Outline
tion in the population under study has a chance of being selected into the sample, and the
likelihood of the selection of any individual is known. Probability sampling pro,;des a
firm basis for generalizing from the sample lo the population. No11probability samples
severely red uce the eval uator’s ability to generalize the results of the study to the larger
population.
limited resources when determining sample size. If a m ajor decision is bei ng based on
data collected, then precisio n and certa inty are critical. Statistical precisio n increases as
the sample s ize increases. When differences in the results are expected to be small, a larger
sampl e guards against confounding variables that might distort the results of a treatment.
The next important meth od decision is to determine how best Lo measure the variables of
interest needed to answer the evaluation questions. These will va ry from evaluation to
the outcome variable of arrests (or rear rests) so as to determine whether the program
reduced criminal justice involvement. In another project, the out come variable mighL be
nmnbcr of hospitalizations or days of hospitalization.
variables must be determined. Using the example of the drug court program above, the deci-
sions might include the following: How will abstinence be m easured? How will reduction in
substance use be measured? How will crimina 1 behavior be measured? llow will employment
be measured? Th is may seem simple at first glance, but there are two complicating factors.
First, there are a variety of ways to measure something as simple as abstinence. One could
measure it by self-report or by actually giving the client a drug test. When looking at reduc-
tion of use, the issu e of measurement becomes a bit more complicated. This will likely need
to be self-report and some kind of comparison (either the same measures must be used
with the same clients before and after the program [this being the best way) or the same mea-
sure must be used with a control group of some kind [like program dropouts)).
need to be included to better understand “who benefits from the program the most and
under what circ umstances” and how those constructs are measured. Again, using the drug
court program as an example, perhaps those clients who are most depressed, have the
most health problems, or have the mos L anxiety do worse in drug court programs because
the program may not address co-occurring disorders. If this is the case, then it will be
important to include measures of depression, anxiety, and health. However, there are
many different measures for each of these constructs, and different measures use different
timeframes as points of reference. T n other words, some depression measures ask abo ul
12-month periods, some ask about 2-week periods, and some ask about 30-day periods.
trade-offs relative to shorter or longer instruments? (For example, the most valid instru-
ment might be so long thal clien ts will get fatigued and refuse to complete it.) Is it better
to measure a reduction in symptoms associated with a standardized test or to employ a
behavioral measure (e.g., counting the n umber of days that patients with chronic mental
illness are compliant with taking their medications)? Is measuring attitudes aboul drug
abuse better than measuring knowledge about the symptoms of d rug addiction?
Evaluators frequently have to struggle with decisions such as these and decide whether it
is better to use instruments that are not “perfect” or to go to the tro u ble of developin g and
validating new ones.
might have to create a new scale or at least modify an existing one. If an evaluator revises
a previ.ously developed measure, then he or she has the burden of demonstrating that the
newly adapted instrument is reliable and valid. Then, there are issues such as the reliabil-
ity of data obtained from clients. Will cl ients be honest in reporting actual d rug and alco-
hol use? How accurate are their memories?
vention programs. Evaluation of prevention programs is especially challenging because
the typical goal of a prevention program is to prevent a particular problem or behavior
from developing. The question then becomes, ” How do you measure something that
never occurs?” In other words, if the prevention program is successful, the problem will
not develop, but it is difficult to dclermine with any certainty that the problem would
have developed in the flrst place absent the prevention program. Ti l uS, measures beco m e
very important as well as the design (s uch as including a control group ).
ies. A good strategy is to include mu ltiple measures and methods if possible, especially
of conclusions across methods and measures.
After the data are collected, the evaluator is faced with a sometimes difficult question of
how to determine whether change had occurred. And, of course, there are several consid-
erations within this overall decision as welL One of the first issues to be decided is what
the unit of analysis will be.
uation of a program. Typically, the basic unit of analysis consists of individual clients but
also may be groups, agencies, communities, schools, or even slates. Fo r example, an evalu-
alor might examine the effectiveness of a drug prevention program by looking for a
decrease in drug-related suspensions or disciplinary actions in high schools in which the
program was imp lemented. In that instance, schools are the primary unit of analysis.
Another eva lu ator might be concerned only with the attitudes toward d rugs and alcohol of
students in one middle school; in that situation, individuals would be the uni l of analysis.
The smal lest unit of ana lysis from which data are gathered often is referred to as a case. The
unit of analysis is critical for determining both the sampling strategy and the data analysis.
groups to be analyzed, the type of dependent var iable (categorical vs. continuous), the
control variables that need to be included, and whether the design is longitudinal. The
literature on similar program eval uation s is also usefullo exam ine so that analysis plans
can consider what has been done in the past. The analysis phase of the evaluation is basi-
cally the end product of the evaluation activities. Therefore, a careful analysis is critical to
the evaluation, the interpretation of the results, and the credibility of the results. Analysis
should be conducted by somebody with adequate experience in statistical methods and
statistical assumptions, and limitations of the study should be carefully examined and
explained to program stakeholders.
Whi le assessing program outcomes is obv iously necessary to gauge the effectiveness of a
program, a more comprehensive understanding of program “success” ca n be a ttained by
examin ing program costs and economic benefits. In general, eco nomic costs and benefits
associated with specific progra ms have received relat ively limited attention. One of the
major challenges in estimating costs of some cornm uni ly-baseu socia l programs is that
slanuard cost est imat ion procedures do not always reflect the true costs of the program.
For example, a drug court program often combines both criminal justice supervision and
substance ab use trcaLrnen L in a comm unity-based envi ronment. And in order for drug
court programs to work effectively, they often use many community and outside agency
resources that are not necessarily directly paid for by the program. For exa mple, although
the drug court program may not directly pay for the jail time incurred as part of client
sanctions, jail time is a central component in many drug cou rt programs. Thus, jail costs
must be considered a drug court program cost.
resources used in providing the program. When resources are donated or subsidized, the
out-of-pocket cost will differ from the oppo rt uni ty cost of the resources for a given
program. Opportunity costs take into account the forgone val ue of an alternative use for
program resources. Other examples of opportunity costs for the drug court program may
include the time and efforts of judges, police officers, probation officers, and prosecutors.
dilemma. The dilemma primarily stems from the trade-off in presenting only out-of-
pocket expenditures for a program (thus the program will have a lower total cost) or
accurately reflecting all of th e costs associated with the program regardless of whether
th ose costs are pa id out of pocket (implying a higher total prog ra m cost). furthe rm ore,
when agencies share resources (e.g., shared overhead costs), the correct proportion of
these resources that are devoted specificall y to a program must be properly specified. To
date, there has been liule discussion in the literature about estimating the opportunity
cost of programs beyond the out of pocket costs. Knowin g which costs to include and
what va lue to place o n certain se rvi ces or items that are not directl y charged Lo the
program can be complica ted.
an econo mic benefit analysis is to determine the monetary value of changes in a range
of program outcomes, mainly derived from changes in client behavior as a result of par-
ticipating in the program. When estimating the benefits of a program such as drug
court, o ne of the most o bvious and important o utco m es is the reduction in c riminal
justice costs (e.g., reduced incarceration and supervision), and these are traditio nally
the only sources of benefit s examined in many drug court evaluations. However, drug
court programs often have a diverse set of goals in addition to reducing criminal justice
costs. For examp le, drug court programs often focus on helping the particip ants
beco m e more productive in society. This includes helping par ticipants take responsibil-
ity fo r their financial o bligations such as child support. In addition, employment is
often an important program goal for drug court clients. If the client is working, he or
she is paying taxes and is less likely to use social welfare programs. Thus, the drug court
program potentially re d uces several di fferent ca tegories of costs that might have
accrued had program participants not r eceived treatment. These “avoided” costs or
benefits are important com po nents to a full eco nomic eva lua tion of drug court
programs.
and the benefits are more di fficult to convert into dollars. For example, Logan et al. (2004)
fo un d that the average direct cost per chug court treatment ep iso de for a grad uate was
$3,319, w hereas the op portunity cost per episode was $5, 132. These differences in costs
due to agency collaboration highlight the importance of clearly defining the perspective
of the cost analysis. As discussed earlier, the trade-off in presenting only out-of-pocket
expenditures for a progra m or accurately refl ect in g all of the costs associated with the
prog ram regardless of whether those costs are paid out of pocket is an important distin c-
tion that should be co nsi dered at the outset of every economic evaluation . On the benefit
side of the program, results suggest that the net economic ben efit was 514,526 for each
graduate of the program. In other words, this translates to a return of $3.83 in economic
benefit for every dolla r invested in the drug court programs for graduates. Obviously,
those who dropped o ut of the program before comple ti ng di d not generate as large of a
r et urn. However, res ul ts suggest that when both gra duates and terminators were exam-
ined together, the net economic benefit of any drug court experience amounted to $5,446
per participant. This translates to a return of $2.71 in economic benefit for every dollar
invested in the drug court programs.
EVALUATION
H o w t o B e c o m e S a v v y E v a l u a t i o n C o n s u m e r s
is among the largest philanthropic foundations in the United States.
Based in Battle Creek, Michv, WKKF engages with communities in priority places across the country
and internationally to create conditions that propel vulnerable children
to realize their full potential in school, work and life.
© 2017 W.K. Kellogg Foundation.
cial purposes. Such use must include acknowledgement that it is “with the permission of the W.K. Kellogg
Foundation.” All other use is prohibited without the prior written permission of the W.K. Kellogg Foundation.
For permission requests, please contact communications@wkkf.org or 269-969-2079.
Foreword
At the W.K. Kellogg Foundation, we believe that evaluation is an effective management tool to both inform
strategy development and track the progress and impact of strategy implementation. We have long been
committed to supporting our grantees’ ability to derive and share lessons learned from their work. To that
end, the foundation published the first Evaluation Handbook (The Kellogg Foundation, 1998) almost two
decades ago to guide evaluation for our grantees. Since that time, as the discipline of evaluation grew and
expanded, the demand for evaluation has risen. More and more nonprofit leaders and practitioners strive
to design evidence-based programs, and more and more funders require their grantees to provide evidence
to demonstrate the success of their funded work. The democratization of evaluation makes it necessary that
evaluation is both rigorous and practical. How to achieve the balance motivated us to update the handbook.
Over the years, the foundation has learned a lot from our grantees about the challenges of evaluation. This
handbook is our continuous effort to demystify evaluation and facilitate its use, for the foundation’s grantees
and for all organizations committed to learning and strengthening their work. It is designed for people who
have little to no exposure to formal evaluation training and provides a starting point for them as they
consider evaluating their work. It is intended to help them become more informed consumers of evaluation.
Evaluations can be simple or extensive depending on the scope and complexity of the work being evaluated.
The scope of the evaluation could potentially include a single program, a multi-site initiative, or a
multifaceted strategy aimed at systems and community change. Regardless of the complexity of the effort,
the basics for evaluating it remain the same and this handbook was written to impart information about
these basics. This handbook complements two other products produced by the Kellogg Foundation: the Logic
Model Development Guide (The Kellogg Foundation, 2004) and the Systems-Oriented Evaluation Guidebook
(The Kellogg Foundation, 2005).
We would like to thank Community Science for writing and HGF, Inc., for designing this handbook, WKKF staff
for their leadership in developing this resource, as well as the following individuals who served as advisors:
Jara Dean Coffey, Traci Endo, Johanna Morariu and Jianping Shen.
Huilan Krenn, Ph.D.
Director of Learning & Impact
W.K. Kellogg Foundation
September 2017
Contents
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Chapter 1: Principles to Guide Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 W.K. Kellogg Foundation’s Principles about Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Community Engagement, Racial Equity and Evaluation . . . . . . . . . . . . . . . . . . . . . 3
1.1.2 Importance of Culture in Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2 American Evaluation Association’s Guiding Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Chapter 2: Definition of Evaluation and the Evolution of Evaluation Practice . . . . . . . . . . . . . . . . . 13
2.1 What is Evaluation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.1.1 Definition of Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.1.2 Definition of Evaluative Thinking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2 Evolution of Evaluation Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Chapter 3: Evaluation Types, Methodologies and Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1 Evaluation Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.2 Evaluation Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.3 Evaluation Methodologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Chapter 4: Overview of the Evaluation Process that Reflects Evaluative Thinking . . . . . . . . . . . . . . 37
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.1 Overview of the Evaluation Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.2 Creating a Learning Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.3 Stages of the Evaluation Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.4 Navigating Evaluation Choices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.4.1 Capacity, Program, or Population Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.4.2 Focus or Unit of Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.4.3 Internal Versus External Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.4.4 Power Dynamics Among Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Chapter 5: Preparing for the Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.1 Why You Might Consider an Evaluation of Your Strategy, Initiative or Program . . . . . . . . 66
5.1.1 Considerations if Your Evaluation
is Part of a National or Statewide Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.1.2 Considerations if the Findings from Your Evaluation
Have Significant Implications for a Theory, Practice or Policy . . . . . . . . . . . . . . . 68
5.1.3 Getting Ready for Evaluation Whether or Not It is a Funding Requirement . . . . . 69
5.2 Who Benefits From the Evaluation and What Are the Potential Risks? . . . . . . . . . . . . . . . 70
5.2.1 Consideration of Different Types of Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.2.2 Use of Evaluation by Different Types of Stakeholders . . . . . . . . . . . . . . . . . . . . . . 73
5.3 Consider Your Organization’s Capacity to Participate in the Evaluation . . . . . . . . . . . . . . 75
5.3.1 A Staff Person Dedicated to Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
5.3.2 Budget for Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
5.3.3 A Culture of Data and Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
5.3.4 Share and Protect Data Across Program Elements or Program Locations . . . . . . 79
Chapter 6: Determine Stakeholders and Engage Them in the Evaluation . . . . . . . . . . . . . . . . . . . . 83
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6.1 Evaluators as Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
6.1.1 Keep Your Evaluator Informed on a Frequent, Regular Basis . . . . . . . . . . . . . . . . 84
6.1.2 Two Types of Evaluators: Internal and External . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
6.1.3 The Evaluator’s Role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
6.1.4 The Evaluator’s Cultural Competency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
6.2 Other Key Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
6.2.1 How to Identify Key Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
6.2.2 How to Engage Key Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Chapter 7: Developing a Logic Model, Evaluation Questions,
Measurement Framework and Evaluation Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
7.1 Overview of Logic Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
7.2 Use the Logic Model throughout Your Effort’s Life Cycle . . . . . . . . . . . . . . . . . . . . . . . . . 109
7.2.1 Use of Logic Models to Design Your Strategy, Initiative or Program . . . . . . . . . . 109
7.2.2 Use of Logic Models to Inform Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 109
7.2.3 Use of Logic Models to Develop Evaluation Questions . . . . . . . . . . . . . . . . . . . . 110
7.2.4 Use of Logic Models to Develop an Evaluation Plan . . . . . . . . . . . . . . . . . . . . . . 110
7.2.5 Use of Logic Models to Create an Outline for Your Evaluation Report . . . . . . . . . 111
7.2.6 Use of Logic Models To Improve Your Strategy, Initiative or Program . . . . . . . . . . 111
7.3 What You Can Expect From Logic Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
7.3.1 It is a Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
7.3.2 The Process Can Surface Differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .113
7.3.3 The Process Can Bring About Innovation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
7.4 How to Go About the Logic Modeling Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
7.4.1 Whom to Involve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
7.4.2 How to Involve Them . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .115
7.5 Components of the Logic Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
7.5.1 Alignment Between Strategies or Activities and Outcomes . . . . . . . . . . . . . . . . 118
7.6 From Logic Model to Evaluation Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
7.6.1 Two Different Ways to Formulate Evaluation Questions
About Your Strategy, Initiative or Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
7.6.2 Typical Scenarios When Formulating Evaluation Questions . . . . . . . . . . . . . . . . 123
7.6.3 Other Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
7.7 From Logic Model to Evaluation Questions to Measurement Framework . . . . . . . . . . . . 124
7.7.1 Key Components of the Measurement Framework. . . . . . . . . . . . . . . . . . . . . . . . 125
7.7.2 How to Use a Measurement Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
7.7.3 Considerations for Developing the Measurement Framework . . . . . . . . . . . . . . 127
7.7.4 Taking a SMART Approach to Measurement Framework Development . . . . . . . 127
7.8 Your Evaluation Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
7.9 Budgeting for Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
7.9.1 Types of Expenses Typically Associated With Evaluation . . . . . . . . . . . . . . . . . . 134
7.9.2 Time Estimates for Specific Evaluation Activities . . . . . . . . . . . . . . . . . . . . . . . . 136
Chapter 8: Data Collection and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
8.1 Determine Data Collection Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
8.1.1 What You Need to Ask Before Collecting Any Data . . . . . . . . . . . . . . . . . . . . . . . 143
8.1.2 Use of Quantitative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
8.1.3 Use of Qualitative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
8.1.4 Use of Mixed Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
8.2 Critical Considerations in Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
8.2.1 Who the Data Collector Is . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
8.2.2 How to Engage Populations Who Have Been Traditionally
Excluded or Treated as Invisible . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
8.2.3 Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
8.2.4 Confidentiality and Anonymity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
8.2.5 Obtaining Consent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
8.2.6 Institutional Review Board (IRB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
8.3 Analyzing and Interpreting Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
8.3.1 Quantitative Data Analysis and Interpretation of Results . . . . . . . . . . . . . . . . . . 167
8.3.2 Qualitative Data Analysis and Interpretation of Results . . . . . . . . . . . . . . . . . . . 175
Chapter 9: Summarize, Communicate and Reflect on Evaluation Findings . . . . . . . . . . . . . . . . . . 183
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
9.1 Communicate and Report Your Evaluation Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
9.1.1 What You Need to Ask Before Putting the Findings Together . . . . . . . . . . . . . . . 184
9.1.2 Different Communicating and Reporting Formats and Options . . . . . . . . . . . . . 187
9.2 Considerations in Developing and Implementing a Communications Plan . . . . . . . . . . 188
9.3 Keeping It Simple . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
9.4 Ways to Display Your Evaluation Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
9.4.1 Displays of Patterns and Trends Over Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
9.4.2 Displays of Distribution and Spread of Responses . . . . . . . . . . . . . . . . . . . . . . . 194
9.4.3 Displays of Comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
9.4.4 Displays of Frequency of Words . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
9.4.5 Displays of Themes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
9.5 Reflecting on Your Evaluation Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
9.5.1 Use of Evaluation Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
9.5.2 Considerations in Preparation for Reflecting on,
Discussing and Using Evaluation Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
Guide to Evaluation Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
What is evaluation? How can it help your organization?
And how can you work more effectively with your evaluator?
This handbook is designed to demystify evaluation and help you
get the most out of evaluation for your organization.
Overview
What is evaluation? How can it help your organization? And how
can you work more effectively with your evaluator? This
handbook is designed to demystify evaluation and help you get
the most out of evaluation for your organization.
The evaluation profession is multi-faceted and can be
characterized partly by its theoretical debates, ethical
considerations and proprietary interests. This handbook will
not attempt to provide in-depth evaluation information that has
little relevance to the day-to-day evaluation needs of nonprofit
directors and staff, nor will it transform anyone into an
evaluator. On the contrary, the goal of this handbook is to
educate busy nonprofit directors and staff, such as yourself,
about the essential elements of evaluation, so you can work
more effectively with trained evaluators; hold evaluators
accountable to the highest standards of quality, integrity and
competency; and maximize the usefulness of evaluation to your
organization.
The scope of your evaluation may include a single program, a
multi-site initiative or a multifaceted strategy aimed at systems
and community change. Regardless of the complexity of the
effort, the basics for evaluating it remain the same and this
handbook was written to impart information about these basics.
Before discussing each chapter in detail, here are a few things to
know about how this handbook is organized:
When the term “you” is used in this guide, it refers to you –
LOOK FOR THESE
This symbol
indicates a new
term is introduced.
The handbook
provides many definitions, all of
which are combined into one list
(“Glossary”) at the end of the
guide. This can be printed as a
reference.
This symbol is
used to summarize
tips.
This symbol refers
to an important
point.
the reader or user of this handbook.
“Highlights” will be included in most chapters to provide a
concise summary of key points.
Brief examples are included within the chapters to make it
easier to understand the information.
Short exercises are included at the end of chapters to help
you retain its key points.
Many publications were consulted in creating this handbook.
You will find the references to these publications at the end
of the handbook.
An annotated resource guide for evaluation was created to
supplement this handbook; you will find it at the end of the
handbook. The resources are organized by evaluation topic
as well as issue such as education, poverty, health and
workforce development. Additionally, relevant and practical
resources cited in each chapter are included here.
This handbook contains nine chapters.
Chapter 1 synthesizes the W.K. Kellogg Foundation’s philosophy
concerning evaluation and the American Evaluation
Association’s guiding principles.
Chapters 2 and 3 explain essential information about the
evaluation field. As mentioned before, the evaluation field is
vast, so only some aspects of evaluation are summarized here.
Chapter 2 lays out the historical development and significant
contributions leading to the current status of the evaluation
field. Chapter 3 describes the various evaluation types, method-
ologies and approaches.
Chapter 4 is a brief overview of the overall evaluation process and its stages, which align with the
subsequent chapters.
Chapters 5 through 9 provide more detail about each stage in the evaluation process and explain
what you should know to work most effectively with an evaluator or how to use evaluation in your
work. Chapter 5 discusses some key considerations when planning your evaluation. Chapter 6
explains how to determine your evaluation stakeholders and engage them in the evaluation
process. Chapter 7 focuses on developing and using a logic model and on generating evaluation
questions, a measurement framework and an evaluation plan. Chapter 8 provides steps for
determining what type of data to collect and guidance on analyzing and interpreting the data.
Finally, in Chapter 9, you will learn considerations for summarizing and communicating your
evaluation findings.
To the extent possible, each chapter is written as a stand-alone section. Depending on what you
want to learn or what you already know, you can select the chapters you want to read. Thus, you
will find some information (e.g., definitions, concepts, resources) repeated in several chapters.
1
Principles to Guide Evaluation
introduction
This chapter is intended to
help you understand the W.K.
Kellogg Foundation’s (WKKF)
principles and the American
Evaluation Association’s
principles concerning
evaluation. As a current or
future grantee of the
foundation, you could find the
foundation’s principles
informative. Accordingly, the
guiding principles—upheld
by the American Evaluation
Association—will help you, as
consumers of evaluation,
to know what to expect of
professional of evaluators as
you work alongside them.
The foundation emphasizes
the necessity of incorporating
culture, community engagement
and racial equity into efforts to
effect community and systems
change; as such, you can see
these topics throughout the
handbook.
1.1
w.k. kellogg foundation’s principles
about evaluation
The Kellogg Foundation believes that all people have the
inherent capacity to effect change in their lives, in their
organizations and in their communities. What often limits the
ability to effect change is the availability of tools and resources
to actualize community desire for change. The foundation is
therefore committed to supporting the change process by
providing tools and resources that assist individuals,
organizations and communities in the generation and practical
application of knowledge that will result in improvements in the
quality of life for children and inform the narratives about certain
groups of people. Evaluation is one such tool.
Evaluation supports the ability to monitor and measure the
quality, pace and direction of change that individuals,
communities and organizations undertake. It does this by
systematically generating knowledge that can support learning,
How this chapter is organized…
1.1 W.K. Kellogg Foundation’s Principles about Evaluation
1.1.1 Community Engagement, Racial Equity and Evaluation
1.1.2 Importance of Culture in Evaluation
1.2 American Evaluation Association’s Guiding Principles
Exercises
quality improvement and good judgment in decision-making. Evaluation also can align purpose,
action and impact to ensure that longer-term change at the societal level unfolds progressively.
The foundation believes the following principles should guide evaluation:
Planning for evaluation should begin the moment new strategies, initiatives and programs are
conceptualized, evaluation findings should be used both to strengthen strategy development
and programming along the way, and to measure the extent of change.
Evaluation should support an organization’s efforts to become stronger and more effective and
should enhance its ability to obtain and effectively use new resources.
Evaluation should be adapted to the contexts of the community being evaluated and to the
important outcomes identified by the community (e.g., policy, impact on equity). Mixed
methods and different perspectives (e.g., a racial equity lens) can help capture the reality and
outcomes experienced by community members.
Evaluation should be designed to address real issues and to provide staff and stakeholders
with reliable information to address problems and to build on strengths and opportunities.
Mixed methods intentionally use two or more kinds of data gathering and analysis
tools — typically a combination of qualitative (e.g., focus groups and
interviews) and quantitative (e.g., multiple choice surveys and assessments) —
in the same evaluation.
A stakeholder is any person or group who has an interest in the strategy, initiative or
program being evaluated or in the results of the evaluation, including the evaluator.
Evaluation should invite multiple perspectives and involve a representation of people who care
about and benefit from the program.
Evaluation should be flexible and adaptable; strategies, initiatives and programs don’t exist
in a vacuum and events such as staff turnover, elections, legislation and economic recession
can affect their implementation and outcomes. Therefore, evaluators and implementers must
be flexible and work together to adapt to such events and respond to the needs of community
members.
Evaluation should build the skills, knowledge and perspectives of individuals to self-reflect,
dialogue and act based on data and knowledge. This strengthens the capacity of all
participants to establish a learning environment and work together to solve problems.
1.1.1
Community Engagement, Racial Equity and Evaluation
Community engagement and racial equity are central to the Kellogg Foundation’s work. In
evaluation, they are interconnected, especially in community change initiatives because people
of color usually make up Kellogg’s stakeholders and are usually the ones left out of discussions
about the evaluation design and implementation. Consider, for instance, an initiative designed to
improve the identification, referral and treatment of children exposed to violence in a Native
American community. Due to historical trauma experienced by Native American communities,
the evaluator must engage tribal leaders in discussions about the best ways to measure the
identification, referral and treatment, along with realistic expectations for change over the
initiative’s grant period. The evaluator should not assume the success measures for this
community are the same as for other communities because of the history, governance structure,
location, socioeconomic conditions and traditional norms of the tribe. This means paying explicit
attention to factors such as:
How tribal leaders and community members talk about the violence they experience.
What tribal and nontribal mental health resources exist.
What nontribal mental health resources are available and how culturally competent are the
resources.
Who needs to give the evaluator permission to collect data.
Who owns the data that is collected.
How to ensure the findings are not misused.
In evaluation, community engagement can look like the following:
Advice on the most appropriate markers of change or indicators for their community as well as
the amount of change that is desired and possible.
Help with determining the best data collection methods and data sources for their community.
Assistance with data collection, provided that those assisting receive proper training
and support.
Discussion about findings and what they mean for the community.
Transformation of the findings into useable knowledge and products that can be leveraged for
advocacy and other types of action.
Community engagement in
evaluation is commonly referred
to as stakeholder engagement.
Some evaluators emphasize
they practice community-based
participatory research or a
participatory approach.
Regardless of what it is called,
the points mentioned above
should still be considered.
Indicators are markers of progress toward the change you hope to make with your
strategy, initiative or program.
Funders, evaluators and nonprofit directors must pay attention to power dynamics when engaging
community in evaluation. Specifically, they should:
Determine who constitutes members of the “community” – for example, residents only;
residents and directors and staff of public, nonprofit and private organizations that live and
work in a place; all the individuals and the elected officials who represent their interests at
local and state levels; or networks of people connected through a shared identity or goal.
Ensure that people who traditionally have been excluded
from decisions that affect their lives – typically people of
color; low-income families; youth; lesbian, gay, bisexual,
transgender and queer (LGBTQ) people; and people with
different disabilities – are engaged in a way that values and
is not dismissive of their perspectives.
Avoid only listening to leaders of large, well-endowed and
established institutions.
Create various avenues for engaging people in the
community, from social media sites to town hall meetings
and small group discussions convened by individuals
trusted in the community, especially by the groups of people mentioned above, and conducted
at safe and convenient locations.
The funders and professionals designing and implementing the evaluation will need to consider
the above points and determine the most feasible, meaningful way to engage the community given
the resources available and timeframe for the evaluation.
Racial equity in evaluation can look like the following:
Framing evaluation questions in a way that doesn’t make assumptions and perpetuate
stereotypes about a particular group of people. For example, if all youth of color are lumped
together and labeled “at-risk youth,” the evaluation could end up looking only at negative
behaviors and how the initiative improved the behaviors and neglect to identify positive
behaviors and how the initiative helped to reinforce these behaviors. Don’t use words in the
questions that make a judgment – good or bad – about a particular group of people.
Ensuring the evaluation process doesn’t maintain the status quo by excluding people of color
as decision-makers, problem-solvers, advisors, information sources or end-users of
the results.
Recognizing and incorporating the knowledge of people of color into the evaluation design
and implementation.
Designing data collection protocols and instruments in the languages spoken by the desired
respondents and at the appropriate reading level.
Accounting for situations and events in the community that could affect racial and ethnic
disparities in health, education, socio-economic status and other well-being outcomes in the
evaluation design if the situations and events are targets of change (e.g., leadership changes,
closing of a factory that was the biggest employer in the town, sudden influx of new
immigrants, budget cuts, etc.). A good evaluation should consider all these possibilities as
“alternative explanations” for the outcomes in order to determine the extent to which the
intervention contributed to the changes observed.
Bringing attention to the systems, policies and social norms that affect racial and ethnic dis-
parities in health, education, socio-economic status and other well-being
outcomes in the evaluation design and implementation. (See the Kellogg Foundation’s 2016
guide on systems-oriented and culturally responsive evaluation.)
Culture is a set of socially transmitted and learned behavior patterns, beliefs,
institutions and all other products of human work and thought that characterize the
functioning of a particular population, profession, organization or community.
Culture is continually evolving.
1.1.2
Importance of Culture in Evaluation
The word culture is often used without fully understanding its meaning. Depending on the
environment or context, it can be used to imply race, ethnicity, sexual orientation, gender identity,
disability and income group. But, “culture” is more than that.
Individuals who share the same culture form relationships that become the basis for a sense of
community. Individuals typically participate in multiple communities simultaneously. These
communities may face racial and other inequities because their cultures have historically been
Resources on designing and managing equity-focused evaluation include:
Borgman-Arboleda, C. & Clark, H. (2010). Considering evaluation: Thoughts for social change
and movement-building groups, by C. Borgman-Arboleda and H. Clark, published by
ActKnowledge in 2010. Available at http://www.actknowledge.org.
How To Design And Manage Equity-Focused Evaluation by M. Bamberger and M. Segone,
published UNICEF in 2011. Available at http://betterevaluation.org/resource/guide/
design_manage_equity_focused_evaluation.
design_manage_equity_focused_evaluation
design_manage_equity_focused_evaluation
suppressed, ignored, dismissed or undervalued. Consequently, people in these communities have
been excluded from decisions that affect their lives and from resources and opportunities that
would enable them to have the best life possible. Culturally competent evaluators are committed
to, and skilled in engaging people in these communities in the planning and implementation of an
evaluation. They also promote racial, gender and other forms of equity through the evaluation.
While training in cultural competency for evaluators has been receiving more attention, culture and
context still are not adequately addressed in evaluation, which can affect the accuracy of findings.
Culture and context can be considered in evaluation design and implementation in ways such as:
Engaging respected leaders in an African American community to facilitate entry into the
community for data collection and other evaluation activities.
Respecting how Muslim women might respond to a female or male evaluator, as their
responses are regulated by a set of behavior patterns and beliefs about gender roles in their
culture.
Bearing in mind the way African Americans and Central American immigrants and refugees
respond to the question, “How many family members live with you?” is shaped by their cultural
values about family, which typically extend beyond the nuclear family.
Understanding that appropriate terms to describe a geographic area vary in urban versus rural
settings (e.g., the term “neighborhood” would not be appropriate in a rural community).
Recognizing the possibility that a white evaluator may not have the skills to respond to racist
comments made by a white respondent who assumed the evaluator and respondent think alike
about other racial groups because they have the same skin color.
Recognizing that in a focus group or in interviews using inappropriate terms such as “black
boys” to refer to African American men can be offensive and will diminish the study’s
effectiveness.
Also, your work and its evaluation don’t exist in a vacuum; organizations and communities have
cultures that are affected by dynamics within, around and between your organization and
community and the formal systems (e.g., education system, health system) and informal systems
(e.g., faith beliefs, extended family networks) within which they are situated. Context refers to the
combination of factors or circumstances surrounding the strategy, initiative or program and the
evaluation that could influence implementation, results and use of the findings.
Make sure that the appropriate terms familiar to the participants and community
members are used in materials for the evaluation, from introductory letters to survey
questionnaires (e.g., the term “household” may be less familiar to some cultural groups than
simply “family”).
Design data collection methods that are easy and comfortable for the participants (e.g.,
an online survey might not be convenient for people with limited or no access to
computers).
Ensure that questionnaires are in a language understood by potential respondents (e.g.,
in the language potential respondents are proficient in, and at a reading level that they
can understand).
Engage an interpreter for focus groups and interviews when necessary.
Interpret the findings to ensure that any community conditions (e.g., leadership chang-
es in the local government, closing a factory that was the biggest employer in the city,
passing of legislation, sudden influx of new immigrants or refugees) that could have
affected the results (e.g., the loss of a major employer in the city that might have
contributed to poor participation in a program for mothers who had to work more hours
to help supplement their household income) are discussed in the evaluation report.
checklist
You Can Work With an Evaluator to:
Useful resources on the role of culture in evaluation include:
The Importance of Culture in Evaluation, by Kien Lee, published by The Colorado Trust in 2007.
Available at http://www.communityscience.com/pdfs/CrossCulturalGuide.r3 .
A Guide to Culturally Response Evaluations, by H.T. Frierson, S. Hood and G.B. Hughes,
included in the National Science Foundation’s 2002 User-Friendly Handbook for Project
Evaluation. Available at http://www.nsf.gov/pubs/2002/nsf02057/nsf02057_5 .
1.2
american evaluation association’s guiding principles
Section 1.1 describes the Kellogg Foundation’s values concerning evaluation. Equally important is
how the American Evaluation Association – the national association for evaluation in the United
States – sets guidelines for evaluators (see www.eval.org for the Association’s guiding principles
for evaluators). The association’s mission is “to improve evaluation practices and methods, in-
crease evaluation use, promote evaluation as a profession and support the contribution of eval-
uation to the generation of theory and knowledge about effective human action.” To this end, the
American Evaluation Association created the following five principles to guide the professional
practice of evaluators and to inform evaluation clients and the general public about the principles
they can expect professional evaluators to uphold. Detailed explanations about the principles can
be downloaded from the association’s website mentioned above; they are abbreviated here.
Systematic Inquiry: Evaluators are expected to conduct systematic, data-based inquiries where
they discuss with you the strengths and limitations of the proposed evaluation design, adhere to
the highest technical standards in employing their methods and communicate their methods and
approaches accurately and in sufficient detail. The inquiries can be based on both qualitative and
quantitative data.
Competence: Evaluators should possess the qualifications, including cultural competency, needed
to undertake the tasks proposed in the evaluation; practice within the limits of their professional
training and competence; and participate in continuing professional development.
Cultural competency is to the commitment and ability (e.g., knowledge, skills) to
respect and engage with diverse segments of communities and to include the
contextual and cultural dimensions relevant to these diverse segments in the
evaluation design and process.
Integrity and Honesty: Evaluators are expected to behave with honesty and integrity during the
entire evaluation process. They are responsible for initiating discussion to clarify the costs, tasks
to be undertaken, limitations of methodology, scope of results likely to be obtained and uses of
data resulting from the evaluation. They should disclose any roles or relationships they have that
could pose a conflict of interest (or appearance of a conflict) with their role as an evaluator. Finally,
they are expected to inform you if certain procedures or activities likely will produce misleading
evaluative information or conclusions.
Respect for People: Evaluators should respect the security, dignity and self-worth of respondents,
program participants, clients and other evaluation stakeholders. This means that they understand
and account for cultural and contextual factors that could influence the results of a study (e.g.,
languages spoken, geographic location, timing, political and social climate, economic conditions
and other relevant activities in progress at the same time). They also are responsible for
communicating clearly the risks, harms and burdens that could affect participants in the
evaluation. Evaluators should help foster social equity by ensuring the people who share
information benefit from their contributions.
Responsibilities for General and Public Welfare: Evaluators are expected to consider relevant
perspectives and interests of the full range of stakeholders and allow all relevant stakeholders
access to evaluative information in ways that respect people and honor promises of confidentiality.
Additionally, evaluators should not ignore any clear threats to the public good that could surface
during the evaluation process.
Why are these principles important? These principles help you, as consumers of evaluation, to
know what is professionally expected of evaluators as you consider and select an evaluator to work
with you.
exercises
1. This group exercise is best done after the staff members in your organization have read Chapter
1 of this handbook. Think about a strategy, initiative or program in which you are involved that
you’d like to be evaluated. Ask yourselves these questions:
A. How can the community be engaged and what factors have to be considered to maximize
members’ engagement?
B. How can you engage the community in conversations about the history of racial
discrimination and oppression, community asset and experiences with larger systems
(e.g., education, law enforcement, health, housing) that can help you understand the
community context from the community members’ perspective?
2. You learned that your evaluator submitted a manuscript for publication to a peer-reviewed
journal based on the evaluation of your initiative. You asked to see the manuscript and after
reading it felt that it misrepresented the initiative’s impact. Which of the guiding principles can
be used to raise and frame your concerns to the evaluator?
3. The evaluation of your collaborative’s advocacy effort for early childhood education is delayed
because several collaborative members did not provide the information needed to the
evaluator. You get a call from your project officer at the funding agency who said he heard from
the evaluator about problems with the collaborative and he wanted to know what was going
on. The project officer suggested he attend the next collaborative meeting to let everyone know
they must cooperate with the evaluator. As the collaborative’s coordinator, how would you han-
dle this situation? (Hint: What information do you need to further understand what is going on?
What power dynamics could be at play here?)
2 Definition of Evaluation and the Evolution of Evaluation Practice
2.1
what is evaluation?
Two important terms are “evaluation” and “evaluative thinking.”
2.1.1
Definition of Evaluation
The purpose of evaluation is to facilitate learning and improve
your strategy, initiative or program. This learning happens
through a process of collecting and summarizing evidence that
leads to conclusions about the value, merit, significance or
quality of an effort. Funders, program implementers and
policymakers often evaluate the strategies, initiatives or pro-
grams they support to determine whether they are worth con-
tinuing support. Although this is not the only use of evaluation,
many people tend to look at evaluation as something that
results in a judgment about the merit of their performance and
work. Often they dislike evaluation because the judgment could
seem unfair for various reasons. For example, they might believe
that inappropriate metrics were used in forming the judgment,
or the evaluator failed to understand what they are trying to do,
or they could resent findings that show their work did not have
the anticipated impact.
You have probably encountered these situations before. You
might have been hesitant to tell your funders anything that
didn’t work; you could lack confidence that your evaluator will
help you frame the findings within the proper context; worst of
all, you might not see evaluation as worthwhile for any purpose
other than to fulfill your funding requirement. Consequently, the
power of evaluation to facilitate learning and improve your strat-
egy, initiative or program is diminished.
How this chapter is organized…
2.1 What is Evaluation?
2.1.1 Definition of Evaluation
2.1.2 Definition of Evaluative
Thinking
2.2 Evolution of Evaluation
Practice
Highlights
Exercises
Some funders will continue to
use evaluation to judge their
grantees’ performance and
worth. This use of
evaluation will not go away, but
it should not be the sole purpose
of evaluation. The aim of this
handbook is to hone your ability
to work with evaluators or with
your staff (if you don’t hire an
external evaluator) to measure
the “right thing”, and to
effectively communicate and use
the results to become better at
what you do. This includes
communicating to funders – and
the people you work with or
serve – about why something did
or did not work.
2.1.2
Definition of Evaluative Thinking
Underlying evaluation is a way of thinking about what results are expected, how results can be
achieved and what data or evidence are needed to inform future actions so that results can be im-
proved. This is called evaluative thinking. At its core are dialogue, reflection, learning and improv-
ing. If you consider evaluation not just as an inquiry that leads to a judgment (did the
program or organization perform as expected?) but also as evaluative thinking, you will become
more comfortable with evaluation and might even embrace it as part of your organization’s culture
and daily operations. Evaluative thinking is a “muscle” that needs to be exercised regularly to
become better and stronger.
Whether you perform an evaluation yourself or work with an external evaluator, in order to benefit
further from evaluative thinking and evaluation you might need a stronger understanding of the
process and a system for planning and implementing it. Also, evaluative thinking should be
integrated into the day-to-day operations of your organization to improve your use of data and
data-informed decision-making and ultimately your efforts, services and products.
collecting and analyzing data instead of a set of disorganized, random opinions —
and telling the story about your strategy, initiative or program. It is based on the
belief that a systematic process is valuable and necessary. This involves identifying
assumptions about what you think works and doesn’t work and why; posing
thoughtful questions about what you expect to see differently during and after you
implement your effort; pursuing deeper understanding through reflection and
dialogue; communicating what was learned without underestimation or
exaggeration; and making informed decisions in preparation for action. Evaluative
thinking also has the potential to shift the narratives about certain issues and groups
of people by challenging widespread assumptions associated with them, providing
data to support alternative explanations, and shifting mindsets through education
and learning.
Create a learning agenda for your staff, board members or partners and designate a staff
person, board member or committee to be responsible for the learning agenda.
Collect, analyze and share data about the performance of the strategy, initiative or
program on a regular basis. This can range from basic data, such as attendance of
program participants, to more complex data, such as reading and math proficiency scores
for third graders in the school in which you work.
Allocate time at staff, team or collaborative meetings for structured discussion about
what everyone learned and how they can apply their new knowledge. Good questions to
ask to structure this discussion include:
What do you think the data tell you about what is happening?
Did the data surprise you? Why or why not?
What should you do differently, if anything?
What support do you need to overcome the challenges you face?
What additional information do you need to know or learn about what you are
doing or not doing?
checklist
Some ways to integrate evaluative thinking into your
organization and day-to-day operations:
A useful resource on evaluative thinking is Evaluative Thinking published by the Bruner
Foundation. Available at http://www.evaluativethinking.org/docs/EvaluativeThinking.
bulletin.2
bulletin.2
bulletin.2
evolution of evaluation practice
You might ask yourself why you need to know about the history of evaluation. One reason is to
understand the motivation behind evaluation and its evolution over time. This information will
increase your insight into how the funding, political and social contexts within which evaluations
are conducted changed over time and will help you know what to expect of evaluators today and as
the field grows.
Evaluation in the United States gained prominence in the 1960s in response to: efforts to
strengthen the U.S. defense system, new laws to serve racial and ethnic minorities and persons
with disabilities equitably and to requirements of the Great Society programs. In the 1970s, eval-
uation was used to assess if educational and social organizations used resources appropriately
and achieved their objectives. In the 1980s, evaluation helped promote excellence in response to
global competition. In the 1990s, evaluation was employed to ensure quality, competitiveness and
equity in service delivery, and the United Way of America started encouraging the nonprofit organi-
zations it funded to use a program outcome evaluation approach.
Evaluation in the 21st century has become increasingly diverse in its range of applications, from
defense policy to social media technology. Also, the world has become more complex with global
migration, widening gaps between rich and poor and threats to democracy. In response, the array
of evaluation methodologies also has had to increase. Innovative approaches, such as
developmental evaluation and culturally responsive evaluation, have emerged.
Developmental evaluation is an approach used to support innovation within an
organization and in its strategies, initiatives and programs. Efforts that are innovative
often are in a state of continuous development and adaptation, and they frequently
unfold in a changing and unpredictable environment.
A useful resource about developmental evaluation is A Developmental Evaluation Primer by
J.A.A. Gamble, published by The J.W. McConnell Family Foundation in 2008. Available at
http://tamarackcci.ca/files/developmental_evaluation_primer .
lie at the heart of any evaluation effort.
Resources about culturally responsive evaluation include:
Continuing the Journey to Reposition Culture and Cultural Context in Evaluation Theory and
Practice, by S. Hood, R. Hopson and H. Frierson, published by Information Age Publishing
in 2015.
Handbook of Critical and Indigenous Methodologies, by N.K. Denzin, Y.S. Lincoln and L.T. Smith,
published by SAGE Publications in 2008.
Practical Strategies for Culturally Competent Evaluation, by the Centers for Disease Control and
Prevention, published in 2014. Available at http://www.cdc.gov/dhdsp/docs/cultural_
competence_guide .
The Center for Culturally Responsive Evaluation and Assessment (http://education.illinois.
edu/crea).
In summary, evaluation today extends beyond the limits of its
original intent and approaches.
Experimental designs with control and treatment groups are
no longer the only or best ways to evaluate programs aimed at
improving the quality of life for children and their families. These
designs are well suited for stable programs in a fairly controlled
environment and where the goal is to determine if the program
led to the anticipated outcomes — to prove the causal link
between the program and outcomes. For example, an
experimental design may be appropriate for determining if an
academic enrichment program implemented by the Boys and
Girls Club led to improved grades. All other things being equal,
the children in the program (“treatment”) can be compared to
the children in another Boys and Girls Club that did not have the
program (“control”).
competence_guide
competence_guide
of people: one group receives the intervention (“treatment group”) and one does not
(“control group”).
Experimental designs, however, are not appropriate for complex community and systems change
initiatives. Cross-case study methodology is more appropriate for such initiatives. This
methodology, which builds on case study methodology, involves several cases and looks for
patterns across them to understand where the commonalities and differences lie and the possible
reasons for them. For example, this design is ideal for evaluating an initiative intended to
strengthen community collaboration to improve early childhood education in five communities
within a state. Each community makes up a “case,” and the five cases can be compared to one
another to look for common and unique patterns rooted in each community’s unique history,
culture and context. This design focuses on determining the initiative’s contribution to changes
in early childhood education. In a complex initiative such as this, determining causality is not the
goal since there are many factors that could contribute to the changes. You may have heard the
phrase “contribution versus attribution,” which refers to this sort of situation.
Another change in the evaluation field is the increasing attention paid to building the knowledge
and skills of nonprofit leaders and staff so they can effectively use evaluation to fulfill funders’
requirements; to inform their program design, implementation and improvement; and to
strengthen the organization. Evaluation capacity-building is an intentional process to increase
individual motivation, knowledge and skills and to enhance a group’s or organization’s ability to
conduct or use evaluation. Evaluation capacity-building efforts have led to positive results;
however, it takes more than knowledge and skills. A shift in organizational culture is needed for
the feedback loop between evaluation and program to become ingrained in the organization.
Finally, a relatively new development in the evaluation field is the emphasis placed on the role of
culture and context, particularly the traditions, histories and norms of people who have
traditionally been excluded or oppressed. This emphasis is important for three reasons:
Evaluation generates knowledge, and knowledge is power. Virtually every phase of the evalua-
tion process has political implications which will affect focus, decisions made, external percep-
tions of the program or organization and determination of those whose interests are advanced
and whose are ignored.
The traditions, histories and norms of a group of people affect the change desired and
achieved. Sometimes, expectations about outcomes are not realistic because more time and
effort are required to lay the groundwork for the change desired.
The attention paid to the role of culture in evaluation also raises questions about how personal
biases and institutional practices may devalue the perspectives of people historically excluded
from decisions that affect their lives. These potential biases and practices can be
acknowledged and factored into the evaluation design, implementation and reporting.
A useful resource about the history of developing culturally responsive evaluators is the New
Directions for Evaluation issue on Building a New Generation of Culturally Responsive Evaluators
Through AEA’s Graduate Education Diversity Internship Program (Number 143, Fall 2014).
HIGHTLIGHTS
Underlying evaluation is evaluative
thinking – using a systematic process of
collecting and analyzing data rather than
a set of disorganized, random opinions
in order to tell the story about a program,
strategy, policy or organization. This
process involves identifying assumptions
about what you think works, what doesn’t
and why; posing thoughtful questions
about the difference(s) you expect to see
during and after you implement your
strategy, initiative or program; pursuing
deeper understanding through reflection
and dialogue; communicating what was
learned without underestimation or
exaggeration; and making informed
decisions in preparation for action.
Recent developments in the evalua-
tion field include culturally responsive
evaluation and evaluation capacity-
building. These developments were
necessary in response to contemporary
issues such as global migration, widening
gaps between rich and poor and threats to
democracy. As consumers of evaluation,
you should be aware that their recent
prominence in the profession means that
not all evaluators are trained to be
culturally responsive or to build
organizations’ evaluation capacity.
exercises
1. Which of the following is a way to integrate evaluative thinking into your day-to-day operations:
A. Allocating time for regular and structured discussion during staff or team meetings
B. Collecting, analyzing and sharing data about the program once a year with program staff
C. Getting the evaluator you contracted to present to the staff once a year about the
evaluation findings
D. Disconnecting the evaluation of programs in your organization from your organization’s
overall learning agenda
E. None of the above
2. Consider the current, day-to-day operations in your organization. What opportunities are there
to integrate evaluative thinking? What are the first steps you can take?
3. You work for the city government. The mayor asked you to evaluate the effectiveness of the new
city plan to invest in mixed-income neighborhoods and more transportation options for resi-
dents. To ensure the inquiry process reflects evaluative thinking, which of the following should
you NOT do:
A. Work with elected and informal leaders who represent the affected areas to organize town
hall meetings and develop guiding questions to facilitate dialogues about the experiences
of residents
B. Convene resident leaders to discuss what a successful mixed-income neighborhood and
effective transportation options look like in order to determine success measures
C. Work with a professional to design and administer a community survey without getting
input on the survey from key stakeholders
D. Contact the community leaders you know, ask them what they think of the plan and ask that
they identify other leaders to whom you can speak
E. In addition to gathering the data, develop a plan to summarize and communicate the
findings to city council members and facilitate a discussion for improvement
4. Think about a time when you evaluated your strategy, initiative or program. Considering the
definition of evaluative thinking and the previous tips on how to integrate evaluative thinking
into your daily operations, what could you have done differently to more effectively use the
evaluation for learning and improvement?
Answers: 1A; 3C
3
introduction
Whether your organization receives funding from a private foundation or public agency, as a leader
within the organization, you want to introduce and integrate evaluative thinking into your daily work
to cultivate a learning organization. This means evaluating your strategy, initiative and program to learn
how to strengthen and improve them. It also means assessing and improving how your organization gen-
erates and applies knowledge. Working with local stakeholders, including staff and board members, and
sometimes community leaders, you typically have to make the following decisions:
The type of evaluation to conduct based on the developmental stage of your strategy, initiative or
program as well as the budget and timeframe.
Considerations of racial equity and culture in your evaluation.
Engagement of community in the evaluation process.
The best evaluator to hire.
How you will learn about the actions you could take based on the evaluation findings.
You can approach evaluation in different ways, depending on your
evaluation goals and the stage of your strategy, initiative or program.
Even if an external evaluator is hired to conduct the evaluation,
you are expected to take an active role in making these decisions.
You and your staff know the program, initiative or strategy better
than anyone else and will be the primary users of the evaluation’s
findings. And there is no “one size fits all” evaluation methodology.
The best evaluations are designed to provide you with the
information you need and to match the scope and complexity of
your strategy, initiative or program’s major activities.
Evaluation Types,
Methodologies and Approaches
How this chapter is organized…
3.1 Evaluation Type
3.2 Evaluation Approaches
3.3 Evaluation Methodologies
Highlights
Exercises
3.1
evaluation type
What is the purpose of your evaluation? To prove that you achieve the outcomes as intended or to
monitor if you are doing what you said you would do in your grant application? How often have you
been confused about whether or not you should be asking for a process evaluation or an outcome
evaluation? What about performance monitoring – how is this different from process or outcome
evaluation?
The type of evaluation you want depends on the stage that your strategy, initiative or program is
in and the purpose of the evaluation. Evaluation type is not the same thing as evaluation
methodology. Once you determine the type you should use, and depending on what you want to
accomplish through the evaluation process (e.g., build staff capacity, empower community
leaders, emphasize community assets), you can decide on the approach you want to take, After
that you can select from different methodologies, such as case study or quasi-experimental design.
Essentially, there are three major types of evaluation and each serves a specific function and
answers certain questions:
Performance monitoring
Process or formative evaluation
Outcome or summative evaluation
Sometimes, it could be necessary to conduct these three types of evaluation simultaneously.
exhibit 3-1: types of evaluation
Purpose Kinds of Questions Answered Timing
Pe
rf
or
m
an
ce
M
on
it
or
in
g
Ensures accountability for
program activities.
Demonstrates that resources
for the strategy, initiative or
program, are used as intend-
ed and managed well.
Monitors and reports on
progress toward pre-estab-
lished goals.
Provides early warning to
funder and management of
problems.
Have activities for the
strategy, initiative or
program been conducted
as planned?
Have products and services
been generated and
provided by the effort as
planned?
Has the effort accomplished
what it set out to do?
Performance monitoring can
be conducted throughout the
strategy, initiative or program
period, from beginning to end.
Pr
oc
es
s
or
F
or
m
at
iv
e
Ev
al
ua
ti
on
Seeks to understand if a
strategy, initiative or
program is being
implemented as planned
and according to schedule.
Assesses if the effort is
producing the intended
outputs.
Identifies strengths and
weaknesses of the effort.
Critical for informing
adjustments to the effort.
Has the strategy, initiative or
program been implemented
as planned and if not, why?
What has worked or not
worked and why?
What needs to be improved
and how?
Process or formative
evaluation should be
conducted at the start-up
period and while the strategy,
initiative or program elements
are still being adapted.
O
ut
co
m
e
or
S
um
m
at
iv
e
al
ua
ti
on
Investigates whether the
strategy, initiative or
program achieved the
desired outcomes and
what made it effective or
ineffective.
Assesses if the effort is
sustainable and replicable.
What changes did the
strategy, initiative or
program cause or
contribute to?
How did the effort cause or
contribute to the changes?
How is the effort going to be
sustained and replicated?
Outcome or summative
evaluation should be
conducted when immediate
and intermediate outcomes
are expected to emerge,
usually after the effort has
been going on for awhile, or
when it is considered
“mature” or “stable” (i.e.,
no longer being adapted and
adjusted).
3.2
evaluation approaches
Once you have determined if you want to conduct performance
monitoring, a process/formative evaluation, an outcome/
summative evaluation, or some combination of the three types
of evaluation, you need to consider how you want to
approach the evaluation. In other words, what do you want to
accomplish through the evaluation process? This decision will
depend on your values, especially about who your stakeholders
are and how you engage them in the evaluation process.
An evaluation approach is the way one goes about designing,
implementing and using the evaluation. Why is it important
for you to know about the various approaches? Evaluators are
trained in certain approaches and have their own values about
how to approach an evaluation. They tend to be partial to certain
approaches whose underlying philosophy they share. You don’t
need to know everything about the various approaches. But
when an evaluator says, “I practice empowerment evaluation,”
or “I practice culturally responsive evaluation,” you do need to
understand the implications for you, your organization, your
program and the people who are supposed to benefit from
the program.
Usually, the data collection
methods used for each type
of evaluation and the various
approaches to evaluation are the
same, such as interviews, focus
groups and surveys. The dif-
ference lies in how the different
methods are combined (e.g., which
is the main method) to answer the
evaluation questions.
Exhibit 3-2 describes some
common evaluation approaches
that you likely will encounter.
While they have similarities,
each approach has important
distinguishing factors. You don’t
have to decide which approach is
the best one to use, but you must
know what to expect for your
organization; your strategy,
initiative or program; and your
evaluation. This material is
reflected in the right-hand
column.
If the evaluator doesn’t specify
any such approach, don’t worry!
It usually means that the
professional is not partial to one
approach and should be willing to
work with you to design an
evaluation that works for you,
your organization and your
community.
exhibit 3-2:
unique attributes of some common evaluation approaches and
what you can expect from them
Brief Description Distinguishing Attributes What to Expect
A
pp
re
ci
at
iv
e
In
qu
ir
y
Focuses on a vision for what
the outcomes should be and
look like and then a plan
toward achieving that vision.
Evaluation users identify where
they have had good practices
in their initiative, strategy or
program and how to increase
these practices. This does
not mean that needs or
deficiencies are not addressed
in this methodology; they just
don’t become the major objects
of inquiry. The emphasis is on
a positive holistic vision versus
addressing discrete problems.
The inquiry process itself is an
intervention for change. The
process involves answering the
following types of questions:
What was your vision for
what you wanted to achieve?
As you reflect on your
experience with the
program, what was a high
point?
What did you feel was most
successful?
What are the most
outstanding stories or
moments that made you
proud?
Questions that focus on
positive topics.
Positive-oriented
vocabulary and rejection of
a deficiency model.
Emphasis on what worked,
how things can be better
and ways to practice and
sustain the solutions.
Cu
lt
ur
al
ly
R
es
po
ns
iv
e
Ev
al
ua
ti
on
Recognizes that cultural values,
beliefs and context lie at the
heart of any evaluation effort.
Explicitly ensures that the
voices of people who have
been historically excluded are
integrated into the design,
planning and implementation
of the evaluation. Incorporates
concepts of oppression into
the design.
resources to intentionally
understand the program,
the people affected by the
program and the history of
the place where the people
and program are located.
Close attention to the
experiences, assumptions
and biases of the evaluator
to ensure genuine
connection to the context
in which the evaluation is
occurring.
Brief Description Distinguishing Attributes What to Expect
D
ev
el
op
m
en
ta
l E
va
lu
at
io
n
Supports the process of
innovation within an
organization and in its
activities. Initiatives that are
innovative often are in a state
of continuous development and
adaptation and they frequently
unfold in a changing and
unpredictable environment.
Emphasizes learning and use
of learning to continuously
refine the innovation and to be
responsive to the environment
and context within which the
innovation is operating.
evaluation on program
agenda to continuously
benefit from the findings
and lessons generated by
the evaluator.
Active engagement of the
evaluator alongside the
program team, technical
assistance providers and
any other supports.
Em
po
w
er
m
en
t E
va
lu
at
io
n
Provides organizations with the
tools and knowledge that allow
them to improve their programs
through self-evaluation and
reflection. The evaluator often
serves as a coach or additional
facilitator, depending on the
organization’s capacity.
Program staff and participants
are involved, through
facilitated dialogues and
discussions, in articulating the
program, setting priorities for
the evaluation and determining
the measures and evidence
needed to monitor the
program’s progress and
success.
Involves time on the part
of staff, organizational
leadership and program
participants.
Builds the skills of those
involved to ask evaluation
questions, collect and
analyze data, interpret the
findings and learn from the
experience and findings.
Sy
st
em
s-
or
ie
nt
ed
al
ua
ti
on
Views a program, initiative,
strategy or policy as part of
social and natural systems and
subsystems that affect and are
affected by the effort’s capacity
to achieve its goals.
Emphasizes boundaries,
relationships and perspectives
within and across the systems
and subsystems, and the
program, initiative, strategy,
policy or other entities.
Attention to the multiple
systems of which the
program, initiative,
strategy, policy or other
entity is a part.
A Developmental Evaluation Primer, by J.A.A. Gamble, published by The J.W. McConnell
Family Foundation in 2008. Available at http://tamarackcci.ca/files/developmental_
evaluation_primer .
An Introduction to Empowerment Evaluation, by B. Sherriff and S. Porter. Available at
http://www.mrc.ac.za/crime/evaluation .
The Center for Appreciative Inquiry (http://www.centerforappreciativeinquiry.net/).
The Center for Culturally Responsive Evaluation and Assessment
(http://education.illinois.edu/crea).
Continuing the Journey to Reposition Culture and Cultural Context in Evaluation Theory and
Practice, by S. Hood, R. Hopson and H. Frierson, published by Information Age Publishing
in 2015.
3.3
evaluation methodologies
Methods refer to qualitative or quantitative techniques for collecting and analyzing data.
Procedures refer to steps that must be followed to ensure an effective evaluation (e.g., selection of
people to include in the evaluation or from whom to collect data). Exhibit 3-3 describes common
evaluation methodologies and what each means for determining outcomes.
A methodology is a set or system of methods and procedures used to answer
evaluation questions.
evaluation_primer
evaluation_primer
exhibit 3-3: types of evaluation methodologies
Brief Description
What It Means for Determining
the Outcomes
Ca
se
S
tu
dy
Focuses on understanding a unit (program, site or
situation) in its context, using a combination of qualitative
and quantitative data. It is ideal for studying a program
in-depth and how it unfolds and performs in a particular
context. The findings cannot be generalized (or applied) to
other situations.
The interaction between the
program and context is explicitly
studied. Multiple data sources are
required to inform the analysis
and findings. The outcomes are
described within the context in
which they occurred.
Cr
os
s-
ca
se
S
tu
dy
Takes several cases and looks for patterns across them to
understand where the commonalities and differences lie
and the possible reasons for the differences. Suitable for
evaluating a program or initiative taking place in multiple
sites, while accounting for the unique conditions in each
site and the common conditions across them.
One or more outcomes can be
explored across a number of cases
to determine how they unfold in
the same way or in different ways
across different contexts.
pe
ri
m
en
ta
l D
es
ig
n Assesses the causal effects of a program by comparing
two groups of participants – one that receives the
intervention (“treatment group”) and one that does not
(“control group”). Also known as randomized control trials
(RCTs). Program participants are assigned at random to the
treatment and control groups. The intervention must be
delivered consistently to everyone in the treatment group.
Demonstrates the outcomes were
achieved because of the program.
ua
si
-e
xp
er
im
en
ta
l D
es
ig
n Is similar to experimental design except there is no
random assignment of participants into treatment and
control groups. Before-and-after program comparison
(sometimes referred to simply as pre- and post-testing) is
one of the most common forms of quasi-experimental
design. Careful understanding of factors that could
influence the results is critical to accurately interpret
the results.
Useful for showing a certain level
of evidence about the degree to
which the program caused the
outcomes, but avoids the ethical
concerns involved in withholding
or delaying treatment or substitut-
ing a less effective treatment for
one group of participants (as in
the case of experimental design).
Brief Description
What It Means for Determining
the Outcomes
O
ut
co
m
e
M
ap
pi
ng
Assists in understanding how a strategy, initiative or
program affects the individuals, organizations and groups
(also referred to as “boundary partners”) it influences or
with whom it interacts. Outcomes are defined as changes
in the behavior, relationships, activities or actions of
the people, groups and organizations of these boundary
partners. Useful for answering four questions about the
program: What is the vision the program is contributing
to? Who are the program’s boundary partners? What are
the changes are being sought and occurring? How is the
program contributing to the change process?
Boundary partners are engaged
in the process of determining the
outcomes and in self-reflection
and monitoring. The outcomes
must be framed in terms of
observable changes in behavior,
relationships, activities and
actions.
program for two years and you now have funding to evaluate it. Here are some suggestions for how
you can prepare for the evaluation:
First, you probably want to conduct a process/formative and outcome/summative evaluation to
help you understand the extent to which the program was implemented as planned and the extent
to which the intended outcomes were achieved.
Second, you might prefer an external evaluator who practices culturally responsive evaluation
because the enrichment program takes place in schools located in communities that vary in
history, demographics and geographic settings. Or you might not have any preference for the
approach as long as the evaluator works with you to answer the evaluation questions and takes
into consideration the culture and context of the participating schools.
Third, you want to combine two methodologies: 1) a quasi-experimental design to compare the
grades of youth across the four schools in the district before and after participation in the
enrichment program and 2) a case study of the youth in two of the schools to understand more
deeply what contributed to the change (or lack of change).
Combining methodologies would strengthen your ability to practice evaluative thinking because
each methodology generates different information to help tell your story. Combining
methodologies, however, requires more resources and an evaluator with expertise in both
qualitative and quantitative methods. Alternatively, if you have resources, you could hire two
separate evaluators, but make sure you bridge their efforts so that at the end, you have one
coherent story and not two separate summaries to share. You also could ask the first evaluator you
hire to subcontract to a second evaluator with the required expertise.
Useful resources about evaluation methodologies include:
Case Study Research: Design and Methods, by R.K. Yin, published by SAGE, Inc. in 2009.
Cross-case Methodology: Bringing Rigour to Community and Systems Change Research and
Evaluation, by K.S. Lee and D.M. Chavis, published in the Journal of Community and
Applied Social Psychology, 2011, Vol. 22, Issue 5. Available at http://onlinelibrary.wiley.com/
doi/10.1002/casp.1131/full.
Outcome Mapping Learning Community at http://www.outcomemapping.ca/.
Outcome Mapping, by S. Earl, F. Carden and T. Smutylo, published by the International
Development Research Centre in 2001. Available at http://www.outcomemapping.ca/
download/OM_English_final .
Study Designs for Program Evaluation, published by Aguirre Division, JBS International in 2006.
Available at http://www.pacenterofexcellence.pitt.edu/documents/study_designs_for_
evaluation .
HIGHLIGHTS
At the start, you should determine what
type(s) of evaluation you need –
performance monitoring, formative or
process evaluation, or summative or
outcome evaluation. The evaluation type
depends on the stage of your program.
Sometimes, evaluators have a preference
for a certain evaluation approach. One
approach is not necessarily better than
the other. It is important to understand
the unique strengths of each approach
and which one best suits your evaluation
goals.
Once you have determined the type(s) of
evaluation you need, you can combine
different evaluation methodologies,
depending on your resources, to answer
your evaluation questions.
download/OM_English_final
download/OM_English_final
evaluation
evaluation
exercises
1. evaluation seeks to know if the program caused or contributed to
behavioral changes among the youth it serves.
A. Outcome
B. Performance monitoring
C. Process
D. Proactive
E. None of the above
2. evaluation seeks to know if the program met the milestones in its
work plan.
A. Outcome
B. Performance monitoring
C. Process
D. Proactive
E. None of the above
3. Which of the following is true about process or formative evaluation?
A. It is less time-consuming
B. It is easy to implement
C. It can be conducted throughout the program period, from beginning to end
D. It should be conducted only when the program is getting started
E. It should be conducted at the end of the program
4. Let’s consider the example below.
The advocacy initiative to fight for early childhood education grew out of an interest shared by
a group of nonprofit organizations. A collaborative was established and your organization
decided to be both the fiscal agent for the grant from the community foundation and the
Answers: 1A; 2B; 3C
coordinator for the collaborative and its activities. According to the work plan approved by the
community foundation, for the first year, you and the collaborating partners would develop an
infrastructure for decision-making, communications and data sharing. Once the infrastructure
has been established, the collaborative would develop an advocacy agenda and action plan.
The remaining two years of the grant would focus on implementing the agenda and action plan,
including affecting policies and practices relevant to early childhood education.
What facilitated or challenged the establishment of the collaborative and its preparation
for action?
How was the community engaged in developing the advocacy agenda?
What policy and other outcomes were achieved or not achieved by the end of three years?
Why?
Given the information above, answer the following:
A. What type of evaluation should you use – performance monitoring, formative, summative or
some combination of these? Why?
B. What evaluation approach do you think would be suitable for this evaluation? Why?
4 Overview of the Evaluation Process that Reflects Evaluative Thinking
introduction
Evaluation can be somewhat intimidating for those unfamiliar with
its usefulness or general process, which has several stages
These stages reflect evaluative thinking:
Identifying assumptions about what you think works and
doesn’t work and why.
Posing thoughtful questions about what change you expect to
see during and after you implement your strategy, initiative or
program.
Pursuing deeper understanding through reflection and dialogue
Communicating what was learned without underestimation or
exaggeration (i.e., without playing down or overstating what
you did or achieved).
Making informed decisions about future action.
Regardless of whether you are evaluating a strategy, initiative,
program, or even your organization – you can apply the same
sequence of evaluation steps. Each step, however, will be more
complicated as the entity to be evaluated becomes more complex;
evaluation of a single project is simpler than evaluating a multi-site
initiative, a strategy, or an effort to change a policy, system or
community. If planned properly, evaluation easily can be understood
and received positively, especially if you engage your stakeholders
throughout the evaluation process.
How this chapter is organized…
4.1 Overview of the
Evaluation Process
4.2 Creating a Learning
Organization
4.3 Stages of the Evaluation
Process
4.4 Navigating Evaluation
Choices
4.4.1 Capacity, Program or
Population Outcomes
4.4.2 Focus or Unit of Analysis
4.4.3 Internal Versus External
Evaluation
4.4.4 Power Dynamics Among
Stakeholders
Highlights
Exercises
4.1
overview of the evaluation process
An evaluation that reflects evaluative thinking is the systematic process of telling your story by:
Identifying assumptions about why you think your strategy, initiative or program will work.
Determining what change you expect to see during and after you implement what you set out
to do.
Collecting and analyzing data to understand what happened.
Communicating, interpreting and reflecting on the results.
Making informed decisions to improve what you plan to do next.
Evaluation should be viewed as a collaborative process that involves any or all of these
stakeholders, depending on their roles: funders (public, private); staff; board members; program
participants or constituents of your organization; community leaders; collaborating organizations
or partners; and policymakers. If done properly, it helps tell the story of your strategy, initiative or
program through a continuous cycle of asking, planning, acting, reflecting and improving.
Evaluation should emphasize utility (i.e., findings should be practical and useful for end users)
and inform decision-making and capacity building strategies for involved parties. Therefore your
evaluation should consider:
Implementation and impact as an ongoing process and what your actions should build on
instead of replace.
Existing information systems to promote improvements that you can sustain over time.
Previous lessons learned from research along with the current values and realities of all the
stakeholders involved.
A logic model to illustrate how your strategy, initiative or program is supposed to create change
with a design that is driven by questions you want to answer, existing capacity,
feasibility and appropriateness of approach and methodology.
linkages among resources, activities, outputs, audiences and short-, intermediate-
and long-term outcomes.
A PREPARE for conducting
an evaluation
B
DETERMINE
stakeholders
and how and
when to
engage them
C
IDENTIFY
assumptions and
determine what
will be di�erent
(theory of change
and logic model)
D DEVELOP evaluation plan (logic
model, measurement,
framework, etc.)E COLLECT and analyze data
F
COMMUNICATE
results and
understand
what happened
(interpret
�ndings and
facilitate
learning)
G
MAKE informed decisions
(improve actions and
next steps)
Stakeholders
Exhibit 4-1: evaluation process that reflects evaluative thinking
Useful resources for planning an evaluation include:
Introduction to Program Evaluation for Public Health Programs: A Self-study Guide, by the Centers
for Disease Control and Prevention, published in 2011. Available at http://www.cdc.gov/
eval/guide/cdcevalmanual .
Planning an Evaluation: Using the Rainbow Framework, by Better Evaluation, published in 2013.
Available at http://betterevaluation.org/sites/default/files/Rainbow%20Framework%20
-%20compact%20version .
Project/Programme Monitoring and Evaluation Guide, by the International Federation of the Red
Cross. Available at http://www.ifrc.org/global/publications/monitoring/ifrc-me-guide-
8-2011 .
Evaluating Community Programs and Initiatives, by The Community Toolbox. Available at
http://ctb.ku.edu/en/tablecontents/index.aspx.
4.2
creating a learning organization
Evaluative thinking and evaluation contribute to an organization’s capacity to become a “learning
organization.” A learning organization is one in which the leadership and staff continually develop
their ability to achieve their desired results through new and betters ways of thinking and
problem-solving.
For successful implementation and use of evaluation, your organization should develop or
strengthen the following:
Knowledge about the purpose, benefits and risks of evaluation
Commitment to use data to support decision-making regularly and consistently and to support
a culture of evaluative thinking
8-2011
8-2011
Internal systems and processes to support learning
Skills in managing, interpreting and using information
Resources to hire an internal evaluator (a staff person with the knowledge and skills to do
evaluation) or an external evaluator (an evaluator who is not an employee) and support staff
and other stakeholder involvement
Relationships with individuals and organizations with expertise in evaluation.
A useful resource about learning organizations is Peter Senge and the Learning Organization,
by M.K. Smith, published in 2001. Available at http://infed.org/mobi/peter-senge-and-the-
learning-organization/
4.3
stages of the evaluation process
The process starts with preparing for the evaluation (A). This involves asking questions, including:
Who is funding the evaluation and why?
Who benefits from the evaluation?
What are the potential risks?
What else is happening at the same time that could affect what you are doing and the
evaluation?
What type of evaluation you need – performance monitoring, formative/process summative/
outcome or all three?
What evaluation approach you prefer, if any, and which evaluation methodology is most
appropriate for your needs?
Do you need an internal or an external evaluator?
learning-organization/
learning-organization/
During this stage, the capacity of the organization or collaborative (if it involves partners) should
also be assessed so the evaluation can be most appropriate and useful. Chapter 5 describes in
more depth what you need to consider when preparing for an evaluation.
Once you have prepared for the evaluation, you want to determine who are the stakeholders of the
evaluation and how and when to engage them (B). For the evaluation process to be a collaborative,
useful learning process, all stakeholders should be identified and engaged accordingly to provide
multiple perspectives about the main issues that could affect the evaluation, and about what they
want to know from the evaluation. Otherwise, the evaluation is likely to be designed based on the
needs and interests of only a select few stakeholders – usually the ones with the most
power – and could miss other important questions and issues of stakeholders who are not included
in the design and planning process.
Critical questions specifically related to the evaluation design that stakeholders can help answer
include, but are not limited to:
What results do you expect from the strategy, initiative or program?
What does success look like? How do you know when you have achieved it?
What factors might help or hinder that success?
Who or what are the best data sources?
Will the evaluation be beneficial or harmful to the community and why?
What are some of the traditions, social norms, or dynamics that should be considered when
the evaluator or staff are collecting data?
Is there potential for the evaluation or evaluator to be perceived negatively, why and what can
be done to reduce this likelihood?
Additional questions that should be asked of you and
stakeholders at the outset also could include:
What do you, other staff, partners and key stakeholders need
to know more about?
What decisions do you feel you need to make, for which you
need more information?
What will you do with the answers to your questions? (Play
out various scenarios, depending on the various answers
you might find.)
Who will make the decisions and when?
What issues related to these decisions are likely to surface?
How are decisions made in this organization or
collaborative?
What other factors may affect the decision-making process?
How will you and others know if you used the evaluation
results and process as planned?
Staff, partners and stakeholders are more likely to use
evaluation if they understand and feel ownership over the
evaluation process. Therefore, the more people who have
information about the evaluation and actively participate in the
process, the easier it will be to use the results for program
improvement and decision-making. However, because of the
many individual and organizational obstacles to using
information and testing assumptions about an effort’s
effectiveness (e.g., fear of being judged, concern about the time
and effort involved, resistance to change, dysfunctional
communication and information-sharing systems, unempowered
staff, turf issues among partners), you will need to engage in
discussion and reflection about the specific obstacles to using
information within your organization or collaborative.
results and decision-making about next steps.
Review list of stakeholders to ensure all appropriate stakeholders are included.
Understand and respect stakeholders’ values.
Create a plan for stakeholder involvement.
Identify areas for stakeholder input.
Bring stakeholders together as needed.
Target key stakeholders for regular participation.
Ask stakeholders to suggest evaluation questions.
checklist
Tips for engaging stakeholders:
The next major stage in the process is to identify the assumptions underlying your idea for why the
strategy, initiative or program will bring about the change you want to see and what will be differ-
ent as a result (C). Developing a theory of change and logic model is very useful during this stage
of the evaluation process.
A theory of change explains the links between activities and outcomes and how
and why the desired change is expected to come about, based on past research or
experiences. A logic model is a graphic representation of the theory of change that
illustrates the linkages among resources, activities, outputs, audiences and short-,
intermediate- and long-term outcomes.
Useful resources for developing a theory of change and logic model include:
Logic Model Development Guide, published by the Kellogg Foundation in 2004. Available at
http://www.wkkf.org/knowledge-center/resources/2006/02/wk-kellogg-foundation-
logic-model-development-guide.aspx.
Point K Tools: Logic Model Builder, by Innovation Network. Available at
Theory of Change: Guided Example – Project Superwomen, published by ActKnowledge and the
Aspen Institute Roundtable on Community Change in 2003. Available at
http://www.theoryofchange.org/pdf/Superwomen_Example .
Theory of Change: A Practical Tool for Action, Results and Learning, by J. Reisman and
A. Gienapp, published by Organizational Research Services in 2004. Available at
http://www.aecf.org/upload/publicationfiles/cc2977k440 .
Many evaluation experts agree that using a logic model is an effective way to help facilitate
success because the model is developed by all stakeholders, including staff, participants and the
evaluator. (For more detail on logic models, see the W.K. Kellogg Foundation’s Logic Model
Development Guide.) Consequently, a model provides participants with the opportunity to share
and discuss their assumptions, develop a common understanding of the change process and
become clearer about terms and definitions. The logic model also:
helps organize and systematize planning, management and evaluation functions;
guides the development of a measurement framework, which identifies the relevant indicators,
measures, data sources, data collection methods and frequency of data collection for each
component of the logic model;
makes explicit the contextual conditions that need to be changed or that could influence the
change process and outcomes; and
guides the reporting of the evaluation findings and reflection process.
It doesn’t matter if you are evaluating a strategy, initiative or program, or even your organization.
If you are trying to impact outcomes at the individual, organization, system or community level, a
logic model is helpful. Developing a logic model with stakeholders helps you, your staff, your board
members and other participants continually learn about what you are trying to accomplish and how
to become better at it.
logic-model-development-guide.aspx
logic-model-development-guide.aspx
determined and a measurement framework can then be created to identify the data sources;
frequency of data collection; and the qualitative and quantitative measures of change (D). All of
this information, along with data collection methods, analysis strategies and plans for reporting
and communicating the findings, are compiled to form the evaluation plan.
After the evaluation plan is completed, data collection can start, followed by analysis (E).
Some analyses can be linear and straightforward. For example, if the goal of the evaluation is to
find out the impact of your organization’s after-school program, then the sequence of activity is
more or less linear, as noted below.
Collect data from the youth and parents before the program starts.
Collect data halfway through the program or at the end of the program (or both).
Analyze the data using statistics (e.g., percentages) and coded qualitative data.
Qualitative data is information that can be
collected or captured in text form.
Other types of analyses can be more complicated. For
example, if the goal of the evaluation is to find out the
impact of your policy advocacy or systems change
strategy, then the sequence of activity would be:
Map the system before your strategy.
Determine what parts of the system you want
to change.
Identify the factors that could facilitate or prevent
that change and whether the factors are within or
outside your control.
Collect data from multiple sources at key junctures to help you understand if, and when, the
change occurred.
Analyze the data using qualitative and/or quantitative methods.
Map the system again to see if and where the change happened.
If you are collecting survey data, review and “clean” data before beginning your analysis.
If you don’t have experience with survey data, you should consult with your evaluator
about cleaning the data and what to do if the data are incomplete or missing (e.g., some
people did not answer all the questions).
If you are collecting qualitative data, review the notes and if necessary, follow up with
the individual or review the situation to fill in any information gaps. If you are working
with an evaluator to collect qualitative data, ask him or her to explain the procedures for
ensuring high quality data.
Leave enough time and money for analysis. Qualitative analysis requires more resources
because it takes more time to read, code and summarize text than it does to enter the data
and calculate the percentages or other statistics involved in quantitative analysis.
Be clear about the use of the word “significant” to describe your findings. If it is not a
statistically significant finding but is still important, you can use the term “practically
significant” or “programmatically significant.”
checklist
Tips for analysis:
The next few stages after data collection and analysis relate to communication and interpretation
of findings (F) and making informed decisions about improvements and next steps (G). Chapter 9
discusses the reporting, interpretation and reflection of evaluation findings in depth. This should
involve your stakeholders who could have insights into findings (e.g., why things turned out the
way they did), facilitating everyone’s learning and determining implications of findings and
lessons learned. The reflection and dialogue, especially about whether the change process
occurred the way everyone expected, can then inform decisions about improvements and next
steps. At this stage, it is helpful to go back to the theory of change and logic model and adjust your
effort based on the new insights.
4.4
navigating evaluation choices
Tensions often arise in evaluation because individual stakeholders have their own priorities for
the evaluation. The funder, evaluator, implementers and other key stakeholders should
continuously communicate to clarify and manage expectations. As events unfold around the
strategy, initiative or program causing it to adapt to the stakeholders’ needs, these discussions are
critical to prevent misunderstandings. The following subsections cover the typical tensions that
emerge and the typical choices the funder, evaluator and other key stakeholders have to make
during evaluation.
Let’s use this example to illustrate the typical tensions: You received a three-year grant to build
your organization’s cultural competency to better respond to the health needs of recent
immigrants and refugees from Southeast Asia (namely Vietnam, Cambodia, Laos and Myanmar)
and help reduce the health disparities they encounter. This grant is part of a national initiative
aimed at building the capacity of health organizations to respond to the growing diversity of
communities across the country. The funder expects to see improved capacity of the funded
organizations to work with racially, ethnically and culturally diverse communities as well as
reductions in health disparities.
4.4.1
Capacity, Program or Population Outcomes
Capacity outcomes. You should consider the following before even starting the evaluation:
Clarify early on the answer to these questions: “Capacity for what or to what end?” and
“Whose capacity?” Your funder, any of the consultants they hire and you must agree about
what capacity outcome is realistic to expect within the grant period and to what end. Using the
example above:
First clarify “Whose capacity is being built?” In the example above, it is clearly your
organization’s capacity.
The “capacity” is your organization’s ability to respond to the health needs of diverse popu-
lations and the “end” is a reduction in health disparities. In three years, it might be feasi-
ble to observe changes in your organization’s board and staff composition that were made
to be more culturally diverse, and in how your organization engages the community (the
capacity outcomes). However, it might not be possible to observe changes in the health
disparities (the ultimate change or difference you want to make) experienced by Southeast
Asians as a population in that time period.
Let’s consider another example – one that reflects a common issue in the field of community and
systems change: Funders in this field often make grants to build community capacity to mobilize
residents to address a problem they face, such as inadequate early childhood education. In this
case, funders, grantees and key community stakeholders must agree on what capacity outcome
is realistic to expect within three years, what difference they expect to see (if any) with early child-
hood education in their community and, equally important in this situation, what constitutes
“community.” Does “community” refer to nonprofit organizations, residents, public agencies,
business owners, etc.? Answering this question early is important because the coalition’s success
could be wrongly judged by the composition of its membership. In one to two years, it may be
feasible to create a community coalition that focuses on early childhood education issues and on
the development and distribution of a report card (posted online and disseminated in print
format) on early childhood education outcomes – a form of community capacity. In three years,
the coalition could have developed the ability to engage and mobilize the larger community into
action. However, it might not be possible within that time period to observe child outcomes (e.g.,
readiness to learn, emotional well-being) that can be traced back to the coalition.
A theory of change and logic model can help show the link between improved capacity and the
changes you anticipate making in people’s lives, in a policy, system or community, and the
sequence of events between the improvement and the changes.
Program (also strategy or initiative) outcomes. These outcomes are the benefits that participants
experience as a result of the strategy, initiative or program; the outcomes are limited to
participants or the people “touched” by the strategy, initiative or program. In the example
above regarding a three-year grant to build your organization’s cultural competency to be able to
respond better to the health needs of recent immigrants and refugees from Southeast Asia:
The outcomes are limited to immigrants and refugees who received services from your
organization.
The outcomes can tell if your organization was effective in engaging them as recipients of your
services and improving health outcomes such as reduction in their blood sugar and blood
pressure levels, fewer asthma attacks and regular and on-time mammograms and pap smears
among the women.
Let’s say you decided to expand your effort to include – with additional resources from another
funder – a community-wide campaign to promote pedestrian safety after learning about the
number of accidents among recent immigrants and refugees. The campaign could result in few-
er pedestrian-related injuries or deaths. The results of the campaign are limited to the people
who were “exposed” to the campaign.
Population outcomes. Population outcomes are:
Changes in the condition or well-being of children, families or communities (e.g., increase in
employment and graduation rates or decreased infant mortality or teen pregnancy rates).
Long-term results of the combined efforts of a number of different strategies, initiatives,
programs and organizations. It takes a comprehensive, multi-year effort to try to change
population outcomes.
If you have a multifaceted strategy aimed at changing systems and that is scalable, sustainable
and impactful at the population level, then the evaluation of your strategy should consider
outcomes such as increased high school graduation or decreased infant mortality rates. If you are
implementing a single program designed to improve the academic achievement of a group of high
school students, it is not realistic to expect outcomes for anyone else beyond the students with
whom you are working. In the example above about a three-year grant to build your organization’s
cultural competency, it would not be realistic to expect changes in health disparities at the popu-
lation level because it would require more than the efforts of your organization to bring about such
change. Instead, it would require a large-scale effort to change systems and community conditions.
4.4.2
Focus or Unit of Analysis
The focus or unit of analysis is the target of change and the
entity that is being analyzed in the evaluation. This entity could be individuals, an organization,
a program model, system, policy or community; the evaluation can focus on one or more of these
units at the same time. In the example previously
mentioned regarding building the cultural competency of health
organizations responding to the needs of Southeast Asian
immigrants and refugees, the units of analysis are both the
organization and the immigrants and refugees served.
Other examples include the following scenarios:
If your program is designed to improve the reading
proficiency of children, the units of analysis are the program and the children.
If you are trying to change a school district policy to
purchase local fresh produce and get rid of vending
machines that sell unhealthy snacks, the units of analysis are the policy, system (from how the
produce is delivered to how the food is prepared and served to children in the schools), the
organization (the schools) and the individual (the children).
Depending on the scope and timeframe of the effort and expectations of all the key stakeholders,
you can decide to focus on the system, policy, organization, individual or all four.
When the focus or unit of analysis is the individual:
You are documenting the changes that individuals experience.
Your evaluation will usually want to assess two things: (1) the degree to which the strategy,
initiative or program is being implemented accordingly and (2) if the individual participants
experienced the desired outcomes. In the earlier example about your organization and the
grant to improve its cultural competency to better serve Southeast Asian immigrants and
refugees, this would include
Individual
Program
Organization
Community
Policy
Larger Formal Systems
CO
M
PL
EX
IT
Y
O
F
M
ET
H
O
D
O
LO
G
IE
S
Exhibit 4-2: different units of analysis
measuring whether individuals served improved their blood sugar and blood pressure
levels or have more regular mammograms. Other types of individual-level outcomes could be:
The young men participating in the program have better grades as a result of the mentoring
activities.
The parents are more involved in their children’s school as a result of the parent leadership
institute.
The heads of households report they buy more fresh fruits and vegetables per week due to
the healthy eating initiative.
The low-income families report using the services offered by the banks in their community.
When the focus or unit of analysis is the organization:
In the example used about increasing cultural competency, you might examine changes in your
organization’s priorities, policies and practices.
To evaluate change at the organization level, you want to hire evaluators skilled in evaluation
and knowledgeable about organizational development. These evaluators should be able to
collect and analyze data on key outcome indicators such as how successful your organization
has been in engaging and sustaining the involvement of different groups of people and the
degree to which staff of diverse racial, ethnic and cultural backgrounds feel valued
and respected.
When the focus or unit of analysis is an initiative or program:
Some grants involve the development and implementation of a new initiative or program
designed to meet the needs of a particular target population. In the cultural competency
culturally competent in working with the Southeast Asian community.
Other grants may adapt existing initiatives or programs for new locations or client groups. For
example, you may work with a community to create and test a cultural and linguistic adaptation
of a financial literacy program tailored to the community’s needs.
The above initiatives or programs share common traits: They are creating and pilot testing new
initiatives or programs without a proven track record of effectiveness. As a result, they need
ongoing monitoring and improvement as the initiative or program is refined.
To evaluate these new or adapted initiatives or programs, you could conduct a case study to
understand:
How the effort was implemented.
What elements made it effective.
What worked and didn’t work and why.
What knowledge, skills and other capacities are required of the staff to implement the
effort.
Most important, how to refine the effort.
Much of the data collected could be qualitative. When the initiative’s or program’s elements
have become final and stable, you can conduct an outcome or summative evaluation and
combine the use of a quasi-experimental design with the case study methodology to assess its
effectiveness.
A quasi-experimental design assesses the causal effects of a program by comparing
two groups of participants (a “treatment” group and a “comparison” group) or by
comparing data collected from one group of participants before and after they par-
ticipated in the program. There is no random assignment of participants into the two
groups, unlike studies using an experimental design.
In a different situation, you might select and implement an initiative or program that already
has been proven effective through rigorous summative/outcome evaluations (also called an
evidence-based program). You could be using a financial literacy curriculum that already has
been tested and shown to be effective. The curriculum is ready for an outcome or summative
evaluation and you can use before-and-after comparisons (i.e., a type of quasi-experimental
design) to determine its effectiveness. You also can combine this methodology with a case
study to collect additional information to answer questions about the quality of the curriculum.
When the focus or unit of analysis is a policy:
These could be situations that involve education campaigns or other mobilization efforts to
advocate for policy change at the local, county or state levels. In the pedestrian safety
campaign example used previously in this chapter, you might discover there is no policy about
use of mobile phones while driving and decide to get involved in an effort to advocate for a
policy to stop their use. This type of initiative calls for evaluators with the policy evaluation
expertise and experience to conduct case studies of the local transportation system, political
engagement, community mobilization process and policy outcomes.
The evaluation will have to rely on multiple data sources and methods (e.g., interviews with
legislators, focus group with advocates, survey of community residents) to assess how the
campaign and external circumstances shaped policy decisions.
Sometimes, when evaluating policy change, you might not observe a new or revised policy,
but instead could find that a policy contrary to what you were advocating for was blocked. This
could still constitute a positive outcome.
Be clear from the outset about your theory of change, logic model and outcome measures.
Useful resources about evaluating policy change include:
Ten Theories to Inform Advocacy and Policy Change Efforts, written by S. Stachowiak and
published by Center for Evaluation Innovation and ORS Impact in 2013. Available at http://
www.evaluationinnovation.org/sites/default/files/Pathways%20for%20Change .
The Evaluation Exchange, Advocacy and Policy Change, Volume XIII, No. 1, published by The
Harvard Family Research Project in 2007. Available at http://www.hfrp.org/var/hfrp/
storage/original/application/6bdf92c3d7e970e7270588109e23b678 .
storage/original/application/6bdf92c3d7e970e7270588109e23b678
storage/original/application/6bdf92c3d7e970e7270588109e23b678
When the focus or unit of analysis is system:
Some grants might be designed to improve a local system. For example, there could be a
health equity initiative that calls for increased access to health care for underserved groups
such as the immigrants and refugees served; low-income families; racial and ethnic
minorities; and people with different sexual orientation. This means changing aspects of the
health system, including how people are reached and informed about their eligibility for health
insurance; how they enroll in insurance plans; their understanding and use of preventive care
services; and the cultural and linguistic competency of health care providers.
Another example could be a food system initiative that requires schools to purchase fresh
produce from local farmers and change the lunch menus for children in subsidized school
lunch programs. This requires changing school district policies, ensuring that local farmers are
trained and equipped to handle and deliver food to schools, and educating children and their
parents about healthy eating.
You can conduct both formative/process and summative/outcome evaluations and employ an
outcome mapping methodology, as many partners must be engaged at different points in the
system to help make the change.
You want to create a picture or map of the parts of the system you are trying to change early in
the effort. Otherwise, you risk evaluating the wrong thing.
You also want to clarify what the change is. For example, the relationship among the parts that
you are attempting to change; new or improved functions of those parts; or new or improved
policies, procedures or practices that support those parts of the system. This clarity will guide
your evaluation questions and data collection and analysis. If you have the time and resources,
you also can assess the experiences of the people who interacted with the system before and
after your effort to change the system or parts of the system.
Useful resources for evaluating systems change include:
A Practical Guide to Evaluating Systems Change in a Human Services System Context, written by
Latham and published by Center for Evaluation Innovation and Learning for Action in 2014.
Available at http://www.evaluationinnovation.org/publications/practical-guide-
evaluating-systems-change-human-services-system-context.
Evaluating Systems Change: A Planning Guide, written by M. Hargreaves and published by
Mathematica Policy Research in 2010. Available at https://www.mathematica-mpr.com/
our-publications-and-findings/publications/evaluating-system-change-a-planning-guide.
When the focus or unit of analysis is a community:
In comprehensive community change initiatives, the unit of analysis is the community. (This
would not be the case in the example about your organization’s cultural competency.)
You want to take the time to define “community” at the beginning of the initiative so the
evaluation does not look for change in the wrong places.
Similar to when the unit of analysis is a system, you want to clarify the nature of the change
– for example, change in relationships among the public, private and nonprofit organizations
that serve the community; or in the physical structure or social norms of the community; or in
the ways different racial and ethnic groups interact, collaborate and relate to each other.
Again, you should have a theory of change and logic model for what you are trying to do, and
the expected outcomes.
Communitywide surveys are commonly used in evaluations where the unit of analysis is the
community, and they can be expensive to administer. Therefore, you must use an appropriate
sampling strategy to select a group of people from the larger community (the sample) that you
believe will reveal information about the larger group (the population).
Useful resources for evaluating community and systems change include:
Evaluating Community Change, published by Grantmakers for Effective Organizations in 2014.
Available at http://docs.geofunders.org/?filename=geo2014_indicators_framework .
Approaches to Measuring: Community Change Indicators, written by L. Weaver, P. Born and
D. Whaley, published by Tamarack – An Institute for Community Engagement in 2010.
Available at http://tamarackcci.ca/resource-library/evaluation/approaches-measuring-
community-change.
evaluating-systems-change-human-services-system-context
evaluating-systems-change-human-services-system-context
community-change
community-change
4.4.3
Internal Versus External Evaluation
Internal evaluation is conducted by a staff person within the
organization that is conducting the program or entity being
evaluated, whereas an external evaluation is conducted by
an evaluator who is not an employee of that organization.
Whether an organization should conduct an internal or
external evaluation usually depends on the available
resources, qualifications of the internal or external
evaluator, scope of the evaluation and the funder’s
requirements. However, other factors are equally important in
making this decision:
External evaluators can bring a broader perspective while
internal evaluators tend to have intimate knowledge about
the context that the strategy, initiative or program is
operating within.
External evaluators can be perceived as threatening
while internal evaluators can be perceived as being less
objective.
Sometimes, you have no choice because the funder requires
an external evaluation. Regardless of whether an internal or
an external evaluator is selected, clear lines of accountability
must be established from the outset.
Useful resources on hiring evaluators include:
• Hiring and Working with an Evaluator, by the
Juvenile Justice Center published in 2016. Available at
http://www.jrsa.org/pubs/juv-justice/evaluator .
• Selecting an Evaluator, by Community Science for the
W.K. Kellogg Foundation. Available from
info@communityscience.com.
4.4.4
Power Dynamics among Stakeholders
As mentioned before, evaluation generates knowledge, and knowledge is power. For example,
when a group of people in a community learn about how well or poorly certain teaching techniques
in an early childhood education program are working, they have the “power” to use this knowledge
to advocate for change. This characteristic of evaluation engenders power dynamics based on:
Who is a part of the evaluation.
How the data were collected and from whom.
Who owns the data.
Who gets to see and use the findings.
Who benefits or fails to benefit from the findings.
Examples of where power affects the evaluation process and evaluation findings include these
actual scenarios:
An evaluator makes decisions about the measures of success without consulting the program
staff or community leaders, and suggests to the staff and leaders they don’t have the
educational qualifications to make the decision.
In a breach of ethics, a program director tells service recipients they will get additional services
if they participate in the evaluation.
Community leaders assume the evaluation report is too “sophisticated” for community mem-
bers and therefore don’t think it’s worth sharing.
Leaders hold discussions to address the findings at times and locations that are inconvenient
for the families who are affected by the findings.
Identify and engage key stakeholders early on.
Place equal value on experiential and theoretical knowledge without diminishing the
effect of either.
Take the time to think carefully as you develop and frame the evaluation questions to
avoid making wrong or biased assumptions about the people who are supposed to
benefit from the program, initiative, strategy or organization.
Integrate into the evaluation plan a communications strategy about how the findings will
be used and who will convey those findings.
checklist
Tips to manage the power dynamics:
HIGHLIGHTS
A good evaluation is useful and
responsive to the organization’s needs;
works within the organization’s capacity
to manage and use information; engages
an organization’s designated staff in deci-
sions about the evaluations; and is scientif-
ically rigorous and honest.
Evaluation can help tell the story of your
program, initiative, strategy or organiza-
tion through a continuous cycle of asking,
planning, acting, reflecting and improving.
Becoming familiar with the cycle of eval-
uation and its stages will help you plan
your evaluation.
Take time to consider the tensions that
often come up in an evaluation – level of
change, focus or unit of analysis, use of an
internal or external evaluator and power
dynamics among stakeholders.
You, your stakeholders and evaluator
should address the question of usefulness
from the outset.
exercises
1. An organization with a culture of data and learning is more likely to benefit from evaluation.
A. True B. False
2. Consider the culture of data and learning in your organization. Does it exist? If so, what are
some of the characteristics of this culture in your organization? If not, what can you do to
promote such a culture in your organization?
3. You can’t develop a logic model until after you see how the program works.
A. True B. False
4. What program or initiative are you involved in that could benefit from a logic model? Why?
5. Let’s consider this example adapted from the American Evaluation Association
(see www.eval.org): A state department of education is funding an initiative to improve
academic outcomes through the development and implementation of school improvement
plans by school districts. The department contracted a technical assistance provider to assist
grantees. You received a two-year grant to implement the initiative in your school district. The
state department of education provided $45,000 for a two-year evaluation to address the
following evaluation questions:
How were schools responding to and using the technical assistance services?
Were schools changing practices with respect to planning, implementation and
monitoring?
Did the use of the technical assistance provider’s services have an effect on academic
outcomes?
methodology included five data collection methods:
A telephone survey of all team members from the five school planning teams in your
school district.
A review of each local school’s pre- and post-initiative school improvement plan.
An analysis of school-level academic outcomes before and after the grant.
Focus groups with principals from all five schools.
Focus groups with a sample of parents from all five schools.
The principals, the parents, the five schools and the school district can be viewed as:
A. Units of analysis
B. Separate entities
C. Stakeholders with conflicting interests
D. All of the above
E. None of the above
6. Consider the choices you have to make about evaluation as described in Section 4.4. Have you
encountered situations where you have had to make decisions about these choices? What did
you decide and why? Knowing what you know now, might you have selected a different choice
and why?
Answers: 1A; 3B; 5D
5
Preparing for the Evaluation
How this chapter is organized…
5.1 Why Might You Consider
an Evaluation of Your
Strategy, Initiative or
Program?
5.1.1 Considerations if Your
Evaluation Is Part of a
National or Statewide
Evaluation
5.1.2 Considerations if the
Findings from Your
Evaluation Have
Significant Implications
on a Theory, Practice
or Policy
5.1.3 Getting Ready for
Evaluation Whether or
Not It Is A Funding
Requirement
5.2 Who Benefits from the
Evaluation and What are
the Potential Risks?
5.2.1 Consideration of Different
Types of Stakeholders
5.2.2 Use of Evaluation by
Different Types of
Stakeholders
continued on following page
introduction
Whether you have conducted
evaluations before or not, you must have asked
yourself, “Who should be involved in the evaluation?” “Who do I
know with the skills and expertise to conduct an evaluation?” “Can
we do it ourselves without hiring someone?” “How much does an
evaluation cost?” These are important questions you should consider
before proceeding with
evaluating your strategy, initiative or program. This is the first step
in evaluative thinking and preparing for conducting an evaluation.
A PREPARE for conducting
an evaluation
B
DETERMINE
stakeholders
and how and
when to
engage them
C
IDENTIFY
assumptions and
determine what
will be di�erent
(theory of change
and logic model)
D DEVELOP evaluation plan (logic
model, measurement,
framework, etc.)E COLLECT and analyze data
F
COMMUNICATE
results and
understand
what happened
(interpret
�ndings and
facilitate
learning)
G
MAKE informed decisions
(improve actions and
next steps)
Stakeholders
5.1
why you might consider an evaluation of
your strategy, initiative or program
Depending on who is funding the evaluation, the requirements
could vary greatly:
Sometimes, if evaluation is a grant requirement, you might
find it difficult to look beyond doing just what is needed to
fulfill the requirement, especially if you have limited
capacity. You collect the data because your funder expects
it and you report the data at the end. You might or might
not take the time to consider how to use the data more
effectively to improve your program.
If it is a not a requirement and you truly see the value of
evaluation, you might need to take time to convince others,
such as decision-makers in the organization or staff, about
the value.
Your program may be a model program that a funder is
interested in replicating. If so, the context of the evaluation
could be more politically charged than usual, so you need to
take extra measures to ensure the findings are
communicated effectively and properly and not
inappropriately taken out of context.
Therefore, here are some questions to ask at this stage:
Is this part of a grant requirement? What data are you
required to report? What type of evaluation are you required
to conduct?
Is this part of a bigger evaluation, such as a national or
statewide evaluation? Are you required to collect data in a
certain way and at a certain time to support the national or
statewide evaluation?
continued from previous page
5.3 Considering Your
Organization’s
Capacity to Participate
in the Evaluation
5.3.1 A Staff Person Dedicated
to Evaluation
5.3.2 Budget for Evaluation
5.3.3 A Culture of Data and
Learning
5.3.4 Sharing and Protecting
Data across Program
Elements or Program
Locations
Highlights
Exercises
If it is part of a national or statewide evaluation, how can you use the required collected data
for your own purposes as well, or supplement the data for your own information needs?
Does your program have significant implications for a theory, practice or policy? If yes, who will
be scrutinizing the results?
If it is not a grant requirement, what needs to be done in your organization to ensure the
decision-makers and staff are on board? What do you need to prepare the organization
for evaluation?
If your organization decided to do the evaluation solely for its own purposes, you will have
more control over the entire evaluation process. If, however, the evaluation is part of a grant
requirement, you must work with the funder’s expectations about reporting, or with the evaluator
assigned by the funder. This does not mean that you give up control of the evaluation process. On
the contrary, you must be just as vigilant about ensuring the responsiveness of the evaluation to
your organization’s information needs.
5.1.1
Considerations if Your Evaluation is Part of a National or Statewide Evaluation
Here are some questions you should ask the funder or the evaluator assigned by the funder, if they
have not already provided this information:
Is there a logic model for the overall initiative, and how does your program’s logic model fit
into it?
What are the evaluation questions for the national or statewide evaluation?
What data are you required to collect? When do you have to report the data?
How does your program evaluation or data inform the larger evaluation?
How will the data you contribute be used?
What are nonnegotiable and negotiable in terms of evaluation? Examples include:
Are there instruments you must use provided by the national or statewide evaluator? Have
they been tested and validated for populations such as the ones you are serving?
Can you tailor these instruments to suit your community context and culture?
When do you have to provide the data to the national or statewide evaluator? How flexible
is the timeline, so it aligns with your program and needs?
When will you be able to get feedback or information about what the national or statewide
evaluation found and learned?
Asking and answering the above questions early is important so you are not surprised by a request
from the funder or national or statewide evaluator. Also, the funder or evaluator may not know
what it takes to carry out the work on the ground or the local considerations that could affect the
national or statewide evaluation. They might not even be aware of how their expectations could
affect your program’s implementation or your organization’s capacity.
On the other hand, you might not have thought about collecting certain types of data, and
understanding what the funder or evaluator wants could trigger an idea that might be very useful to
your strategy, initiative, program or organization. It is not unusual for a funder and
national or statewide evaluator to consider adding questions or adjusting their design after
speaking with local program directors and staff.
5.1.2
Considerations if the Findings from Your Evaluation Have Significant Implications for a
Theory, Practice or Policy
Sometimes, your strategy, initiative or program may be primed for replication or expansion
because someone thinks it works or is partial to what you are trying to do. The evaluation findings
could have implications on a theory, practice or policy. If this is the situation, the evaluation could
be considered a high-stakes one with lots of potential users. Knowing this early can help you plan
how the findings will be communicated, by whom and when. Such planning is crucial to ensure the
findings are shared properly – not taken out of context, exaggerated or dismissed. Here are some
questions to guide your approach:
How interactive should the communication be for each of the intended audiences? For exam-
ple, written and print materials are least interactive, while discussions and working
sessions are most interactive. Verbal and video presentations fall somewhere in the middle.
A useful resource about communicating evaluation findings is Evaluating Strategies for Communicating
and Reporting, by R. Torres, H. Preskill and M. Piontek, M., published by SAGE in 2005.
What are the risks in sharing the findings that could affect the strategy, initiative, program,
organization or the community you serve?
Can the findings be taken out of context and harm the effort, organization or community?
What can you do to mitigate the risks and consequences?
What roles do you, your staff, your board members and your internal or external evaluator have
in summarizing and communicating the findings and insights?
Who would be the most effective messenger of the information?
Do you need a facilitator skilled in adult learning techniques to assist with discussions,
working sessions and even verbal presentations?
Should you provide training for the messengers as spokespersons for the media and other
types of inquiries?
5.1.3
Getting Ready for Evaluation Whether or Not It is a Funding Requirement
You may realize the value of evaluation and choose to pursue it even if not required by your funder.
If this is the case, you need to make sure your board members, other decision-makers and staff
have equally bought into its value. Attending to staff commitment is especially important because
implementing an evaluation likely will add to their responsibilities and, perhaps, stretch their
already limited resources. Some steps to prepare your organization for evaluation include:
Look at the functions within the organization and discuss with the staff responsible for each
function what information they wish they had to perform their roles more effectively.
Introduce the idea of evaluation and how it could benefit the organization and its efforts.
Ask staff about their concerns regarding evaluation and find out what negative or positive
experiences they have had with evaluation.
Explore ways in which data collection can be integrated into the staff’s daily functions without
overburdening them and how they might use the data to support their work.
Form an evaluation and learning committee with staff from different levels of the organization
or with different functions, and charge the committee with finding out more about evaluation.
Address the concerns that were raised in both driving and guiding the evaluation
planning process.
5.2
who benefits from the evaluation and
what are the potential risks?
As multiple stakeholders are involved in evaluation, you must
carefully consider how the information and findings you share
with the stakeholders, including your evaluator, can be used (or
misused). Check thoroughly whether anything in the data you
collect for the evaluation can be taken out of context or
misconstrued in a way that is culturally incompetent or
harmful to your community. If you have a designated evaluator,
you should work with him or her on this issue.
To begin to answer the question, “Who benefits from the
evaluation and what are the potential risks?” consider who is
interested in learning about the evaluation of your strategy,
initiative or program. Additionally who cares about the answers
to questions such as:
Are you running your effort as you had planned? Who stands
to benefit or lose from the way you run your effort?
Are the participants benefiting from your effort as desired?
Who stands to benefit or lose if they do or do not?
Is your effort making an impact? Who stands to benefit or
lose from the way your effort does or does not make an
impact?
Might the results affect policies? Who stands to benefit or
lose if it does or does not?
5.2.1
Consideration of Different Types of Stakeholders
Who are the individuals who could affect or be affected by the
strategy, initiative or program and its evaluation?
They can include:
Funders (public, private, nonprofit)
Program staff
Board members
Program participants or constituents of your organization
Community leaders
Collaborating organizations or partners
Other organizations serving the same target population
Policymakers
exhibit 5-1: examples of reasons why different stakeholders might be interested in the
evaluation results of the after-school mentoring program for youth living in low-income
neighborhoods in an urban area
Stakeholders Why are they interested in learning about how your program is doing?
Funders They want to know if their investments were put to good use. If the
mentoring program is not reaching expectations, such as increasing
academic achievement, they would want to know why and make
decisions about continuing to fund the program or building the
program’s capacity (or not) based on the findings.
Program staff They want to know if they are doing their work properly and if they are
bringing changes to the youth’s lives as planned. If not, they would want
to use the information to improve their program and revisit its theory and
logic model.
Board members They want to learn about the progress of the program and the impact it is
making (or not) in the lives of the youth served. If it is having an impact,
they would want to use this information for their fundraising efforts. If
not, they would want to address the issues the program is facing.
Program participants or
constituents
The youth want to tell you if and how the program is helping them meet
their academic or career goals. They want to know if the findings actually
match what they experienced. If not, they would want to help the
program capture more accurately the outcomes they experienced.
Stakeholders Why are they interested in learning about how your program is doing?
Community leaders They want to know if the program is making a difference in their
communities or if it is doing more harm than good. For example, is it
shedding a positive light on the community or bringing bad publicity to
the community by emphasizing that there are many at risk-youth present?
If the program is working, they could use the information to advocate for
sustaining or expanding the program. If not, they could advocate
for changes such as bringing in a new executive team to manage
the program.
Collaborating organizations
or partners
They want to know if their support is bringing changes to the lives of
the youth as planned. If not, they would want to use the information to
improve their support for the program.
Other organizations serving
the same target populations
They want to know whether the program achieved the intended goals
and compare the results to their program to determine if your program
supports or does not support their agenda.
Policymakers They want to know whether the program supports or opposes their policy
positions about after-school mentoring programs. They would want to
know if there are supporting stories they can use for their advocacy or
alternative explanations to make their arguments stronger.
Useful resources for engaging stakeholders in evaluation include:
A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions, by H. Preskill
and N. Jones, published by the Robert Wood Johnson Foundation in 2009. Available at
http://www.rwjf.org/en/library/research/2009/12/a-practical-guide-for-engaging-
stakeholders-in-developing-evalua.html.
Do Nothing About Me Without Me, by J. C. Bourns, published by Grantmakers for Effective
Organizations and Interaction Institute for Social Change in 2010. Available at http://docs.
geofunders.org/?filename=do_nothing_about_me_without_me .
Stakeholder Engagement Tool, by S. Salentine and A. Johnston, published in MEASURE
Evaluation and USAID in 2011. Available at http://www.cpc.unc.edu/measure/resources/
publications/ms-11-46-e.
stakeholders-in-developing-evalua.html
stakeholders-in-developing-evalua.html
5.2.2
Use of Evaluation by Different Types of Stakeholders
Different stakeholders probably would use evaluation for different purposes, producing certain
benefits but also potential risks that could be avoided if you had a strategy to deal with them first.
For example, if you are evaluating one of your after-school mentoring programs for youth living in
low-income neighborhoods, consider how the results would affect your stakeholders.
Exhibit 5-2 is a table to help you think through the benefits and risks for each group and how you
can avoid the risks, to the best of your ability, as every situation is different. Exhibit 5-2 does not
include every potential scenario but gives general examples. Please take the time to complete such
a table carefully in your evaluations and customize with specific action steps for your context. You
and your evaluator should complete the table together and work with each group of stakeholders
to plan and take steps to prevent misunderstandings about the evaluation and misuse of the
findings.
exhibit 5-2: benefits and potential risks of the evaluation
Stakeholders
group
The evaluation results can
help this group by …
The evaluation results can
put this group at risk by …
To prevent potential harm,
you should …
Funders Demonstrating to the
foundation staff that the
grants are effective.
Highlighting that the
foundation may be funding
grants that do not work, and
the program officers who
favor the grant programs
may have to end them.
Engage the program officers
early on to understand what
they consider success and
discuss their information
needs.
Program staff Showing that their work is
making a difference.
Showing that their work is
not making a difference.
Consequently, some of the
grants may not be renewed,
which could lead to budget
and staff cuts.
Work with the staff early
on to ensure the evaluation
questions align with grant
requirements.
Board members Identifying areas where the
nonprofit organization is
making a difference.
Identifying areas of
weaknesses in the
nonprofit organization’s
leadership, administrative
and financial systems and
day-to-day operations.
Engage board members in
the design and
implementation of the
evaluation.
Stakeholders
group
The evaluation results can
help this group by …
The evaluation results can
put this group at risk by …
To prevent potential harm,
you should …
Program
participants or
constituents
Strengthening, continuing
or expanding programs that
they can continue to
participate in and benefit
from.
Eliminating opportunities
afforded by the programs.
Engage them early on to
help determine what
success looks like as a
result of their participation
in the programs and ensure
their vision is aligned with
the funder’s
expectations.
Community
leaders
Bringing resources to the
community.
Eliminating or reducing
existing resources.
Obtain their perspectives
about larger community
forces related to the issues
the programs are intended
to address.
Collaborating
organizations
and partners
Demonstrating that they
can and should work
together to achieve change.
Surfacing conflicts that
make it less desirable for
them to work together in
the future.
Work with all the partners
to ensure their information
needs, roles and respon-
sibilities are considered in
the evaluation design and
implementation.
Other
organizations
that serve the
same target
populations
Demonstrating your
program supports their
agenda and the target
populations they serve.
Diminishing the need or
impact of their programs
and competing with them
for grants.
Conduct an environmental
scan of organizations that
serve the same target pop-
ulations and understand
how their agendas relate to
yours.
Policymakers Elevating what works and
providing data that support
their policy agenda.
Bringing attention to
problems in the districts
or communities for which
they are responsible and
challenging their leadership
and policies.
Speak to policymakers to
understand their policy
agenda and how the
evaluation findings might or
might not support their
information needs and
agendas.
5.3
consider your organization’s capacity to participate in the
evaluation
To make the most of an evaluation, invest in building your organization’s capacity not only to
become an informed consumer of evaluation but also to develop and maintain a learning system
and culture. You have to set this tone at your organization; the evaluator cannot do this for you.
However, you can work with the evaluator to discuss when findings may be available and how to
best present the findings.
The following checklist in Exhibit 5-3 can help determine your organization’s evaluation capacity.
For each question, research the organization’s history of doing these tasks. Whether the staff had a
positive or negative experience the last time they attempted these activities, likely there are
lessons to build on. By finding out what might have been done before, you can avoid reinventing
the wheel or repeating the same mistakes.
exhibit 5-3: checklist to help your organization assess its evaluation capacity
Questions Yes No
Does your organization have
leaders who are willing to take
risks; share their learning with
funders, board members, and
staff; and adapt based on data
and knowledge?
Ask leaders to acknowledge the
importance of evaluation. Engage
leaders in sharing and reflecting on
the evaluation findings.
Prepare a presentation to leaders
about how data and information
could improve the organization’s
efforts to become more impactful.
Engage leaders in a discussion
about their concerns and fears
about sharing the evaluation
findings and actions that can be
taken to alleviate their concerns.
Does your organization have
a staff person who has some
percentage of time dedicated
to providing information to the
evaluator or funder as well as
summarizing and sharing the
information in-house?
Engage this person in the evaluation
you are considering.
Find out what helps or challenges
his or her tasks and consider how
to build on what is working and
develop strategies to deal with the
challenges.
Assess the existing staff’s skills
to determine if someone from
within the organization has the
capacity to fulfill this function.
Develop a job description.
Does your organization
allocate funds to support
evaluation?
Find out if the funds are a set-aside
and can be used for evaluation in
general.
If not, and it is dependent on the
grant, do a budget analysis to
determine how much has been spent
historically to do the evaluation.
Determine if the funds can be used
more efficiently to expand the
evaluation work, and if not, speak to
your board about what you can do to
expand the budget.
Speak to your board about
allocating funds for evaluation.
Speak to your funder to find out if
they can supplement the grant or
allow you to reallocate a portion
of the grant for evaluation.
Questions Yes No
Does your organization have
a culture of using data and
learning?
Establish a learning committee and
task it with developing and
implementing a learning plan by:
Speaking to staff and finding out
what is facilitating the culture and
what is challenging it.
Building on what is working and
developing strategies to deal with
the challenges.
Establish a learning committee.
Task it with developing and
implementing a learning plan by
conducting an assessment and
asking staff when and how they
use data in their work and the
challenges for making best use of
the data.
Does your organization share
data across program or
program locations and have
a system or process for
protecting the data?
Using the learning committee, find
out what is working and not working,
including what tools are being used
to share data and, most important,
to what degree the data are
protected to prevent loss and
violation of confidentiality.
Using the learning committee,
find out what type of software
program or other tool is needed
for data sharing and protection.
5.3.1
A Staff Person Dedicated to Evaluation
If your organization has an internal evaluator, that is great. However, many organizations cannot
afford an internal evaluator, so they hire an evaluator or work with an evaluator appointed by
the funder. Often, program staff are overstretched and overburdened by data requirements. This
problem becomes greater when staff does not have the capacity to handle data issues. In this case,
having a dedicated staff member who has the capacity to work with data would benefit the
organization. This staff person should be knowledgeable about various types of data, data
collection methods, data requirements, data analyses and data reporting. This person serves as a
data expert who works with the evaluator or the funder to provide the needed information
(e.g., raw data, summaries of data). This person can attend webinars or workshops conducted
by organizations such as the American Evaluation Association (http://www.eval.org) and
The Evaluator’s Institute (http://tei.gwu.edu).
5.3.2
Budget for Evaluation
An organization serious about evaluation will have a line item
in their budget for evaluation. This may include salaries for an
internal evaluator or data staff. This line item also could include
a budget to offset the costs for conducting an evaluation.
Expenses might include hiring an evaluator, cost of data system
or software, training of staff or volunteers, travel costs for
evaluation (e.g., travel expenses for evaluation participants),
translator or interpreter costs, etc. Section 7.9 in Chapter 7
provides more detailed explanation about how to budget for
certain types of expenses and activities in evaluation.
5.3.3
A Culture of Data and Learning
An organization with a culture of data and learning is more likely
to benefit from evaluation. It becomes a learning organization,
i.e., an organization in which leadership and staff continually
develop their abilities to achieve the results desired. In such
an organization, staff has a strong commitment to using data
for program improvement. An organization that emphasizes
learning will be more likely to use evaluation findings to improve
their programs or services. Learning organizations tend to have
systems already in place for data collection. If yours is not such
an organization, you should work with staff to build this culture,
which starts with organization leadership who models the be-
havior by asking questions, using data to inform decisions and
making improvements based on data. Leadership also
must sanction staff time to collect data where necessary, to
attend training about data use and to conduct reflections and
learning meetings.
5.3.4
Share and Protect Data Across Program Elements or Program Locations
Your organization can become a better consumer of evaluation by investing in a system that will
allow data sharing across program elements or program locations. This also will prevent your staff
from working in isolation and will benefit program participants by giving you a more
comprehensive understanding of their needs or progress.
When your organization participates in evaluation, you will be collecting and sharing a lot of data.
You should consider ways to protect the participants’ data – a step that is both ethical and
reassuring, as it also will help ease your participants’ worry about the safety of their data. This
might involve having password-protected computers or locked cabinets for data storage. You also
could consider having a data-sharing plan with evaluators or funders.
HIGHLIGHTS
Depending on who is funding the
evaluation, the requirements might vary
greatly. Be vigilant about understanding
these requirements and be engaged in
the process to prepare for the evaluation –
even when there is an external evaluator
assigned to your organization by
the funder.
Carefully consider how the information
and findings you share with the
stakeholders, including your evaluator,
can be used (or misused). Check
thoroughly whether anything in what you
collect for the evaluation can be taken out
of context or misconstrued in a way that
is culturally incompetent or harmful to
your community.
Invest in building your organization’s
capacity both to become an informed
consumer of evaluation and to develop
and maintain a learning system and
culture.
Whether you hire an internal or external
evaluation depends on the resources you
have, your staff members’ knowledge and
skills related to evaluation, your funder’s
requirements and the importance of an
independent perspective.
exercises
1. When you hire an external evaluator, you can give up control of the evaluation to him/her and
participate in the evaluation process only when asked by the evaluator.
A. True B. False
2. Consider a time when you hired an external evaluator or used an internal evaluator. What were
the pros and cons? If you could do it all over again, would you make a different decision after
reading Chapter 5? Why or why not?
3. It is perfectly fine to share data with anyone in your organization regardless of whether or not
they work on the program.
A. True B. False
4. What policies and procedures do you have in place to protect the data your organization col-
lects and uses? If none, what policies and procedures might you consider?
Answers: 1B; 3B
5. Your organization received a grant to increase low-income families’ access to healthy and fresh
food. The grant was awarded as part of a national initiative to decrease obesity through the use
of environmental change strategies. The award letter stated that you would be required to
participate in a national evaluation and nothing else. You have a conference call with the
funder in a week. What questions should you ask the funder about your organization’s
participation in the national evaluation?
6
Engage Them in the Evaluation
How this chapter is organized…
6.1 Evaluators as
Stakeholders
6.1.1 Keeping Your Evaluator
Informed on a Frequent,
Regular Basis
6.1.2 Two Types of Evaluators:
Internal and External
6.1.3 The Evaluator’s Role
6.1.4 The Evaluator’s Cultural
Competency
6.2 Other Key Stakeholders
6.2.1 How To Identify Key
Stakeholders
6.2.2 How To Engage Key
Stakeholders
Highlights
Exercises
A PREPARE for conducting
an evaluation
B
DETERMINE
stakeholders
and how and
when to
engage them
C
IDENTIFY
assumptions and
determine what
will be di�erent
(theory of change
and logic model)
D DEVELOP evaluation plan (logic
model, measurement,
framework, etc.)E COLLECT and analyze data
F
COMMUNICATE
results and
understand
what happened
(interpret
�ndings and
facilitate
learning)
G
MAKE informed decisions
(improve actions and
next steps)
Stakeholders
introduction
An evaluation is not simply
a matter of looking at your
organization, strategy,
initiative or program
and saying, “Here is
what we want to
know.” The evaluation
should include
understanding and
considering the priorities
and concerns of the various
people who have some vested
interest in what you do. Typical
stakeholders include your board of directors, staff, current or
potential funders, technical assistance providers, policymakers and
program participants. In most cases, evaluators also are considered
stakeholders because they are directly involved with the effort.
A stakeholder is any person or group who has an
interest in the strategy, initiative or program
being evaluated or in the results of the evaluation,
including the evaluator.
Involving stakeholders is important because they can help (or
hinder) an evaluation before it is conducted, while it is being
conducted and after the results are produced and ready for use.
Because nonprofit organizations often tackle complex issues,
stakeholders take on particular importance in ensuring the right
evaluation questions are identified and that evaluation results will
be used to make a difference. Stakeholders are much more likely to
support the evaluation and act on the results and recommendations
if they are involved in the evaluation process from the beginning.
if they are involved in the evaluation process from the beginning.
6.1
evaluators as stakeholders
As mentioned earlier, you will want to engage various
stakeholders in your evaluation. This includes the evaluator, who is not often thought of as a stake-
holder. However,
evaluators are not neutral third parties. Evaluators have a vested interest in what they are doing
and care about doing it well. With help from the organization’s executive director and staff, they
also can bring together all the other stakeholders needed for the evaluation.
6.1.1
Keep Your Evaluator Informed on a Frequent, Regular Basis
Evaluators play a very important role in your efforts, from helping you design the research to inter-
preting and communicating the results. You have a responsibility to keep the evaluator informed
on a frequent and regular basis about what is happening in your strategy, initiative or program. You
are immersed in your effort’s implementation and pay close attention to the need to adjust when
you see something is not working. Communicating these adjustments to your evaluator will make
his or her work more efficient and effective.
Let’s say you are implementing a financial literacy program. One aspect involves partnering with
a financial institution to host and conduct a 12-week-long financial literacy academy. However,
after you received the grant and were ready to start, the financial institution that committed to the
partnership pulled out of the agreement. You approached several banks and credit unions in the
community, but none had the staff capacity to do it. In the end, you decided to implement the cur-
riculum yourself and shortened it to 10 weeks to fit the program schedule.
In the middle of this challenging situation, while spending more time than you anticipated seeking
a partner and then revising the curriculum, you might have forgotten to inform the evaluator of the
change. Or you might not have realized the implications for the evaluation and therefore didn’t
inform the evaluator.
Here’s why it is important to have spoken to the evaluator: A survey questionnaire – already tested
and validated by the people who designed the academy – was used to evaluate the academy. Once
the model for the academy had been changed, the questionnaire was no longer valid. It needed
to be revised to fit your new approach and the results cannot be compared to those from another
study that used the academy model. The evaluation design would have to be adjusted.
An evaluator sometimes lives elsewhere and you won’t see him or her frequently. In such cases,
maintain regular contact by telephone and email.
6.1.2
Two Types of Evaluators: Internal and External
You might have heard the terms “internal” or “external” evaluator. You should understand the
difference and when to use an internal or external evaluator or, in some cases, both.
Internal Evaluator
It is perfectly fine to assign the responsibility for evaluation to a current staff member or to hire an
evaluator to join your staff. This internal evaluator can serve as both an evaluator and a staff
member with other responsibilities. This option has a number of advantages:
An internal evaluator can be less expensive because you’re not spending resources to hire
someone.
You are using existing resources and building the evaluation capacity within your own
organization.
Most importantly, as the internal evaluator works within the program, he or she might be more
familiar with the program, its staff and community members; could have access to
organizational resources; and might have more opportunities for informal feedback with
program stakeholders.
However, keep in mind the following disadvantages to hiring an internal evaluator:
The evaluator might be constrained by demands of other existing job responsibilities, leaving
insufficient time to focus on the evaluation.
The internal evaluator could lack the outside perspective and technical skills of an external
evaluator.
The internal evaluator might find it harder to remain objective, especially if evaluating a pro-
gram that he or she is working on.
The internal evaluator could also want to avoid negative conclusions, which reflect badly on
that person’s own work, other people in the program or the organization as a whole.
Nonetheless, if the strategy, initiative or program is not too complex and has budget constraints,
assigning or hiring a staff person with the right skills to be an internal evaluator is advisable.
checklist
Things to consider for an internal evaluator:
University degrees in evaluation are not common. Many people now working as
evaluators have previously held managerial and administrative roles or conducted applied
social research. Therefore, consider hiring those who do not label themselves as
professional evaluators, but who have conducted evaluation tasks for similar programs.
If you cannot afford to hire someone for evaluation alone, then consider what other skills
the person needs. Assess the candidates’ prior experience working with the sectors and
populations you serve.
Also assess candidates’ ability to build bridges with program staff. Look for experience
with facilitating learning with groups. However, even when staff members have the above
qualities, you should still consider some of these questions:
Do they have the technical expertise?
Can they be objective? Do their roles in the organization create some bias,
predisposition or even prejudice?
Could they be influenced by others in the organization who desire a certain
evaluation outcome?
Are the internal politics such that a valid and fair evaluation is difficult, if
not impossible?
Will their findings have credibility, both within the organization and to
external parties?
The following principles are important for an effective internal evaluation:
Involve staff as much as possible.
Link the evaluation to the organization’s learning agenda. (If you don’t have a learning agenda,
you might want to develop one.)
Dedicate a staff person or form a group of staff members to be responsible for the day-to-day
management of the evaluation.
Strive for consensus on the evaluation plan and make sure your staff are involved in the plan.
Provide necessary resources for the evaluation and the staff person or persons responsible for
the evaluation.
Develop a process to regularly share updates about the evaluation’s progress.
Allocate time for staff to reflect on the evaluation’s findings and to discuss the findings’
implications on their work.
Develop a systematic plan by which improvements to your organization’s strategies, programs,
initiatives and policies can be made.
External Evaluator
In many cases when you have larger grants or a complex strategy, initiative or program, you might
want or need to hire external evaluators because:
These professionals often have assets including technical expertise and experience not
available in your organization.
They also could have networks that bring additional useful expertise and knowledge.
External evaluators might have broader evaluation expertise than internal evaluators,
particularly if they specialize in program evaluation or have conducted extensive research on
the subject matter related to your effort.
External evaluators also could bring a different perspective to the evaluation because they are
not directly affiliated with your organization or effort.
At the same time, here are some drawbacks to hiring external evaluators:
External evaluators could be detached from the daily operations of the strategy, initiative or
program and thus have limited knowledge of the effort’s needs and goals and limited access to
the effort’s activities.
External evaluators might not understand the community as well as you do.
External evaluators can be more costly than internal evaluators.
External evaluators have less opportunity to develop internal evaluation capacity.
You and your external evaluator will need time to build trust and good communications. It is
common for external evaluators to create some initial anxiety for you and your staff; everyone will
need to work hard at developing a working relationship.
checklist
Considerations for selecting an external evaluator:
Assess the candidates’ prior experiences working with the sectors and populations you
serve and their understanding of the socioeconomic, demographic, historical, cultural and
political factors that shape the conditions in which the strategy, initiative or program is
operating. This means ensuring culturally competent evaluators – those who understand
how various groups perceive an intervention, communicate their views and experiences
and are affected by the factors mentioned above. Culturally competent evaluators are
particularly effective because they:
Keep an open mind.
Avoid making assumptions about a particular group of people.
Understand how their own cultural background, biases and worldviews could affect
their interactions with program participants and other stakeholders.
Gather appropriate data to draw conclusions by using methods that are respectful of
other cultures.
Look for candidates whose experience suggests the capability to devise and manage a
variety of evaluation designs and tasks, and who can clearly articulate which design is
most appropriate for your effort. Ask questions that require the candidates to describe
their experiences working with similar populations, and how they would apply their
understanding of the nuances of working with these groups to your evaluation.
exhibit 6-1: advantages and disadvantages of internal and external evaluators
Advantages Disadvantages
Internal Evaluator Costs less.
Knows the effort being evaluated.
Can easily help coordinate key
stakeholders.
Helps build capacity of organization to
conduct future evaluations.
Might not have enough time to work
on the evaluation due to other
responsibilities in the organization.
Could lack expertise and technical
skills of an external evaluator.
Might lack objectivity.
Could raise doubts among potential
funders about the objectivity and
validity of the evaluation.
Might inhibit candor of clients who do
not express themselves honestly to
someone on the staff whom they will
see again.
External Evaluator Brings technical expertise that might
not be available within your
organization.
Could have networks with additional
expertise.
Could be more efficient due to
experience with evaluation.
Could be more objective.
Could have greater credibility with
potential funders.
Might not have enough understanding
of the effort or context.
Can be more expensive.
Might have less opportunity to devel-
op internal evaluation capacity.
Could cause anxiety among staff
members who feel they are under a
microscope from an outsider.
Might cause staff to be less engaged,
seeing the evaluation as the external
evaluator’s responsibility.
Useful resources on hiring evaluators include:
Hiring and Working with an Evaluator, by the Juvenile Justice Center, published in 2016.
Available at http://www.jrsa.org/pubs/juv-justice/evaluator .
How Do I Work with An Internal Evaluator or External Evaluator Contractor, by E.F. Chappelle,
published in 2011. Available at http://www.cdc.gov/dhdsp/pubs/docs/
cb_january_11_2011 .
Selecting an Evaluator, by Community Science for the W.K. Kellogg Foundation. Available
from info@communityscience.com.
cb_january_11_2011
cb_january_11_2011
6.1.3
The Evaluator’s Role
With your staff and stakeholders, think through all of the potential evaluator roles and
relationships and determine which configuration best fits your particular situation, the purpose of
the evaluation and the questions you are attempting to address.
Remember the primary goals of evaluation are (a) that stakeholders are engaged, active
participants in the process; and (b) that the evaluation process and findings will be meaningful
and useful to those ultimately responsible for improving and assessing the strategy, initiative or
program. In the end, there is no one way to conduct an evaluation. Given that premise, the critical
skills of the evaluator you choose should include:
The ability to listen, negotiate and bring together multiple perspectives, particularly those that
have been historically excluded.
The ability to analyze the specific situation.
The ability to assist in developing a design with the program team that leads to the most useful
and important information and final products.
Another important evaluator role is
the relationship between the evaluator
and primary stakeholders, including the
program team. Questions to consider
include:
How can this relationship be highly
interactive?
How much control should the
evaluator have over the evaluation
process relative to that of the
stakeholders or program team?
How actively involved should key staff
and stakeholders be in the evaluation
process?
Depending on the primary purpose of the evaluation, you might
also look for an evaluator with different strengths. For instance:
If the evaluation purpose is to determine the worth or merit
of a program, you could look for an evaluator with
methodological expertise and experience.
If the evaluation is focused on facilitating improvements to a
new program, you could look for someone with strong
facilitation skills and knowledge of organizational
development.
6.1.4
The Evaluator’s Cultural Competency
Most importantly, the evaluator should be culturally competent.
This means you should hire someone who has skills that enable
him or her to:
Have an open mind about people who may seem different.
Avoid making assumptions about other people’s worldviews
and behaviors.
Understand how historical and contextual factors (e.g.,
structural racism, poverty, sense of hopelessness) could
affect the evaluation process and results.
Appreciate and account for the strengths and assets of
people who, despite the inequities they experience, strive to
overcome their day-to-day challenges and aspire to have a
higher quality of life.
Gather the right data to draw appropriate conclusions that
consider the context within which the program is operating.
Overall, among
the skills and
expertise wanted
from a good evaluator are the
ability to work with you,
program staff and other
stakeholders, and the ability to
practice cultural competency.
The selected evaluator must be
able to listen, bring together
several perspectives, integrate
stakeholder input in the evalua-
tion design and prioritize anal-
yses that will be most applicable
to your organization’s
information needs.
However, remember that hiring
an external evaluator should
not mean the organization
suspends its engagement in the
evaluation. The evaluator brings
requisite methodological
skills, competencies and
experiences, but your staff and
evaluation committee (if you
have one) will remain critical
players in the evaluation process
by determining the measures of
success, “making meaning” of
the data analysis, extracting and
discerning lessons learned and
making decisions based on the
evaluation findings. Indeed, all
of the organization’s
stakeholders need to be active
and engaged in the process.
In addition to these skills, culturally competent evaluators typically:
Go out of their way to demystify evaluations by explaining the evaluation design and process in
terms that people with no training in evaluation can understand.
Ensure the results are explained in plain and simple language so people with no social science
background can understand.
Encourage people to ask questions and share their views about what the results might mean
to them.
Useful resources for the role of culture in evaluation and culturally competent evaluations
include:
The Importance of Culture in Evaluation, by Kien Lee, published by The Colorado Trust in 2007.
Available at http://www.communityscience.com/pdfs/CrossCulturalGuide.r3 .
A Guide to Conducting Culturally Responsive Evaluations by H.T. Frierson, S. Hood and
G.B. Hughes, included in the National Science Foundation’s 2002 User-Friendly Handbook
for Project Evaluation. Available at https://www.nsf.gov/pubs/2002/nsf02057/
nsf02057_5 .
6.2
other key stakeholders
As mentioned before, in addition to the evaluator, all evaluations have multiple stakeholders. They
include:
funders (public, private);
program staff;
board members;
program participants or constituents of your organization;
community leaders;
collaborating organizations or partners; and
policymakers.
For example, if your strategy, initiative or program focuses on improving the competencies of
high school teachers in your community, the range of stakeholders would include the program
implementers, participating teachers, members of the school board, school administrators,
parents, participating students, funders and the evaluator.
exhibit 6-2: types of stakeholders
Types of
Stakeholders Definition Examples
Implementers Those directly involved in operating and
implementing the program.
Program directors and managers
Trainers who work with the teachers
Partners Those who actively support and are
invested in the program.
Organizations that are advocating for a
higher quality education system
Parents
Representatives of the school system
Policymakers Those in a position to decide something
about your program (e.g., whether or not
it should be expanded to other school
districts).
School board
School administrators (e.g., principal)
State education department director
Program
participants
Those being served or affected by your
programs.
Students in the schools with
the program
Teachers
Funders Those who fund the program. Public agencies
Private foundations
Remember to ensure you have gathered multiple perspectives about the main issues by involving
as many stakeholders as possible in initial evaluation discussions. Otherwise, the evaluation is
likely to be designed based on the needs and interests of only a few stakeholders – usually the
ones with the most power – and may miss other important questions and issues of stakeholders
not included at the table. There are many advantages to involving stakeholders:
Stakeholders can help improve the quality of decision-
making, since those with a vested interest contribute from
the initial stages.
Stakeholders can build trust that leads to acceptance of both
positive and negative feedback.
Involving stakeholders ensures the information
gathered is more reliable and comes from different
perspectives.
Involving stakeholders creates opportunities for you, your
stakeholders and decision-makers to learn from each other
by exchanging information and experiences.
6.2.1
How to Identify Key Stakeholders
Roles of various stakeholders
Determine what your needs are. At this point, begin to assess
whom you should involve. Since stakeholders offer different
kinds of value, your reasons for inviting certain individuals or
groups to participate should include the following:
They have content knowledge of the program being
evaluated (e.g., staff and any expert consultants you hired).
They represent diverse perspectives and experiences, so
they can raise questions and ideas that reflect all sides of
the issue (e.g., community leaders, political representatives,
target population, partner organizations, etc.).
They are affected by the program (e.g., program
participants).
They are in positions of influence and can raise questions
relevant to politicians, elected officials and other change
agents (e.g., respected community leaders, advocates).
When involving
stakeholders from
the beginning of
the evaluation process, you are
more likely to:
Reduce stakeholders’
distrust and fear
of evaluation.
Increase stakeholders’
awareness of, and
commitment to, the
evaluation process.
Increase the chances
stakeholders will support
your evaluation efforts,
advocate for your effort and
adhere to subsequent
recommendations.
Increase the chances the
evaluation findings will be
used.
Increase the credibility of
your evaluation findings.
If you do not engage
stakeholders, your evaluation
runs the risk of missing
important elements of the
strategy, initiative or program.
In that case, evaluation
findings might be ignored,
criticized or resisted because
your evaluation did not
consider stakeholder concerns
or priorities.
They are proponents of evaluation and offer support throughout the evaluation’s design and
implementation (e.g., your funders and board of directors).
They are responsible for decisions about the evaluation and program (e.g., organizational
leadership, program director).
6.2.2
How to Engage Key Stakeholders
Several factors should be considered in determining the best strategy for engaging your
stakeholders. These considerations will help you decide whether to work with stakeholders in
person or virtually, and which engagement methods – individual meetings, group meetings or
surveys – best fit your circumstances.
Imagine that your funders have asked you, as part of the grant requirement, to evaluate the
curriculum they funded to improve the competencies of high school teachers. You have
considered who could benefit from the evaluation (e.g., students, the designer of the curriculum)
and whose interests could be at risk due to the evaluation (e.g., principals, teachers). You need
stakeholders to support your evaluation, so at this point here are some considerations when
engaging stakeholders:
a. Budget: Do you have the budget to cover the costs of gathering input from stakeholders? While
you might want to engage every stakeholder in your community who is vested in improving the
quality of education, you should consider what financial resources you have to engage these
stakeholders. For example, it will cost more money to travel to engage some stakeholders.
Consequently, you might consider limiting the number of stakeholders or conducting virtual
meetings to gather their input.
b. Time: Some stakeholders, such as legislators and elected officials, require a fair amount of
time to engage as key informants. Determine whether you have the time to wait (in some cases
several months) to book an appointment to see them. In this case, you could consider sending
them a short list of questions and asking them to respond by email, or requesting a 15-minute
telephone call to discuss their perspectives.
c. Geographic locations: Bringing people physically together
can be valuable. Visual cues and building relationships are
critical throughout an initiative and its evaluation. However,
depending on the budget, time constraints and locations
of stakeholders, getting them t ogether in the same room
might not be efficient or feasible. Using a virtual engagement
strategy such as a webinar could be your best option.
d. Engagement methods: Another consideration when
engaging stakeholders is the methods you would use to
bring them on board. You should determine whether you
need to engage the stakeholders face-to-face, or via email,
webinar or a survey. Your budget and time should factor into
your decisions about these methods. For example, if you
have limited resources, you might want to engage
stakeholders who are easier to access (e.g., via email) and
require minimal resources to engage. You might have to
allocate funds to reimburse people for their transportation
and childcare expenses if you involve certain types of
stakeholders (e.g., youth, parents) face-to-face.
No magic
formula exists to
determine how
many stakeholders should be
included. However, keep in
mind that larger groups (more
than nine) could take longer to
reach decisions than
smaller groups and might make
the process more complicated.
Also, consider carefully what
will enable and motivate the
stakeholder to participate.
Community members might
need transportation or childcare.
Elected leaders could need
specific and clear explanation
about how their involvement
relates to concerns of their
constituents. Program staff
might need an explanation
about how their involvement
can make their jobs easier.
Exhibit 6-3: template for engaging stakeholders
Who is the
stakeholder?
What is the best
way to engage the
stakeholder?
What expenses
and other special
considerations are
there?
What is the
stakeholder’s role?
How frequently
should you
communicate
with them?
Now that you have identified other key stakeholders for the evaluation, how do you invite them to
participate? Here is one strategy:
Invite the identified stakeholders to a meeting, or series of meetings, depending on their avail-
able time. At this meeting ensure that you:
Brief stakeholders on the strategy, initiative or program and the evaluation. One major
objective of this meeting should be to obtain a clear understanding of stakeholder
interests, perceptions and concerns related to the effort and evaluation.
Have all stakeholders clearly identify and agree to their roles and responsibilities related to
the evaluation before it begins.
Assure stakeholders that you will strive to maintain open communications and address
their concerns. Clarify to the group when they will hear from you, including when you will
seek their input and involvement during the evaluation process.
Emphasize to the stakeholder group that while their input and recommendations are very
valuable, it may not be feasible to implement all of their recommendations. Emphasize that
these decisions are driven by the availability of program resources (i.e., program staff, time
and budget priorities).
After this initial meeting, and when the selected stakeholders are on board and the theory of
change and logic model have been developed, you can begin identifying the most important
questions you want the evaluation to answer.
Involving the stakeholders in developing the questions is important. Effective evaluation
questions can guide how to examine your data to determine if your program is accomplishing what
it should. Having stakeholders assist in developing the questions will not only strengthen their
buy-in and support, but their perspectives could allow you to look at the strategy, initiative or
program and evaluation from a new angle. Also, some stakeholders may “hear” the questions
differently, especially if the questions’ phrasings suggest inappropriate assumptions about the
program participants or the community.
Consider this question, for example: “How has the program affected the academic performance of
high-risk youth?” This question labels the program youth as “high-risk,” which usually raises
concerns among advocates and leaders who work with the youth because of the general and
negative connotation associated with the term. For that reason, the question could be better
phrased as, “How has the program improved the academic performance of participating youth?”
Engaging different stakeholders in developing the evaluation questions provides an opportunity
for this sort of discussion, which also fosters learning by the organization staff, the evaluator and
the stakeholders.
HIGHLIGHTS
Evaluators are stakeholders, too.
Consider: What are their interests? How
might this affect how the evaluation is
designed – questions posed from a
particular focus and what interpretations
are made?
There are advantages and
disadvantages to hiring an external
evaluator or an internal evaluator. You
should consider the type of evaluation you
plan to conduct, the funds available for
the evaluation, the staff’s current capacity
to participate in the evaluation and the
intended use of the findings.
Stakeholders will have different,
sometimes even contradictory, interests
and views. They also hold different levels
of power. Program directors have more
power than staff. Legislators have more
power than primary grade students.
Funders have a particular kind of power.
Ask yourself: Which stakeholders are not
being heard in this process? Why not?
Where can we build consensus and how
can we prioritize the issues?
Examine the values embedded in the
questions being asked. Whose values are
they? How do other stakeholders,
particularly program participants, think
and feel about this set of values? Are there
different or better questions the evaluation
team members and other stakeholders
could build consensus around?
Useful resources for engaging stakeholders in evaluation include:
A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions, by H. Preskill
and N. Jones, published by the Robert Wood Johnson Foundation in 2009. Available at
http://www.rwjf.org/en/library/research/2009/12/a-practical-guide-for-engaging-
stakeholders-in-developing-evalua.html.
Do Nothing About Me Without Me, by J. C. Bourns, published by Grantmakers for Effective
Organizations and Interaction Institute for Social Change in 2010. Available at http://docs.
geofunders.org/?filename=do_nothing_about_me_without_me .
Stakeholder Engagement Tool, by S. Salentine and A. Johnston, published in MEASURE
Evaluation and USAID in 2011. Available at http://www.cpc.unc.edu/measure/resources/
publications/ms-11-46-e.
exercises
1. When engaging stakeholders
A. Ask stakeholders about the best way to involve and communicate with them
B. Respect only those stakeholders’ values that are close to yours
C. Be clear about their role in the evaluation
D. Both A and B
E. Both A and C
2. If your organization hires an experienced evaluator with an excellent track record, there is no
need to involve program stakeholders.
A. True B. False
3. Stakeholders would more likely fully support your evaluation, if you involve them during the
stage.
A. Planning
B. Data collection
C. Analysis
D. Interpretation
E. All of the above
4. Consider a program, initiative, strategy or policy that you would like to evaluate. Who are the
stakeholders and what are the best ways to engage them?
5. You are working with an external evaluator to evaluate a curriculum to improve the quality of
teaching in high schools. A recent report by an independent research association has shown
decreasing high school graduation rates during the past five years for your community. The
report has raised a lot of concerns among parents, principals and local elected officials. Your
Answers: 1E; 2B; 3E
organization adopted the evidence-based curriculum from a national organization. If success-
ful, the curriculum may be expanded to other school districts. To evaluate the curriculum’s
implementation and effectiveness, your evaluator proposed the following activities: interviews
with principals; focus groups with teachers and parents; observations during instruction; and
content analysis of lesson plans before and after the curriculum’s implementation. Apply the
template intruduced previously [Exhibit 6-3] to determine how to engage the stakeholders in
this question.
Who is the
stakeholder?
What is the best
way to engage the
stakeholder?
What expenses
and other special
considerations are
there?
What is the
stakeholder’s role?
How frequently
should you
communicate
with them?
7
Developing a Logic Model, Evaluation
Questions, Measurement Framework
and Evaluation Plan
A PREPARE for conducting
an evaluation
B
DETERMINE
stakeholders
and how and
when to
engage them
C
IDENTIFY
assumptions and
determine what
will be di�erent
(theory of change
and logic model)
D DEVELOP evaluation plan (logic
model, measurement,
framework, etc.)E COLLECT and analyze data
F
COMMUNICATE
results and
understand
what happened
(interpret
�ndings and
facilitate
learning)
G
MAKE informed decisions
(improve actions and
next steps)
Stakeholders
The Kellogg Foundation Logic Model Development Guide (available at www.wkkf.org/resource-
directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide) is an excellent
companion for this chapter Rather than repeat the information in that guide, this chapter focuses on the
use and process of developing a logic model and how to generate evaluation questions and a
measurement framework from it. You should combine all of these components to form an evaluation
plan.
Evaluation – especially logic models – is a learning
and management tool that can be used throughout the
life of a strategy, initiative or program, whatever
your stake in it. Using the logic model results in
effective design of the effort and offers greater
learning opportunities, better documentation of
outcomes and shared knowledge about what
works and why.
More important, the logic model helps to
ensure that evaluative thinking is integrated
into your evaluation design and implementation.
Evaluative thinking is a systematic process of collecting and analyzing data in order
to tell the story about your strategy, initiative, program, policy or organization. It
is based on the belief that a systematic process is valuable and necessary. Such a
process involves identifying assumptions about what you think works and doesn’t
work and why; posing thoughtful questions about what change you expect to see
during and after you implement your effort; pursuing deeper understanding through
reflection and dialogue; communicating what was learned without underestimation
or exaggeration; and making informed decisions in preparation for action.
directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide
directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide
directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide
How this chapter is organized…
7.1 Overview of Logic Model
7.2 Use The Logic Model Throughout Your
Effort’s Life Cycle
7.2.1 Use of Logic Models to Design Your
Strategy, Initiative or Program
7.2.2 Use of Logic Models to Inform
Implementation
7.2.3 Use of Logic Models to Develop Evaluation
Questions
7.2.4 Use of Logic Models to Develop an
Evaluation Plan
7.2.5 Use of Logic Models to Create an Outline for
Your Evaluation Report
7.2.6 Use of Logic Models to Improve Your
Strategy, Initiative or Program
7.3 What You Can Expect from Logic Modeling
7.3.1 It is a Process
7.3.2 The Process Can Surface Differences
7.3.3 The Process Can Bring About Innovation
7.4 How to Go About the Logic Modeling
Process
7.4.1 Whom to Involve
7.4.2 How to Involve Them
7.5 Components of the Logic Model
7.5.1 Alignment between Strategies or Activities
and Outcomes
7.6 From Logic Model to Evaluation Questions
7.6.1 Two Different Ways to Formulate Evaluation
Questions About Your Strategy, Initiative
or Program
7.6.2 Typical Scenarios When Formulating
Evaluation Questions
7.6.3 Other Considerations
7.7 From Logic Model to Evaluation Questions
to Measurement Framework
7.7.1 Key Components of the Measurement
Framework
7.7.2 How to Use a Measurement Framework
7.7.3 Considerations for Developing the
Measurement Framework
7.7.4 Taking a SMART Approach to Measurement
Framework Development
7.8 Your Evaluation Plan
7.9 Budgeting for Evaluation
7.9.1 Types of Expenses Typically
Associated with Evaluation
7.9.2 Time Estimates for Specific Evaluation
Activities
Highlights
Exercises
overview of logic model
You design and implement your strategy, initiative or program the way you do for a reason. For
example based on your experience and findings from past research studies, you believe that
adequate information about how to purchase a home and the services offered by financial
institutions are not accessible to working class African-American families. Your effort in Rainbow
County seeks to increase the number and percent of working class African-American
homeowners in the county. To this end, your effort includes services to educate families about
budgeting, savings and improving their credit scores; acquiring financing; and understanding what
it means to be a homeowner. You also plan to work with the lending institutions in the county to
improve their lending policies, practices and services. Your theory of change is that by working
with the families and lending institutions, more residents will be able to purchase homes;
homeownership supports asset building and family economic security. When you draw the
pathway of change for the effort, you are in fact creating a logic model.
A theory of change is a narrative that explains the links between program strategies
or activities and outcomes, and how and why the desired change is expected to
come about.
In the above example, the rationale for your effort would be explained in more detail, including
citations of supporting studies and assumptions you are making. This information becomes the
basis of your theory of change.
A logic model is a graphic representation of the theory of change that illustrates the
linkages among program resources, activities, outputs, audiences and short-,
intermediate- and long-term outcomes related to a specific problem or situation.
Useful resources for developing a theory of change and logic model include:
Logic Model Development Guide published by the Kellogg Foundation in 2004. Available at
http://www.wkkf.org/knowledge-center/resources/2006/02/wk-kellogg-foundation-
logic-model-development-guide.aspx.
Point K Tools: Logic Model Builder, by Innovation Network. Available at
Theory of Change: Guided Example – Project Superwomen published by ActKnowledge and the
Aspen Institute Roundtable on Community Change in 2003. Available at
http://www.theoryofchange.org/pdf/Superwomen_Example .
Theory of Change: A Practical Tool for Action, Results and Learning, by J. Reisman and A.
Gienapp, published by Organizational Research Services in 2004. Available at
http://www.aecf.org/upload/publicationfiles/cc2977k440 .
Your logic model for the above theory could look like. The previous the logic model that shows
how (A) leads to (B) and (B) to (C) and (C) to (D) and so on through (G). The effort may be affected
by the context, history, culture and other factors surrounding it (G). A logic model usually includes
components such as resources/inputs, activities, outputs and outcomes, and the outcomes can be
divided into immediate, intermediate and long-term outcomes (explained further in section 7.2).
The logic model operationalizes your theory of change. During the evaluation, you collect data to
test the theory. In most cases, practitioners, evaluators and even funders tend to use “logic mod-
el” and “theory of change” interchangeably. However, they are complementary and should be used
together, not separately or as a substitute for each other.
logic-model-development-guide.aspx
logic-model-development-guide.aspx
G
Context, History, Culture, and Other Factors
Exhibit 7-1: sample program logic model
A
Funding
Staff
Consultation
from a housing
intermediary
Resources/
Inputs
Activities Outputs
Immediate
Outcomes
Intermediate
Outcomes
Long-term
Outcomes
B
Design and
conduct 10
courses for
working class
African
American
families
Reach out to
and work with
lending
institutions
Review existing
lending policies
C
Number of
participants
who
completed the
courses
Participants’
needs were
met
Number of
partnerships
with lending
institutions
D
Increased
knowledge
among families
about process
and resources
for purchasing
a home
Improved
financial
literacy among
families
Actions by
lending
institutions to
reassess their
policies and
practices
E1
Increased
savings
E2
Improved
credit scores
E3
Increased loan
approval rate
E4
Better terms for
mortgage loans
E5
Changes in
lending policies
and practices
F
Increase in
percentage of
homeowner-
ship among
working class
African
Americans in
the county
Theories of change link outcomes and activities to explain HOW and WHY the desired
change is expected to come about, while logic models graphically illustrate program com-
ponents such as inputs, activities and outcomes; creating one will help you and your stake-
holders clearly identify inputs, activities and outcomes. Also, the arrows show the connections between
program components should you decide to include them in the logic model. Some people could interpret
the arrows as causal links (i.e., component D causes component E) but your evaluation might not be able
to prove this because 1) the design you selected (e.g., you and your evaluator decided a case study design
was the best one to answer the evaluation questions) or 2) your initiative is complex and many factors
contribute to the desired outcomes. Additionally, arrows that point in only one direction could suggest a
linear flow in how events and outcomes unfold and that is probably not how they occur. Once you have
completed the evaluation and have better insight into what happened, you could revisit the logic model
and illustrate the arrows or flow and then test the logic model again.
7.2
use the logic model throughout your effort’s life cycle
Using a logic model is an effective way to ensure a strategy, initiative or program’s success. A logic
model is a “living” document that should not be shelved and forgotten once developed but used
and reviewed regularly throughout the life of the effort. The logic model, in particular, is useful for
informing the design of a strategy, initiative or program; its implementation; the plan for
evaluating it; communication of the evaluation findings; and its improvement.
7.2.1
Use of Logic Models to Design Your Strategy, Initiative or Program
When you are designing your strategy, initiative or program or writing a funding proposal for it,
developing a theory of change and logic model as part of the process can be helpful. Some funders
request a logic model as part of the application. At this stage, you might need to do some research
about the theories and best practices that support the outcomes you want to achieve: What
strategies and activities have been shown to be most effective in achieving those outcomes?
Once you have that information, you can begin to sketch out a logic model to show the path and
progression of change. The proposal writing should become easier if you can describe the logic
model and the theory underlying it.
Sometimes after you are awarded the grant, you could have an opportunity to review the logic mod-
el with other grant stakeholders (e.g., evaluator, technical assistance provider, foundation or
government program officer). This is a great opportunity to take advantage of the expertise of these
stakeholders. They could serve as a sounding board. They might have questions about your logic
model and even counterarguments to your logic. If you are working with an external evaluator at
this point, involve that person in this logic modeling process, so you and the evaluator have a
shared understanding of your strategy, initiative or program.
7.2.2
Use of Logic Models to Inform Implementation
A logic model is not an evaluation tool, but a planning tool to ensure you are implementing the
strategy, initiative or program as illustrated in the logic model. The components of the logic
model – inputs; strategies/activities; outputs;
immediate, intermediate and long-term outcomes;
and contextual conditions – act like ingredients in
a recipe for change. You also can use the model to
determine if your effort is being implemented as
planned. If not, you can use the data gathered to
make decisions for midcourse adjustments. This is
frequently referred to as a process of “formative
evaluation”.
7.2.3
Use of Logic Models to Develop Evaluation
Questions
Evaluation questions can be generated from a logic
model. The logic model acts like a hypothesis: It
allows you to see which stage of development you are
in with regard to your strategy, initiative or program,
and therefore what types of questions to ask. It also
helps you craft specific questions. For instance, without a logic model, you could simply ask the
question, “Did the program achieve the intended outcomes?” With a logic model, can see that only
certain outcomes should be expected after one year of implementation. Thus, your questions can
be more precise and easier to answer.
7.2.4
Use of Logic Models to Develop an Evaluation Plan
Out of the logic model, you can create a measurement framework (see Section 7.7) and then
explain how you will collect and analyze data about the measures in the framework. This forms the
basis of your evaluation plan. Also, it’s important to include in your plan ways to use and
communicate the findings. Otherwise, the findings can be used inappropriately without your
consent or knowledge.
7.2.5
Use of Logic Models to Create an Outline for Your Evaluation Report
Other applications of the logic model could be used to organize your reporting, and also used like
a hypothesis. In your report, you can use the model to show how you tested the hypothesis and
how your evaluation findings support, or do not support, the hypothesis. The typical sections of
your report could include:
Underlying assumptions, components and context for the strategy, initiative or program
Resources used to support the effort
Activities
Results (outputs, immediate and intermediate outcomes)
Facilitating factors and challenges
Recommendations for improvement
7.2.6
Use of Logic Models to Improve Your Strategy, Initiative or Program
The evaluation findings tell you the degree to which your theory of change, illustrated in the logic
model, is true. Remember the earlier recommendation that the logic model should not be shelved
and forgotten? At this point, bring it out again and see if what you expected to happen when you
created the logic model actually happened. Were the activities implemented as planned? Did
your strategy, initiative or program benefit people the way you thought it would? Did the activities
contribute to the results you desired? If not, assess where the effort did not unfold as expected and
what improvements are needed. After you make the improvements to the effort, you should revisit
the measurement framework and evaluation plan, and make the necessary and corresponding
modifications.
The logic modeling process also can happen during a strategic planning process for your
organization. while the text focuses more on a strategy, initiative or program logic model,
an organization can have a logic model, too. This logic model would show how the multiple
strategies and programs of your organization contribute to the organization’s mission and goals. If your
organization is going through a strategic planning process and you are working with an evaluator, make
sure the evaluator and strategic planner work together, becuase they could have different skills. The
strategic planner might be skilled in the process and mapping out the strategies and programs while the
evaluator could be better at identifying the metrics for monitoring your organization’s goals and
objectives.
Involving an evaluator during your organization’s logic modeling or strategic planning process can
benefit your organization. An evaluator working with this process should have good facilitation skills,
such as:
Ability to guide people through a process of learning or planning together
Ability to create a safe space for various stakeholders to share different perspectives
Ability to shift roles between facilitator and expert
Knowledge of logic modeling and strategic planning process
Expertise in topic areas of interest (e.g., health, education, economic development)
Logic modeling can occur at various times in the organization and program cycles. Logic modeling
should happen, at a minimum at the beginning of the program cycle. However, logic modeling
should not end there. You should revisit the logic model as often as needed. Remember, your logic
model is a “living” document. Revisit, adapt or change when appropriate.
7.3
what you can expect from logic modeling
7.3.1
It is a Process
Often times the two words “logic model” could make people cringe! This could be due to their
previous experience, often involving painstakingly long and frustrating conversations with an
“outsider” trying to describe things that seem obvious. Sometimes the process is difficult because
the evaluator asks question after question trying to understand what the plan is. The evaluator
might seem to be criticizing every part of the program, but he or she really is just trying to gain a
clearer picture of the theory of change that may only exist in someone’s mind. Or frustrations could
arise because the evaluator dismisses the staff’s input or jumps to wrong conclusions about the
strategy. In some situations, program leaders might believe everyone understands the program the
same way, only to find out during the first logic modeling meeting they are wrong. Do these
scenarios seem familiar?
Developing a logic model does not happen in just one meeting. Several meetings and iterations of
the logic model probably will be needed to get the logic model just right for your strategy, initiative
or program. Contrary to popular belief, logic modeling is a process that involves time, energy and
thought; it is not a task to complete in a two-hour meeting.
Logic models are not evaluation tools; they are learning and management tools that should be
used throughout the life of a strategy, initiative or program. A logic modeling process should facili-
tate effective planning, implementation, evaluation and improvement of your effort.
7.3.2
The Process Can Surface Differences
The logic modeling process will surface differences in viewpoints about everything from definitions
to assumptions about the change process. This dynamic is normal. In fact, it’s helpful when dis-
agreements arise while creating a logic model for your strategy, initiative or program.
Precision is required regarding definitions and concepts.
For example, if the activity consists of instructional courses about homeownership, you will be
asked during the logic modeling process to identify the instructors, how frequently each course
occurs (number of sessions per week and number of hours per session), how many participants per
session and how improvement in knowledge about homeownership is measured. Achieving this
sort of precision requires time to discuss and decide. You could be surprised how many different
ways of thinking might exist among your staff and other stakeholders. Hence, good facilitation is
necessary when developing a logic model.
7.3.3
The Process Can Bring about Innovation
This process is intended to explore the pathway to change from various angles, and even surface
innovative ideas on how the strategy, initiative or program can be better designed or otherwise
improved. For instance, with the previous homeownership example, someone might suggest that
homeownership alone cannot lead to family economic security; economic and workforce
development must also be addressed. That person could suggest adding a component to include
job training.
7.4
how to go about the logic modeling process
7.4.1
Whom to Involve
Ideally, logic modeling should be done with key stakeholders of your strategy, initiative or program,
such as key staff, the organization’s leadership, program participants, community leaders and, of
course, the evaluator. Sometimes your board members and funders might have to be involved in
the process.
7.4.2
How to Involve Them
You should work closely with the evaluator to design the process because you know which
stakeholders to involve and the evaluator understands how the process should unfold. Together,
you can design an effective process and agenda. There are different ways to engage stakeholders
in the process, all of which require dialogue and interaction. Here are three examples:
If you decide to bring everyone together for several hours and it’s a fairly large group, you
should design a process that allows for small group discussions. These small groups can be
organized by stakeholder type (e.g., all the program participants in one group) or as a mix of
different stakeholders (e.g., two program participants, two staff members, two board
members).
Keep in Mind: Differences in power among stakeholders can cause some people to be less
vocal and others to be inadvertently “louder.” You can do four things to manage this:
Make sure the stakeholders with less power (e.g., program participants) outnumber the
ones with more power (e.g., staff or board members).
Orient and prepare the stakeholder group with less power before the meeting so they know
what to expect and feel confident about their contribution to the process.
Use a discussion technique that gives everyone a chance to talk.
Use exercises that allow participants to write down their thoughts and turn in the written
notes; this could help increase participation from those who are shy about speaking in
front of a group.
If you have participants who speak a language other than English, use an interpreter and
simultaneous interpretation equipment. Also, conduct half the meeting in one language
and the other half in the other spoken language and require people not proficient in the
language to wear wireless headsets. This way, everyone experiences what it is like to not
understand another person’s language, which could help level the playing field.
If you decide not to bring everyone together, you can design a process consisting of several
separate discussions and then combine the feedback to formulate the logic model. Then, share
the draft logic model with everyone, get comments and revise the logic model accordingly.
been integrated into the revised version.
Sometimes people engage better when they can react to something. If you think this approach
will work for your strategy, initiative or program, you can create a draft logic model and then
convene the stakeholders to discuss it. Work with the evaluator to develop a set of questions to
structure and guide the discussion.
Keep in Mind: If you do it this way, be sure to stress the version they are reacting to is a draft
and be clear about the parameters for revisions. You don’t want people to think they can
change whatever they want and risk disengaging them when their input is not integrated into
the revised version.
7.5
components of the logic model
Basic components of the logic model, as described in the
Kellogg Foundation’s Logic Model Development Guide, include:
Resources/Inputs: The human, financial, organizational and
community resources which a program, initiative, strategy or
policy has for the work.
Strategies or Activities: The processes, tools, events,
technology and actions that are an intentional part of the
implementation. These interventions are used to bring about
the intended changes or results.
Strategies or activities are the processes,
techniques, tools, events, technology and actions
of the planned program, used to bring about the
intended changes or outcomes.
Outputs: The direct products of activities; might include
types, levels and targets of services to be delivered by the
strategy, initiative or program.
outcomes: The specific changes in participants’ behavior, knowledge, skills, status and level of
functioning, or in policies, procedures and practices. Short-term outcomes should be
attainable within one to three years, while longer-term outcomes should be achievable within a
four- to six-year timeframe. These timeframes reflect the range of typical funding cycles.
Immediate Outcomes or Short-Term Outcomes: Immediate changes or benefits
expected – usually within one to two years – as a result of successful implementation
of the strategy.
Intermediate Outcomes, which also can be considered Short-Term Outcomes:
Changes or benefits, usually within one to two years of the immediate outcomes.
Long-Term Outcomes: Lasting changes with organizational, community or
systems-level benefits (e.g., improved social conditions, reduced rate of a particular
health outcome).
Context: The relevant demographic, economic, community, historical, cultural, political or other
social factors that influence the activities and outcomes of a strategy, initiative or program.
Contextual factors can include the racial and ethnic makeup of a community, geographic
region, economic opportunity, the history of discrimination, language barriers, political
climate, access to health care, quality education or employment and impoverished conditions.
The context informs the types of activities planned. In place-based work, these contextual vari-
ables might serve as outcome variables as well (i.e., the efforts are intended to change these
variables as they exist in the targeted place).
The components could vary based on the theory (or the change process) of the strategy, initiative
or program. Each term is explained more in detail in the W.K. Kellogg Foundation Logic Model
Development Guide (http://www.smartgivers.org/uploads/logicmodelguidepdf ).
7.5.1
Alignment between Strategies or Activities and Outcomes
One of the most useful things about the logic model is that it provides a visual tool to ensure that
your strategies or activities and immediate, intermediate and long-term outcomes are aligned.
In other words, you can trace the connection between all these components. Sometimes, one
strategy or activity can lead to several immediate outcomes and one intermediate outcome. You
can have different combinations; what is important is the “logic” of their connections.
Let’s use a pervious example. You are implementing an initiative in Rainbow County to increase
the number and percent of working class African-American homeowners in the county. Your effort
includes services to educate families about budgeting, saving and improving their credit scores;
acquiring financing; and understanding what it means to be a homeowner. You also plan to work
with the county lending institutions to improve their lending policies, practices and services.
exhibit 7-2: alignment between activities and outcomes
Where there is alignment …
Activity Immediate Outcome Intermediate Outcome Long-Term Outcome
Design and conduct
10 courses for working
class African American
families
Increased knowledge
among families about
process and resources
for purchasing a home
Increased savings
Improved credit
scores
Increased loan
approval rate
Increase in percentage of
homeownership among
working class African-
American families in
the county
Where there is misalignment that requires adjustment …
Activity Immediate Outcome Intermediate Outcome Long-Term Outcome
Design and conduct
10 courses for working
class African American
families
Increased knowledge
among families about
process and resources
for purchasing a home
Residents purchase
homes
Decrease in poverty
This might
not be realistic to achieve;
rather, it could be more feasible to
observe residents engaged in steps
toward homeownership.
Homeownership
alone will not lead to
decrease in poverty.
7.6
from logic model to evaluation questions
Drafting evaluation questions most likely will require numerous meetings with the evaluator and
other key stakeholders, as appropriate. Evaluation questions will depend on the phase the
strategy, initiative or program is in, the particular local circumstances and the ultimate purpose of
the evaluation. For example, your effort could benefit from a process or formative evaluation, but
may not be ready for an outcome or summative evaluation at this time.
A logic model can help generate evaluation questions. You can see what phase the effort is in and
therefore, which questions to ask and when it is best to answer them. Also, the model helps clarify
which variables are critical to achieving the desired outcomes. The array of questions you might
want to answer about your effort could be vast. A logic model helps narrow the array in a systemat-
ic way by highlighting the connections between strategies or activities and outcomes, and between
strategies or activities and the assumptions underlying the program. You will be better able to
address questions such as:
How is the strategy, initiative or program supposed to work?
Where do the assumptions in the model hold and where do they break down?
Where are the gaps or unrealistic assumptions in the model?
Which pieces of the model seem to be yielding the strongest outcomes or relationships to one
another?
Which pieces of the model are not functioning in practice?
Are there key assumptions that have not been embedded in the effort that should be?
By organizing evaluation questions based on the logic model, you are better able to determine
which questions to target in an evaluation.
There is another consideration in developing evaluation questions. The questions also depend
on the type of evaluation you want to conduct – a performance evaluation, a process or formative
evaluation or an outcome or summative evaluation.
A performance evaluation is concerned with:
Ensuring accountability
Demonstrating that resources are used as intended and managed well
Monitoring and reporting on progress toward established goals
Providing early warning of problems to funder and management
A process or formative evaluation is concerned with:
Understanding if a strategy, initiative or program is being implemented as planned and
according to schedule
Assessing if the effort is producing the intended outputs
Identifying strengths and weaknesses of the effort
Informing mid-course adjustments
An outcome or summative evaluation is concerned with:
Investigating whether the strategy, initiative or program achieved the desired outcomes and
what made it effective or ineffective
Making mid-course adjustments to the effort
Assessing if the effort is sustainable and replicable
A useful resource for engaging stakeholders in developing evaluation questions is:
A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions, by H. Preskill
and N. Jones, published by the Robert Wood Johnson Foundation in 2009. Available at
http://www.rwjf.org/en/library/research/2009/12/a-practical-guide-for-engaging-
stakeholders-in-developing-evalua.html.
7.6.1
Two Different Ways to Formulate Evaluation Questions About Your Strategy, Initiative
or Program
Here is a method to help formulate evaluation questions. Look at the logic model and start with the
following five elements:
Who – Who was your strategy initiative, or program intended to benefit?
What – What was the effort intended to do? What was the context within which the effort took
place and how could it have affected its implementation and outcomes?
When – When did activities take place? When did the desired changes start to occur?
Why – Why is the effort important to your organization or community? Why might it be important
to people in other organizations or communities?
How – How is the effort intended to affect the desired changes or bring about the desired out-
comes?
Here are examples of evaluation questions to ask for the previous sample scenario:
Who – Who, among the African American families reached, successfully completed the courses?
Who, among the lending institutions in your county, were open to partnering with your
organization?
stakeholders-in-developing-evalua.html
stakeholders-in-developing-evalua.html
What – What components of the program were successfully implemented? What components
were most useful and least useful to families? What are the lessons learned from the program
that could help other nonprofit organizations with a similar goal?
When – When (at what stage in the partnership) did a lending institution begin to reassess its
policies, practices and services? When did families begin to take steps toward homeowner-
ship?
Why – Why did the lending institutions decide to participate in your program? Why did some
families complete or not complete all the courses?
How – How did the economic crisis affect the families’ ability to improve their financial situation
enough to purchase a home? Did the initiative contribute to the desired intermediate
outcomes (i.e., families took steps toward purchasing a house) and if “yes,” how?
Another method for formulating evaluation questions is to consider different aspects of your
strategy, initiative or program and generate questions about each of these aspects.
exhibit 7-3: examples of evaluation questions about different aspects of your effort
Aspect of Your Effort Example of Evaluation Question
Theory of Change and
Logic Model
Was the theory of change and logic model correct?
What aspects of the theory and logic model did not happen in practice
and why or why not?
Implementation Was the effort implemented as intended?
Why or why not?
Results and Outcomes To what extent did the effort lead to the anticipated results?
What was the change and to what extent did the effort contribute to the
change?
What difference did the effort make to the organization, participants
and community?
Aspect of Your Effort Example of Evaluation Question
Context What demographic, economic, community, historical, cultural, political
or other social factors could have influenced the effort’s implementation
and outcomes?
What organizational factors (e.g., staff capacity, leadership, resources)
might have affected the effort’s implementation and outcomes?
Learnings What worked and what did not?
What were unintended consequences or benefits?
Investment Was the effort cost effective?
Was there another alternative that could have represented a better
investment?
What Next Can the effort be scaled up?
Can the effort be replicated elsewhere?
Is the change self-sustaining or does it require continued intervention?
7.6.2
Typical Scenarios When Formulating Evaluation Questions
What happens when different stakeholders have different questions and your list of questions
becomes too long?
At this point you want to remind your stakeholders about the objective of the evaluation. You can
point to and use the logic model to focus their questions. You also can estimate the cost of the
data collection and analysis needed to answer the questions, and explain that your evaluation
funds are limited.
How many evaluation questions should there be?
Again, remind stakeholders that questions should focus on and align with the information you
want to obtain about the program. Therefore, try to avoid unrelated questions and questions you
cannot address due to time and budget constraints.
Typically, three to five evaluation questions are recommended.
7.6.3
Other Considerations When Formulating Evaluation Questions
Commitment letter — Check your funding or grant agreement, which could contain a list of
evaluation questions the funder would like the program or initiative to address. This practice is
typical of most foundations and public agencies when making grants. However, the evaluation
does not have to be limited to answering just these questions. You can add your own according to
your own learning needs.
National evaluation — Sometimes, you are part of a larger multi-site initiative where an evaluator
is hired by the funder and is responsible for evaluating the entire initiative. That evaluator also will
have a set of evaluation questions. You could ask the same questions about your program site
(versus the entire initiative) or you might add questions specific to your program. It’s best not to
have completely different questions. That way, you can be efficient and collect data for both the
national and your own evaluation.
7.7
from logic model to evaluation questions to measurement
framework
Logic modeling is the first and most important step in planning for your evaluation. With your logic
model and evaluation questions, you then can plan your evaluation. Using your logic model and
knowing the questions you want answered, you can develop a measurement framework, as an
evaluation planning tool.
Developing a measurement framework will allow you to determine how to assess progress toward
achieving outcomes and answer the evaluation questions.
With a measurement framework for your effort in hand, you get a clear picture of how to conduct
your evaluation. The measurement framework provides another opportunity for stakeholders to
further define outcomes. With it, you can consider what the outcome means in more concrete
terms.
7.7.1
Key Components of the Measurement Framework
Seven key components make up the measurement framework:
Outputs are direct products of activities and may include types, levels and targets of services
to be delivered by the strategy, initiative or program.
Outcomes are the immediate, intermediate and long-term changes or benefits you need to
document. These outcomes should be the same ones identified in the logic model.
Indicators are markers of progress toward the change you hope to make with your strategy,
initiative or program.
Measures of change are values – quantitative and qualitative – that can be used to assess
whether the progress was made.
Data collection methods are the strategies for collecting data. This could include quantitative
methods, such as conducting surveys or analyzing existing data, or qualitative methods, such
as conducting interviews or a document analysis.
Data sources are the locations from which (e.g., national database, program survey), or people
from whom, (e.g., program participants), you will obtain data.
Data collection frequency is how often you plan to collect data.
Indicators and measures of change are sometimes used interchangeably by various people,
including funders and evaluators. Don’t be confused. What is most important to know is
when there is progress toward the desired outcomes and how you will measure that progress.
The measure can be expressed in numerical form (e.g., percent change) or in words (e.g., first-time events,
such as the first time participants felt empowered to save for a house because they now understood the
process and had the skills to go through the steps).
7.7.2
How to Use a Measurement Framework
Once you have identified your outputs and immediate, intermediate and long-term outcomes,
you can list each output and outcome on the measurement framework in the first column. After
you have listed each one, you can make a clear plan for assessing progress toward that particular
output or outcome. This involves moving across the rows of the measurement framework from left
to right to identify indicators, measures of change, data collection methods, data sources and data
collection frequency for each outcome. Please note that as you complete the measurement
framework, some components could contain overlapping responses. For example, the data source
for two outcomes may be the same.
7.7.3
Considerations for Developing the Measurement Framework
Take contextual factors into account: Identify realistic indicators and measures of change. As
previously discussed, it is essential to consider positive and negative contextual conditions
that could affect the strategy, initiative or program’s success.
Involve key stakeholders: Involve individuals who are key stakeholders in the effort’s
implementation and evaluation. This will provide another opportunity to incorporate feedback
from individuals with diverse perspectives.
Understand your capacity: Be sure you have the needed capacity (human power, skills, time,
etc.) to collect and analyze data to measure the outcomes identified. If necessary,
consider how you can acquire additional resources, personnel or training to do so.
Create a living document: Like your logic model, your measurement framework is a living
document. It is a tool for planning, but should be regularly modified it based on changes in
your goals, activities, organization’s capacity or information gained from the data you are
collecting. When you adjust the logic model and evaluation questions, you also should adjust
the measurement framework.
7.7.4
Taking a SMART Approach to Measurement Framework Development
Let’s return to the previous example of the effort in Rainbow County to increase the number and
percent of working class African American homeowners in the county. Based on your experience
and findings from past research studies, you believe that instruction about what it takes to
purchase a home and about the services offered by financial institutions are not accessible to
working class African American families. To remedy those circumstances, your effort includes
services to educate families about budgeting, savings and improving their credit scores; acquiring
financing; and understanding what it means to be a homeowner. You also plan to work with the
county institutions to improve their lending policies, practices and services. Your theory of change
is that by working with the families and lending institutions, more residents will be able to
purchase homes. Hence, homeownership supports asset building and family economic security.
Specific outputs and outcomes clearly state the issue of focus, target group and
timeframe. For example, in the above scenario, the outputs are the number of
courses per week or per month, the length of the courses, the content taught
and the average number of attendees. The outcomes specifically focus on the
individuals who are enrolled in the program.
Measurable outputs and outcomes are ones where you can clearly assess the
change that has occurred among the people affected by the effort. For instance,
in the Rainbow County example, participants’ understanding of steps in the
homeownership preparation process can be assessed using pre- and post-tests.
Achievable outputs and outcomes take into account the scale and scope of
outcomes that can be achieved based on time and resources available. For
instance, you can expect the outcomes are achievable because the program has
the resources to provide X number of courses over Y number of months for
families to learn the steps in the homeownership process.
Relevant outputs and outcomes (immediate, intermediate and long-term) work
toward your desired change in an incremental manner. For the effort, this
involves small steps to increasing the number of homeowners in Rainbow
County.
Time-specific outputs and outcomes mean the expected timeframe for changes
should be clear and realistic. In the Rainbow County example, the
program is implemented over the course of two years, taking into consideration
the amount of time potential homeowners might need to learn about and
engage in the homeownership process, and the amount of time to engage
lending institutions. Additionally, the measurement framework provides a tool
for laying out how impacts unfold and evolve. You also could find it helpful to
conduct research to determine how long to implement programs before you can
to observe and measure changes.
exhibit 7-4: sample measurement framework tool
Output/
Outcome Indicator
Measures of
Change
Data Collection
Methods Data Sources
Frequency of
Data Collection
Specific outputs
or changes
(should be same
information in
your logic model)
Markers toward
progress
Value for
assessing
progress
How data will be
collected
Where data will
be obtained from
How often data
will be collected
Output
Number of
participants
who completed
courses
Completion of
courses
Number of
participants
who completed
X percentage of
courses
Average
number of
participants per
course
Program staff
records and
attendance logs
Program staff
records and
attendance logs
Monthly (or
according to
the course
schedule)
Immediate Outcome
Improved
financial
literacy among
families
Use of banking
services
A family budget
First time
families opened
a bank account
and created a
family budget
Interviews Course
participants
Twice
throughout the
program
Intermediate Outcome
Increased
savings
More money in
the bank
Percentage of
change in funds
in the bank
Adherence to
family budget
Interviews
Log to track
monthly
savings
Course
graduates
Every six
months after
family
graduates from
program
Output/
Outcome Indicator
Measures of
Change
Data Collection
Methods Data Sources
Frequency of
Data Collection
Long-Term Outcome
Increase the
number of
working class
African-
American home
owners in
Rainbow County
Percent
homeowner-
ship among
African-
American
families
Homeowner-
ship rates,
disaggregated
by race
Collect home
ownership data
from the county
County records Every two years
A useful resource on developing measurement frameworks is Building a Common Outcome
Framework to Measure Nonprofit Performance, published by The Urban Institute in 2006. Available at
http://www.urban.org/uploadedpdf/411404_nonprofit_performance .
7.8
your evaluation plan
Using the logic model, the evaluation questions, and the measurement framework tool as the basic
components, you can develop an evaluation plan that pulls all of these and more together. A good
evaluation plan should have the following elements:
Background Information about the strategy, initiative or program: Purpose of the evaluation,
origins of the effort, goals, theory of change and logic model.
Evaluation Questions: Specific questions that are measurable. These might need to be
prioritized in order to focus resources and keep evaluation manageable.
Evaluation Design: Data collection methods; types of data that will be collected; sampling
procedures; analysis approach; steps taken to ensure accuracy, validity and reliability; and
limitations. The measurement framework should be incorporated into this section.
Timeline: Completion dates and time ranges for key steps and deliverables.
Plan for communicating findings and using results to inform work: Details regarding what
products will be developed and what will be included in each product.
Budgetary Information (see Section 7.9 for more detailed information about how to develop an
evaluation budget): Could include expenses for staff time, consultants’ time, travel,
communications, supplies and other costs (e.g., incentives for participants, translation and
interpretation time).
Evaluator/evaluation team: Specify who is responsible for conducting the evaluation process
and what this role entails. This might be an external evaluator, internal evaluator or internal
evaluator with an external consultant.
Parts of your evaluation plan can be copied and used to write your evaluation report.
exhibit 7-5: side-by-side comparison of an evaluation plan and an evaluation report
Evaluation Plan Evaluation Report
Cover Page: Includes clear title; name and location
of the strategy, initiative or program; period to be
covered by evaluation.
Cover Page: Includes clear title; name and location
of the strategy, initiative or program; period covered
by evaluation or date evaluation was completed.
Background Information of the Effort: Purpose of
the evaluation, origins and goals of effort, activities
and services, including theory of change and logic
model.
Executive Summary: Brief, stand-alone
description of program, outline of evaluation
purpose and goals, methods, summary of findings
and recommendations.
Evaluation Questions: Specific questions that are
measurable, might need to be prioritized to focus
resources and keep evaluation manageable.
Introduction and Background: Purpose of the
evaluation, origins and goals of effort, target
population, activities and services, review of related
research, evaluation questions and overview and
description of report.
Evaluation Design: Data collection methods; types
of data to be collected; sampling procedures;
analysis approach; steps taken to ensure accuracy,
validity and reliability; and limitations.
Evaluation Design: Data collection methods, types
of data collected, sampling procedures; analysis
approach; steps taken to ensure accuracy, validity,
and reliability; and limitations (include the theory of
change and logic model).
Timeline:Completion dates or time ranges for key
steps and deliverables.
Evaluation Results: Evaluation findings, evaluation
questions addressed, visual representations of
results (e.g., charts, graphs, etc.).
Plan for Communicating Findings and Using Results
to Inform Work: Details regarding what products
will be developed and what will be included in each
product. Potential users and intended audience
include program staff and administration, program
participants, community leaders, funders, public
officials and partner organizations.
Summary, Conclusion and Recommendations:
Summary of results, implications of findings,
factors that could have shaped results, clear and
actionable recommendations.
Budgetary Information: Might include expenses
for staff time, consultants’ time, travel,
communications, supplies and other administrative
costs. (If the plan is to be shared with a wider group
of people, information about staff and consultants’
time may be excluded.)
Evaluator and Evaluation Team: Specifications
regarding who is responsible for conducting the
evaluation process and what this role entails. This
may be an external evaluator, internal evaluator or
internal evaluator with an external consultant.
Various, surprising things can affect how long it takes to implement and complete an evaluation.
When developing the timeline of the evaluation, you should consider the following:
Factors that could affect respondent availability (school year cycles, summer months, etc.)
Factors that might affect data collection (data collector trainings, translation time of instru-
ments, Institutional Review Board (IRB) approval, data availability, etc.)
Upcoming opportunities to share findings (e.g., board meetings, annual meetings or funding
cycles, etc.)
Institutional Review Boards (IRBs) are entities set up to protect the rights and welfare
of people who participate in research. Evaluations of programs involving Native
Americans/Alaska Natives also require permission from their tribal governments.
Data collection that puts participants at any type of risk requires IRB review and approval.
Universities usually have their own IRBs and faculty or students must submit an application to the
IRB for approval before starting research. American Indian/Alaska Native communities usually have
their own IRBs and procedures to protect their members from being harmed by research. There also
are independent review boards if you (or your evaluator) are not affiliated with a university or
working with a tribal nation. Plan for this step and the amount of time it could take to get
approval, which can affect the schedule for data collection and analysis. This topic is addressed in
more detail in Chapter 8.
7.9
budgeting for evaluation
How much will the evaluation of your strategy, initiative or program cost? How much should you
budget for the evaluation when you submit your grant application? The cost of an evaluation
can vary widely, depending on the type of evaluation you want to conduct (e.g., performance
monitoring, process or formative evaluation, outcome or summative evaluation), the approach you
want to take (e.g., empowerment evaluation, developmental evaluation) and the most appropriate
evaluation methodology for your strategy, initiative or program (e.g., case study, quasi-
experimental design). Chapter 3 provides a more detailed explanation of what the different types,
approaches and methodologies mean for your organization.
A general guideline is to allocate 5 to 10 percent of your total program cost for performance
monitoring and/or a process or formative evaluation. That is, if your strategy, initiative or program
requires $100,000 to operate, you can estimate evaluation to cost between $5,000 and $10,000.
Consequently, your overall estimated budget for program and evaluation would range from
$105,000 to $110,000. If you plan to conduct an outcome or summative evaluation, consider
allocating 15 to 20 percent of your total program cost to evaluation.
Useful resources on budgeting for evaluation include:
The Evaluation Part of a Proposal Budget. Available at http://www.opportunitystudies.org/
repository/File/evaluation/evaluation_budgeting .
Evaluation on a Shoestring Budget, by Wilder Research, published in 2009. Available at
https://www.wilder.org/Wilder-Research/Publications/Studies/Program%20Evaluation
%20and%20Research%20Tips/Evaluation%20on%20a%20Shoestring%20Budget%20-%20
Tips%20From%20the%20Wilder%20Research%20Program%20Evaluation%20Series,%20
Fact%20Sheet .
7.9.1
Types of Expenses Typically Associated with Evaluation
If you hire an independent evaluator, you want to ensure these expenses, where applicable, are
covered in the budget the evaluator submits. This list can help guide your allocation of resources
and review of budgets from evaluators.
Consider labor costs for external evaluator, internal evaluator and other staff and consultants.
External evaluator (hourly or daily rate multiplied by the number of hours or days required
for the work)
Internal evaluator or staff person responsible for the evaluation (hourly rate multiplied by
the number of hours required for the work)
Staff person or other consultants who will assist with tasks such as data collection, facilita-
tion of organizational learning or transformation of findings into products to share with the
public (hourly rate multiplied by the number of hours required for the work)
repository/File/evaluation/evaluation_budgeting
repository/File/evaluation/evaluation_budgeting
%20and%20Research%20Tips/Evaluation%20on%20a%20Shoestring%20Budget%20-%20Tips%20From%20the%20Wilder%20Research%20Program%20Evaluation%20Series,%20Fact%20Sheet
%20and%20Research%20Tips/Evaluation%20on%20a%20Shoestring%20Budget%20-%20Tips%20From%20the%20Wilder%20Research%20Program%20Evaluation%20Series,%20Fact%20Sheet
%20and%20Research%20Tips/Evaluation%20on%20a%20Shoestring%20Budget%20-%20Tips%20From%20the%20Wilder%20Research%20Program%20Evaluation%20Series,%20Fact%20Sheet
%20and%20Research%20Tips/Evaluation%20on%20a%20Shoestring%20Budget%20-%20Tips%20From%20the%20Wilder%20Research%20Program%20Evaluation%20Series,%20Fact%20Sheet
Compensate people from whom you collect data.
Appreciation for their time in the form of cash or gift
cards (make sure it is for supplies they need and from a
store they can easily access)
Travel expenses for bus or cab fare
Childcare expenses, either included in the cash incentive
or value of the gift card or as payment for someone to
look after the participants’ children during meetings and
focus groups
Translation or interpreting fees if you need to hire translators
or interpreters to reach populations whose primary language
is not English. You might need interpreters to be present
during interviews, focus groups or other types of data
collection activities.
Most translators charge by the word or page or a flat fee.
The translation fees will vary depending on the
complexity of the subject matter, number of words to
be translated into another language, amount of time to
complete the
assignment and language combination.
Most interpreters charge by the hour or day. The fees will
depend on the geographical location, the nature of the
assignment and interpreter’s qualifications.
Supplies will be needed
Duplication of paper-and-pencil survey questionnaires,
consent forms and any other materials needed for
data collection
Refreshments and food at data collection venues
Postage if the questionnaires are distributed by mail and
returned in self-addressed stamped envelopes
It’s important to
appropriately
compensate, based
on a living wage, for the
respondents’ time. There might
be some implications for people
who receive public benefits, so
be sure to look into this and, if
necessary, find alternative and
appropriate ways to compensate
them for their time and effort.
A useful resource on
compensation is
Compensation for Research
Subjects, published by the
Committee for Protection
of Human Subjects at the
University of California,
Berkeley. Available at
http://cphs.berkeley.edu/
compensation . Your IRB
should also have guidelines
that you can follow.
A useful resource on
translation fees can be
found at http://www.
affordablelanguageservices.
com/translation-costs.
Some state government
websites also contain
information about
interpreting fees.
Institutional Review Board (IRB) fees could be an expense.
If the evaluator or somebody in your organization is affiliated with a university, the fees may
be waived
If not, you can use an independent, private IRB that charges a fee for its services and
typical charges range from about $500 to more than $1,000
7.9.2
Time Estimates for Specific Evaluation Activities
Proposed budgets, including evaluation budgets, are based on a set of assumptions. If you are
developing an evaluation budget, keep track of your assumptions. If you are working with an
external evaluator, it might be helpful to ask the evaluator to clarify his or her assumptions. In
addition to time cost and the expense of designing instruments, preparing for data collection and/
or analyzing the data, typical data collection activities include:
Administration of paper-and-pencil surveys.
Depending on the length of the survey questionnaire and the amount of time required to
help the respondent complete the questionnaire, it could take between 15 and 30 minutes,
on average, to administer each questionnaire. It imight take longer if the questionnaire is
complex.
For example, if you administer the survey at a community event where you expect to get 30
people to complete the questionnaire, the total time required for the survey can range from
approximately 7.5 hours (15 minutes x 30 people) to 15 hours (30 minutes x 30 people). If
you engage two administrators, the total number of hours for the survey won’t change, but
the time spent by each administrator will be shorter.
Administration of online surveys.
The cost of administering an online survey can be divided into three parts:
i. Compiling the list of email addresses to send the questionnaire, which might take
30 minutes or more, depending on the total number of recipients.
ii. Identifying the nonrespondents and sending them reminders, which could take
30 minutes or more, depending on the number of nonrespondents.
iii. Identifying the people who remained unresponsive even after sending them email
reminders and contacting them by telephone to encourage them to respond; this could
take five to 10 minutes per call, on average.
For example, if you administer an online survey to 30 people, the activity will require
approximately 2.5 hours (30 minutes to compile their email addresses, 15 minutes to send
one reminder and 10 minutes to contact 10 people, assuming that one-third of the sample
will remain unresponsive).
Implementation of focus groups.
Scheduling a focus group could take two to four hours, depending on the number of
schedules to coordinate, and it might take one hour to secure a location for the focus
group.
A focus group usually takes one to two hours, on average, not including set up and
take down.
There should be a facilitator and a note-taker at the focus group, which typically translates
into two to four hours for staff time per focus group.
Some evaluators also might want to record the focus group discussion and have a formal
transcription of the discussion, which could increase the cost of the focus group.
Conducting interviews.
Scheduling an in-person or telephone interview usually takes 15 minutes on average per
subject.
An interview typically takes between 30 minutes and one hour.
There should be an interviewer and a note-taker, which typically averages one to two hours
of staff time per interview.
Some evaluators also might want to record the interview and have a formal transcription of
the interview, which could increase the cost.
HIGHLIGHTS
A logic model is an important living
document for your strategy, initiative or
program. However, there should be equal, if
not more, emphasis on the process of logic
modeling.
Engaging stakeholders in the logic modeling
process strengthens your strategy, initiative
or program.
The logic modeling process will help surface
the various assumptions stakeholders might
hold, promote shared understanding,
resolve disagreements and generate new
and innovative ideas.
Good facilitation is key to a successful logic
modeling process.
A logic model can help generate evaluation
questions. You can see what phase the
program is in and therefore, which
questions to ask and when it is best to
answer them.
To formulate evaluation questions, look at
your logic model and ask:
Who your effort benefits
What the effort is intended to do
When activities occur
Why the effort is important
How the effort will bring about the
desired change
A measurement framework can be
generated from the logic model. A
measurement framework enables you to
determine how to assess your progress
toward achieving outcomes. The
measurement framework includes seven
main components: outputs; outcomes;
indicators; measures of change; data
collection methods; data sources; and data
collection frequency.
Using the logic model, evaluation questions
and the measurement framework as basic
components, you can develop an evaluation
plan, which pulls this information – and
more – together.
The cost of an evaluation can vary
widely, depending on the type evaluation
you conduct, the approach you take and the
most appropriate evaluation methodology
for your strategy, initiative or program.
A proposed evaluation budget is based
on a set of assumptions. It is important to
document the assumptions or clarify the
assumptions in your budget or with your
external evaluator.
exercises
1. Processes, tools, events, technology and actions that are parts of a program are examples of
.
A. Outputs
B. Resources/inputs
C. Program impact
D. Short-term outcomes
E. Activities
2. A logic model can be used to .
A. Design your program
B. Inform implementation
C. Develop evaluation questions
D. Create an outline for your evaluation report
E. All the above
3. Outcomes are .
A. Attendance rates for a parenting program
B. Total number of sessions conducted for a leadership program
C. Changes in youths’ attitudes and behaviors related to eating healthy
D. All of the above
E. None of the above
4. Long-term outcomes are typically achieved in a year or less.
A. True B. False
Answers: 1E; 2E; 3C; 4B
5. Consider your program or initiative. Are the strategies or activities aligned with the outcomes
you are trying to achieve? Is there a logical connection between them? If not, what needs to be
adjusted?
6. Consider a measurement framework that you may have developed for a program or initiative.
After reading Chapter 7, is there anything you would change in the measurement framework?
7. Consider an evaluation that you conducted in the past. Were there any expenses you wished
you had budgeted for? If you were to develop a budget for a similar evaluation today, what
labor and other expenses would you include in the budget?
8 Data Collection and Analysis
an evaluation
B
DETERMINE
stakeholders
and how and
when to
engage them
C
IDENTIFY
assumptions and
determine what
will be di�erent
(theory of change
and logic model)
D DEVELOP evaluation plan (logic
model, measurement,
framework, etc.)E COLLECT and analyze data
F
COMMUNICATE
results and
understand
what happened
(interpret
�ndings and
facilitate
learning)
G
MAKE informed decisions
(improve actions and
next steps)
Stakeholders
introduction
You might have wondered why some evaluations
use surveys
while others use focus groups. Is one better than
the other? Not always. The data collection method
depends on what you want to know, how much
money and time you have and what the people
involved are comfortable with. This is your part
in the decision-making process. You should be
aware, nevertheless, of the tension among
funders, policymakers and evaluators regarding
quantitative and qualitative methods and data.
Some people prefer numbers over narratives and
might discount one form of data over the other.
You should not let their preference sway you from
what is best for your effort and evaluation;
nevertheless, you should be prepared to justify
your choice of method or methods. Systematic data collection and analysis is
essential to evaluative thinking and brings rigor to the story you can tell about your program, strategy,
policy or organization.
Evaluative thinking is a systematic process of collecting and analyzing data to tell the
story about your strategy, initiative, program, policy or organization. It is based on the
belief that a systematic process is valuable and necessary, which involves identifying
assumptions about what you think works and doesn’t work and why; posing thoughtful
questions about what you expect to see change during and after implementing your
effort; pursuing deeper understanding through reflection and dialogue; communicat-
ing what was learned without underestimation or exaggeration; and making informed
decisions in preparation for action.
Important considerations to include when designing your data collection or reviewing your evaluator’s
design:
Who will collect the data – your program staff or the evaluator? If the former, do they have the neces-
sary skills? If the latter, are funds available to hire the person?
How will you reach certain populations who are uncomfortable with, and distrustful of, people who
want information from them?
What knowledge, skills and other resources (e.g., staff capacity,
money and technology) are needed and available for data collec-
tion?
How much time will it take and what will be the final result?
What does it mean to maintain confidentiality and anonymity for
your participants?
What is an Institutional Review Board (IRB), do you need such a
review and how do you get it?
8.1
determine data collection methods
8.1.1
What You Need to Ask Before Collecting Any Data
There are many ways to collect data, some more costly and
time-consuming than others. Therefore, before you start
collecting data, it is imperative to consider the following
questions with stakeholders such as your organization’s
leadership, key staff, board members or an advisory committee.
How this chapter is organized…
8.1 Determine Data
Collection Methods
8.1.1 What You Need to Ask
Before Collecting Any
Data
8.1.2 Use of Quantitative
Methods
8.1.3 Use of Qualitative
Methods
8.1.4 Use of Mixed Methods
8.2 Critical Considerations
in Data Collection
8.2.1 Who the Data Collector Is
8.2.2 Howto Engage
Populations Who Have
Been Traditionally
Excluded or Treated
as Invisible
8.2.3 Resources
8.2.4 Confidentiality and
Anonymity
8.2.5 Obtaining Consent
8.2.6 Institutional Review
Board (IRB)
8.3. Analyzing and
Interpreting Data
8.3.1 Quantitative Data
Analysis and
Interpretation of Results
8.3.2 Qualitative Data
Analysis and
Interpretation of Results
Highlights
Exercises
exhibit 8-1: essential questions to ask before collecting data
Evaluation Purpose
What are the evaluation questions? Which methods will help answer them and provide the most
reliable and valid data?
Who is the primary audience for the findings? What types of data will make the most sense and be most
useful to them?
Data Sources
Who is providing the data? Who are the potential respondents? How many respondents are needed?
Timing
How long is the strategy, initiative or program?
How much time was budgeted for data collection and analysis?
When is the right time to collect the data?
When will the results become available for reporting to the funder, board members, community leaders
and other stakeholders?
Resources
What is the evaluation budget? How much of it is for data collection and analysis versus reporting and
dissemination of findings?
Who should collect the data – staff or someone else? Does staff have the time and skills? Should an
external evaluator be hired?
Source: The table was adapted the Evaluation Toolkit, published by the Pell Institute and Pathways to College Network.
Available at http://toolkit.pellinstitute.org/evaluation-guide/collect-data/
A resource on data collection methods is the User-Friendly Handbook for Project Evaluation,
published by the National Science Foundation in 2002. Available at http://www.nsf.gov/
pubs/2002/nsf02057/nsf02057_5 .
8.1.2
Use of Quantitative Methods
Determining if you should use quantitative methods.
Depending on the answers to the questions raised in Exhibit 8-1, quantitative methods such as
surveys or assessments could be right for you. Or qualitative methods such as interviews or focus
groups might suit your needs better.
Types of questions that quantitative methods can help answer
At the most basic level, quantitative methods are concerned with what, who and when. Therefore,
you should consider quantitative methods if your evaluation questions include inquiries about
who participated and benefited from your program; what they learned or got from your program;
what changes were brought about by your strategy, initiative or program; and when the changes
occurred. Also, if you need to generalize (apply to other settings) the findings or make predictions
about the results of your organization’s work, quantitative methods are better.
Examples of quantitative data include:
Number of people who attended program activities over the course of the year – participation
rate.
Whether or not program participants developed new knowledge and skills – percent change in
knowledge and skills before and after participation.
Number of students in your school district who graduated in any given year, including those
who enrolled in ninth grade for the first time, plus number of transfer students who joined the
class, minus the students who left – high school graduation rate.
Number of deaths of infants younger than 1 year old per 1,000 live births in your county – infant
mortality rate.
Primary audience for the findings
Quantitative methods generate data that appeal to people who prefer information that quantifies
impact and provides the “bottom line.”
Potential respondents and sample size
Quantitative methods can be helpful because it’s relatively less expensive to administer a survey
than to conduct interviews or focus groups with a lot of people (think about the time it takes to
send a survey questionnaire to 50 people, compared with talking to 50 people for 30 to 45 minutes
each). Simultaneously, you should consider whether potential respondents will be uncomfortable
with surveys and questions that ask them to rate something on a scale of 1 to 10. This is usually
true for people with low literacy, very limited English proficiency and distrust of surveys because
they don’t know where the data go and how the data are used. If the potential respondents are
uncomfortable, you might not get a good response rate or high quality data. In this case, conduct-
ing interviews or focus groups with fewer respondents could be preferable because it would yield
better quality data.
Amount of time for data collection and analysis
Quantitative methods are useful if the amount of time to collect and analyze data is very limited.
A survey with close-ended questions (i.e., you provide response options that respondents can
select) that ask people to rate something takes less time to administer than scheduling and
conducting interviews and focus groups. Running calculations of frequencies, averages or
percentages also will take less time than reviewing, coding and analyzing qualitative data (i.e.,
notes from discussions and interviews). Quantitative methods are equally useful if the program
extends over two or more years or if the change is expected to occur over the course of several
years (e.g., changes in high school graduation rate), because you can compare baseline data with
subsequent data.
Closed-ended questions provide discrete, multiple-choice answers that respondents
can select.
Budget and other resources
In general, under comparable conditions (i.e., amount of time for data collection, sample size),
quantitative methods can be less expensive than qualitative or mixed methods (which use a
combination of quantitative and qualitative methods) for several reasons. Secondary data such
as number of emergency room visits or crimes committed are usually available at minimal or no
charge. Surveys can be administered online or by mail, which costs less than the typical resources
needed to travel to an interview or focus group. An analyst needs less time to calculate
frequencies and percentages than to read and code text from interview and focus group
transcripts. Also, a different set of skills is required to do these calculations than to code text and
generate themes.
exhibit 8-2: advantages and disadvantages of quantitative methods
Advantages Disadvantages
You can collect data from a large sample of people. Sensitive information on topics such domestic
violence, drug use, racism, immigration status and
other personal matters are difficult to obtain.
You can analyze the data relatively quickly and
easily, especially if you are using software packages
such as Excel, STATA, SPSS, etc. .
They generally do not explain the reasons for
responses.
It does not require a lot of money if a survey is used
and administered online or by mail.
They may not be comfortable or familiar for certain
groups of people who distrust mail or telephone
surveys, have low literacy, or come from oral
societies where written words are not part of
their traditions.
The results can be generalized if the sample is
representative of the study population. (There are
specific statistical methods to calculate the rep-
resentativeness of a sample; ask someone with
expertise in sampling.)
There are two major ways for collecting quantitative data: (a) surveys and (b) tests and
assessments.
Surveys
Surveys are one of the most popular ways to collect quantitative data. In a survey, a questionnaire
is distributed to a group of people to complete. While such questionnaires could include
open-ended questions, closed-ended questions are typically used to collect quantitative data.
Statistical analysis can be applied easily to responses to closed-ended questions.
Make the questions short and clear, ideally no more than 20 words. Be sure to give the
respondents all the information they will need to answer the questions (e.g., select the best
answer, select all the options that apply).
Start with questions that are not personal and fairly easy to answer.
Avoid questions that have more than one central idea or theme (sometimes referred to as
“double-barreled” questions). Example of a double-barreled question:
Did the voter education campaign increase your knowledge about how to register to
vote and to learn about the candidates running for governor?
This question contains two ideas; the voter education campaign could have affected the
person’s knowledge about each one differently.
Do not use jargon or vocabulary that your respondents might not know.
Avoid inexact words that are open to interpretation (e.g., “generally,” “usually,”
“average,” “typically,” “often” and “rarely”).
Make sure the answer to one question relates smoothly to the next. For example, if
necessary add, “If yes, did you … ?” or “If no, did you … ?”
Give a time period if asking respondents to think about something they did before (e.g.,
during the last six months, how often did you … ?).
Think carefully about terms or concepts that could be interpreted differently in other
cultures. If needed, consult someone who shares the same culture as your potential
respondents (e.g., family in some cultures refers to both nuclear and extended family
members; “house” and “home” mean the same thing in some cultures and languages while
you, in fact, might be interested in the structure of a building – “house” – versus the feel-
ing of belonging – “home”).
checklist
Simple rules to follow when developing a survey:
exhibit 8-3: when to use various survey methods
Survey Method When to Use
Mail surveys You have a complete and accurate mailing list.
The people you want to survey have some interest in the survey topic and
are likely to respond.
The people you want to survey prefer this method (e.g., less tech-savvy
individuals).
You want to give respondents time to consider their answers or other
information when completing the survey.
Telephone surveys You need results relatively quickly.
The people you want to survey have telephones.
The people you want to survey have difficulty using written or online
surveys (e.g., respondents with low literacy rates, poor eyesight, or no
access to computers).
The survey is relatively short to discuss by telephone.
Internet or web-based
surveys
You need results relatively quickly.
The people you want to survey are regular Internet users.
Your survey is short and simple.
Your survey is more complex with skip patterns (e.g., responses to a
question determine which questions to be answered later, etc.). This is
easier to do when the questionnaire is in electronic format than in
hardcopy format.
Handout surveys (the
questionnaire is handed
out for people to complete
in paper form or on a tablet
[e.g., iPad])
You want to capitalize on who is available (e.g., people at a conference,
community festival, etc.).
The people you want to survey might not be available or accessible to
you again.
Survey Method When to Use
Face-to-face surveys (you
go over the questionnaire
in person)
Your survey questions are too complex and may need in-person
explanation.
There is concern that people would not respond willingly unless
someone they trust is present to reassure them about the content of
the questions.
The people you want to survey are unreachable by phone, email or
Internet.
Budget is not an issue and you have trained interviewers who can
administer the survey consistently and properly.
Source: E. Taylor-Powell & C. Herman (2000). Collecting Evaluation Data: Surveys. Available:
http://learningstore.uwex.edu/assets/pdfs/g3658-10
Tests and assessments
Tests and assessments can be useful tools in evaluation that can generate information about
changes in health status or behavior that your organization, strategy or program helped to make.
However, developing the right tests and assessments requires expertise and specialized training
to properly design, administer and analyze. Some tests and assessments may already exist, and
the data are available for use, (e.g., SAT scores, blood sugar levels and community walkability
indexes).
exhibit 8-4: different types of tests and assessments
Physiological health status tests
Can be used to reveal the health needs of a certain population or indicate the extent of a particular
health problem in a target population or community. Examples of these tests are broad-based
screenings — such as cholesterol, blood sugar or blood-pressure readings — and birthweights
of babies.
Physiological tests also can be used to measure your strategy, initiative or program’s outcomes. For
instance, if you are operating a prenatal care program, you can compare the birthweights of infants
born to mothers in your effort with mothers who are not in a prenatal care program. Statistical tests for
significance can be applied to this kind of data to further confirm the effects of your effort.
Knowledge or achievement tests
Can be used to measure changes in participants’ knowledge. Through testing before and after the
program, you can assess what the participants need to learn, and then measure what they have actually
learned. Be aware, however, that a person’s knowledge does not mean the person is using that knowl-
edge in everyday life.
Another type of knowledge or achievement testing is done through observation, as when a staff
member observes a mother interacting with her child during a home visit. The observation includes a
set of behaviors the staff member must be trained to look for and record.
Other types of assessments and inventories
Can be used to measure need and assess outcomes. You may develop your own instruments to
determine, for example, if your clients are satisfied with the services you offer. Standardized
questionnaires developed by health researchers on such topics as patient satisfaction, general health
(including items on physical, emotional and social function), mental health, depression and disability
status also can be used. Such standardized tests also exist for studying children’s experiences with
schools, qualifications of teachers and other education-related matters.
There are advantages to using a questionnaire that has already been developed and field-tested. For
example, you don’t have to spend time and money creating your own. However, keep in mind that
standardized assessments might not adequately reflect the important and unique aspects of your
initiative program or the situation or culture of the people you serve. If you are using an existing
assessment, find out where it was tested before and if it is culturally appropriate for the group of people
you hope to survey.
Useful resources on analyzing quantitative data include:
Analyzing Quantitative Data, by E. Taylor-Powell, published by the University of Wisconsin
Cooperative Extension. Available at http://learningstore.uwex.edu/assets/pdfs/G3658-6.
pdf.
Analyzing and Interpreting Data, published by Wilder Research. Available at http://www.
evaluatod.org/assets/resources/evaluation-guides/analyzing-interpretingdata-8-09 .
8.1.3
Use of Qualitative Methods
Determining if you should use qualitative methods.
Depending on the answers to the questions raised in subsection 8.1.1, qualitative methods could
be right for you.
Types of questions that qualitative methods can help answer
At the most basic level, qualitative methods are concerned with why and how and are useful for
in-depth study of a particular issue rather than a broad study. Therefore, if your evaluation
questions include inquiries about how the participants in your strategy, initiative or program
applied what they learned in their daily life, or why the community leaders had difficulty mobilizing
the residents to take action on an issue, you should consider qualitative methods. Qualitative data
also provide contextual information about your organization, effort and community, which could
mean more to the program director who must make recommendations for improvement or to the
policymaker who must revise an existing policy.
Examples of qualitative data include:
Factors that motivated participants to attend (or not) activities during the year
Ways in which program participants applied their new knowledge and skills
Reasons for the decrease in high school dropout rate over the past two years
Reasons for the increase in number of deaths in your county over the past five years among
infants younger than one year old
Primary audience for the findings
Qualitative methods generate data that appeal to audiences who prefer information in the form
of case studies, stories and rich descriptions, and who are curious to know what lies behind data
trends and statistics.
Potential respondents and sample size
Qualitative methods are helpful if you are working with a smaller number of people, mainly
because conducting interviews and focus groups with a lot of people can be expensive and time
consuming. Qualitative methods also help if the people
you want to collect information from feel more comfortable
expressing their opinions verbally than in written form.
Finally, qualitative methods are useful when the information
to be collected contains a lot of nuances due to cultural and
language differences.
Program length and amount of time for data collection
Qualitative methods can be useful regardless of program
length (e.g., a summer youth program, a six-month training
institute, or a year-long prevention program). The evaluation
plan should allow enough time to analyze the data because
coding text and identifying the themes that emerge from the
coding is more time-consuming than calculating
frequencies, averages or percentages.
Budget and other resources
In general, under comparable conditions (i.e., length of
program, amount of time for data collection, sample size),
qualitative methods can be more expensive than
quantitative or mixed methods for several reasons. There
may be travel costs for the data collector and for the
respondents, childcare expenses for parents who
participate in a focus group, or interpreter fees for
people who don’t speak English. An analyst also needs
more time to read and code text from interview and focus
group transcripts and observation notes. Finally, a different
set of skills is required to code text and generate themes
than to calculate frequencies and percentages.
exhibit 8-5: advantages and disadvantages of qualitative methods
Advantages Disadvantages
Provides understanding and description of in-depth
experiences by individuals in your strategy,
initiative or program.
Not useful if you want to generalize findings to the
whole study population or community (i.e., findings
may be relevant only to the people your
organization or program serves).
Provides you or the evaluator with an opportunity to
explain definitions or questions that are unclear to
participants.
Participants may not feel comfortable verbalizing
and discussing sensitive topics.
You or the evaluator can easily guide and redirect
questions in real time.
Collecting and analyzing data can be expensive and
time consuming.
Findings may be easier to interpret for some of your
stakeholders who are uncomfortable with numbers
and other forms of quantitative data.
field-tested survey questionnaires or assessment
tools exist for the topic you want to explore.
Different types of qualitative data collection methods
There are three major ways for collecting qualitative data: interviews, focus groups and
observations. Each method has pros and cons, depending on whom you want to collect data from
and other factors.
Interviews
There are two types of interviews: structured and semi-structured. In structured interviews, the
questions are written out exactly the way they should be asked, and the interviewer should ask
every respondent the questions in the same order.
In a semi-structured interview, topics are listed and examples of probes are provided, and the
interview becomes more of a discussion. The interviewer can tailor the questions – to a certain
degree – to the respondent’s role in the program and cultural and language background.
Interviews also can be conducted by telephone or in person. The former can save time and money
(e.g., no travel to the interview site). The latter is helpful if respondents’ body language is
important to observe and if you expect respondents to be more engaged by encountering the inter-
viewer face-to-face.
Be very familiar with the interview questions. Read them several times before the
interview to ensure comfort with the questions and their flow.
Be prepared to rephrase questions or ask them out of order if you are conducting a
semi-structured interview. Respondents may need questions reworded to understand
their meaning. They also could answer a question listed later in the interview guide in
their initial response; if this happens, you don’t want to ask that particular question when
you come to it in the guide and act as if you didn’t hear the respondent’s answer in the
first place. You want to either rephrase the question to see if you get the same answer,
skip it, or reflect on what the respondent said earlier and ask him or her if you
understood it correctly.
Be relaxed. Respondents can sense your comfort or anxiety. The more relaxed you are, the
more relaxed respondents will be. This helps the interview flow smoothly.
Avoid rushing respondents. Allow enough time for respondents to answer questions
before assuming they do not have something to say.
Maintain eye contact. Don’t be so involved with your notes that you appear disengaged.
Avoid making any faces or sounds that could be perceived by respondents as
disapproval, approval, or any sort of judgment about what they said.
If you want to record an interview, ask for permission first. If you see indications that the
presence of a tape recorder makes the respondent uncomfortable, consider turning off
the recorder.
checklist
Tips for conducting interviews:
Focus groups
Focus groups are structured discussions to understand people’s perspectives, experiences or
knowledge about a particular topic. A moderator suggests topics and facilitates the discussion.
The goal is to discover the how and why of something, to get contextual responses rather than
“yes” or “no” answers. You can use a focus group to answer questions that you might ask in an
interview, such as:
What motivated you to participate in the program?
Why do you think more and more infants younger than 1 year old are dying in your county over
the past five years (after you show them the infant mortality rate)?
How are you using the knowledge and skills you learned from the program?
Generally, you can ask the same questions in focus groups that you might ask an individual during
an interview. However, focus groups are different from interviews in that:
Participants can bounce or build off each other’s comments and provide richer information.
The setting provides an opportunity to observe interactions and dynamics among people that
mirror real life.
People can get uncomfortable when they have to answer certain questions in front of other
people (e.g., personal questions about their families or anything about finances or
legal matters).
The unit of analysis is the group and not individuals in the group. Consequently, you can’t
summarize the findings by stating things like, “Four people indicated they now know how to
read food labels while the remaining six people said they continue to struggle with
understanding the labels.” Instead, the finding should read, “Participants were mixed in their
ability to understand food labels after the program.”
Focus groups usually consist of six to 12 people. These groups are not directed at, or focused on,
getting consensus.
Ask no more than five questions and make sure they are not phrased in a way that solic-
its a “yes” or “no” response, but instead generates discussion.
Be very familiar with the questions. Read them several times before the focus group to
ensure comfort with the questions and their flow.
Have a process for managing the group dynamics. For example, if someone in the group
doesn’t say anything, plan how to engage that person. Alternatively, if someone
dominates the discussion, plan how to give others a chance to talk.
If you want to record the focus group, make sure everyone in the group gives permission.
Position the recorder so everyone’s voices will be captured clearly. If it appears the
presence of a tape recorder makes even one respondent uncomfortable, consider taking
handwritten notes or turning off the recorder.
checklist
Tips for conducting focus groups:
Focus groups require excellent facilitation skills. Sometimes, focus groups can surface
tensions, conflicts or other emotions, and the person conducting the focus group must be
prepared to deal with the situation. To do this, you or your staff must carefully consider
what could go wrong during the focus group, work with the evaluator or facilitator to prepare for
uncomfortable moments (e.g., two people getting into an argument, someone bursts into tears) and agree
on how to handle them.
Observations
Observations are structured means of recording the actions and interactions of participants in an
evaluation. They provide an opportunity to collect data on a range of behaviors, capture
interactions and openly explore the topic of interest in the evaluation. Observations can be
quantified or described qualitatively. Observations as a data collection method can be useful if you
want to understand the context within which an activity for the strategy, initiative or program takes
place. For instance, observations are common in evaluations of classroom instruction techniques
and child development activities.
Use a structured protocol that makes it easier for the observer to gather and record
observations. The protocol could include criteria for the setting, quality of interactions,
content of activities, behaviors of participants, and anything else that is supposed to be
part of the program and, most important, observable.
Avoid asking the observer to record things that cannot be observed, such as what people
are thinking or why they did something.
Be clear about the role of the observer and how the observer should behave in the setting.
checklist
Tips on using observations:
Review of artifacts (e.g., documents, recordings, videos)
This method is important because artifacts can be a source of data for the evaluation. Artifacts
include mission statements, organizational charts, annual reports, activity schedules, audio
recordings, diaries, videos, grant proposals, participant utilization records, promotional
literature, etc.
Such materials can enable you to learn about the history, philosophy, goals, outcomes and
challenges of a particular program; these materials provide clues about important shifts in
program development or maturation. Document reviews also can help you formulate questions for
a survey or an interview. Bear in mind that written documents do not necessarily provide
comprehensive or correct answers to specific problems because they could contain errors,
omissions or exaggerations. They simply provide one form of information and should be used
carefully and in connection with other types of data.
Useful resources on qualitative data analysis include:
Analyzing and Interpreting Data, published by Wilder Research. Available at http://www.
evaluatod.org/assets/resources/evaluation-guides/analyzing-interpretingdata-8-09 .
Qualitative Data Analysis, by M. Miles, M. Huberman and J. Saldaña, published by SAGE
Publications in 2014.
8.1.4
Use of Mixed Methods
Over the past three decades, a trend in evaluation has been to shift toward mixing quantitative and
qualitative methods into a single evaluation called mixed method evaluation. This approach makes
sense because each method has its own strengths and weaknesses, as discussed in
previous subsections. Combining them can lead to a stronger, more complete evaluation than a
conventional evaluation that uses only one method.
A mixed method evaluation systematically integrates two or more evaluation methods,
usually drawing on both quantitative and qualitative data, such as using surveys and
focus groups in one evaluation study.
Using examples provided previously about quantitative and qualitative methods, you could use
mixed methods in the following ways:
Track and analyze the attendance of people in your program over the past six months. You
notice participants from a particular neighborhood don’t attend regularly and see a dip in
attendance in January and February. You can conduct focus groups to find out why.
Administer a survey questionnaire and ask program participants about a set of behaviors. You
can then conduct interviews with a subset of participants to find out how your program helped,
or did not help, them change those behaviors.
Monitor the high school graduation rate in your community. You see a decrease over the
past five years and conduct a series of focus groups with various people such as principals,
teachers, parents and students, to find out what factors they believe are contributing to the
decrease.
Observe an increase in the number of deaths in your county of infants younger than 1 year per
1,000 live births. You can conduct interviews with parents whose infants have died (remember,
don’t conduct a focus group because of the sensitivity of the subject) and focus groups with
service providers to find out what they believe is contributing to the increase.
exhibit 8-6: advantages and disadvantages of mixed methods
Advantages Disadvantages
Provides a more complete understanding of the
program, its implementation and outcomes than
quantitative or qualitative methods alone.
Can be more expensive and time consuming than
quantitative or qualitative methods alone.
Can be more efficient when the findings from one
method inform the content for another method. For
instance, the answers gathered from a focus group
can be used to develop response options for a
survey questionnaire.
It can be difficult to find an internal or external
evaluator with expertise in both qualitative and
quantitative methods.
Enables triangulation where a finding can be con-
firmed by multiple data sources.
8.2
critical considerations in data
collection
Several considerations in data collection don’t always get
enough attention early on in the evaluation process. These
considerations are especially important when working
in communities that have traditionally been excluded or
devalued in evaluation and research studies.
8.2.1
Who the Data Collector Is
The decision about who will collect the data must be
carefully made. You want respondents to feel comfortable
with the person asking for data and to respond honestly and
confidently. You don’t want respondents to be intimidated
or fearful. A data collector can be someone external to your organization or someone within your
organization (e.g., staff person, board member or volunteer). You should make your decision based
on several characteristics about the data collector. The data collector should:
Not be invested in getting positive or negative answers from respondents and thus be
compelled to sway the respondent.
Not be someone a participant feels compelled to please (e.g., someone who has power to
provide or withhold service).
Have the skills to interact respectfully with respondents (e.g., not “talk down” to people) and
have the humility to recognize that he or she does not know everything there is to know about
another group of people, especially if they come from a different cultural background
(e.g., don’t assume that everyone greets each other with a handshake or a hug).
Be able to follow instructions and protocols consistently.
Be discreet and not share any information provided by one respondent with another
respondent or with people other than the evaluation team.
8.2.2
How to Engage Populations Who Have Been Traditionally Excluded or Treated as Invisible
If your evaluation involves such populations (e.g., undocumented immigrants, low-income
families, people of color, people with different sexual orientation, people with disabilities, victims
of violence and incarcerated populations), keep in mind several critical considerations throughout
the evaluation process, including:
Their past experiences with researchers and evaluators (e.g., is this a community that has been
studied before by all sorts of research or evaluation groups).
Logistical and other challenges faced if they have to travel to another location for an
interview or focus group (e.g., they have to take two buses to get there, they don’t have money
for bus fare, they need someone to watch their children while attending the focus group).
Cultural differences that could cause discomfort (e.g., women from some cultures are not
allowed to be left alone to talk to male data collectors; undocumented immigrants might be
afraid to speak to anyone they don’t know; some words, phrases and gestures could be
disrespectful in another culture).
Language differences that could cause discomfort and misunderstandings (e.g., is an
interpreter needed, will the interpreter keep the information he or she hears confidential, is
the translation accurate and appropriate for that particular cultural group).
Literacy level, which could cause discomfort and embarrassment if someone cannot read or
write well.
Framing of findings within context to not perpetuate stereotypes or suggest the victims are to
be blamed for their situations.
8.2.3
Resources
Certain expenses must be carefully accounted for and not forgotten when you create the evaluation
budget, including:
Professional translation or interpreting fees
Most translators charge by the word or page while interpreters charge by the hour. They
also could charge a fixed fee depending on the assignment. Get two to three quotes for
comparison.
Expert review of the instrument’s reading level
Teachers of English as a Second Language (ESL), Adult Basic Education (ABE) and General
Education Development (GED®) test (or other high school equivalency preparation) are
good resources in your community for this purpose. Yhey typically charge by the hour or you
can negotiate a fixed fee for services.
Transportation
Bus and subway fares vary across cities. Budget a little bit more in case a few people
missed the bus and have to take a taxi.
Childcare assistance
Consider how many childcare providers you could need depending on the number of
children, or include the cost in the stipends or incentives (cash or gift cards)
for participants.
Refreshments and food
Try to negotiate with a local restaurant to cater your event for free or at a discounted price.
Remember to provide healthy and fresh options.
Stipends or incentives (e.g., gift cards, cash)
This can vary depending on how much time and what level of involvement you are asking of
participants. If you decide to use American Express or Visa gift cards, remember to budget
for the fee associated with each card.
Postage if it’s a mail survey
Don’t forget to weigh the introduction letter and the survey questionnaire together to make
sure the postage is sufficient.
8.2.4
Confidentiality and Anonymity
Confidentiality and anonymity are essential considerations for you and the evaluator. Your
respondents’ privacy should be protected vigilantly. For example, names of participants should
never be revealed in an evaluation report. The terms “anonymity” and “confidentiality” have
different meanings and should not be used interchangeably.
Anonymity requires you and your evaluator to not know who the participants
are. For instance, you don’t ask respondents to put their names in a survey or identify
themselves in a focus group.
Confidentiality means you and your evaluator know who the participants are,
but you don’t link any answers to the respondents. Any information you have
that contains the person’s name or personal information must be kept in a locked
drawer or stored in a password-protected electronic file.
8.2.5
Obtaining Consent
One of the most important ethical rules governing evaluation is that potential respondents must
give their permission to participate in the evaluation before you survey, interview or observe them.
When you are collecting data from anyone under 18 years, you also must ask their parents for
written consent. You can ask for permission in writing or verbally. However, the request should
include information about:
What you are asking them to do
The benefits and risks of their participation
Who will see the data and how the data will be used
How much time it will take for them to answer the questions
How their participation is voluntary and they can decline or stop in the middle of the process if
it makes them uncomfortable
8.2.6
Institutional Review Board (IRB)
An Institutional Review Board (IRB) is a committee established
to review and approve research involving human subjects. The
IRB’s purpose is to ensure that all human subject research is
conducted in accordance with federal, institutional and ethical
guidelines.
To determine whether your evaluation needs IRB approval,
you must consult the guidelines on protecting human research
subjects from the following sources: your funding agency; your
agency, organization or institution; and your evaluator (if using
an external evaluator).
Universities have their own IRBs and you have to be affiliated
with the university to use the IRB there. Native American/
Alaskan Native tribes tend to have their own IRBs, and you must
get permission from the tribal government to collect data from
their members. Otherwise, you can pay a private IRB to review
and certify your evaluation design, plan and instruments.
If your evaluation
requires IRB
approval, make
sure you submit your IRB
application well in advance
before you begin collecting data
(approvals can take anywhere
between several days to several
months). Also, IRB requires
a processing fee, so make sure
you allocate resources for IRB
approval in your budget.
8.3
analyzing and interpreting data
Now that you have collected the data you need, it’s time to analyze, interpret and make sense of the
data, and then act on what you learned. This process can be complicated and, at times, technical.
For example, take a look at the following excerpt from a study by the Pew Research Center.
The wave of incoming Asians pushed the total number of Asian Americans to a
record 18.2 million, or 5.8 percent, of the total U.S. population, according to census
data. By comparison, non-Hispanic whites (197.5 million) account for 63.3 percent
of the U.S. population, while Hispanics (52 million) and non-Hispanic blacks (38.3
million) account for 16.7 percent and 12.3 percent, respectively.
The influx of Asians reflects a slowdown in illegal immigration while American
employers increase their demand for high-skilled workers.
If you used data collected by the U.S. Census as part of your evaluation, what does the above
passage suggest?
The trends, based on census data, are facts. However, what the Associated Press said – which
attempts to make sense of the data – leaves the impression that all Hispanics may be “illegal” and
all Asians may be “high-skilled workers.” It could create the perception that one group is “good”
and the other is “bad.” Yet, nothing in the data presented suggests this conclusion.
An interpretation like this can affect – in a negative way – how people think about Hispanics and
Asians, as well as how policymakers deal with immigration. Therefore, making sure that you
analyze, interpret and present your data correctly is a critical responsibility.
8.3.1
Quantitative Data Analysis and Interpretation of Results
Descriptive statistical analysis.
Quantitative analysis is often associated with statistical analysis, and program staff without a
background in statistics could be intimidated by the thought of doing this type of analysis. Yet, this
is the most basic form of quantitative analysis. When you calculate the number and
percentage of responses to a particular question or the average rating for questions about the
usefulness of the training, you are starting to do descriptive statistical analysis. It is used to
examine the responses to a question by calculating and looking at the following things:
Distribution of responses or frequency distribution (e.g., how many people checked response
option 1, response option 2, response option 3, etc.).
Average value, or the mean (i.e., looking at the average rating across the participants’ ratings).
The most common response, or the mode.
The number in the exact middle of the data set, or the median.
The mean, median and mode combined are also referred to as the central tendency and provide
information about the “typical response or score.” Most people don’t use the term central
tendency. You usually hear the terms “frequency of responses,” “mean response” and “most
common response.”
Descriptive statistical analysis also provides another piece of information technically referred to as
Obviously you
cannot calculate
mean values for
variables such as race and
ethnicity, gender, sexual ori-
entation or religion because a
numerical value for them has no
meaning.
variability. This statistic refers to the following:
Spread of your results, including the range (difference
between the highest and lowest scores).
Variance (shows how widely individuals in a group vary in
their responses).
Standard deviation (how close or far a particular response is
from the average response).
Unless you have statistical skills or know how to use a statistical analysis software package such
as SPSS or SAS, you should rely on someone with training in quantitative methods to do this analy-
sis. Nevertheless, it is helpful for you to understand what these terms mean to accurately interpret
your evaluation findings.
You could have heard about the 65, 95 and 99 percent rule. It means that in a normal or bell-
shaped distribution, if you go up and down (or left and right) one standard unit (or one standard
deviation), you will include roughly 65 percent of the cases or responses in the distribution. If you
go up and down two units, you will include roughly 95 percent of the cases or responses, and if you
go up and down three units, you will include roughly 99 percent of the cases or responses.
A normal or bell-shaped distribution is a symmetrical curve that is mathematically
defined based on the mean and standard deviation, as shown below.
−2 −1 +1−3 +30 +2
34.1% 34.1%
13.6%
2.2%
13.6% 0.1%0.1%
2.2%
68% of data
99.7% of data
Standard
Deviations
Mean
One
Standard
Deviation
95% of data
Thus, when working with someone to interpret quantitative data
using descriptive statistical analysis, you might want to ponder
the following:
When looking only at the average, or mean value (e.g., value
of 3 on a 1 to 5 scale), are most of the responses close to the
average value or are the responses falling on the extreme
ends of the continuum (values of 1 and 5)?
If the standard deviation is a high number, which means
most of the participants responded very differently from
each other, what could have contributed to the wide
variation in their responses?
If the standard deviation is a very low number, it means that
most of the participants responded very similarly to each
other. Should and can the participants be divided into
subgroups and further analysis be conducted to see if there
is a difference between subgroups?
Example #1: You are implementing a leadership institute to
improve parents’ ability to advocate for their children’s
education. As part of the institute, you invite speakers to
present information about particular policies that affect
children’s ability to learn and succeed academically. After each
information session, you survey parents about their new
knowledge. Here is a question from the survey.
Question:
Please indicate the amount of new knowledge you developed
about suspension policies in your child’s school after attending
the information session.
1 2 3 4
None A little Some A lot
You might want
to assign a small
number to
negative responses so the bigger
the number, the more positive
the results. You should avoid
changing this configuration
across questions so respondents
don’t get confused.
Here is how you can display the responses to the above question:
NEW KNOWLEDGE ABOUT SUSPENSION POLICIES
N = 30
Amount of New Knowledge Developed No. (%)
None 16 (53.3)
A little 7 (23.3)
Some 3 (10.0)
A lot 2 (6.7)
Missing 2 (6.7)
Total 30 (100.0)
The title should be clear.
Always show the total number of
respondents so the reader knows
the number the percentage is based
on.
R round any figures to at least one decimal
point because it is more precise.
Always show the number and the percentage;
one without the other doesn’t convey the
proportion sufficiently (e.g., 50 percent is half
of the proportion, which seems large, but
depending on the sample size, 50 percent
could be only 16 people, which could be a
relatively small number if you are
attempting to serve 100 people.
Mean (Average) = 1.68, which means that your participants developed a little new knowledge about
the suspension policies in their child’s school.
In this example, calculating the median does not provide a meaningful piece of information.
No new knowledge developed was the most common response (or mode).
Example #2: You are implementing a city-wide strategy to promote immigrant integration, which
includes encouraging new immigrants to use public resources in the community and, during the
process, to interact with long-time residents. Here is a question you can ask in a community survey
of immigrants.
Question:
Please indicate how many times in the last three months you visited the public library.
The data you collected from 20 people indicated the following number of visits:
1, 1, 4, 5, 7, 10, 11, 11, 13, 14, 15, 16, 16, 17, 18, 19, 20, 20, 20, 20
You can create a frequency distribution with this basic display:
NUMBER OF VISITS TO THE PUBLIC LIBRARY LAST THREE MONTHS
N = 20
Number of Visits/Month No. of Respondents (%)
1 2 (10.0)
4 1 (5.0)
etc. etc.
The mean (average) number of visits in the last three months = 12.9 visits, or almost 13 visits;
another way to describe this statistic could be “on average, about one time per week.”
If you don’t convert the values into a standardized number, you can’t tell how spread out the
responses are just by looking at them. It would be even harder if you had, say, 100 responses. So,
here is where statisticians (or the evaluator you work with) apply mathematical formulas to
calculate the variance (45.31) and the standard deviation (6.73).
What do the numbers mean?
In this example, the mean is 12.9 and the standard deviation is 6.73. This means that about 65
percent of the responses fall between 6.17 (12.9 minus 6.73) and 19.63 (12.9 plus 6.73). In plain
English, this implies that 65 percent of people surveyed visited the public library between
approximately six and 20 times over the last three months.
If the standard deviation is a small value relative to the average value, it means most respondents
tend to have scores close to the average. This means the variability is low or the respondents
tended to behave similarly. If the standard deviation is a large value relative to the average, it
means there is a lot of variability and most of the respondents tend to have scores that are farther
away from the average (thus the distribution is wider). In the example above, it means the number
of times most of the respondents visited the public library in the last three months varied widely
from the average number.
One can handle missing data in different ways. If you have little missing data – less than
10 percent of the total responses – then you can just report on the number and percentage of
missing responses. If there is a higher proportion of missing data, then you might want to
perform imputation, which is the process of replacing missing data with substituted values. These
values can be based on different conditions that make sense for the data set. For instance, you can
substitute the missing responses from women who make less than $15,000 annual income with the
average response of all the women who make less than $15,000. If you have a lot of missing responses in
your data set, and you cannot generate decent results without performing imputation, you should consult
with a statistician, a trained quantitative analyst, or your evaluator to find out how best to handle the
missing data.
Imputation is the process of replacing missing data with substituted values.
Inferential statistics
Evaluators often use inferential statistics to determine if the results and conclusion extend beyond
the data you collected. Inferential statistics can be used to try to infer from the sample data what
the trend might look like in the larger population. It also can be used to make judgments about
whether the difference between two groups of people or two data sets happened by chance (i.e., is
the result statistically significant or not). It takes someone trained in quantitative analytical
techniques or statistics to perform inferential analysis; nevertheless, you should know the basics
to be informed consumers of the conclusion.
Statistical significance means that the difference between two or more groups of
people (or the same group of people at two different times) is not due to chance. More
precisely, the hypothesis or theory that there would be no difference between the two
groups of people is not true.
Things you to know about inferential statistics:
There are five common types of inferential analysis that you are most likely to come across:
Chi squares – comparing two categorical variables, like gender and voting preference
among community members.
Correlations – indicates that a relationship or pattern exists; it does not mean that one
variable “causes” the other (e.g., you could see a strong positive correlation between
healthy eating and weight loss among your program participants; however, the correlation
will not tell you if healthy eating is the cause for their weight loss).
T-tests – comparing the average values (mean scores) of two groups of people (e.g.,
between men and women) or of two data sets (e.g., scores on a pretest and post-test on
healthy eating behaviors).
Analysis of variance – comparing the average of three or more groups (e.g., between people
of three different age groups) or of three data sets (e.g., scores on healthy eating behaviors
for the same group of people in year one, year two and year three).
Regression – determining whether one variable is a predictor of another (e.g., if the length
of participation [number of weeks] in a healthy eating program is actually a predictor of
weight loss).
It requires a certain sample size or number of respondents to be able to conduct inferential
analysis. The smaller the sample size, the less reliable the results. If your evaluator wants to
conduct inferential analysis, ask if the sample size is large enough for this sort of analysis.
Find out how missing data were handled in the analysis – were the cases with missing data
excluded from the analysis or was imputation conducted to be able to include them? This
affects the sample size used in the analysis.
Sometimes, your evaluator or quantitative data expert may get excited about running
inferential analysis, but it might not be efficient or helpful to do this type of analysis depending
on what you want to know about your organization, strategy or program’s impact. Don’t assume
it is the best way to analyze quantitative data. Ask this question of your evaluator or the staff
person assigned to the evaluation and make sure you get a clear answer: What will the finding
tell you about your strategy, initiative or program’s impact? What will you learn from the find-
ing? Request the answer in plain English and non-statistical terms.
Interpretation
Quantitative findings must be interpreted with the organization, strategy, initiative, program and
other contextual factors in mind — factors that you and your staff know better than most. Therefore,
you absolutely must not leave the interpretation to the evaluator you hire, but work together with
the evaluator to review and interpret the findings.
These questions can guide your interpretation:
What is the magnitude of the effort’s impact, or how effective
was the organization, strategy, initiative or program? This is
typically expressed in terms of effect size.
Effect size is a standard measure that shows the
difference in the mean or average values between
two groups of people or data sets.
If you had a lot of missing data or insufficient responses,
why and what can be done differently to increase the
response rate in the future?
Are the results what you expected when you planned the
strategy or program? If not, what do you think affected the
results? Do you have qualitative data that can provide
insights into the results?
Are the results programmatically significant or not,
regardless of statistical significance, and what does it mean?
For instance, the difference in responses from two groups of
people might not be statistically significant, but could still
be large enough to warrant more tailored approaches or
interventions for each group. It is especially important to
tailor your approaches or interventions to the history,
cultural background and experience of different groups of
people.
What implications do the results have on the organization,
strategy or program? What actions do you need to take,
if any?
Statistical
significance is the
probability that the
observed difference between two
groups of people or data sets is
due to chance, but this
information is not enough to
fully understand the impact of
your organization, strategy,
initiative or program. This is
where knowing the effect size
can be useful. The general rule
is that a value of 0.8 suggests
a relatively large effect (the
highest value is 1.0), 0.5
means a moderate effect, and 0.2
means a small effect.
Your evaluation
should support
your organization,
strategy, initiative
or program improvement.
Complex statistical analyses of
well-designed experimental
investigations that are costly
but do not lead to improvements
are unnecessary It is better to
keep it simple. Sometimes,
looking at the frequency of
responses, average value, most
common response and the
maximum and minimum
responses is enough to give deep
insights into your work and
help you make decisions about
next steps for your organization
and your effort.
8.3.2
Qualitative Data Analysis and Interpretation of Results
Qualitative data usually take the form of text. There are four major steps in qualitative data
analysis are described below.
Review the data
Before performing any qualitative data analysis, you must read and understand the data you have
collected. Look at the question or questions in the interview, focus group or observation the data
are supposed to answer. Check for unclear or incomplete sentences, and get clarity before you
code the data.
Organize the data
You can organize your data in various ways to make it easier to code. For example, you can organize
the data by question or by type of respondent or both.
Code the data
When coding data, you should identify and label the string of text that answers the question or
provides insight about a certain topic. You should label the text consistently across respondents.
There are two basic methods of coding and you could use one or both of these:
Open coding — When you assign codes based on what emerges from the data. Here you must
put aside your presuppositions, expectations of what the findings should look like and
previous knowledge of the subject matter, so themes actually emerge from your data.
Closed coding — When you already have codes prepared beforehand based on the questions
you want to answer.
Sometimes, even after you have coded the data, you might need to revisit the codes to refine them,
especially if you performed closed coding and found something in the data that you did not expect.
Also, sometimes codes flourish in a way that leaves you with too much data. If so, the code needs
to be broken down into subcodes to better organize the data. The rule for coding is to make the
codes fit the data, rather than trying to make your data fit your codes.
Identify and generate themes
After the data have been coded, study the text that were coded and develop a
general statement or theme that reflects what was said or observed. This is usually the hardest
part because you have to read the coded text several times and generate a theme that captures
the essence of the statements or observations collectively. If you want to explain the specifics, you
can provide examples of statements and observations to support the theme. Also, to capture how
strong a theme is, you can report on the number and percentage of responses you coded that
supported the theme. You also might need to note any differences in themes if you group
respondents by race and ethnicity, gender, or any other demographic characteristics.
Question for program staff:
What do you think a high-quality youth mentoring program should have?
Responses:
1. People who look like the youth and can relate to them.
2. Adults who are committed to putting in time to help the youth outside of the program.
3. Structured activities that teach youth life skills.
4. People who are from the neighborhoods the youth come from.
5. People who can relate to the youths’ experiences.
The relevant text can be coded to answer more than one question.
Don’t create long strings of codes, and always keep a list of codes, their definitions and
when and why you revised the code and definition.
Make sure you code enough text to provide the full context of the data (not just a few
words with no meaning when separated from the rest of the text).
Consider connections between codes as well and look for patterns in relationships.
checklist
Tips on coding data:
6. Role models for the youth, people in whom the youth can see something is familiar to them,
maybe they grew up in the same neighborhood or remind them of someone they admire
growing up.
7. At least two hours per week that mentors must spend with the youth, if not more if they care
about their mentee.
8. Adults the youth can call any time, even outside of program activities, when they have
questions.
9. Some of our adult mentors actually take their mentees to baseball games and other kinds of
events, which has really helped build their relationships.
10. Our youth tend to enjoy spending time with people who they can talk to and who take the time
to talk to the youth, mainly because they have experiences they can relate to.
Examples of the types of closed codes and definitions you can develop before analysis:
CARE_ADULTS = Caring adults
STRUC_ACT = Structured activities such as college preparation, understanding of different career
pathways, how to deal with peer pressure, etc. Structured activities mean there is an instructor or
facilitator and a clear start and end time for the activity.
At first glance, it looks like responses 2, 7, 8, 9 and 10 can clearly be coded CARE_ADULTS. But
when you look at the responses more carefully, you would probably realize that “Caring Adults”
may be insufficient to capture the essence of some of the statements; they’re not just about caring
adults, but caring adults the youth can relate to or with whom they’ve shared experiences. So, you
can create a new code, ADULTS_RELATETO, and then code responses 1, 4, 5, 6 and 10 with this new
code. Note that response #10 is coded both as CARE_ADULTS and ADULTS_RELATETO.
Here’s how you can summarize this particular finding:
A total of 10 program staff members were interviewed for their perspectives about the attributes of
a high-quality mentoring program. Adults who really care about their mentees was an important
characteristic mentioned by half of the staff. Equally important was the fact the adults were people
the youth could relate to because they share similar backgrounds or interests.
Interpret the findings
The next step involves making sense of what you found. This means comparing your results to your
expected outcomes, original evaluation questions, the goals and objectives of your program and
current state-of-the-art knowledge (e.g., research about mentoring programs). Some questions to
guide your interpretation include:
Did any of the patterns and themes surprise you?
What are the factors that might explain the deviations?
If you collected quantitative data (that is, you used mixed methods), do the qualitative findings
support the quantitative findings? If not, what are the factors that could explain the differences
(e.g., sampling, the way the questions were asked in the survey compared to the interviews,
etc.)?
Do any interesting stories emerge from the responses?
Do the results suggest any recommendations for improving the program?
Do the results lead to additional questions about the program? Do they suggest additional data
could be needed?
Do you need to change the way the data are collected next time?
Be thoughtful when you are making sense of the data. Don’t rush to conclusions or make
assumptions about what your participants meant to say. Involving other people (e.g., program
staff ) or working with your evaluator to discuss what the findings mean will help you make sense of
the data.
HIGHLIGHTS
At the most basic level, quantitative
methods are concerned about what, who
and when, and qualitative methods are
useful for determining how and why.
Your questions or what you want to learn
should drive the decision about whether
you should use quantitative or qualitative
methods. One type of method is not
necessarily better than the other. You should
weigh what you want to know and how
much money, staff capacity and time you
have for the evaluation. Sometimes, you
might hire an evaluator who is partial to one
method, but don’t let his or her preference
drive your decision.
Quantitative data collection methods
include surveys, tests and assessments.
Qualitative data collection methods include
interviews, focus groups, observations and
review of artifacts.
Mixed methods evaluation, which combines
quantitative and qualitative methods, can
lead to a stronger, more complete evaluation
than only one method.
Critical considerations in data collection
include who will collect the data, how to
engage populations who have been
traditionally excluded or treated as
invisible, allocation of funds for certain
expenses that you might not think about
(e.g., childcare assistance, transportation,
translation and interpretation), how to
protect the confidentiality and anonymity of
your respondents, permission from
potential respondents to participate in the
evaluation (especially anyone under 18
years) and review of research or evaluation
protocol by an Institutional Review
Board (IRB).
In quantitative analysis, three things are
important to examine: frequency
distribution, central tendency and
variability.
In qualitative analysis, open and closed
coding can be used to analyze the data and
generate themes.
Interpretation of findings means comparing
your results to your expected outcomes,
original evaluation questions, goals and
objectives of your program and current
state-of-the-art knowledge (e.g., research
about mentoring programs).
When collecting, analyzing and interpreting both quantitative and qualitative data, remember that:
exercises
1. Which of the following is an advantage of qualitative methods?
A. Requires strong mathematical skills
B. Can be used to find out why your program works or does not work
C. Likely to cost less than quantitative methods
D. Takes less time than in quantitative methods to analyze the same number of responses
E. None of the above
2. An evaluation design that utilizes both qualitative and quantitative methods is called
.
A. Ethnography
B. Mixed methods evaluation
C. Quasi-experimental evaluation
D. Comprehensive evaluation
E. Summative evaluation
3. When evaluating a disability program with participants who have poor eyesight, which survey
method is recommended? .
A. Internet/computer survey
B. Mail survey
C. Telephone survey
D. Questionnaire survey
E. None of the above
4. Anonymity requires that .
A. You and the evaluator know the participant
B. All program staff know the participant
C. You know the participant but keep it a secret
D. You don’t link any of the answers to the participant
E. You and evaluator don’t know the participant
5. Think about an evaluation that you’d like to conduct for a program or initiative. What are the
best data collection methods to help answer your evaluation questions and why?
6. Consider an evaluation you conducted in the past. Would you do anything differently after
reading this chapter and why?
Answers: 1B, 2B, 3C, 4E
9
Summarize, Communicate and
Reflect on Evaluation Findings
A PREPARE for conducting
an evaluation
B
DETERMINE
stakeholders
and how and
when to
engage them
C
IDENTIFY
assumptions and
determine what
will be di�erent
(theory of change
and logic model)
D DEVELOP evaluation plan (logic
model, measurement,
framework, etc.)E COLLECT and analyze data
F
COMMUNICATE
results and
understand
what happened
(interpret
�ndings and
facilitate
learning)
G
MAKE informed decisions
(improve actions and
next steps)
Stakeholders
introduction
Evaluation findings can be communicated in
many different ways to tell the story of your
strategy, initiative or program. You might
have a typical format for presenting your
evaluation findings, nevertheless, you
should know why some formats for
displaying and communicating your
findings could be more effective than
others. One end of the spectrum features
traditional reports. The other end uses
creative means such as photography and
drama, which can help communicate
findings to people who have lower levels of
literacy or come from cultures with strong
oral traditions.
There also are various options for displaying
your evaluation findings to sharpen the story
you want to tell. The field of data visualization and visual analytics has grown due to the availability of
large amounts of data along with technology advancements in accessing, handling and displaying data.
You easily can get caught up in all the options, but need to remain focused on what you want to
communicate, why and to whom.
Attending to this stage of the evaluation process is key because effective summary and communication of
evaluation findings helps:
Convey knowledge
Facilitate understanding
Create meaning
Confirm or challenge theories or previous ways of thinking
Inform decision-making and action
How this chapter is organized…
9.1 Communicate and
Report Your
Evaluation Findings
9.1.1 What You Need to Ask
Before Putting the
Findings Together
9.1.2 Different Communicating
and Reporting Formats
and Options
9.2 Considerations in
Developing and
Implementing a
Communications Plan
9.3 Keeping It Simple
9.4 Ways to Display Your
Evaluation Findings
9.4.1 Displays of Patterns and
Trends over Time
9.4.2 Displays of Distribution
and Spread of Responses
9.4.3 Displays of Comparisons
9.4.4 Displays of Frequency of
Words
9.4.5 Displays of Themes
9.5 Reflecting on Your
Evaluation Findings
9.5.1 Using Evaluation
Findings
9.5.2 Considerations in
Preparation for Reflecting
on, Discussing, and Using
Evaluation Findings
Highlights
Exercises
findings and insights. You can share findings in the middle of your
program as long as you clarify that they are interim, preliminary
insights.
9.1
communicate and report your evaluation
findings
9.1.1
What to Ask Before Putting the Findings Together
After data collection and analysis, you need to determine how to
summarize and communicate the findings and tell the story
effectively to your stakeholders, whether they are funders, board
members, community leaders, staff, volunteers or program
participants. Much of the material for this chapter comes from
Evaluation Strategies for Communicating and Reporting, by R.
Torres, H. Preskill and M. Piontek, published 2005 by SAGE Pub-
lications, and is adapted for this handbook.
You can use many strategies to summarize and communicate your findings. Sometimes, you could
have much more data than you possibly can share effectively. Therefore, you should begin by
asking the following questions before putting the findings together:
Are you required to submit an evaluation report to the funder about the results and impact of
the strategy, initiative or program?
What are the reporting requirements (where and when the report is due, questions to be
answered, sections that must be included, page limitations, inclusion of graphs and
tables, etc.)?
Who else needs to know the results and impact of the effort and why?
Do you want to inform them about their investments, give them tools to make decisions,
encourage support for the effort, or inform them for other reasons?
Depending on why they’re being informed, which aspects of the results and impact could
be particularly interesting to them?
When is the best time to share the findings and impact with the intended audience?
How much does each intended audience know about the effort?
How interactive do you want the communication to be for each intended audience? For
example, written and print materials are least interactive while discussions and working
sessions are most interactive. Verbal and video presentations fall somewhere in the middle.
What are the risks in sharing findings that could lead to bad consequences for your
organization and the community you serve?
Can the findings be taken out of context and harm the organization or community?
What can you do to mitigate the risks and consequences?
What roles do you, your staff, your board members and your internal or external evaluator have
in summarizing and communicating the findings and insights?
Who are the most effective messengers for the information?
Do you need a facilitator skilled in adult learning techniques to assist with discussions,
working sessions and verbal presentations?
Your answers to these questions will help determine the content and format for summarizing and
communicating the results and impact of the program.
From the outset, emphasize the use of evaluation for learning.
Involve key stakeholders in the evaluation’s design and implementation and
communicate throughout the evaluation process so there are no surprises.
Think about what to say and how to say it from the perspective of the stakeholders
hearing about the evaluation findings.
Share any negative findings through a discussion format so you can effectively facilitate
the learning process and reduce the likelihood of misunderstandings.
Don’t start the report or discussion with negative findings. Instead, lead with positive
findings and use words and phrases such as “accomplishments,” “how we can do better”
and “work in progress.”
checklist
Tips for dealing with potentially negative findings:
9.1.2
Different Communicating and Reporting Formats and Options
exhibit 9-1: overview of communicating and reporting formats
(adapted from torres, preskill and piontek, 2005, p. 27)
Need or Purpose
Format Ed
uc
at
e
or
b
ui
ld
ar
en
es
s
Im
pr
ov
e
pr
og
ra
m
En
su
re
a
cc
ou
nt
ab
ili
ty
Le
ve
ra
ge
s
up
po
rt
G
en
er
at
e
ne
w
ow
le
dg
e
R
ep
lic
at
e
an
d
sc
al
e
pr
og
ra
m
M
ak
e
re
co
m
m
en
da
ti
on
s
ab
ou
t f
ut
ur
e
di
re
ct
io
ns
A
dj
us
t e
va
lu
at
io
n
si
gn
a
nd
pl
em
en
ta
ti
on
Short communications
(e.g., memos, emails,
blogs, tweets)
Reports (interim and final)
Executive summaries
Newsletters, briefs,
brochures, bulletins
Posters
Webinars
Blogs
Discussions (group or
individual)
Photography
Drama
When communicating to policymakers, don’t use language that is too technical or present
too many caveats about the statistical results. For this particular audience, a distinction
between statistical and practical importance could be too much to provide. Instead, only
present findings of practical and policy importance.
checklist
Tip for communications:
9.2
considerations in developing and implementing a
communications plan
Now that you have determined the best way to summarize and communicate the evaluation
findings and insights, you might want to develop a communications plan. The plan doesn’t have
to be extensive or long; its main purpose is to help you organize and put in writing the process for
sharing your evaluation findings and insights. You might want to involve your staff and other
stakeholders in developing and implementing the plan. The plan is an important tool to:
Stay focused on how you ultimately want to use the information. Without a plan, it is easy to
lose sight of this focus. Other priorities such as collecting the data and responding to your
funder’s reporting requirements could overshadow everything else toward the end of your
strategy, initiative or program.
Prompt you to think about the best messengers for your evaluation, potential risks in
sharing the information and strategies you can develop to mitigate the risks and
consequences. For example, if you share information about the vulnerabilities of the children
or youth your effort serves or if you use youth to deliver the message in a presentation, do you
risk further stereotyping about them?
Plan ahead about what resources you could need, including funding and technical expertise.
At a minimum, your communications plan should address the following:
Background, organization and context of the program, initiative, strategy, or policy
Purpose of the evaluation
Audiences and their specific information needs
Evaluation products, by type of audience
Potential risks and strategies for mitigating them
Budget for products
Additional knowledge, skills, or resources needed
Timeline
9.3
keep it simple
A key principle about effectively communicating your findings
is to keep the displays simple. Avoid cluttering the displays
with lines, colors, shades or anything else that could make it
attractive, but draws attention away from the content. To make
the material easy for the reader to understand, include only the
essential, critical information.
Let’s consider the example that has been used previously. An
initiative in Rainbow County seeks to increase the number and
percent of working class African American homeowners in the
county. The program includes services to educate families about
budgeting, savings and improving their credit scores; acquiring
financing; and understanding what it means to be a homeowner.
It also works with lending institutions to make their products
and services more accessible to working class families. The
initiative tracks where the graduates are in the homeownership
process. Exhibit 9-2 is a typical chart that shows this
information and highlights the people who are in the most advanced stage (i.e., their loan
application has been approved). This chart can be created easily in Microsoft Word. Now, look at
the table in Exhibit 9-3. It’s the same table, but without lines and shading. Ask yourself:
Which one is easier on your eyes?
Do you notice the difference between the two tables?
Does one give you more information than the other?
Does one present the data in a more manageable way than the other?
What is the key point of the tables? Is one table clearer about its message? If so, which one?
Exhibit 9-2: a typical chart in the form of a table
Year Course Was Completed Name Current Stage of Homeownership
Spring 2014 John Sykes Improved credit score
Spring 2014 Antwon Bates Increased savings
Spring 2014 Tony Austin Loan approved
Spring 2014 Laura Garcia Loan approved
Fall 2014 Mary Crawford Improved credit score
Fall 2014 Sarah Jones Loan approved
Fall 2014 Bob Love Increased savings
Fall 2014 Laura Mills Applied for loan
Spring 2015 Cindy Jones Applied for loan
Spring 2015 John Simmons Applied for loan
Fall 2015 Tanesha Williams Have family budget
Fall 2015 Michelle Simms Have family budget
Fall 2015 Mark Sifford Increased savings
Exhibit 9-3: typical chart in simplified table
Year Course Was Completed Name Current Stage of Homeownership
Spring 2014 John Sykes Improved credit score
Antwon Bates Increased savings
Tony Austin Loan approved
Laura Garcia Loan approved
Fall 2014 Mary Crawford Improved credit score
Sarah Jones Loan approved
Bob Love Increased savings
Laura Mills Applied for loan
Spring 2015 Cindy Jones Applied for loan
John Simmons Applied for loan
Fall 2015 Tanesha Williams Have family budget
Michelle Simms Have family budget
Mark Sifford Increased savings
9.4
ways to display your evaluation findings
After you have a communications plan and understand the importance of presenting findings sim-
ply, you and the evaluator can discuss the best way to display the findings based on
quantitative and qualitative data. Data visualization is a growing field and there are lots of
resources about how to convey your data effectively.
It is impossible to cover the topic fully in this handbook, but some excellent information is
available from the following resources:
Workshops and other instructional materials, published by A.K. Emery. Available at
http://annkemery.com/introducing-the-essentials/.
Presenting Data Effectively, by S. Evergreen, published by SAGE Publications in 2014.
Additional resources are available at http://stephanieevergreen.com.
When displaying findings, your intention is to:
Draw the viewer’s to ponder the content rather than the method, graphic design or
something else.
Avoid any misrepresentation of what the data say.
Provide clear labels to help the viewer understand what he or she is seeing.
Avoid small print that causes the viewer trouble in reading and understanding the data.
9.4.1
Displays of Patterns and Trends Over Time
Displays that are effective for showing patterns or trends over time include graphics that show
the time on one dimension (typically, the horizontal dimension or x-axis), and the item for which
change is being observed on another dimension (typically, the vertical dimension or y-axis).
Examples of such displays, line graphs and slope graphs, are illustrated below. Both graphs
convey the same findings. In the first graph, each line in the slope graph represents one of the
three groups of program participants and changes in each group’s use of banking services before
and after a six-month financial literacy program. At a glance, the viewer can quickly see that groups
A and C increased their use while group B decreased their use. The graph also shows that group
A benefited the most from the program because the slope of the line is steeper than the line for
group C. This should prompt the viewer to ask why there is a difference between the groups, and
explore what circumstances led to group B’s decreased use of banking services. Answers to these
questions could lead to improvements in the program.
exhibit 9-4a: display of patterns and trends over time using a line graph
C
C
B
B
A
A
exhibit 9-4b: display of patterns and trends over time using a slope graph
Frequency of use of banking
services before the program
Frequency of use of banking
services after the program
9.4.2
Displays of Distribution and Spread of Responses
Displays that effectively show how responses are distributed along two dimensions include scatter
plots and histograms (see Exhibits 9-5a and 9-5b). For example, you operate an initiative to
prepare young men and boys of color for college. In the scatterplot below, the dots would represent
your program participants. The horizontal dimension or x-axis would represent the
number of times the young men attended the ACT tutorials over a six-month period. The vertical
dimension, or y-axis, would represent their ACT scores in a mock test.
The display does
not show that
attending tutorials
regularly and consistently
will increase ACT test scores,
which would suggest a causal
relationship. Instead it shows
a positive relationship between
two behaviors – not that one
behavior causes the other. Other
factors could be contributing to
the increased ACT scores.
At a glance, the viewer can see that a line is implied. This
suggests that attending tutorials and ACT scores are related – the
more times the participant attended the tutorials, the higher
his or her ACT scores. The viewer also can see that a handful of
participants who attended almost all the tutorials earned very
high ACT scores. The reverse is just as clearly true for those
participants who attended very few tutorials and scored low
on their mock ACT tests. This display can be used to make the
case that attending tutorials regularly and consistently can help
increase ACT test scores.
The histogram below also shows distribution of data. For in-
stance, the histogram could be used to show the number of
infants born with certain weight over the course of your effort.
The horizontal, x-axis, would represent frequency of cases for
each of the weight values. A viewer can look at the histogram
and see that most of the infants were born with weights that fell
in the middle range. To take the analysis one step further, you
could compare the values in the middle of the range with the
average weight of a newborn in the state to determine if your
program participants tend to have worse, better, or similar birth
outcomes relative to the state average.
exhibit 9-5a: displays of distribution and spread of responses using a scatterplot
exhibit 9-5b: displays of distribution and spread of responses using a histogram
Least Frequent Most Frequent
Infant Weight (lbs)
Lowest Score
N
um
b
er
o
f C
as
es
Highest Score
Respondents who
attended all the
tutorials and scored
the highest on
their tests.
9.4.3
Displays of Comparisons
Displays that help compare two or more groups include bar charts, clustered bar charts, side-by-
side bar charts and stacked bar charts. In these displays, the bars usually represent a categorical
variable (e.g., frequency of breastfeeding) while the length of the bar represents a quantitative val-
ue (e.g., frequency, percentage or rating). The clustered, side-by-side and stacked bar charts can
show a third dimension by using different color bars to represent different groups of people (e.g.,
women in your program compared with women in another program).
For example, in the clustered bar shown in below, the viewer can see that most women in the study
breastfed for more than 12 months. It’s also clear that slightly more women in your program (X)
did this compared with women in program Y. This finding may prompt the viewer to ask “why?” A
discussion about what worked well in your program based on other data you might have collected
could follow.
exhibit 9-6: examples of bar, clustered bar, side-by-side bar and stacked bar charts
> 12
months
> 12
months
> 12
months
> 12
months
6- 12
months
6- 12
months
6- 12
months
6- 12
months
<6 months <6 months
<6 months<6 months
Women in program X Women in program Y Women in program Z
9-6a bar
9-6c side-by-side bar
9-6b clustered bar
9-6d stacked bar
The Step-by-Step Guide to Evaluation | page 197
9.4.4
Displays of Frequency of Words
Word clouds allow you to show your qualitative findings in the simplest way. A word cloud
displays how many times a word has been mentioned in the given text or texts. The words
mentioned more often appear larger on the figure and the words mentioned less often appear
smaller on the figure, as shown below. Word clouds can be created using online applications such
as Wordle (www.wordle.net) or TagCrowd (www.tagcrowd.com). Word clouds are most useful for
displaying findings about responses to the question, “What one word would you use to describe
…?”
exhibit 9-7: example of a word cloud
(Taken from Henderson, S., & Segal, E. H. (2013). Visualizing qualitative data in evaluation
research. New Directions on Evaluation, p. 56)
www.wordle.net
www.tagcrowd.com
Page 198 | The Step-by-Step Guide to Evaluation
For example, the viewer can see in the above graphic that the word “data” was mentioned most fre-
quently in your staff members’ responses to the question, “What is the one word you think about
when we talk about evaluation?” After your staff received extensive training in evaluation, you
could ask the question again and see how they respond by comparing the two word clouds.
What the word cloud doesn’t tell you, however, is whether the word “data” was mentioned in a
positive or negative way. Nor does it show how “data” related to other words used during the
interviews. Therefore, word clouds, while attractive and easy to create, should not be used as a
stand-alone display. You should provide an explanation about the word cloud, going beyond what
the viewer sees.
9.4.5
Displays of Themes
If you would like to convey more about your qualitative findings, you can summarize and display
themes from your qualitative data.
Themes are patterns tfound in qualitative data. The general rule is that a theme is
formed when three or more pieces of evidence point to the same idea.
For example, if three interviewees identified challenges to accessing services, that
would be a theme, whether the challenges were due to restricted service hours (no
extended office hours available) or language barriers (no interpreters or translated
materials available).
A matrix is an effective format for displaying themes derived from qualitative data.
Let’s use this example; Program staff was asked, “What are the three most important
qualities of a youth mentoring program?” The staff also was asked to rate their responses in order
of importance.
Below is how you can describe the findings based on the exhibit. [Note: The darker the shade the
more important it is.]
The Step-by-Step Guide to Evaluation | page 199
Three qualities are essential for a youth mentoring program: structured activities, caring adults in
general and caring adults youth can relate to. Among these three qualities, more than half of the
staff (N=6, 60 percent) believed that “caring adults youth can relate to” is the most important
quality. While “structured activities” is an important quality in youth mentoring programs, it was
rated by staff as the least important among the three qualities.
exhibit 9-8: matrix for communicating qualitative findings
Respondents
Qualities of a Youth
Mentoring Program
1 2 3 4 5 6 7 8 9 10
Structured activities
Caring adults in general
Caring adults youth can
relate to
9.5
reflecting on your evaluation findings
As stated previously, staff and other stakeholders are more likely to use the evaluation findings if
they understand the purpose of the evaluation and contributed to its design, implementation,
interpretation and use of the findings. To summarize and communicate your findings is not
sufficient. It’s important to reflect upon the findings and their implications and plan ways to put
them to use. Remember, evaluation must provide usable information to equip you to make
informed decisions and shape your programs to be as effective as possible.
However, there are obstacles to reflecting upon the findings and planning ways to use them, such
as:
Fear of being judged by board members, staff or people outside your organization.
Concern about the time and effort involved to convene stakeholders to discuss and reflect on
the findings.
Page 200 | The Step-by-Step Guide to Evaluation
Resistance to change that could impact the way things have
been done in the past.
Inadequate communication and knowledge sharing systems
that affect how, when and with whom information is shared.
Staff who are not interested in the findings for various
reasons.
Organizational limitations such as limited budget and staff
capacity to carry out other functions deemed more
important.
Concern about negative findings.
9.5.1
Use of Evaluation Findings
Improving your strategy, initiative or program
A goal of every evaluation should be to improve the strategy,
initiative or program, and evaluation findings should support
decisions and actions about what makes the effort more
effective. You, your staff and other stakeholders might want to
discuss what the findings say about the strengths and
weaknesses of the effort. Together, you should determine what
to do to adjust and improve the effort.
Being accountable
Evaluation is frequently used to hold the implementers account-
able. Sometimes, you have to adjust your strategy, initiative or
program because you learned that something wasn’t working
(e.g., you had trouble engaging parents with the current out-
reach strategy, or the industry for which you were developing a
workforce suddenly collapsed). If this happens, communicate
with your funder and other stakeholders so they understand why
the outcomes for which you are accountable are no longer valid.
The Step-by-Step Guide to Evaluation | page 201
Educating or building awareness
Evaluation can be used to educate or build stakeholders’ or the public’s awareness of your
strategy, initiative or program, and the issues it addresses. In reflecting on and discussing your
findings with the intent of educating and building awareness, you must think carefully about how
you frame the findings to avoid inadvertently perpetuating stereotypes or painting a negative
image of the people or community you serve. Also, you might want to combine quantitative and
qualitative data to show the numbers and provide the story behind the numbers to make the
information come alive.
Leveraging support
You can also seize opportunities to reflect on and discuss your evaluation findings to leverage
support from stakeholders such as community leaders and funders. In these situations, you want
to communicate clearly the evidence that supports the benefits of your strategy, initiative or
program. In these scenarios, put any negative findings in the context of learning and improvement.
Generating new knowledge
Evaluation can be used to discover and share new knowledge about effective practice. Evaluation
frequently tests a theory of change and the results can generate insights about whether, and under
what conditions the theory holds true. The new knowledge can inform decisions that affect the
lives of the people your strategy, initiative or program serves.
Replicating and scaling the strategy, initiative or program
If the evaluation shows that your strategy, initiative or program is effective and provides insight
into what makes it effective, you could want to consider replicating it in other communities or
increasing the number of people exposed to it. You have to be careful, however, about replicating
it in other communities that have different community and cultural contexts. You should ascertain
the degree to which the elements that worked in your program, initiative or strategy are culturally
appropriate for another group of people living under different conditions.
Developing recommendations for next steps
Your evaluation findings can help you develop recommendations for the future. These
recommendations could pertain to a wide range of issues such as type of support needed from
funders, training and other technical assistance needed to help your organization, staff capacity,
programming and other matters.
Page 202 | The Step-by-Step Guide to Evaluation
Making adjustments to the evaluation design and process, if necessary
In some cases, you could learn the evaluation design or implementation process was not
optimal for answering your evaluation questions. For example, you might have discovered that very
few people responded to the online questionnaire. Your evaluation could have been more effective
if you had used interviews or focus groups, even if it meant fewer respondents. Adjusting the
design of the evaluation midway likely would be difficult, but you should weigh the pros and cons
of continuing the evaluation with limitations or making midcourse corrections. This situation could
be a good learning opportunity for you and your organization. Also, if you see problems with the
way the external evaluator has designed the evaluation, such as not having translated materials
available for people with limited English proficiency, then you should bring up the issue even
though the evaluation is well under way. Flagging these types of limitations is important so the
problem can be resolved immediately and without harming the effectiveness of your evaluation.
9.5.2
Considerations in Preparation for Reflecting on, Discussing and Using Evaluation Findings
Communication avenues for different types of stakeholders
An effective process provides multiple avenues to impact staff, program participants and other
stakeholders in positive ways. Some avenues for consideration include:
Set aside time during staff meetings for reflection and discussion about the evaluation find-
ings. Align the timing of the reflections and discussions with the evaluation’s timeline for
reporting findings.
Host “open houses” or coffee or tea chats to share evaluation findings with program
participants or other stakeholders. This helps keep the communication of findings in a
nonthreatening and less formal discussion format.
Organize and conduct roundtables with various stakeholders to discuss and reflect on
the findings.
Host a webinar and invite participants to ask questions.
Blog about the highlights of the evaluation findings or post them on social media sites. Pose
questions and invite comments from readers and followers.
The Step-by-Step Guide to Evaluation | page 203
An effective process also helps:
Improve communication and understanding between different groups (e.g., between line staff
and managers or between staff and volunteers) about the strategy, initiative
or program.
Improve understanding about the population being served, particularly disenfranchised groups
who are often unheard, misrepresented and misjudged.
Leverage additional resources and other support for the effort.
Facilitate development of knowledge, skills and other competencies among staff, program
participants and other stakeholders.
Below are examples of scenarios where the reflection process and discussion facilitated some of
the above changes.
Example 1
An initial evaluation of a program providing educational services to families and children in an
economically disadvantaged urban community helped program staff discover they were
operating the program based on a set of implicit and unspoken assumptions. The fact that these
assumptions were not put in writing or discussed explicitly as part of the program seemed to be
contributing to problems with new staff members’ ability to understand the program, its goals and
underlying principles. Founding staff used the evaluation findings and insights to create a
Page 204 | The Step-by-Step Guide to Evaluation
historical overview of the program’s origins, the path it had taken to get where it was and the
assumptions underlying the program and its mission. The discussion helped the new staff better
understand the program and created shared understanding between new and longer-serving staff.
Example 2
The evaluator provided a preliminary summary to the staff about what she was learning from the
interviews conducted. One preliminary finding revealed that interviewees had varying
understandings about what the program meant by “creating a more inclusive community.”
African-American interviewees, in particular, questioned the likelihood of this outcome when they
didn’t see their leaders involved in the steering committee for the program. Communicating this
finding to the program staff led to a discussion about what inclusiveness means and how the
African-American interviewees were unaware that the NAACP director and the high school
counselor – both African Americans – served on the committee. The evaluator used this opportu-
nity to ask the staff about what an inclusive community looks like to them, what representation
means and how aware they were of other types of African American leaders in the community. The
discussion resulted in a series of learning sessions in the organization about what it takes to build
an inclusive community.
Requirements for an effective process
Excellent facilitation: If you or your evaluator do not have strong facilitation skills, consider
hiring a professional.
Clear meeting objectives and outcomes: If you are facilitating the reflections and use of
evaluation findings through a meeting or discussion, set clear objectives and learning
outcomes for the process.
Rules of engagement: As in any good meeting, have rules about how participants should en-
gage one another (don’t interrupt when someone is talking, be respectful, etc.).
Attention to power differences: If the meeting involves people with different levels of power –
whether based on rank, position, race, ethnicity or any other characteristic – have procedures
in place to prevent the more powerful from dominating the discussion. Also give those with
less power the opportunity to contribute (e.g., small group discussions, written feedback, go-
ing around the room and inviting each person to say something).
The Step-by-Step Guide to Evaluation | page 205
Potential disagreements: You also want to ensure you have
a process for handling potential disagreements, tensions or
conflicts due to the findings. Take time to identify findings,
especially negative findings, which could surface such
challenges, then work with the facilitator (if you have one) to
plan how to deal with them when they emerge.
Next steps: Summarize the decisions and next steps at the
end of the reflections and discussion.
You might want to train a few people to serve as spokespersons for the program and
evaluation findings if the program is a very visible one (e.g., it’s the first program of
its kind in the community, it is testing a theory that has been heavily publicized in the
media) or if findings are somewhat controversial (e.g., they don’t support the theory, they
bring attention to issues such as structural racism and other inequities). In this way, you
can manage the communications and limit it to the few people who have deep
understanding of the program and the evaluation and who are prepared and skilled in
speaking to journalists and others. You also could plan a press release or a press
conference, so you have more control over the framing of the findings and prevent
their misuse.
checklist
Tip for communicating evaluation findings:
Page 206 | The Step-by-Step Guide to Evaluation
HIGHLIGHTS
Before you begin putting the findings
together, be clear about: your funder’s
reporting requirements; stakeholders who
should know about the results and what
you want them to do with the information;
which aspects of the results may be
particularly interesting to them; and the best
time to share the information with them.
Pay close attention to negative findings and
have a strategy for dealing with them.
Develop a communications plan to help you
organize and put in writing the process for
sharing your findings and insights.
Keep displays about your findings simple
to enable viewer to focus on the content.
Start with all the information in the display,
take a step back and then strip the display
of any unnecessary and distracting graph-
ics and other information. Put yourself in
the viewer’s shoes and if the finding could
be misunderstood and make the necessary
adjustments.
If you are replicating your program,
initiative or strategy, check thoroughly to
see if you have sufficient evaluation
information to determine what parts might
need to be adapted for another cultural or
community context.
If you choose to reflect on findings in
a meeting, ensure you plan for good
facilitation; set an agenda with clear
meeting objectives and learning outcomes;
set rules of engagement; include procedures
for handling power differences and dis-
agreements, and identify next steps.
You might want to train a few
spokespersons for your program or
evaluation findings, especially if the
program is a high-stakes or visible program
or if the findings could stir controversy.
The Step-by-Step Guide to Evaluation | page 207
exercises
1. If the evaluation does not produce favorable results, which of the following can help ease the
fear of reporting such findings:
A. Seeing this as an opportunity to learn and make improvements
B. Keeping the findings a secret
C. Starting the report with a list of the negative results
D. Making sure the decision-makers know about the negative results right away
E. None of the above
2. Can you identify three things that are missing from the following data display?
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Health
Bene�ts/
Services
Types of
Policies
How Health
Insurance
Works
Ways to
Buy
Insurance
Tax Credit/
Other
Programs
Average
Excellent
Fair
1.76
2.62
1.63
2.53
1.56
2.50
1.62 1.68
2.56 2.56
1.81
2.50
Le
ve
l o
f K
no
w
le
dg
e
figure 1: average ratings of knowledge, by gender
Answers:1A; 2 - explanation of what the scores mean, legend to explain the gray and black bars
(which one refers to males and which ones to females), and total number of respondents.
Page 208 | The Step-by-Step Guide to Evaluation
3. Take a set of qualitative or quantitative findings from one of your evaluation reports. Consider
how the findings could be displayed to better communicate the story you want to tell about the
program or initiative that was evaluated.
The Step-by-Step Guide to Evaluation | page 209
Conclusion
This handbook was designed to demystify evaluation and familiarize you with its basic elements so
you can integrate evaluative thinking into your daily work and partner more effectively with trained
evaluators. While you may choose to hire an independent evaluator, you can still use the basic
knowledge you learned about evaluation to make decisions regarding:
The most appropriate evaluation type, approach and methodology to assess your strategy,
initiative, program or policy
Ways to engage key stakeholders throughout the evaluation process
Amount of funds to allocate for the evaluation
Culturally appropriate data collection methods
Analysis strategies
Interpretation, use and communication of findings
Practicing evaluative thinking can help strengthen the impact of your organization’s work. By
becoming a more informed consumer of evaluation, you can participate in the evaluation of your
strategy, initiative or program in a more meaningful manner. As a result, you can conduct an
effective evaluation that generates knowledge that will not only benefit your organization but also
the families and communities you work with and serve.
Page 210 | The Step-by-Step Guide to Evaluation
The Step-by-Step Guide to Evaluation | page 211
Glossary
Anonymity Anonymity requires you and your evaluator do not know who the
participants are. For instance, you don’t ask respondents to put their
names on or identify themselves in a survey or focus group.
Closed-ended Questions Closed-ended questions provide discrete, multiple-choice answers
that respondents can select.
Confidentiality Confidentiality means you and your evaluator know who the
participants are, but you don’t link any of the answers to the
respondent. Any information you have that contains the person’s
name or personal information must be kept in a locked drawer or
stored in a password-protected electronic file.
Cultural Competency Refers to the ability – stance, knowledge, skills and commitment –
to respect and engage with diverse segments of communities and
to include the contextual and cultural dimensions relevant to these
diverse segments in the evaluation design and process.
Culture A set of socially transmitted and learned behavior patterns, beliefs,
institutions and all other products of human work and thought that
characterize the functioning of a particular population, profession,
organization or community. Culture is continually evolving.
Culturally Responsive
Evaluation
Culturally responsive evaluation recognizes that cultural values,
beliefs and context lie at the heart of any evaluation effort.
Developmental Evaluation An approach used to support innovation within an organization and
in its strategies, initiatives and programs. Efforts that are innovative
are often in a state of continuous development and adaptation, and
they frequently unfold in a changing and unpredictable environment.
Page 212 | The Step-by-Step Guide to Evaluation
Effect Size A standard measure that shows the difference in the mean or average
values between two groups of people or datasets.
Evaluative Thinking Evaluative thinking is a cognitive process, motivated by
inquisitiveness and a belief in the value of evidence, which involves
identifying assumptions, posing thoughtful questions, pursuing
deeper understanding through reflection and perspective taking, and
making informed decisions in preparation for action.
Experimental Design Experimental designs assess the causal effects of a program by com-
paring two groups of people - one group receives the
intervention (“treatment group”) and one does not (“control group”).
Immediate Outcomes or
Short-Term Outcomes
Immediate changes or benefits expected — usually within one to two
years — as a result of successful implementation of the strategy.
Imputation The process of replacing missing data with substituted values.
Intermediate Outcomes or
Short-Term Outcomes
Changes or benefits, usually within one to two years of the immediate
outcomes.
Indicators Indicators are markers of progress toward the change you hope to
make with your strategy, initiative, or program.
Institutional Review Boards Institutional Review Boards (IRBs) are entities set up to protect the
rights and welfare of people who participate in research. Evaluations
of programs involving Native Americans/Alaska Natives also require
permission from their tribal governments.
Logic Model A logic model is a graphic representation of the theory of change that
illustrates the linkages among program resources, activities, outputs,
audiences, and short-, intermediate-, and long-term outcomes relat-
ed to a specific problem or situation.
The Step-by-Step Guide to Evaluation | page 213
Long-Term Outcomes Lasting changes with organizational, community, or systems-level
benefits (e.g., organizational practices or policies, new or modified
legislation, improved social conditions). Sometimes, these outcomes
might be referred to as impact.
Methodology A set or system of methods and procedures that you use to answer
your evaluation questions.
Mixed Methods Study Involve the intentional use of two or more different kinds of data
gathering and analysis tools — typically a combination of qualitative
(e.g., focus groups and interviews) and quantitative (e.g., multiple
choice surveys and assessments) — in the same evaluation.
Normal or Bell-shaped
Distribution
A normal or bell-shaped distribution refers to a symmetrical curve
that is mathematically defined based on the mean and standard
deviation.
Open-ended Questions Open-ended questions are questions that ask respondents to
respond through written text.
Qualitative Data Qualitative data is any information that can be collected or captured
in text form.
Quasi-experimental Design Assesses the causal effects of a program by comparing two groups of
participants (a “treatment” group and a “comparison” group) or by
comparing data collected from one group of participants before and
after they participated in the program. There is no random
assignment of participants into the two groups, unlike studies using
an experimental design.
SMART Metrics Specific, Measurable, Attainable, Realistic and Timely metrics
Stakeholder A stakeholder is defined as any person or group who has an interest
in the strategy, initiative or program being evaluated or in the results
of your evaluation, including your evaluator.
Page 214 | The Step-by-Step Guide to Evaluation
Statistical Significance The difference between two or more groups of people is a result that
is not due to chance, or more precisely, that the hypothesis or theory
that there would be no difference between the two groups of people
is not true.
Strategies or Activities Strategies or activities are the processes, techniques, tools, events,
technology and actions of the planned program, used to bring about
the intended program changes or outcomes.
Themes Themes are patterns that you find in your qualitative data. The
general rule is that a theme is formed when there are three or more
pieces of evidence pointing to the same idea. For example, if three
interviewees identified challenges to accessing services, that would
be a theme, whether the challenges were due to restricted services
hours (no extended office hours available) or language barriers (no
interpreters or translated materials available).
Theory of Change A theory of change is a narrative that explains the links between
activities and outcomes and how and why the desired change is
expected to come about.
The Step-by-Step Guide to Evaluation | page 215
Bibliography
Bamberger M, & Segone M. (2011). How to Design and Manage Equity-Focused Evaluation. New York:
UNICEF Evaluation Office.
Chelimsky, E. & Shadish, W. (Eds.) (1997). Evaluation for the 21st Century. Thousand Oaks, CA:
SAGE Publications.
Clark, H. & Anderson, A.A. (2004). Theories of Change and Logic Models: Telling Them Apart. Paper presented
at the American Evaluation Association Conference Presentation, Atlanta, GA.
Child Trends. (2007). Five Steps for Selecting an Evaluator: A Guide for Out-Of-School Time Practitioners.
Retrieved August 26, 2015, from http://www.childtrends.org/wp-content/uploads/2013/04/Child_
Trends-2007_10_01_RB_SelectingEvaluator .
Cross, E.Y., Katz, J.H., Miller, F.A., & Seashore E.W. (1994). The Promise of Diversity. Burr Ridge, IL: Irwin
Professional Publishing.
Dalton, J., Elias, M., & Wandersman, A. (2001). Community Psychology: Linking Individuals and
Communities. Belmont, CA: Wadsworth.
Earl, S., Carden, F., & Smutylo, T. (2001). Outcome mapping. Ottawa, Canada: International Development
Research Center.
Fetterman, D., Kaftarian, S., & Wandersman, A. (Eds.) (2001). Empowerment Evaluation: Knowledge and Tools
for Self-Assessment and Accountability. Thousand Oaks, CA: SAGE Publications.
Fitzpatrick, J. (2012). An introduction to context and its role in evaluation practice. New Directions for
Evaluation, 135, 7-24.
Frierson, H.T., Hood, S., & Hughes, G.B. (2002). A guide to conducting culturally responsive evaluations.
In The 2002 User-friendly Handbook for Evaluation (pp. 63-73). Arlington, VA: National Science
Foundation.
https://www.childtrends.org/wp-content/uploads/2013/04/Child_Trends-2007_10_01_RB_SelectingEvaluator
https://www.childtrends.org/wp-content/uploads/2013/04/Child_Trends-2007_10_01_RB_SelectingEvaluator
Page 216 | The Step-by-Step Guide to Evaluation
Gitlin, A. (Ed.). (1994). Power and Method. New York, NY: Routledge.
Hendricks, M., Plantz, M., & Pritchard, K. (2008). Measuring outcomes of United Way-funded programs:
Expectations and reality. New Directions for Evaluation, 119, 13-35.
Hofstede G. (1997). Culture and Organizations. New York, NY: McGraw-Hill.
Hood, S., Hopson, R. & Frierson, H.T. (Eds.) (2015). Continuing the Journey to Reposition Culture and
Cultural Context in Evaluation Theory and Practice. Greenwich, CT: Information Age.
Labin, S., Duffy, J., Meyers, D., Wandersman, A., & Lesesne, C. (2012). A research synthesis of the
evaluation capacity building literature. American Journal of Evaluation, 33, 1-32.
Lee, K. (2007). The Importance of Culture in Evaluation. Denver, CO: The Colorado Trust.
Lee, K. S., & Chavis, D. M. (2012), Cross-case methodology: Bringing rigour to community and systems
change research and evaluation. Journal of Community and Applied Social Psychology, 22 (5): 428-438.
Mathison, S. (2005). Encyclopedia of Evaluation. Thousand Oaks, CA: SAGE Publications.
Newcomer, K., Hatry, H., & Wholey, J. (2015). Handbook of Practical Program Evaluation, 4th Ed., New
Jersey: John Wiley & Sons, Inc.
Newcomer, K. & Wirtz, P. (2004). Using statistics in evaluation. In J. Wholey, H. Hatry, & K. Newcomer
(Eds.), Handbook of Practical Program Evaluation, 2nd Ed. (pp. 439-459). San Francisco, CA: Jossey Bass.
Owens, J.M. (2007). Program Evaluation. New York, NY: The Guilford Press.
Patrizi, P. & Patton, M.Q. (Eds.) (2010). Evaluating strategy. New Directions for Evaluation, No. 128.
Pawson, R. (2003). Nothing as practical as a good theory. Evaluation, 9, 471-490. doi:
10.1177/1356389003094007.
Posovac, E.J. & Carey, R. G. (2003). Program Evaluation: Methods and Case Studies. Upper Saddle River, NJ:
Prentice Hall.
Revels, M. (February 16, 2016). Focus Group Research: Workshop Material. Atlanta, GA: Author.
The Step-by-Step Guide to Evaluation | page 217
Stufflebeam, D. (2001). Evaluation models. New Directions for Evaluation, 89, 7-98.
Taylor-Powell, E. & Herman, C. (2000). Collecting Evaluation Data: Surveys. Retrieved August 1, 2015
http://learningstore.uwex.edu/assets/pdfs/g3658-10 .
The American Heritage College Dictionary, 3rd Ed. (2000) Boston, MA: Houghton Mifflin.
The Urban Institute. (2006). Building A Common Outcome Framework to Measure Nonprofit Performance.
Washington, DC: The Urban Institute.
Torres, R., Preskill, H., & Piontek, M. (2005). Evaluating Strategies for Communicating and Reporting, 2nd Ed.
Thousand Oaks, CA: SAGE Publications.
Trochim, W. (1999). The Research Methods Knowledge Base. Ithaca, NY: Cornell University.
Voget, W.P. (1999). Dictionary of Statistics & Methodology: A Nontechnical Guide for the Social Sciences.
Thousand Oaks, CA: SAGE Publications.
Yin, R.K. (2003). Case Study Research: Design and Methods, 3rd Ed. Thousand Oaks, CA: SAGE Publications.
http://learningstore.uwex.edu/assets/pdfs/g3658-10
Page 218 | The Step-by-Step Guide to Evaluation
The Step-by-Step Guide to Evaluation | page 219
Guide to Evaluation
Resources
OVERVIEW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
EVALUATION BASICS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Introductory Evaluation Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Evaluation for Planning, Development
and Continuous Quality Assurance and Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
Community Based and Social Change Evaluation Resources . . . . . . . . . . . . . . . . . . . . . . . 223
Theories of Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
Logic Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
Engaging Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
Cultural Competence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
Racial Equity Lens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
Hiring an Evaluator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
Using and Presenting Evaluation Findings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
APPLIED EVALUATION RESOURCES BY TOPIC AREAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
Advocacy and Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
Workforce Development and Employment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
Financial Literacy and Family Asset-building . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
Early Childhood Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
K-12 and Higher Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
Health . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
Poverty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
Youth Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
SECONDARY DATA SOURCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
Juvenile Justice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
Early Childhood Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
K-12 and Higher Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Health . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
OTHER DATA-RELATED RESOURCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
Page 220 | The Step-by-Step Guide to Evaluation
OVERVIEW
This resource guide includes major resources about evaluation in general, and about specific topics of
particular interest related to evaluation in the W.K. Kellogg Foundation’s priority areas. Resources marked
with the symbol “” are resources recommended by the Kellogg Foundation because they are particularly
useful and relevant to both those with limited experience with evaluation as well as those with ample
experience.
EVALUATION BASICS
Introductory Evaluation Resources
Better Evaluation (2013). Manage an evaluation or evaluation system. Retrieved from
http://betterevaluation.org/sites/default/files/Manage%20-%20Compact
This web-based resource provides practical assistance to nonprofit organizations engaged in evaluation.
It is intended to help users’ access information on the important aspects of program evaluation. The
sources are listed in the order by which conducting an effective evaluation should be done. Topics range
from engaging stakeholders to building evaluation capacity. The hyperlinks allow users to continuously
go deeper on a particular topic (e.g., from understanding the basic components of a budget for
evaluation to sample budget forms).
Better Evaluation (2013). Planning an evaluation: Using the rainbow framework. Retrieved from http://
betterevaluation.org/sites/default/files/Rainbow%20Framework%20-%20compact%20version
This web-based resource offers a framework that can be used to develop an evaluation plan by
prompting the reader to consider a series of key questions. It is broken down into seven evaluation cate-
gories: managing an evaluation; defining what is to be evaluated; framing the boundaries of evaluation;
describing activities, outcomes, impacts and contexts; understanding causes of outcomes; synthesizing
data from evaluation; and reporting and supporting use of evaluation findings.
Centers for Disease Control and Prevention. (2011). Introduction to program evaluation for public health
programs: A self-study guide. Retrieved from http://www.cdc.gov/eval/guide/cdcevalmanual
This report, intended for a beginner-level audience, provides a glossary of key evaluation terms. Several
evaluation methods are discussed, including formative, summative, midterm and final evaluation. The
Centers for Disease Control and Prevention (CDC) deem triangulation as the most effective data collection
method. The publication addresses the need to expand social equity and cultural sensitivity, the benefits
of evaluation and the use of evaluation to build capacity.
Glenaffric Ltd. (2007). Six steps to effective evaluation: A Handbook for programme and project
managers. Joint Information Systems Committee. Retrieved from http://www.jisc.ac.uk/media/
documents/programmes/reppres/evaluationhandbook
This handbook outlines an approach to evaluation that is applicable to all JISC development activity and
http://betterevaluation.org/sites/default/files/Manage%20-%20Compact
http://betterevaluation.org/sites/default/files/Rainbow%20Framework%20-%20compact%20version
http://betterevaluation.org/sites/default/files/Rainbow%20Framework%20-%20compact%20version
http://www.cdc.gov/eval/guide/cdcevalmanual
https://www.webarchive.org.uk/wayback/archive/20140615150400/http://www.jisc.ac.uk/media/documents/programmes/reppres/evaluationhandbook
https://www.webarchive.org.uk/wayback/archive/20140615150400/http://www.jisc.ac.uk/media/documents/programmes/reppres/evaluationhandbook
The Step-by-Step Guide to Evaluation | page 221
relevant to both program and project managers. Although the handbook is intended for JISC managers,
those with intermediate level experience with evaluation will find it useful. The handbook provides guid-
ance in conducting the following six steps: (1) identify stakeholders, (2) describe project and understand
program, (3) design evaluation, (4) gather evidence, (5) analyze results and (6) report findings. The six-
step approach promotes the use of formative and summative evaluation and describes effective
data collection.
JBS International. (2006). Study designs for program evaluations. Retrieved from http://www.pacenterof
excellence.pitt.edu/documents/study_designs_for_evaluation
This document discusses the following evaluation approaches: exploratory, descriptive, experimental
and quasi-experimental. The report is geared more toward evaluators as it describes and provides ex-
amples of randomized controlled trials and a number of quasi-experimental designs, such as propensity
score matching, pretest and posttest comparisons, and simple differences. Also included in this
document is a discussion of factors that may affect internal and external validity of an evaluation.
Pawson, R. (2003). Nothing as practical as a good theory. Evaluation, 9, 471-490. doi:
10.1177/1356389003094007
This article, written for evaluation beginners, explains what evaluation is. Methods of evaluation are
discussed in great detail and are supplemented with real examples. The benefits of evaluation are also
explained.
Shackman, G. (2009). What is program evaluation? Retrieved from http://gsociology.icaap.org/methods/
basicguideshandouts.html
This short guide provides users with a basic overview of program evaluation, discussing logic models,
methods of evaluation, data collection methods and ways to determine whether a program caused its
intended outcome.
The Urban Institute. (2006). Building a common outcome framework to measure nonprofit performance.
Retrieved from http://www.urban.org/uploadedpdf/411404_nonprofit_performance
This document, written for nonprofit organizations, provides basic indicators to assess different types of
nonprofit programs. It also guides nonprofits in developing their own indicators. Ultimately, this
framework provides an approach for assessing programs in a way that is specific, observable,
understandable, relevant, time bound and valid.
United Nations Educational, Scientific, and Cultural Organization (2007). Internal Oversight Service
Evaluation Section. Retrieved from http://unesdoc.unesco.org/images/0015/001557/155748E
This handbook, designed to be used by a beginner-level audience, describes the basic evaluation
approaches and methods. While it does not touch on social equity or cultural sensitivity, it explains the
benefits of evaluation and the use of evaluation to build capacity.
http://www.pacenterofexcellence.pitt.edu/documents/study_designs_for_evaluation
http://www.pacenterofexcellence.pitt.edu/documents/study_designs_for_evaluation
http://gsociology.icaap.org/methods/basicguideshandouts.html
http://gsociology.icaap.org/methods/basicguideshandouts.html
http://www.urban.org/uploadedpdf/411404_nonprofit_performance
http://unesdoc.unesco.org/images/0015/001557/155748E
Page 222 | The Step-by-Step Guide to Evaluation
U.S. Health and Human Services, Administration for Children and Families. (2010). The Program manager’s
guide to evaluation: Second edition. Retrieved from http://www.acf.hhs.gov/sites/default/files/opre/
program_managers_guide_to_eval2010
This guide, for people with little to some evaluation experience, explains what program evaluation is, why
evaluation is important, how to conduct an evaluation, how to report evaluation findings and how to use
evaluation results to improve programs that benefit children and families.
Wall, J. (n.d.). Program evaluation model: 9-step process. Sage Solutions. Retrieved from
http://www.janetwall.net/attachments/File/9_Step_Evaluation_Model_Paper
This publication provides readers with a nine-step model for conducting effective program evaluation. Af-
ter a brief introduction to program evaluation, the document describes each step. Sample questions and
data collection models are included. This document also lists resources on various topics (e.g., assess-
ment and evaluation standards, ethical standards, response rates.).
Evaluation for Planning, Development and Continuous Quality Assurance and Monitoring
Corporation for National and Community Service. (n.d.). Evaluation resources. Available at
http://www.nationalservice.gov/resources/evaluation/all-evaluation-resources
This web-based resource contains a series of hyperlinks to the federal agency’s evaluation core
curriculum course. The courses are designed to assist nonprofit organizations as they move through
each stage of the evaluation process, from planning to use of the results. The narrative instructions are
accompanied by slides and samples of evaluation products, such as evaluation plans and statements of
work for evaluation.
International Federation of the Red Cross. (n.d.). Project/programme monitoring and evaluation guide.
Retrieved by http://www.ifrc.org/global/publications/monitoring/ifrc-me-guide-8-2011
This handbook aims to promote a common understanding of monitoring and evaluation for the
International Federation of Red Cross and Red Crescent Societies (IFRC) projects. The handbook also
provides general evaluation information useful to individuals with a novice level of evaluation
experience. Several methods of evaluation are discussed in the publication and the IFRC encourages a
triangulation method (using several types of data collection methods such as surveys, interviews and
self-report) for data collection. This publication stresses the importance of cultural sensitivity when
collecting data on sensitive topics.
Prevention by Design. (2006). Evaluation terminology tip sheet. Retrieved from http://socrates.berkeley.
edu/~pbd/pdfs/evaluation_terminology
This document, which can be used by anyone, provides the definitions for common evaluation terminolo-
gies that are essential to planning and conducting evaluation work.
http://www.acf.hhs.gov/sites/default/files/opre/program_managers_guide_to_eval2010
http://www.acf.hhs.gov/sites/default/files/opre/program_managers_guide_to_eval2010
http://www.janetwall.net/attachments/File/9_Step_Evaluation_Model_Paper
https://www.nationalservice.gov/resources/evaluation/all-evaluation-resources
http://www.ifrc.org/global/publications/monitoring/ifrc-me-guide-8-2011
http://socrates.berkeley.edu/~pbd/pdfs/Evaluation_Terminology
http://socrates.berkeley.edu/~pbd/pdfs/Evaluation_Terminology
The Step-by-Step Guide to Evaluation | page 223
United Nations Development Programme. (2009). Handbook on planning, monitoring and evaluating for
development results. Retrieved from http://web.undp.org/evaluation/handbook/documents/english/
pme-handbook
This handbook is intended to support the United Nations Development Programme (UNDP) in
becoming more results-oriented and to improve its focus on development changes and real
improvements in people’s lives. The handbook is useful for UNDP staff, managers and executive board,
as well as independent evaluators and members of the national, regional and global evaluation
community. The handbook categorizes evaluation methods by timing—ex-ante, midterm, final or terminal
and ex-post evaluation. According to the document, the data collection method depends upon certain
criteria (accessibility to surveys, interviews, self-report, etc.). Though social equity and cultural
sensitivity are not addressed, the benefits of evaluation and the use of evaluation to build capacity are
discussed in great detail.
Community based and Social Change Evaluation Resources
Borgman-Arboleda, C. & Clark, H. (2010). Considering evaluation: Thoughts for social change and
movement-building groups. New York, NY: ActKnowledge. Retrieved from http://www.actknowledge.org
This booklet presents an easy-to-use resource for evaluation and assessments of social justice, social
change and movement building work. Different types of evaluation, as well as various methods, are
discussed in this publication, from how to collect data to recommending the use of mixed methods in
evaluation. Social equity is briefly mentioned in this piece while the various benefits of evaluation are
clearly noted.
Crystal Foster, C. & Louie, J. (2010). Grassroots action and learning for social change. Center for Evaluation
Innovation. Retrieved from http://www.evaluationinnovation.org/sites/default/files/foster%20louie%20
brief
This guide examines the similarities and differences between advocacy and organizing, then presents a
framework for evaluating community organizing. The guide details what evaluation should be—
participatory, prospective, learning-based, real-time, respectful of the culture of organizing, attentive to
leadership development and evidence based.
Gelmon, S., Foucek, A., & Waterbury, A. (2005). Program evaluation: Principles and practices (second
edition). Portland, OR: Northwest Health Foundation. Retrieved from http://www.northwesthealth.org/
search?q=evaluation%20handbook
This handbook provides a framework for community-based organizations to evaluate and understand the
effectiveness of their programs. The handbook provides an overview of basic program evaluation prin-
ciples and practices for documenting program progress. Cultural competency and building capacity are
briefly discussed.
http://web.undp.org/evaluation/handbook/documents/english/pme-handbook
http://web.undp.org/evaluation/handbook/documents/english/pme-handbook
http://www.actknowledge.org
http://www.evaluationinnovation.org/sites/default/files/foster
http://20brief
http://20brief
http://www.northwesthealth.org/search?q=evaluation%20handbook
http://www.northwesthealth.org/search?q=evaluation%20handbook
Page 224 | The Step-by-Step Guide to Evaluation
Grantmakers for Effective Organizations. (n.d.). Four essentials for evaluation. Retrieved from
http://docs.geofunders.org/?filename=2012_geo_evaluation_essentials
This document is aimed at helping grantmakers further their evaluation efforts. The target audience is
supporters of evaluation who want to embed these practices more deeply in the work of their
organizations. According to this piece, there are four steps critical for evaluations: lead, plan, organize,
and share. Readers can also use this text to gain an understanding of evaluation methods and data
collection. The document acknowledges the importance of social equity and cultural competence when
conducting an evaluation. The benefits of conducting evaluations are discussed throughout this guide
as well.
The Community Toolbox. (n.d.). Evaluating community programs and initiatives. Retrieved from
http://ctb.ku.edu/en/tablecontents/index.aspx
This toolkit guide addresses how to evaluate a specific program and standards for good evaluations. The
piece is geared toward a beginner-level audience and describes evaluation approaches and methods. An
entire section of the Community Toolbox is devoted to social equity and cultural competency.
Theories of Changes
Anderson, A. (n.d.). The community builder’s approach to theory of change: A practical guide to theory
development. The Aspen Institute Roundtable on Community Change. Retrieved from
http://www has.ie/Shared/Files/4/TOC_fac_guide
This document provides all the information needed to facilitate a theory of change process with a
community group by reviewing the major definitions of theories of change, providing important
background information for facilitators before they enter a planning session, and offering practical
guidance for facilitating planning sessions. The resource guide also provides a resource toolbox for the
theory of change facilitator.
Clark, H. & Anderson, A. (2004). Theories of change and logic models: Telling them apart. ActKnowledge
& Aspen Roundtable on Community Change. Retrieved from http://www.theoryofchange.org/wp-content/
uploads/toco_library/pdf/TOCs_and_Logic_Models_forAEA
This presentation explains the differences between theories of change and logic models and when to use
each.
Community Vision. (2011). Developing a theory of change for your structural change grantmaking. Retrieved
from http://www.lgbtfunders.org/commonvision/resources/developing%20a%20theory%20of%20
change
This guide defines theory of change for evaluation beginners and offers steps for developing an
effective theory of change. The presentation also discusses the differences between theories of change
and logic models.
http://docs.geofunders.org/?filename=2012_geo_evaluation_essentials
http://ctb.ku.edu/en/tablecontents/index.aspx
http://www has.ie/Shared/Files/4/TOC_fac_guide
http://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/TOCs_and_Logic_Models_forAEA
http://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/TOCs_and_Logic_Models_forAEA
http://www.lgbtfunders.org/commonvision/resources/developing
http://20change
http://20change
The Step-by-Step Guide to Evaluation | page 225
Reisman, J. & Gienapp, A. (2004). Theory of change: A practical tool for action, results and learning.
Organizational Research Services. Retrieved from http://www.aecf.org/upload/publicationfiles/
cc2977k440
This handbook created for the Annie E. Casey Foundation gives an overview of what a theory of change
is and how to create one. Included in this handbook are examples of theories of change and tables and
graphs that can be used as samples to create theories of change.
The Aspen Institute: Aspen Planning and Evaluation Program. (n.d.). Advocacy progress planner. Retrieved
from http://planning.continuousprogress.org/
This online tool can be used by anyone interested in advocacy planning and evaluation.
Logic Models
Centers for Disease Control and Prevention, Division for Heart Disease and Stroke Prevention. (n.d.).
Evaluation guide: Developing and using a logic model. Retrieved from http://www.cdc.gov/dhdsp/
programs/spha/evaluation_guides/docs/logic_model
These guides are a series of evaluation technical assistance tools developed by the CDC Division for Heart
Disease and Stroke Prevention to assist in the evaluation of heart disease and stroke prevention activities
within states. The guides are intended to offer guidance, provide consistent definitions of terms, and aid
skill building on logic modelling.
Center for Program Evaluation and Performance Measurement. (n.d.). Planning the evaluation. Retrieved from
http://www.cdc.gov/dhdsp/pubs/docs/cb_feb2014
This online resource provides a brief overview of what logic models are, why they are used, and how to
develop them.
Child Welfare Information Gateway. Logic model builder. (n.d.). Retrieved from
https://toolkit.childwelfare.gov/toolkit/
This online toolkit provides step-by-step assistance in developing a logic model to use for program
evaluation. This tool allows for the logic model to be downloaded to Microsoft Word so that it can be
customized and reformatted.
Hawkins, B., Taylor-Powell, E., & MacDonald, G. (n.d.) Logic Models Library Guide. U.S. Centers for Disease
Control and Prevention. Retrieved from http://libguides.library.cdc.gov/logic_model
This web-based library includes a selection of no-cost, open-access materials on logic models for various
public health activities, domestic and international. The resource includes materials in four categories:
(1) logic models for program planning and implementation, (2) logic models for program evaluation, (3)
developing logic models and (4) instructional materials (i.e., how to develop a logic model).
http://www.aecf.org/upload/publicationfiles/cc2977k440
http://www.aecf.org/upload/publicationfiles/cc2977k440
http://planning.continuousprogress.org
http://www.cdc.gov/dhdsp/programs/spha/evaluation_guides/docs/logic_model
http://www.cdc.gov/dhdsp/programs/spha/evaluation_guides/docs/logic_model
http://www.cdc.gov/dhdsp/pubs/docs/cb_feb2014
https://toolkit.childwelfare.gov/toolkit
http://libguides.library.cdc.gov/logic_model
Page 226 | The Step-by-Step Guide to Evaluation
Hyndman, B., Hershfield, L., & Thesenvitz, J. (2001). Logic models workbook. The Health Communication Unit
at the Centre for Health Promotion at the University of Toronto. Retrieved from
http://www.blazesports.org/wp-content/uploads/2011/02/thcu-logic-model-workbook
This workbook uses a four-step approach to provide an overview of key concepts and methods to assist
health promotion practitioners in the development of program logic models.
Innovation Network. (n.d.). Point K Tools: Logic model builder. Retrieved from
http://www.innonet.org/?section_id=64&content_id=185
This web-based workbook assists nonprofit directors and other interested individuals in building a logic
model for their program. Registration is required.
Knowlton, L. W. & Phillips, C.C. (2012). The logic model guidebook: Better strategies for great results
(Second edition). Los Angeles, CA: SAGE Publications.
This guide provides students, practitioners, and beginning researchers with practical support to develop
and improve logic models that reflect knowledge, practice and beliefs.
Milstein, B. & Chapel, T. Developing a logic model or theory of change. The Community Tool Box. Retrieved
from http://ctb.ku.edu/en/tablecontents/sub_section_main_1877.aspx
This toolkit, created for nonprofit organizations, provides an overview of what a logic model is, when it
can be used, how to create one, what makes the process effective, and the benefits and limitations of
logic modeling.
Sundra, D., Scherer, J., & Anderson, L. (2003). A guide on logic model development for CDC’s Prevention
Research Centers. Centers for Disease Control and Prevention. Retrieved from
https://www.bja.gov/evaluation/guide/documents/cdc-logic-model-development
This guide assists users in acquiring knowledge and skills for developing logic models for their centers.
This guide examines what a logic model is and the benefits of using one, the importance of involving
stakeholders in logic model development and the major components of a logic model.
Taylor-Powell, E. & Henert, E. (2008). Developing a logic model: Teaching and training guide. Madison, WI:
University of Wisconsin - Extension. Retrieved from http://www.uwex.edu/ces/pdande/evaluation/pdf/
lmguidecomplete
This guide provides an extensive overview of logic modeling. The guide explains what a logic model is,
logic model components and language, benefits of logic models, and how to develop a logic model.
Some excerpts of this piece are aimed at an advanced-level audience.
University of Wisconsin-Extension. (2003). Enhancing program performance with logic models. Retrieved
from http://www.uwex.edu/ces/pdande/evaluation/pdf/lmcourseall
This document, which is a companion for an online course, provides recommendations for planning and
evaluating education and outreach programs, as well as helping program practitioners use and apply
http://www.blazesports.org/wp-content/uploads/2011/02/thcu-logic-model-workbook
http://www.innonet.org/?section_id=64&content_id=185
http://ctb.ku.edu/en/tablecontents/sub_section_main_1877.aspx
https://www.bja.gov/evaluation/guide/documents/cdc-logic-model-development
http://www.uwex.edu/ces/pdande/evaluation/pdf/lmguidecomplete
http://www.uwex.edu/ces/pdande/evaluation/pdf/lmguidecomplete
http://www.uwex.edu/ces/pdande/evaluation/pdf/lmcourseall
The Step-by-Step Guide to Evaluation | page 227
logic models. Throughout the PDF, icons are used to indicate additional resources (e.g., link to play audio
file, practice activity). With over 200 pages of material, this document provides an in-depth look into
logic models.
The World Bank. (n.d.). The LogFrame handbook: A logical framework approach to project cycle management.
Retrieved from http://documents.worldbank.org/curated/en/783001468134383368/pdf/
31240b0LFhandbook
This handbook summarizes the essential elements of logic model development. Although it provides
guidelines for World Bank usage, the handbook is useful for a beginner-level audience.
Westmoreland, H., Lopez, E., & Rosenberg, H. (2009). How to develop a logic model for district-wide family
engagement strategies. Harvard Family Research Project. Retrieved from http://www.hfrp.org/
publications-resources/browse-our-publications/how-to-develop-a-logic-model-for-districtwide-
family-engagement-strategies
This short step-by-step guide is intended to help users to develop logic models for district-wide family
engagement efforts. This piece can be used to help determine which goals are realistic, develop a
strategy to achieve those goals and chart progress.
W.K. Kellogg Foundation. (2004). Logic model development guide. Retrieved from http://www.wkkf.org/
knowledge-center/resources/2006/02/wk-kellogg-foundation-logic-model-development-guide.aspx
This guide provides practical assistance to nonprofits engaged in evaluation. It aims to give staff of
nonprofits and community members alike sufficient orientation to the underlying principles of “logic
modeling” so that they can use this tool to enhance their program planning, implementation and
dissemination activities.
Engaging Stakeholders
Albritton, E., Edmunds, M., Thomas, V., Petersen, D., Ferry, G., Brach, C., & Bergofsky, L. (n.d.). The national
evaluation of the CHIPRA quality demonstration grant program, implementation guide number 1:
Engaging stakeholders to improve the quality of children’s health care. Agency for Healthcare Research
and Quality. Retrieved from http://www.ahrq.gov/policymakers/chipra/demoeval/what-we-learned/
implementation-guides/implementation-guide1/index.html
This guide was designed to help government officials and other program administrators engage and
partner with stakeholders in initiatives to improve the quality of child health care.
Bourns, C. (2010). Do nothing about me without me: An action guide for engaging stakeholders. Grantmakers
for Effective Organizations. Retrieved from http://docs.geofunders.org/?filename=do_nothing_about_
me_without_me
http://documents.worldbank.org/curated/en/783001468134383368/pdf/31240b0LFhandbook
http://documents.worldbank.org/curated/en/783001468134383368/pdf/31240b0LFhandbook
http://www.hfrp.org/publications-resources/browse-our-publications/how
http://www.hfrp.org/publications-resources/browse-our-publications/how
http://www.wkkf.org/knowledge-center/resources/2006/02/wk-kellogg-foundation-logic-model-development-guide.aspx
http://www.wkkf.org/knowledge-center/resources/2006/02/wk-kellogg-foundation-logic-model-development-guide.aspx
http://www.ahrq.gov/policymakers/chipra/demoeval/what-we-learned/implementation-guides/implementation-guide1/index.html
http://www.ahrq.gov/policymakers/chipra/demoeval/what-we-learned/implementation-guides/implementation-guide1/index.html
http://docs.geofunders.org/?filename=do_nothing_about_me_without_me
http://docs.geofunders.org/?filename=do_nothing_about_me_without_me
Page 228 | The Step-by-Step Guide to Evaluation
This guide, developed for nonprofit organizations, defines stakeholder engagement, makes the case
that involving stakeholders leads to improved results, provides a variety of options for engaging
stakeholders, and supplies examples of the positive impact stakeholder engagement has
on grantmaking.
Preskill, H. & Jones, N. (2009). A practical guide for engaging stakeholders in developing evaluation
questions. Retrieved from http://www.rwjf.org/content/dam/web-assets/2009/01/a-practical-guide-
for-engaging-stakeholders-in-developing-evalua
This guide aims to assist evaluators and their clients in engaging stakeholders—those with a stake or
interest in the program, policy, or initiative being evaluated. The guide should assist philanthropy, but
also the field of evaluation more generally, as it seeks to increase the value and usefulness of evaluation.
U.S. Agency for International Development, FHI360, PEPFAR, PTA, AVAC. (2014). Stakeholder engagement
toolkit quick guide. Retrieved from http://www.fhi360.org/sites/default/files/webpages/se-toolkit/
quick-guide
This guide was designed based on feedback from community liaison officers, community educators and
others working in community outreach in research settings. Its purpose is to give users quick and easy
access to modifiable (Microsoft Word or Excel) files of Stakeholder Engagement Toolkit tools with instruc-
tions for using them.
Cultural Competence
Centers for Disease Control and Prevention. (2014). Practical strategies for culturally competent
evaluation. Atlanta, GA: U.S. Department of Health and Human Services. Retrieved from
http://www.cdc.gov/dhdsp/docs/cultural_competence_guide
This guide was developed as an introduction and resource for state partners to use to promote cultural
competence in the evaluation of public health programs and initiatives. Designed for program staff and
evaluators, this guide highlights the prominent role of culture in our work. It provides important strat-
egies for approaching an evaluation with a critical cultural lens to ensure that evaluation efforts have
cultural relevance and generate meaningful findings that stakeholders—individuals who are invested in
the program or potentially affected by the evaluation—ultimately will value and use.
Throughout this guide, aspects of cultural competence in evaluation are discussed within the context
of CDC’s Framework for Program Evaluation in Public Health to highlight opportunities for integrating
cultural competence during each of the six steps of the evaluation process. A list of related resources and
tools and an abbreviated version of this guide, titled Program Evaluation Tip Sheet: Integrating Cultural
Competence into Evaluation, are available as an appendix (http://www.cdc.gov/dhdsp/docs/
cultural_competence_tip_sheet ).
http://www.rwjf.org/content/dam/web-assets/2009/01
http://www.fhi360.org/sites/default/files/webpages/se-toolkit/quick-guide
http://www.fhi360.org/sites/default/files/webpages/se-toolkit/quick-guide
http://www.cdc.gov/dhdsp/docs/cultural_competence_guide
http://www.cdc.gov/dhdsp/docs/cultural_competence_tip_sheet
http://www.cdc.gov/dhdsp/docs/cultural_competence_tip_sheet
The Step-by-Step Guide to Evaluation | page 229
Frierson, H.T., Hood, S. & Hughes, G.B. (2002). A guide to conducting culturally responsive evaluations. In
The 2002 User-friendly Handbook for Evaluation (pp. 63-73). Arlington, VA: National Science Foundation.
Retrieved from http://www.nsf.gov/pubs/2002/nsf02057/nsf02057_5
This handbook, developed for evaluators and nonprofit organizations, reviews why evaluations must
consider the cultural context within which programs occur, and provides strategies for culturally
responsive evaluations.
Hood, S., Hopson, R. & Frierson, H.T. (Eds.) (2015). Continuing the journey to reposition culture and cultural
context in evaluation theory and practice. Greenwich, CT: Information Age.
This book contains one of the largest collections of works on evaluation in indigenous contexts and
settings to be found in a single edited volume. The authors attempt to answer questions about the
attributes of culturally responsive evaluation and how evaluators should exhibit cultural competence in
addition to their technical knowledge.
Inouye, T.E., Yu, H.C. & Adefuin, J. (2005). Commissioning multicultural evaluation: A foundation resource
guide. Oakland, CA: Social Policy Research Associates. Retrieved from http://leadershiplearning.org/
system/files/multicult_eval_rpt
This resource guide was developed to help foundation executives, program staff, and other stakeholders
to integrate a multicultural focus within their initiative and program evaluations. It synthesizes some of
the best learning about multicultural evaluation from field experience.
Lee, K. (2007). The Importance of Culture in Evaluation: A Practical Guide for Evaluators. Denver, CO:
Colorado Trust Foundation. Retrieved from http://www.communityscience.com/pubs/
crossculturalguide.r3
This guide written for evaluators focuses on three characteristics that influence interactions among
people and between people and evaluators: culture, social identity and privilege or power. There are
sample questions to help evaluators not make assumptions about people and instead, approach people
in a respectful way to find out about the norms and traditions that shape their worldviews. This guide
does not go into detail about evaluation methods or data collection.
Lee, K. (2009). The journey continues: Ensuring a cross-culturally competent evaluation. Denver, CO:
Colorado Trust Foundation. Retrieved from http://www.coloradotrust.org/attachments/0001/0089/
ensurecrossculturalcompetent-evaluation.indd
This document provides fictional case studies that show how evaluation can be cultural responsive as
well as sensitive to the people involved in the evaluation.
http://www.nsf.gov/pubs/2002/nsf02057/nsf02057_5
http://leadershiplearning.org/system/files/multicult_eval_rpt
http://leadershiplearning.org/system/files/multicult_eval_rpt
http://www.communityscience.com/pubs/crossculturalguide.r3
http://www.communityscience.com/pubs/crossculturalguide.r3
https://www.wcasa.org/file_open.php?id=861
https://www.wcasa.org/file_open.php?id=861
Page 230 | The Step-by-Step Guide to Evaluation
Racial Equity Lens
Bamberger, M. & Segone M. (2011). How to design and manage equity-focused evaluation. Retrieved
from http://betterevaluation.org/resource/guide/design_manage_equity_focused_evaluation
The document also explains how to conduct and design equity-focused evaluations under real-world
constraints.
Denzin, N.K., Lincoln, Y. S., & Smith, L.T. (2008). Handbook of critical and indigenous methodologies.
Thousand Oaks, CA: SAGE Publications.
This handbook covers everything from the history of critical and indigenous theory and how it came to
inform and impact qualitative research and indigenous peoples to the critical constructs themselves,
including race and diversity, gender representation (queer theory, feminism), culture, and politics to the
meaning of “critical” concepts within specific disciplines (critical psychology, critical communication/
mass communication, media studies, cultural studies, political economy, education, sociology, anthro-
pology, history, etc.)—all in an effort to define emancipatory research and explore what critical qualita-
tive research can do for social change and social justice.
Portland’s Partnership for Racial Equity. (2011). Racial equity strategy guide. Retrieved from
https://www.portlandoregon.gov/oni/article/505494
This guide was created to build capacity within the city of Portland to achieve equity on a day-to-day
basis. The guide was developed by engaging staff from key city bureaus to discuss how to
successfully operationalize the goal of equity into their daily work and pursue intentional goals and
measurable outcomes. This publication is meant to assist bureaus and decision-makers in developing
and using effective tools, which inform the city’s day-to-day actions of policy-making, resource
allocation, planning, program development and implementation, and evaluation.
Race and Social Justice Initiative. (2010). Racial equity toolkit for policies, programs, and budget. Retrieved
from http://www.seattle.gov/documents/departments/rsji/racialequitytoolkit_final_august2012
This toolkit, written for Seattle government employees, can be adapted by nonprofit organizations for
their evaluation. It lays out a process and a set of questions to guide the development, implementation,
and evaluation of policies, initiatives, programs, and budget issues to address the impacts on
racial equity.
Racial tools. (n.d.). Retrieved March 7, 2016 from http://www.racialequitytools.org/home
This website is a good resource for individuals and community groups working for change in their
communities, specifically with issues relating to race and racial equity. The site is written for users who
are new to evaluation, or perhaps those who give funds related to racial equity, but who are not yet clear
on how to evaluate it. The site discusses how to apply a “racialized” perspective to evaluation, meaning
using the ideas of racism, oppression, privilege and access to power as a lens through which evaluation
questions are developed and results are analyzed. The site is organized around the typical stages of
http://betterevaluation.org/resource/guide/design_manage_equity_focused_evaluation
https://www.portlandoregon.gov/oni/article/505494
http://www.seattle.gov/documents/departments/rsji/racialequitytoolkit_final_august2012
http://www.racialequitytools.org/home
The Step-by-Step Guide to Evaluation | page 231
evaluation: (A) Getting Ready: Defining Your Work; (B)Designing: Developing Evaluation Design and
Plan; (C) Collecting Information: Finding and Collecting Data; (D) Analyzing: Examining and Interpreting
Evaluation Information; (E) Sharing Findings: Sharing Outcome Information and Evaluation Findings;
and (F) Reflecting: Ways Evaluation Findings Can Improve the Work.
Hiring an Evaluator
Community Toolbox. (n.d.). Choosing evaluators. Retrieved March 7, 2016 from http://ctb.ku.edu/en/
table-of-contents/evaluate/evaluation/choose-evaluators/main
This section from the Community Toolbox is about starting the process of setting up an evaluation, i.e.,
choosing the evaluators who will carry it out, and planning what it will look like.
Hoefer, R. (2012). Successful program evaluation: Hiring an evaluator. Amazon Digital Services [Kindle]
Downloaded from Amazon.com.
This manuscript addresses one question that is very important to nonprofit managers and executive di-
rectors “How to hire a program evaluator?” It covers topics such as “who is an evaluator, why a nonprofit
needs an evaluator, interviewing an evaluator and mistakes that can get nonprofit organizations, its
director and board, and the evaluator in very serious trouble with the federal government.”
Juvenile Justice Evaluation Center (n.d.). Hiring and working and with an evaluator. Retrieved March 7, 2016
from http://www.jrsa.org/pubs/juv-justice/evaluator
The purpose of this briefing is to provide information to juvenile justice program managers about how
to go about hiring an evaluator. The briefing discusses how a qualified evaluator can assist a program
manager in assessing her or his program’s performance, what characteristics to look for in hiring a qual-
ified evaluator, and how to go about finding such a person.
Office of Minority Health. (n.d.). How to hire an evaluator. Retrieved March 3, 2016 from http://minority-
health.hhs.gov/assets/pdf/checked/1/howtohireanevaluator
This publication offers guiding principles for hiring an evaluator that is appropriate for an organization’s
project needs.
Using and Presenting Evaluation Findings
Baker, A. & Bruner, B. (n.d.). Evaluative thinking: Using evaluation findings. Cambridge, MA: Bruner
Foundation. Retrieved from http://www.evaluativethinking.org/docs/evaluativethinking.bulletin.6
This document explains multiple ways to communicate and use evaluation findings. It highlights how
results from surveys, interviews, observations, etc., can be incorporated into planning processes.
Finally, the document provides guidelines to staff or external evaluators on completing a formal
evaluation report.
http://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/choose-evaluators/main
http://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/choose-evaluators/main
http://Amazon.com
http://www.jrsa.org/pubs/juv-justice/evaluator
http://minorityhealth.hhs.gov/assets/pdf/checked/1/howtohireanevaluator
http://minorityhealth.hhs.gov/assets/pdf/checked/1/howtohireanevaluator
http://www.evaluativethinking.org/docs/evaluativethinking.bulletin.6
Page 232 | The Step-by-Step Guide to Evaluation
Centers for Disease Control and Prevention. (2013). Evaluation reporting: A guide to help ensure use of
evaluation findings. Retrieved from http://www.cdc.gov/dhdsp/docs/evaluation_reporting_guide
This guide is one in a series of Program Evaluation Guides developed by the CDC’s Division for Heart
Disease and Stroke Prevention (DHDSP) to assist CDC grantees in evaluating heart disease and stroke
prevention activities. While the guide is written for CDC grantees, nonprofit organizations will still find it
useful, as it focuses on ensuring evaluation use through evaluation reporting and addresses the
following topics: (1) key considerations for effectively reporting evaluation findings; (2) essential
elements for evaluation reporting; (3) importance of dissemination and (4) tools and resources.
Emery, A. (n.d.). Ann’s blog: Equipping you to collect, analyze, and visualize data [Blog archives].
Retrieved from http://annkemery.com/blog/
This blog about data visualization is useful to nonprofit organizations, as the writer provides tips on
presenting data in a pictorial or graphical format.
Evergreen, S. (n.d.). Evergreen data: Intentional reporting and data visualization [Blog archives].
Retrieved from http://stephanieevergreen.com/category/blog/
This blog about data visualization provides users with ways to make slopes in Excel and display data in
reports.
Evergreen, S. (2013). Presenting data effectively: Communicating your findings for maximum impact.
Thousand Oaks, CA: SAGE Publications.
This book focuses on the guiding principles of presenting data in evidence-based ways so that
audiences are effectively engaged and researchers are better understood. The author draws on her
extensive experience in the study of research reporting, interdisciplinary evaluation and data
visualization, as well as from diverse interdisciplinary fields, including cognitive psychology,
communications and graphic design, to extract tangible and practical data-reporting communication
lessons and insights. She then demonstrates how to apply those principles to the design of data
presentations to make it easier for the audience to understand, remember, and use the data.
Holm-Hansen, C. (2008). Communicating evaluation results. Wilder Research. Retrieved from
http://www.wilder.org/wilder-research/publications/studies/program%20evaluation%20and%20
research%20tips/communicating%20evaluation%20results%20-%20tips%20for%20conducting%20
program%20evaluation%20issue%2014,%20fact%20sheet
This tip sheet provides basic options for organizing and analyzing quantitative and qualitative data. It
also provides tips for writing reports.
The Canadian Institute of Chartered Accountants. (2011). Improved annual reporting by not-for-profit
organizations. Retrieved from http://www.cica.ca/focus-on-practice-areas/reporting-and-capital-
markets/item54324
https://www.cdc.gov/dhdsp/docs/evaluation_reporting_guide
http://annkemery.com/blog
http://stephanieevergreen.com/category/blog
http://www.wilder.org/wilder-research/publications/studies/program%20evaluation%20and%20research%20tips/communicating%20evaluation%20results%20-%20tips%20for%20conducting%20program%20evaluation%20issue%2014,%20fact%20sheet
http://www.wilder.org/wilder-research/publications/studies/program%20evaluation%20and%20research%20tips/communicating%20evaluation%20results%20-%20tips%20for%20conducting%20program%20evaluation%20issue%2014,%20fact%20sheet
http://www.wilder.org/wilder-research/publications/studies/program%20evaluation%20and%20research%20tips/communicating%20evaluation%20results%20-%20tips%20for%20conducting%20program%20evaluation%20issue%2014,%20fact%20sheet
https://www.cpacanada.ca/en/business-and-accounting-resources/financial-and-non-financial-reporting/not-for-profit-organizations/publications/annual-reporting-for-not-for-profit-organizations
https://www.cpacanada.ca/en/business-and-accounting-resources/financial-and-non-financial-reporting/not-for-profit-organizations/publications/annual-reporting-for-not-for-profit-organizations
The Step-by-Step Guide to Evaluation | page 233
This guide is designed to assist staff or volunteers responsible for preparing and approving the annual
report of a not-for-profit organization. It provides guiding principles for reporting, key evaluation
elements that should be addressed, and examples of report components from organizations of
different sizes.
The Institute for Urban Research. (2012). Making data driven decisions. Retrieved from http://community-
science.com/knowledge4equity/datadrivendecisionmakingworksheets
This presentation and supplementary piece educate users about the components of a data strategy:
accessing data, analyzing data, and using data to make data driven decisions.
Torres, R.T. (2012). Drawing conclusions from data and presenting them to others. Torres Consulting Group.
Retrieved from http://www.communityscience.com/knowledge4equity/DrawingConclusions
FromData
This presentation teaches users the basics on interpreting graphs and charts of data related to health
disparities, with the goal of presenting conclusions to others. This session also covers how
communication is used as a strategy for ending health disparities.
Torres, R.T., Preskill, H.S. & Piontek, M.E. (2004). Evaluation strategies for communicating and reporting:
Enhancing learning in organizations (Second edition). Thousand Oaks, CA: SAGE Publications.
This book includes worksheets and instructions for creating a detailed communicating and reporting
plan based on audience needs and characteristics. It also covers advances in technology including
website communications, web and videoconferencing and Internet chat rooms.
APPLIED EVALUATION RESOURCES BY TOPIC AREAS
This section provides resources on targeted evaluation methods and approaches that are applied to specific
bodies of work.
Advocacy and Policy
Beer, T., Ingargiola, P.S. & Beer, M.F. (2012). Advocacy and public policy grantmaking: Matching process to
purpose. Denver, CO: The Colorado Trust. Retrieved from http://www.evaluationinnovation.org/sites/
default/files/advocacy public policy grantmaking
This report is intended for both funders who are new to advocacy funding, and those who have been at
it for years. Using the information found within this report, funders can embark on an advocacy funding
strategy understanding more clearly what to expect for all stakeholders.
Centers for Disease Control and Prevention. (n.d.). Brief 1: Overview of policy evaluation. Retrieved from
https://www.cdc.gov/injury/pdfs/policy/brief%201-a
http://communityscience.com/knowledge4equity/datadrivendecisionmakingworksheets
http://communityscience.com/knowledge4equity/datadrivendecisionmakingworksheets
http://www.communityscience.com/knowledge4equity/DrawingConclusionsFromData
http://www.communityscience.com/knowledge4equity/DrawingConclusionsFromData
http://www.evaluationinnovation.org/sites/default/files/advocacy
http://www.evaluationinnovation.org/sites/default/files/advocacy
http://grantmaking
https://www.cdc.gov/injury/pdfs/policy/brief%201-a
Page 234 | The Step-by-Step Guide to Evaluation
This brief provides users with an overview of policy evaluation, including a description of the evaluation
framework used and a definition of policy. It also presents information about the general value and
potential challenges of conducting policy evaluation.
Coffman, J. (2009). A user’s guide to advocacy evaluation planning. Harvard Family Research Project.
Retrieved from http://www.hfrp.org/evaluation/publications-resources/a-user-s-guide-to-advocacy-
evaluation-planning
This guide was developed for advocates, evaluators, and funders who want guidance on how to evaluate
advocacy and policy change efforts. This guide is broken down into four steps: (1) identify how evaluation
will be used, (2) map the strategy, (3) prioritize the components and (4) identify measures and methods.
The guide recommends using data collection strategies, such as surveys, document review, observation
polling, focus groups, and case studies.
Coffman, J. (n.d.). Monitoring and evaluating advocacy: companion to the advocacy toolkit. UNICEF. Re-
trieved from http://www.unicef.org/cbsc/files/advocacy_toolkit_companion
This toolkit provides detailed steps, guidance and tools for developing and implementing an advocacy
strategy. The toolkit also outlines eight fundamental areas that can help strengthen an office’s capacity
for advocacy and covers several cross cutting aspects of advocacy including monitoring and evaluating
advocacy.
Coffman, J. & Beer, T. (2015). The Advocacy Strategy Framework: A tool for articulating an advocacy theory of
change. Retrieved from http://www.evaluationinnovation.org/sites/default/files/Adocacy%20
Strategy%20Framework
This brief offers a simple one-page tool for thinking about the theories of change that underlie public
policy advocacy strategies. It first presents the tool and then offers six questions that advocates, and
funders working with advocates, can work through to articulate better their theories of change.
Coffman, J. & Reed, E. (n.d.). Unique methods in advocacy evaluation. Retrieved from http://www.innonet.
org/resources/files/unique_methods_brief
This brief describes four methods (Bellwether methodology, policymaker ratings, intense period debriefs,
and system mapping) that were developed to respond to advocacy’s unique measurement challenges.
All four methods have been tested in real-life evaluations. The brief is targeted to a more advanced
evaluation audience.
Guthrie, K., Louie, J., David, T. & Foster, C.C. (2005). The challenge of assessing policy and advocacy
activities: Strategies for a prospective evaluation approach. Blueprint Research & Design, Inc. (funded by
and prepared for The California Endowment). Retrieved from http://www.arabellaadvisors.com/
wp-content/uploads/2012/03/challenge-policy-advocacy-activities
This paper presents a recommended approach to policy change evaluation and is written for
evaluation experts.
http://www.hfrp.org/evaluation/publications-resources/a-user-s-guide-to-advocacy-evaluation-planning
http://www.hfrp.org/evaluation/publications-resources/a-user-s-guide-to-advocacy-evaluation-planning
http://www.unicef.org/cbsc/files/advocacy_toolkit_companion
http://www.evaluationinnovation.org/sites/default/files/Adocacy%20Strategy%20Framework
http://www.evaluationinnovation.org/sites/default/files/Adocacy%20Strategy%20Framework
http://www.innonet.org/resources/files/unique_methods_brief
http://www.innonet.org/resources/files/unique_methods_brief
http://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/2005_-_Guthrie_-_The_challenge_of_assessing_policy_advocacy
http://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/2005_-_Guthrie_-_The_challenge_of_assessing_policy_advocacy
The Step-by-Step Guide to Evaluation | page 235
Guthrie, K., Louie, J., David, T. & Foster, C. C. (2006). The challenge of assessing policy and advocacy
activities: Part II - Moving from theory to practice. Blueprint Research & Design, Inc. (funded by and
prepared for The California Endowment). Retrieved from http://www.arabellaadvisors.com/wp-content/
uploads/2012/03/challenge-assessing-policy-advocacy-activities
Drawing on interviews with evaluation experts and endowment stakeholders, as well as a literature
review, this study identifies some of the key issues in evaluating work on policy change and advocacy.
The authors then present a framework for monitoring progress, assessing impact and deriving lessons
from this type of grantmaking.
Innovation Network. (n.d.). Pathfinder: A practical guide to advocacy evaluation. Retrieved from
http://www.innonet.org/resources/files/pathfinder_advocate_web
This guide is an introduction to advocacy evaluation from the advocate’s perspective. The guide seeks to
provide a sense of what is involved in advocacy evaluation. It also helps readers to know what to look for
during an evaluation.
John S. and James L. Knight Foundation. (2011). IMPACT: A Practical Guide to Evaluation Community
Information Projects. Retrieved from https://www.knightfoundation.org/media/uploads/publication_
pdfs/Impact-a-guide-to-Evaluating_Community_Info_Projects
This guide aims to help organizations collect useful information about the effectiveness and impact of
their community information projects by highlighting aspects of the evaluation process that are unique,
challenging, or critical in a community information context. It also describes and includes relevant and
meaningful tools that assess community information projects.
Lynn, J. (2014). Assessing and Evaluating Change in Advocacy Fields. Spark Policy Institute. Retrieved from
http://www.evaluationinnovation.org/sites/default/files/Spark-Evaluating_Change_In_Advocacy_
Fields
This white paper aims to bring together emerging ideas about how to assess advocacy fields and
evaluate advocacy field building initiatives.
Mansfield, C. (2010). Monitoring & evaluation of advocacy campaigns literature review. Ecumenical Advocacy
Alliance. Retrieved from http://actknowledge.org/resources/documents/monitoringandevaluationofad-
vocacycampaignsliteraturereview .
This report summarizes a literature review that was undertaken by the Ecumenical Advocacy Alliance
and its members in preparation for developing a monitoring and evaluation tool for advocacy work. The
topics covered range from the basics of organizing and conducting an evaluation to theories of policy
and social change. Advocacy strategies and the best practices for assessing progress on indicators that
are appropriate for a given campaign are also discussed. Realizing the importance of social equity and
cultural competency during evaluation is briefly mentioned.
http://www.pointk.org/resources/files/challenge_assessing_policy_advocacy2
http://www.pointk.org/resources/files/challenge_assessing_policy_advocacy2
http://www.pointk.org/resources/files/pathfinder_advocate_web
https://www.knightfoundation.org/media/uploads/publication_pdfs/Impact-a-guide-to-Evaluating_Community_Info_Projects
https://www.knightfoundation.org/media/uploads/publication_pdfs/Impact-a-guide-to-Evaluating_Community_Info_Projects
http://www.evaluationinnovation.org/sites/default/files/Spark-Evaluating_Change_In_Advocacy_Fields
http://www.evaluationinnovation.org/sites/default/files/Spark-Evaluating_Change_In_Advocacy_Fields
http://actknowledge.org/resources/documents/monitoringandevaluationofadvocacycampaignsliteraturereview
http://actknowledge.org/resources/documents/monitoringandevaluationofadvocacycampaignsliteraturereview
Page 236 | The Step-by-Step Guide to Evaluation
Organizational Research Services. (2009). Ten considerations for advocacy evaluation planning: Lessons
leared from Kids Count grantee experiences. Anne E. Casey Foundation. Retrieved from
http://www.organizationalresearch.com/publicationsandresources/ten_considerations_for_
advocacy_evaluation_planning
This guide explores ways to think about evaluation of advocacy and policy work and presents a
framework to name advocacy and policy outcomes as well as broad directions for evaluation of advocacy
and policy efforts.
Pankaj, V., Athanasiades, K., & Emery, A. (2014). Coalition assessment: Approaches for measuring capacity
and impact. Retrieved from http://www.innonet.org/resources/files/innonet-coalition-assessment
This resource guide provides practitioners and funders with insights into the coalition assessment
process along with concrete examples and lessons learned from other studies.
Reisman, J., Gienapp, A., & Stachowiak, S. (2007). A Guide to Measuring Advocacy and Policy.
Organizational Research Services for Annie E. Casey Foundation. Retrieved from http://www.aecf.org/m/
resourcedoc/aecf-aguidetomeasuringpolicyandadvocacy-2007
This guide helps grantmakers think and talk about measurement of advocacy and policy. The guide puts
forth a framework for naming outcomes associated with advocacy and policy work as well as directions
for evaluation design. The framework is intended to provide common ways to identify and talk about
outcomes, providing philanthropic and nonprofit audiences an opportunity to react to, refine, and adopt
the outcome categories presented. In addition, grantmakers can consider some key directions for
evaluation design that include a broad range of methodologies, intensities, audiences, timeframes
and purposes.
Reisman, J., Gienapp, A., & Stachowiak, S. (n.d.). A handbook of data collection tools: A companion to “A
guide to measuring advocacy and policy.” Annie E. Casey Foundation. http://orsimpact.com/wp-
content/uploads/2013/08/a_handbook_of_data_collection_tools
This guide provides perspectives on evaluation of advocacy and policy. It also provides users with
examples of practical tools and processes for collecting useful information from policy and advocacy
efforts.
Workforce Development and Employment
Austrian, Z. & Norton, J. (2002). What Works in Economic Development Practice? John S. and James L. Knight
Foundation. Retrieved from http://cua6.urban.csuohio.edu/economicdevelopment/knight/what_works_
final
This document reviews evaluation literature relevant to economic development strategies. A “what
works” matrix follows the review of the evaluation studies. The matrix highlights significant findings and
key lessons learned from this work.
http://www.organizationalresearch.com/publicationsandresources/ten_considerations_for_advocacy_evaluation_planning
http://www.organizationalresearch.com/publicationsandresources/ten_considerations_for_advocacy_evaluation_planning
http://www.innonet.org/resources/files/innonet-coalition-assessment
http://www.aecf.org/m/resourcedoc/aecf-aguidetomeasuringpolicyandadvocacy-2007
http://www.aecf.org/m/resourcedoc/aecf-aguidetomeasuringpolicyandadvocacy-2007
http://orsimpact.com/wp-content/uploads/2013/08/a_handbook_of_data_collection_tools
http://orsimpact.com/wp-content/uploads/2013/08/a_handbook_of_data_collection_tools
http://cua6.urban.csuohio.edu/economicdevelopment/knight/what_works_final
http://cua6.urban.csuohio.edu/economicdevelopment/knight/what_works_final
The Step-by-Step Guide to Evaluation | page 237
Center for Energy Workforce Development (n.d.). Workforce development evaluation toolkit. Retrieved
from http://www.cewd.org/workforce/evaltoolkit
This toolkit is designed to evaluate workforce development efforts. The evaluation process described in
this tool kit can be used as a framework for measuring any workforce strategy. Included in this
publication are steps on how to use the toolkit, definitions of key terms, a sample survey, and examples
of data.
Weigensberg, E., Schlecht, C., Laken, F., Goerge, Ro., Stagner, M., Ballard, P., & DeCoursey, J. (2012). Inside
the Black Box: What Makes Workforce Development Programs Successful? Chapin Hall at the University
of Chicago. Retrieved from http://www.chapinhall.org/sites/default/files/Inside%20the%20Black%20
Box_04_23_12_0
This document can be used as a reference by community organizations as it reports on a study that looks
at several successful programs in Chicago. It attempt to understand the factors that may explain why the
programs achieved different types of successes, and how these factors may be quantified or measured to
help improve the system.
Financial Literacy and Family Asset-building
Anders, J., Graddy, S., Grieve, M., & Visser, D. (2011). Measuring outcomes of financial capability
programs: Success Measures tools for practitioners. Success Measures. Retrieved from
http://www.successmeasures.org/sites/all/files/financialcapabilityonlinefinaloct2011
This guide introduces new Success Measures® outcome indicators and data collection tools that can be
used by nonprofit practitioners to measure the impact and effectiveness of an array of financial capability
programs. The guide also describes how and why these tools were created.
The Government of Canada. (2010). Detailed Guide to Evaluating Financial Education Programmes.
Organisation for Economic Cooperation and Development (OECD) International Network on Financial
Education (INFE). Retrieved from http://www.oecd.org/finance/financial-education/49994090
This web document provides users with a better understanding of some key evaluation concepts and how
they can be applied to financial education projects, programs and initiatives. The document also
supplies information on how to design and implement an evaluation.
Jayaratne, K.S.U., Lyons, A.C., & Palmer, L. (n.d.). Financial education evaluation manual. National
Endowment for Financial Education. Retrieved from http://toolkit.nefe.org/portals/0/toolkit-manual
This online database and companion manual (available in print or electronic format) is designed to help
financial educators understand evaluation concepts and apply them in educational program evaluation.
Specifically, the toolkit provides assistance to financial educators who are seeking help to evaluate and
document the impact of their educational programs.
http://www.cewd.org/workforce/evaltoolkit
http://www.chapinhall.org/sites/default/files/Inside%20the%20Black%20Box_04_23_12_0
http://www.chapinhall.org/sites/default/files/Inside%20the%20Black%20Box_04_23_12_0
http://www.successmeasures.org/sites/all/files/financialcapabilityonlinefinaloct2011
http://www.oecd.org/finance/financial-education/49994090
http://toolkit.nefe.org/portals/0/toolkit-manual
Page 238 | The Step-by-Step Guide to Evaluation
Ruskin, L., & Chiem, N. (2013). Implementing evaluation in financial coaching: An exploratory case study.
Retrieved from https://www.earn.org/wp-content/uploads/2015/06/130920_earn_financial_coaching_
evaluation_case_study_vf
This report discusses some challenges that can arise when evaluating financial programs and provides
solutions.
Early Childhood Education
California Department of Education. (2000). Handbook on assessment and evaluation in early childhood
special education programs. Retrieved from http://www.cde.ca.gov/sp/se/fp/documents/ecassmnt .
This handbook is intended to provide users with a resource that presents quality criteria for best
practices in program development, ideas, concepts, and evaluation in the context of the statutory
requirements for early childhood special education programs.
Friese, S., King, C., & Tout, K. (2013). INQUIRE Data Toolkit. OPRE Report # 2013-58. Washington, DC: Office of
Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health
and Human Services. Retrieved from http://www.acf.hhs.gov/sites/default/files/opre/inquire_data_
toolkit_final_dec_2013_submitted_1_8_13
This toolkit is designed to provide tools that can be used by staff to support effective data collection and
the use of data to answer important policy and reporting questions through the use of common data
elements.
Hepburn, K.S., Kaufmann, R.K., Perry, D.F., Allen, M.D., Brennan, E.M., & Green, B. L. (2007). Early
childhood mental health consultation: An evaluation tool kit. Washington, DC: Georgetown
University, Technical Assistance Center for Children’s Mental Health; Johns Hopkins University,
Women’s and Children’s Health Policy Center; and Portland State University, Research and Training
Center on Family Support and Children’s Mental Health. Retrieved from http://gucchd.georgetown.edu/
products/ecmhctoolkit
This toolkit is intended to increase the capacity for high-quality evaluation of early childhood mental
health consultation (ECMHC) in community based settings. The toolkit also provides states,
communities, programs and grant-funded projects that are developing or have developed early
childhood mental health consultation programs, guidance, tools and resources that will assist them in
designing and implementing program evaluations.
Lugo-Gil, J., Sattar, S., Ross, C., Boller, K., Tout, K, & Kirby, G. (2011). The Quality Rating and Improvement
System (QRIS) evaluation toolkit. OPRE Report 2011-31. Washington, DC: U.S. Department of Health and
Human Services, Administration for Children and Families, Office of Planning, Research and Evaluation.
Retrieved from https://www.acf.hhs.gov/sites/default/files/opre/qris_toolkit
This toolkit is intended to serve as an informational resource for state administrators, child care and early
education practitioners, and other stakeholders on how Quality Rating and Improvement Systems (QRISs)
https://www.earn.org/wp-content/uploads/2015/06/130920_earn_financial_coaching_evaluation_case_study_vf
https://www.earn.org/wp-content/uploads/2015/06/130920_earn_financial_coaching_evaluation_case_study_vf
http://www.seedsofpartnership.org/pdf/assessmentEvaluation
http://www.acf.hhs.gov/sites/default/files/opre/inquire_data_toolkit_final_dec_2013_submitted_1_8_13
http://www.acf.hhs.gov/sites/default/files/opre/inquire_data_toolkit_final_dec_2013_submitted_1_8_13
http://gucchd.georgetown.edu/products/ecmhctoolkit
http://gucchd.georgetown.edu/products/ecmhctoolkit
https://www.acf.hhs.gov/sites/default/files/opre/qris_toolkit
The Step-by-Step Guide to Evaluation | page 239
work. The authors also discuss why it is important to conduct evaluation of QRISs and on how to plan and
design an evaluation of QRISs.
Slentz, K.L., Early, D.M., & McKenna, M. (2008). A guide to assessment in early childhood: Infancy to age
eight. Washington State Office of Superintendent of Public Instruction. Retrieved from
http://www.k12.wa.us/earlylearning/pubdocs/assessment_print
This guide is intended as a resource for designing professional development activities for program
administrators and direct service staff responsible for gathering and interpreting assessment
information.
UNICEF. (2012). A framework and tool box for monitoring and improving quality. Retrieved from
http://www.unicef.org/ceecis/ecd_framework_part_ii_june3
The objective of this document is to provide a comprehensive framework with tools for monitoring and
evaluating the quality of early childhood education services as a means to improving access to and avail-
ability of such services.
K-12 and Higher Education
Baltimore City Public School System. (2003). Performance-based evaluation handbook. Retrieved from
http://www.nctq.org/evaluation_handbook/34-07
This handbook provides teacher level employees with an overview of the benefits of evaluation for
improving education. While the handbook is written for teacher level employees, it is still useful to
nonprofit organizations seeking to evaluate their education programs.
Greene, J., Boyce, A., & Ahn J. A values-engaged, educative approach for evaluating education programs: A
guidebook for practice. University of Illinois. Retrieved from http://comm.eval.org/higherlogic/system/
downloaddocumentfile.ashx?documentfilekey=75bc9c3b-b169-4529-b2d3-642056d95f35
This guidebook presents practical guidelines for evaluators of education programs. It presents these
guidelines within a “values-engaged, educative” framework for evaluation. The guide is targeted toward
a beginner level audience and guides the reader through a step-by-step lesson on designing a successful
evaluation. Data collection methods are not discussed in this publication, but the benefits of evaluation
are highlighted throughout the guidebook.
Kauerz, K., & Coffman, J. (2013). Framework for planning, implementing, and evaluating PreK-3rd grade
approaches. Retrieved from http://depts.washington.edu/pthru3/PreK-3rd_Framework_11x17
This document offers a framework that is intended to be referenced and used over an extended period
of time for reflection, self-evaluation, and improvement of PreK-3rd grade efforts. This framework helps
to address key questions facing those who are developing PreK-3rd grade approaches in their school,
districts and communities.
http://www.k12.wa.us/earlylearning/pubdocs/assessment_print
http://www.unicef.org/ceecis/ecd_framework_part_ii_june3
http://www.nctq.org/evaluation_handbook/34-07
http://comm.eval.org/higherlogic/system/downloaddocumentfile.ashx?documentfilekey=75bc9c3b-b169-4529-b2d3-642056d95f35
http://comm.eval.org/higherlogic/system/downloaddocumentfile.ashx?documentfilekey=75bc9c3b-b169-4529-b2d3-642056d95f35
http://depts.washington.edu/pthru3/PreK-3rd_Framework_11x17
Page 240 | The Step-by-Step Guide to Evaluation
Opportunity Culture. (2012). Redesigning schools to reach every student with excellent teachers. Retrieved
from http://opportunityculture.org/wp-content/uploads/2012/05/selection_development_evaluation_
toolkit-public_impact
This toolkit includes job descriptions, competencies, and companion tools that can be used to select,
evaluate and develop teachers and staff.
Paulsen, C.A. & Dailey, D. (2002). A guide for education personnel: Evaluating a program or intervention.
Elementary and Middle Schools Technical Assistance Center (EMSTAC). Washington, DC: American
Institutes for Research. Retrieved from http://www.emstac.org/resources/eval
This guide provides school, district, and state personnel with an overview of the evaluation process. The
guide describes what is entailed in an evaluation and issues to be aware of when planning one. It also
provides steps to help evaluators get started in the planning process and identify areas where one may
need assistance.
Rockwood School District. (2007). Rockwood School District program evaluation plan. Retrieved from
http://www.rsdmo.org/dataquality/Program%20Evaluation%20Documents/Program%20Evaluation
%20Plan
This evaluation plan document is intended to be used as a sample document by evaluators working with
school district programs.
State University of New York. (2012). Guide for the Evaluation of Undergraduate Programs. Retrieved from
http://www.oneonta.edu/academics/assessment/pdfs/ufsguidelines
This guide outlines how to proceed with an academic program evaluation. The document provides
guidance, advice, and direction for every individual, department, governance, and administrator
involved in the evaluation process and a set of useful references and highly relevant appendices.
University of North Carolina Chapel Hill. (2012). Instruction manual: Toolkit for evaluating K-12
International Outreach Programs. Retrieved from http://cgi.unc.edu/uploads/media_items/k-12-
outreach-program-evaluation-manual.original
This manual is designed to help K-12 International Outreach Programs utilize evaluation to answer
questions about their programs. Used in conjunction with the companion Evaluation Tool Kit on their
website, the manual helps users develop basic competence and confidence in designing and
implementing evaluations that address particular program needs.
Health
Centers for Disease Control and Prevention. Framework for program evaluation in public health. (MMWR
1999:48, No. RR-11). Retrieved from ftp://ftp.cdc.gov/pub/Publications/mmwr/rr/rr4811
This document provides a framework to guide public health professionals in their use of program
evaluation. The framework is a practical, non-prescriptive tool, designed to summarize and organize
http://opportunityculture.org/wp-content/uploads/2012/05/selection_development_evaluation_toolkit-public_impact
http://opportunityculture.org/wp-content/uploads/2012/05/selection_development_evaluation_toolkit-public_impact
http://www.emstac.org/resources/eval
http://www.rsdmo.org/dataquality/Program%20Evaluation%20Documents/Program%20Evaluation%20Plan
http://www.rsdmo.org/dataquality/Program%20Evaluation%20Documents/Program%20Evaluation%20Plan
http://www.oneonta.edu/academics/assessment/pdfs/ufsguidelines
http://cgi.unc.edu/uploads/media_items/k-12-outreach-program-evaluation-manual.original
http://cgi.unc.edu/uploads/media_items/k-12-outreach-program-evaluation-manual.original
ftp://ftp.cdc.gov/pub/Publications/mmwr/rr/rr4811
The Step-by-Step Guide to Evaluation | page 241
essential elements of program evaluation. It comprises steps in program evaluation practice and
standards for effective program evaluation.
Centers for Disease Control and Prevention. Introduction to program evaluation for public health
programs: A self-study guide. Retrieved from http://www.cdc.gov/eval/guide/cdcevalmanual
This “how to” guide is intended to assist managers and staff of public, private, and community public
health programs to plan, design, implement and use comprehensive evaluations in practical ways. The
strategies presented in this manual will help ensure that evaluations meet the diverse needs of internal
and external stakeholders. Such needs include (A) assessing and documenting program implementation,
outcomes, efficiency and cost-effectiveness of activities and (B) taking action based on evaluation results
to increase the impact of programs.
Cullen, A., Gilles, T., & Rosenthal, J. (2006). Evaluating community-based child health promotion pro-
grams: A snapshot of strategies and methods. Retrieved from http://www.nashp.org/sites/default/files/
community_health_promotion
This report provides practical information for community groups and states developing evaluation
components of community-based projects that focus on children’s health promotion. The report gives a
snapshot of seven projects nationwide. It is not intended to provide a theoretical discussion of evaluation
methods or to serve as a comprehensive review of evaluation programs.
Gage, A.J., Ali, D., & Suzuki, C. (2005). A Guide for Monitoring and Evaluating Child Health Programs.
Retrieved from http://www.coregroup.org/storage/documents/Workingpapers/ms-05-15
This guide for international organizations provides a comprehensive listing of commonly used indica-
tors for monitoring and evaluating child health programs in developing countries. The guide provides a
succinct but thorough overview of monitoring and evaluation, describing program components (inputs,
processes, outputs, and outcomes), evaluation approaches, indicators and data sources. Commonly
used indicators and a description for each are listed for various child health topics, such as newborn
health, immunization and mortality.
National Network for Oral Health Access. (2015). A user’s guide for implementation of interprofessional oral
health core clinical competencies: Results of a pilot project. Retrieved from http://www.nnoha.org/
nnoha-content/uploads/2015/01/IPOHCCC-Users-Guide-Final_01-23-2015
This guide helps public health professionals in their use of program evaluation. This guide provides a
structure, options and suggestions to help health centers develop programs to implement oral health
competencies that integrate oral health care into primary health care, increasing access to oral health
care and improving the oral health status of the populations the health centers serve.
Organizational Research Services. (2000). Community roots for oral health - Guidelines for successful coa-
litions, supplement: Selecting and evaluating outcomes for oral health coalition efforts. Retrieved from
http://www.doh.wa.gov/portals/1/documents/pubs/160-142-coalitions-apx4
https://www.cdc.gov/eval/guide/cdcevalmanual
http://www.nashp.org/sites/default/files/community_health_promotion
http://www.nashp.org/sites/default/files/community_health_promotion
http://www.coregroup.org/storage/documents/Workingpapers/ms-05-15
http://www.nnoha.org/nnoha-content/uploads/2015/01/IPOHCCC-Users-Guide-Final_01-23-2015
http://www.nnoha.org/nnoha-content/uploads/2015/01/IPOHCCC-Users-Guide-Final_01-23-2015
http://www.doh.wa.gov/portals/1/documents/pubs/160-142-coalitions-apx4
Page 242 | The Step-by-Step Guide to Evaluation
This guide is intended to accompany the Community Roots for Oral health—Guidelines for Successful
Coalitions and serves as a primer on selecting and evaluating outcomes for community-based oral health
coalition efforts.
The Center for the Advancement of Community Based Public Health (2000). An evaluation framework for
community health programs. Retrieved from http://prevention.sph.sc.edu/Documents/CENTERED%20
Eval_Framework
This document presents a framework that emphasizes program evaluation as a practical and ongoing
process that involves program staff and community members along with evaluation experts. The overall
goal of the framework is to help guide and inform the evaluation process.
Van Marris, B. & King, B. (2007). Evaluating health promotion programs. Centre for Health Promotion at
the University of Toronto. Retrieved from http://www.blazesports.org/wp-content/uploads/2011/02/
THCU-Evaluation-Workbook
This document assists health promotion practitioners in the development and implementation of
program evaluations and is applicable to all other evaluators. The workbook describes three main types
of evaluation: formative, process and summative. Readers can use this document to develop an
understanding of evaluation methods, data, and data collection tools such as interviews and
questionnaires. Social equity is only briefly mentioned while the benefits of evaluation are discussed
throughout the document.
World Health Organization, UNICEF, UC Davis, USAID, & IFPRI. (2007). Indicators for assessing infant
and young child practices, part 1: Definitions. Retrieved from http://whqlibdoc.who.int/
publications/2008/9789241596664_eng
This report summarizes the discussions and consensus reached on eight core indicators and seven
optional indicators for assessing infant and young child feeding practices that are population-based and
can be derived from household survey data.
Poverty
Baker, J.L. (2000). Evaluating the Impact of Development Projects on Poverty. World Bank. Retrieved from
http://siteresources.worldbank.org/INTISPMA/Resources/handbook
This guide provides project managers and policy analysts with the tools needed for evaluating project
impact when working with individuals living in poverty. It is aimed at readers with a general knowledge
of statistics. The publication focuses on the main components needed in planning impact evaluation.
It discusses various data collection methods although no mention is made about social equity or
cultural sensitivity. A large portion of the report focuses on the benefits of evaluation in regards to
poverty reduction.
http://prevention.sph.sc.edu/Documents/CENTERED%20Eval_Framework
http://prevention.sph.sc.edu/Documents/CENTERED%20Eval_Framework
http://www.blazesports.org/wp-content/uploads/2011/02/THCU-Evaluation-Workbook
http://www.blazesports.org/wp-content/uploads/2011/02/THCU-Evaluation-Workbook
http://whqlibdoc.who.int/publications/2008/9789241596664_eng
http://whqlibdoc.who.int/publications/2008/9789241596664_eng
http://siteresources.worldbank.org/INTISPMA/Resources/handbook
The Step-by-Step Guide to Evaluation | page 243
Ravallion, M., (1996) Issues in measuring and modeling poverty. Economic Journal, 106, 1328-44.
This journal article discusses the empirical problems that confound measures of poverty. The authors
then provide recommendations to resolving this and other additional measures of poverty.
Youth Development
Harris, E. (2011). Afterschool evaluation 101: How to evaluate an expanded learning program. Harvard
Family Research Project. Retrieved from http://www.k12.wa.us/21stcenturylearning/pubdocs/after-
schoolevaluation101
This document is designed to help out-of-school time program directors who have little or no evaluation
experience develop an evaluation strategy. The guide recommends using a variety of data collection
methods, including case studies, document review, observation, tests and interviews. Evaluation
approach and methods are clearly stated in this document, although no information on social equity
or cultural sensitivity is included. Throughout this publication, the benefits of evaluation are
frequently discussed.
Lorimer, J. (2006). Demystifying outcomes: Demonstrating results in youth development programs. Fund for
the City of New York, Youth Development Institute. Retrieved from http://fred.fcny.org/ydipubs/pubs/
demystifyingoutcomes
This paper explains the importance of evaluation and presents strategies for identifying appropriate
outcomes for youth development programs.
Yohalem, N. and Wilson-Ahlstrom, A. (with Fischer, S. and Shinn, M.). (2009). Measuring youth program
quality: A guide to assessment tools (Second edition). Washington, DC: The Forum for Youth Investment.
Retrieved from http://forumfyi.org/files/MeasuringYouthProgramQuality_2ndEd
This compendium provides useful guidance to practitioners, policy makers, and evaluators as to what
options are available and what issues to consider when selecting and using a quality assessment tool.
It focuses on the purpose and history, content structure and methodology, technical properties and user
considerations for each of the instruments included, as well as a brief description of how they are being
used in the field.
SECONDARY DATA SOURCES
Juvenile Justice
Office of Juvenile Justice and Delinquency Prevention (OJJDP)
Easy Access to Juvenile Populations. http://www.ojjdp.gov/ojstatbb/ezapop/default.asp
This site provides access to national, state and county level population data detailed by age, sex, race,
and ethnicity. Users can create detailed population profiles for a single jurisdiction or create state com-
parison or county comparison tables.
http://www.k12.wa.us/21stcenturylearning/pubdocs/afterschoolevaluation101
http://www.k12.wa.us/21stcenturylearning/pubdocs/afterschoolevaluation101
https://static1.squarespace.com/static/55e5e13ee4b0300afc316341/t/56731785d8af10a9410d0b2a/1450383237029/Demystifying+Outcomes+Booklet+Layout+5+14+07+
https://static1.squarespace.com/static/55e5e13ee4b0300afc316341/t/56731785d8af10a9410d0b2a/1450383237029/Demystifying+Outcomes+Booklet+Layout+5+14+07+
http://forumfyi.org/files/MeasuringYouthProgramQuality_2ndEd
http://www.ojjdp.gov/ojstatbb/ezapop/default.asp
Page 244 | The Step-by-Step Guide to Evaluation
Statistical Briefing Book. https://www.ojjdp.gov/ojstatbb/
This site includes links to data on juveniles as victims, juveniles as offenders, juvenile justice system
structure and process, law enforcement and juvenile crime, juveniles in court, juveniles on probation,
juveniles in corrections and juvenile reentry and aftercare.
The National Juvenile Court Data Archive. https://www.ojjdp.gov/ojstatbb/njcda/default.asp
This site is a good source for statistics on juvenile court, available in number form on the Easy Access
page listed above, but the National Juvenile Court Data Archive provides the stat book for download in
PDF, which is a little easier to digest. The data can also be downloaded for any state at
http://www.ojjdp.gov/ojstatbb/njcda/default.asp.
The Juvenile Justice Information Exchange. http://jjie.org/
This site provides data and basic information on every juvenile justice system in the country.
U.S. Department of Education, Office of Civil Rights. http://ocrdata.ed.gov/
This site provides data collected on key education and civil rights issues in our nation’s public schools
for use by Office of Civil Rights in its enforcement and monitoring efforts regarding equal educational
opportunity. The Civil Rights Dara Collection (CRDC) is also a tool for other department offices, federal
agencies, policymakers, researchers, educators, school officials, and the public to analyze student equity
and opportunity.
Early Childhood Education
Child Care and Early Education Research Connections.http://www.researchconnections.org/childcare/
welcome
This site offers a broad spectrum of research data on child care and early education and related policies.
To make use of this data, the user needs statistical software and a good understanding of statistical
methodology.
The Early Childhood Data Collaborative. http://www.ecedata.org/
This site provides tools and data resources to encourage state policy change. The site also provides a
national forum to support the development and use of coordinated state early childhood data
collaborative data systems.
Education Resource Information Center (ERIC). http://eric.ed.gov/
This database provides access to information from journals included in the Current Index of Journals in
Education and Resources in Education Index.
National Center for Education Statistics, Early Childhood Longitudinal Program. https://nces.ed.gov/ecls/
This site provides national data on children’s status at birth and at various points thereafter; children’s
https://www.ojjdp.gov/ojstatbb/
https://www.ojjdp.gov/ojstatbb/njcda/default.asp
http://www.ojjdp.gov/ojstatbb/njcda/default.asp
http://jjie.org/
http://ocrdata.ed.gov
http://www.researchconnections.org/childcare/welcome
http://www.researchconnections.org/childcare/welcome
http://www.ecedata.org/
http://eric.ed.gov
https://nces.ed.gov/ecls
The Step-by-Step Guide to Evaluation | page 245
transitions to nonparental care, early education programs, and school; and children’s experiences and
growth through the eighth grade. The site also provides data that can be used to analyze the
relationships among a wide range of family, school, community, and individual variables with children’s
development, early learning, and performance in school.
National Center for Education Statistics, Institute of Education Sciences’ DAS: Data Analysis System.
http://nces.ed.gov/das/
This web-based data analysis software provides public access to mostly individual-level education
survey data collected by the U.S. Department of Education. Users can build their own analysis tables and
covariance matrices for regression analysis. With DAS 2.0, users are able to perform weighted
least-squares and logistic regression analysis. Step-by-step tutorials are available for using these data.
National Center for Education Statistics, Institute of Education Sciences Peer Analysis System (PAS) and
Dataset Cutting Tool (DCT). http://nces.ed.gov/ipeds/
This site provides an essential tool for performing institutional level comparisons and for creating
specific data “slices” using the Integrated Postsecondary Data System (IPEDS). PAS provides a variety of
analytical features for peer analysis. This includes the ability to create new calculated variables, to sort
and rank schools based on the data items selected, and to view standard report templates. DCT allows
users to download IPEDS data to use with software packages for analysis and comparisons. The DCT can
also be used to obtain complete data files and codes or to create customized datasets according to your
specifications. Users may select their schools of interest with the DCT or may upload a list previously
created in PAS.
UNICEF. http://data.unicef.org/ecd/early-childhood-education
This site acts as the main source of nationally representative and comparable data on attendance in early
childhood care and education programs. Users can download Microsoft Excel files containing data from
the site.
K-12 and Higher Education
Data.Gov — Education http://www.data.gov/education/
This site includes multiple databases. Datasets cover education at all levels.
Education Resource Information Center (ERIC). http://eric.ed.gov/
This database provides access to information from journals included in the Current Index of Journals in
Education and Resources in Education Index.
National Center for Education Statistics. http://nces.ed.gov/datatools/
This site acts as the primary federal entity for collecting and analyzing data related to education. Under
the tab for Data & Tools, NCES offers custom datasets and tools, state/district profiles, and more.
http://nces.ed.gov/das
http://nces.ed.gov/ipeds
http://data.unicef.org/ecd/early
http://Data.Gov
http://www.data.gov/education
http://eric.ed.gov
http://nces.ed.gov/datatools
Page 246 | The Step-by-Step Guide to Evaluation
U.S. Department of Education, Office of Civil Rights. http://ocrdata.ed.gov/
This site includes results from data on key education and civil rights issues in our nation’s public schools
for use by Office of Civil Rights in its enforcement and monitoring efforts regarding equal educational
opportunity. The Civil Rights Data Collection is also a tool for other department offices, federal agencies,
policymakers, researchers, educators, school officials, and the public who wish to access and analyze
data on student equity and opportunity.
Health
CDC Behavioral Risk Factor Surveillance System (BRFSS).
http://www.cdc.gov/brfss/data_documentation/
This resource offers information and links to the BRFSS data. The BRFSS collects health-related state data
through telephone surveys of U.S. residents regarding their health-related risk behaviors, chronic health
conditions and use of preventive services. BRFSS collects data in all 50 states as well as the District of
Columbia and three U.S. territories. BRFSS is the largest continuously conducted health survey system in
the world, interviewing over 400,000 adults each year.
CDC Wide-ranging Online Data for Epidemiologic Research (WONDER). http://wonder.cdc.gov
This site is an easy-to-use, menu driven system that makes the information resources of the CDC
available to public health professionals and the public at large. The system allows users to access
statistical research data published by CDC, as well as reference materials, reports and guidelines on
health-related topics.
Community Health Status Indicators (CHSI). http://wwwn.cdc.gov/communityhealth
This an interactive web application that produces health profiles for all 3,143 counties in the United
States. Each profile includes key indicators of health outcomes, which describe the population health
status of a county and factors that have the potential to influence health outcomes, such as health care
access and quality, health behaviors, social factors and the physical environment.
County Health Rankings. http://www.countyhealthrankings.org/
This website provides access to 50 state reports, ranking each county within the 50 states according to its
health outcomes and the multiple health factors that determine a county’s health.
Data.Gov — Health. https://www.data.gov/health/
This site includes links to multiple databases on health.
Data Set Directory of Social Determinants of Health at the Local Level. http://www.communityscience.com/
knowledge4equity/DataSetDirectory
This guide contains a list of existing data sets that exist and can be used to address determinants of
health. The data are organized into 12 dimensions (categories) of the social environment—economy,
http://ocrdata.ed.gov/
http://www.cdc.gov/brfss/data_documentation
http://wonder.cdc.gov
http://wwwn.cdc.gov/communityhealth
http://www.countyhealthrankings.org
http://Data.Gov
https://www.data.gov/health/
The Step-by-Step Guide to Evaluation | page 247
employment, education, political, environmental, housing, medical, governmental, public health, psy-
chosocial, behavioral and transport.
FastStats. https://www.cdc.gov/nchs/fastats/
This site provides access to statistics on topics of public health importance and is organized
alphabetically. Links are provided to publications that include the statistics presented and to sources to
more data and related web pages.
Food Environment Atlas. http://www.ers.usda.gov/foodatlas/
This site provides links to data on food choice and health. Users can create maps showing variation in a
single indicator across the United States, view all county-level indicators for a selected county and
identify counties sharing the same degree of multiple indicators.
Health Data Interactive (HDI). www.cdc.gov/nchs/hdi.htm
This web-based application provides access to pretabulated national data. The site provides interactive
access to a broad set of key public health statistics. The primary objective is to provide national estimates
of public health measures cross tabulated by a common set of variables. HDI also aims to educate users
about the data and data systems available from the National Center of Health Statistics.
Health Indicators Warehouse. http://www.healthindicators.gov/
This site provides access to high quality data that can be used to improve the users understanding of
community’s health status and health determinants. The purpose of the site is to (1) provide a single
source for national, state, and community health indicators; (2) meet needs of multiple population health
initiatives; (3) facilitate harmonization of indicators across initiatives and (4) link indicators with evi-
dence-based interventions.
Health Resources and Services Administration’s (HRSA) Community Health Status Indicators (CHSI).
http://www.communityhealth.hhs.gov/homepage.aspx?j=1
This site provides over 200 health indicators at the county level. Within this database, there are a range
of summary tables. CHSI allows users to select a county and view a range of data published by different
federal agencies. The site also suggests “peer counties” that have similar demographics.
ICPSR. https://www.icpsr.umich.edu/icpsrweb/icpsr/index.jsp
This site maintains and provides access to a vast archive of social science data for research and instruc-
tion (over 8,000 discrete studies or surveys with more than 65,000 data sets), and offers training in
analyzing quantitative data in order to facilitate effective data use. This is the best site to access both
national and internal data related to health.
https://www.cdc.gov/nchs/fastats/
http://www.ers.usda.gov/foodatlas
http://www.cdc.gov/nchs/hdi.htm
http://www.healthindicators.gov
http://www.communityhealth.hhs.gov/homepage.aspx?j=1
https://www.icpsr.umich.edu/icpsrweb/icpsr/index.jsp
Page 248 | The Step-by-Step Guide to Evaluation
National Center for Health Statistics (NCHS) Summary of Available Data Sets. https://www.cdc.gov/nchs/
data/factsheets/factsheet_summary.htm
This site provides a list of current surveys and data collection systems used by the NCHS. Each source is
accompanied with a description, data collection method, target sample size, disparity variables, fre-
quency and FY 2012-2013 plans. The website organizes the surveys by population surveys, vital records,
provider surveys, and telephone surveys.
Office on Women’s Health: Quick Health Data Online. http://www.healthstatus2020.com/owh/
This interactive web system provides reliable, easily accessible health data to help assess needs,
develop programs and inform policies.
The National Longitudinal Study of Adolescent to Adult Health (Add Health). http://www.cpc.unc.edu/
projects/addhealth
This site provides access to data on a longitudinal survey that began in 1994, which collects data on
respondents’ social, economic, psychological, and physical well-being. The site also provides data on
family, neighborhood, community, school, friendships, peer groups, and romantic relationships,
providing unique opportunities to study how social environments and behaviors in adolescence are
linked to health and achievement outcomes in young adulthood.
OTHER DATA-RELATED RESOURCES
American Community Survey (ACS). http://www.census.gov/acs/www/
This U.S. Census site provides data on an ongoing survey. Information from the survey generates data
that helps determine how more than $400 billion in federal and state funds are distributed each year.
The U.S. Census Bureau sends the survey to about three million addresses every year.
American Fact Finder. http://factfinder2.census.gov/faces/nav/jsf/pages/index.xhtml
This online tool allows users to search for “popular” facts on a city, county, or state level. These include
data on population, age, business, education, housing, income, origins and language, poverty, and
veterans.
Annie E. Casey Foundation’s Kid Count Data Center. http://datacenter.kidscount.org/
This is a national and state-by-state effort to improve the well-being of children in the U.S. This program
aims to provoke local, state, and national discussions concerning ways to secure better futures for all
children by providing high quality data and trend analysis.
Bureau of Labor Statistics (BLS). http://www.bls.gov/
The BLS is a data reporting agency for the U.S. government in the field of labor economics and
statistics. This unit collects, processes, analyzes, and disseminates essential statistical data to the
https://www.cdc.gov/nchs/data/factsheets/factsheet_summary.htm
https://www.cdc.gov/nchs/data/factsheets/factsheet_summary.htm
http://www.healthstatus2020.com/owh
http://www.cpc.unc.edu/projects/addhealth
http://www.cpc.unc.edu/projects/addhealth
http://www.census.gov/acs/www
http://factfinder2.census.gov/faces/nav/jsf/pages/index.xhtml
http://datacenter.kidscount.org
http://www.bls.gov
The Step-by-Step Guide to Evaluation | page 249
American public, the U.S. Congress, other federal agencies, state and local governments, business, and
labor representatives. Data must meet a number of criteria, including relevance to current social and
economic issues, timeliness in reflecting today’s rapidly changing economic conditions, accuracy and
consistently high statistical quality, and impartiality in both subject matter and presentation.
Burns, J., Paul, D., & Paz, S. (2012). Participatory asset mapping. Advancement Project. Retrieved from
http://www.communityscience.com/knowledge4equity/AssetMappingToolkit
This toolbox presents research concepts, methods, and tools through topical guides such as Commu-
nity Research Participatory Asset Mapping. The focus is on community-based organizations that bring
together community members to visualize and actualize research and its outcomes.
CDC Youth Behavioral Surveillance System (YRBSS). http://www.cdc.gov/healthyyouth/yrbs/index.htm
This site monitors six types of health-risk behaviors that contribute to the leading causes of death and
disability among youth and adults. YRBSS also measures the prevalence of obesity and asthma among
youth and young adults. Also included, is a national school-based survey conducted by CDC and state,
territorial, tribal, and local surveys.
Census Data Federated Electronic Research, Review, Extraction, and Tabulation Tool (FERRETT).
http://dataferrett.census.gov/
This data analysis and extraction tool offers recoding capabilities to customize federal, state, and local
data depending on the desired search. Regardless of where the data reside, DataFerrett helps locate the
data across the Internet.
Community Commons. http://www.communitycommons.org/
This is an interactive mapping, networking, and learning tool geared toward the broad-based healthy,
sustainable, and livable communities’ movement. Registered users have free access to a geographic
information system (GIS) mapping tool that utilizes over 7,000 GIS data layers at state, county, zip code,
block group, tract, and point-levels. The website also provides peer learning forums to help users interact
with colleagues exploring similar interests and challenges.
Easy Access to the Census of Juveniles in Residential Placement: 1997-2013. http://www.ojjdp.gov/ojstatbb/
ezacjrp/
This site is updated almost annually although a few years ago, the site had some major gaps. However,
the information there is incredibly useful for looking at number of youth in state facilities (a one-day
count), both pre- and post-conviction. It also uses U.S. Census estimates and calculates a rate of commit-
ment, making the data comparable from one state to another.
FBI Uniform Crime Reports. http://www.fbi.gov/about-us/cjis/ucr
This site is useful for retrieving juvenile arrest rates, as the FBI uses a standard definition of juvenile as
anyone arrested under the age of 18. It makes the numbers from one region or state to the next more
http://www.communityscience.com/knowledge4equity/AssetMappingToolkit
http://www.cdc.gov/healthyyouth/yrbs/index.htm
http://dataferrett.census.gov
http://www.communitycommons.org
http://www.ojjdp.gov/ojstatbb/ezacjrp
http://www.ojjdp.gov/ojstatbb/ezacjrp
http://www.fbi.gov/about-us/cjis/ucr
Page 250 | The Step-by-Step Guide to Evaluation
comparable since the definition varies in some places. In this report, one can find year-by-year numbers.
For example, Table 69 provides the same numbers every year, is fully downloadable, and looks at state
arrest statistics for all offenses and offense categories for youth under 18.
Health, United States Annual Report. www.cdc.gov/nchs/hus.htm
This document can be used by people who want to access data on health status and its determinants,
health care resources, health care utilization, and health insurance and expenditures. This report has
two major components, a chart book that illustrates with text and figures major trends in health and 134
detailed trend tables.
Homeless Data Exchange (HRE). www.hudhdx.info
This online tool is designed to allow the Continuum of Care programs for the homeless to submit data to
the U.S. Department of Housing and Urban Development for the Annual Housing Inventory Count,
Homeless Point-in-Time Counts and the Annual Homelessness Assessment Report.
Juvenile Justice Geography, Policy, Practice & Statistics. http://www.jjgps.org/
This online repository provides visitors with a sweeping view of the juvenile justice landscape across
states and a place to make comparisons and chart changes. The site layers the most relevant national
and state level statistics with information on state laws and practice and charts juvenile justice system
change.
Mclean, J., Bell, J., & Rubin, V. Getting Equity Advocacy Results Guide. Policy Link. Retrieved from
http://www.communityscience.com/knowledge4equity/GEARSnapshotBuildTheBase.PDF
This guide identifies essential components of successful equity advocacy for policy change.
National Data Archive on Child Abuse and Neglect. http://www.ndacan.cornell.edu/
This site provides statistics on the occurrence of child abuse and neglect across the U.S.
National Institute on Alcohol Abuse and Alcoholism (NIAAA). http://www.niaaa.nih.gov/research/
guidelines-and-resources/epidemiologic-data
This site provides alcohol-related trend data in the U.S. for apparent per capita alcohol consumption,
liver cirrhosis mortality, alcohol-related morbidity among short-stay community hospital discharges, and
underage drinking behavior. The data reference manuals provide detailed data tables on alcohol
consumption and related conditions by demographic characteristics, including age, sex and race or
ethnicity.
National Organization for Research at the University of Chicago—Neighborhoods and Families Making
Connections Data. http://mcstudy.norc.org/data-access/
This site provides access to multiple databases related on various social issues. One needs to apply for
access to the data.
http://www.cdc.gov/nchs/hus.htm
http://www.hudhdx.info
http://www.jjgps.org/
http://www.ndacan.cornell.edu
http://www.niaaa.nih.gov/research/guidelines-and-resources/epidemiologic
http://www.niaaa.nih.gov/research/guidelines-and-resources/epidemiologic
http://mcstudy.norc.org/data
The Step-by-Step Guide to Evaluation | page 251
Pew Research Center. http://www.pewresearch.org/topics/
This site provides access to multiple data bases related to diverse topics. Users can download data from
the site.
Promoting Health Equity. http://communityscience.com/knowledge4equity/cdcpromotinghealthequity
This CDC resource provides results of a diverse forum of community participants who have experienced
developing, implementing and evaluating interventions to address conditions contributing to health
inequalities.
Stories: Using information in community building and local policies. Retrieved from http://community-
science.com/knowledge4equity/usinginformationincommunitybuildingandlocalpolicy
This guide presents cases from 18 cities where National Neighborhoods Indicators Partnerships (NNIP)
partners have been working to achieve improvements in community conditions.
Wider Opportunities for Women, Economic Security Database. http://www.basiceconomicsecurity.org/
This site is about the Basic Economic Security Tables™ (BEST) Index and the Elder Economic Security
Standard™ Index (Elder Index) that measure the incomes workers and retired elders need to achieve
economic security. The BEST and Elder Index present local expenses, savings requirements and economic
security incomes by family type, and at the city, county and state levels. The BEST and Elder Index are
comprehensive definitions of, and blueprints for, economic security. Database users can
Find an index for a location and family type
Compare their own families’ expenses to the local BEST Index
Compare indexes or single expenses across locations and family types
Download national, state, county and city index data
Access additional information on economic security and the work supported by the BEST and
Elder Index
http://www.pewresearch.org/topics
http://communityscience.com/knowledge4equity/cdcpromotinghealthequity
http://communityscience.com/knowledge4equity/usinginformationincommunitybuildingandlocalpolicy
http://communityscience.com/knowledge4equity/usinginformationincommunitybuildingandlocalpolicy
http://www.basiceconomicsecurity.org/
Page 252 | The Step-by-Step Guide to Evaluation
Conclusion
Glossary
1.1
Order an Essay Now & Get These Features For Free:
