Midterm Assignment “Chapter 1 – 8”Chapter 1
Activity: Defining Evaluation
In small groups, discuss the characteristics you think should be included in a
definition of evaluation.
Egon Guba (1985) defined evaluation as “disciplined inquiry” with a target to improve a
program by determining its value.
The recent definitions include such as by Rossi et al. (2019) a reference to context and an explicit
goal to “inform social action to improve social conditions” (p. 6).
Based on the definition we should include in the definition,
An evaluand, the purpose of evaluation, process, or procedure and the context in which results
will be interpreted.
Activity: Applying Evaluative Thinking
“What are we assuming? Do we know that?”
From the text about money, we are assuming that money determines the socio-economic status
of race, ethnicity, and family education which we don’t know actually. We can’t determine race
or ethnicity-based on money.
“How do we know that?”
1
We know from prior studies which have established a relationship between money and
race/ethnicity and education of the family.
“What makes you say that?”
The evidence related to money as sources of power and access to resources and association with
some race and ethnic groups as available in the literature make us say that.
Seek Out Blind Spots
“What are we missing?”
We don’t know other financial resources, total assets, family sizes, number of people working in
the family, and types of jobs.
“Whose perspective isn’t represented?”
People from rich families are not presented here. In addition, the evaluator’s views are given but
not the respondent’s views. There should be the perspective of respondents who would-be
participants of the study.
“What other explanations could there be?”
The other explanations include the relationships of other demographics such as gender, family
size, and type of work. Further, it is required to add, the type of support poor families are
obtaining and what are their rights and obligations in participating in these surveys/studies.
Capture Musings & Learning Questions
“I wonder if …” people of poor families refused to participate in the study.
2
“I bet if we …” give right of voluntary participation, many could withdraw.
“If I knew __________, I could _________.” (IllumiLab, 2018b, “Asking Questions”)
If I knew people have fears, I could allow them to express their fears and participate voluntarily.
Reflect and Discuss: My Biases
In small groups, discuss some of the fairly superficial preferences and biases you might have.
Then, either speaking in general or personally, discuss preferences that might impact how
someone approaches a project.
From an evaluation perspective, I don’t have any specific superficial preferences, however, as the
text of the book reveals some group preferences such as the supremacy of whites over blacks. I
might prefer Caucasians or people of the Arabian Peninsula to others. My other potential
preferences include women over men and especially married student women who are taking care
of their families and meeting their academic requirements. Similarly, biases can be mostly
cultural, which could prevent reasonable, knowledge consideration of a question. The
international biases in my case would be the inclusion of international women students with
children, and sympathies for such families who are struggling with their education as well as
meeting their family needs.
The preferences could reduce development and reasonable consideration of a question.
The biases and preferences can result in the selection of inappropriate methods, sample (sample
biases), interpretation of results, and concluding. Most preferences and biases affect the quality
of evaluations in terms of validity and reliability of data and results.
3
Voices from the Field
In his 2019 Voices From the Field interview, as follows, Melvin E. Hall discusses objectivity,
bias, and why he became an evaluator.
He discussed built-in structural biases, which are little b-biases and big B biases. Melvin chose
the option of balancing biases with other perspectives out of two options. Melvin became an
evaluator to make other people realize and understand their own biases as well as the evaluator’s
biases. The biases are not only unavoidable but in many instances, these shouldn’t be avoided as
some biases are positive and others may be negative, For example, he expressed his positive bias
towards HBCUs being an affiliate of HBCU. The best thing he mentioned, is to know and
understand your own biases and entangle your own biases with other peoples’ biases.
Activity: Reflections on Working with White People and Antiracism
Read the following three statements and write a short paragraph about your response to them and
any impact they may have on your response to race and racism.
To three responses, I would endorse DiAngelo’s (2018) comments with an endorsement that
racial inequalities cannot be diminished from society just because of being nicer and having a
smile on the face for people of color. The internalized superiority of Whites in the system cannot
simply go away and effects of superiority can be seen in the society regardless of the positive
behavior of Whites with people of color. Similarly, following Tenney’s point of view, intentions
4
do not matter solely but what matters is the supremacy of ruling race and fight against racial
disparities.
Racial inequities are not contemporary and have a long history in the world. The racial
disparities were started in the US since the transportation of Negro slaves from Africa to Europe
and then to America. There is induced inferiority among people of color and superiority in
Whites since the inception of this society. The racial disparities can be seen in various areas of
public interests such as education, health, and justice. It is very quick and easy to understand
such behavior and therefore regardless of good behavior, the major concern is the fight against
racism.
Case Studies: Evaluation Results vs. Politicians
As the following case studies indicate, evaluation results can and at times do lose out to
politics.
In the first example of 21st Century Learning Centers, evaluation results were not politically
approved and the California governor got public approval in favor of the program regardless of
its failure to meet proposed objectives. The program was found to be not effective for students’
outcomes after evaluative study, however, general public opinion did not accept the evidence and
justified the existence of the program in the community.
Evaluations are performed under certain circumstances and public opinion is also
accounted for evaluations. A researcher should consider political policies and forces while
5
evaluating programs. Evaluation results lose the attention of political attention when evaluation
results don’t mitigate with political policies and general public opinion.
In the case of DARE which targeted school-aged youths was found to be ineffective through
multiple evaluations and meta-analysis; however Federal funding remained continued for the
program. The program continued to be used in some schools for a variety of reasons including
the belief that a short-term program can’t change students’ drug-taking behavior.
Sometimes government wants to fund some programs even with their short-term failures
as politicians have their vested interest in their communities. The community service programs
are not fully stopped based on evaluations, but can gradually be minimized with approval of
political opinion. Political disapproval appears when evaluation results do not negate public
opinion at large.
6
Chapter 2:
Reflect and Discuss: What Ethics Means to You
Most adults have a vision or image in their mind of an ethical business, ethical organization,
ethical government, or ethical society. On the individual level, ethics has a specific and
oftentimes unique meaning and source. Using the prompts that follow, reflect on and discuss
what ethics means to you.
•
What does ethics mean to you as a student, parent, spouse/partner, employee, and/or
other roles?
As a student my ethics are
Be fair and honest in my work, and don’t use unfair means to my academic work. I was a
student would practice my autonomy of work and follow institutional guidelines in exercising
my autonomy. I value my work and would like to not be harmful to anyone and work
independently within my limits. I believe in fair treatment, justice, and equality which is are
legitimate ethical considerations for any student, even beyond studentship.
•
What is the basis of your ethical decision-making and behaviors?
My ethical values come from my religious beliefs and cultural affiliations. The culture I
belong to is an ideal Muslim society that practices all good Islamic values and traditions,
therefore a source of my ethical decisions comes from my religion and culture. My family is
also values-oriented that inspires my ethical considerations.
7
Reflect and Discuss: The Tuskegee Timeline
Review the timeline of the Tuskegee Syphilis Study and then reflect on and discuss the questions
that follow.
•
Do you think the Tuskegee Syphilis Study has a current and/or lasting societal impact? If
so, what?
The study has a long-lasting effect of increased mistrust and mortality among people of color
due to violating ethical rules. They were never well informed about the purpose of the study
and were not treated for their disease even the treatment was available for Syphilis. In the
Tuskegee Syphilis Study, men were informed they were treated for their disease and given
other benefits such as a free exam, free meals, and burial insurance.
•
Might certain individuals or groups still be suspicious of evaluators and other researchers
because of experiments like the Tuskegee Syphilis Study? Elaborate on your response.
Other studies (e.g. The Radiation Studies, The HeLa Story, and others) revealed
vulnerable people who are potentially suspicious about evaluators and other researchers.
For example in the radiation study, most of the participants were not aware of the nature
and type of experiments that would be performed on them. The participants were hospital
patients, pregnant women, children with intellectual disabilities, and military personnel.
There could be any vulnerable pupation who could be recruited for research without
giving them informed consent and the right to withdraw.
8
Reflect and Discuss: Ethical Considerations and Authority Figures
In contrast to the 1940s and 1950s, in today’s society do you think everyday citizens are more
aware of ethical behavior related to social and behavioral sciences research—that is, right versus
wrong—or do you believe that an authority figure will always be able to sway people’s judgment
toward unethical behavior? Under what conditions do you believe this is less or more likely to be
the case? Provide examples.
Yes, at present people are aware of their rights and ethical behaviors; certain laws secure the
rights of participants and lower the risk of being misused or maltreated. For example, Belmont
Report established three basic ethical principles, respect for persons, beneficence, and justice.
People should be aware of their rights and it is the responsibility of researchers to inform
individuals of their safety and wellbeing beforehand. It is also practical to have an authority
figure to protect peoples’ judgment towards unethical behavior. For example, governmental
agencies and other non-profit organizations that look after public rights and safety issues could
be there to monitor public rights. Institutional review boards (IRBs) are defined authorities to
protect such rights and look for ethical behaviors.
Case Study: Identifying Hidden Agendas and Ethical Land Mines
•
What do you see as the major ethical issues that the evaluator is facing in this case?
•
The program director has perceptions that findings of evaluation could influence his post
and staff member can use it against the director.
•
The program director is not agreed with the survey questions and has apprehensions of
negative opinions of the angry students about HSC.
9
•
The director wants evaluators to modify the survey to minimize the negative responses of
students and other respondents.
•
The directors want to use items measuring negative aspects of these items that are used in
evaluations of other programs at the University. However, those programs are evaluated
by different evaluators.
All these factors created an ethical dilemma by affecting the evaluation procedure and
challenging evaluators’ integrity and honesty for evaluation (violations under sections C and E).
Evaluators should apprehend potential threats and influence of staff members in bringing
negative opinions of students, therefore it is recommended to use multiple ways of data
collections (Quantitative and Qualitative; Likert-type scales, date review, focus groups,
interviews, and or observations) and should not rely solely on the survey. Individual interviews
or focus groups can provide a better insight into the program and also can minimize the
apprehensions of the program director. An evaluator is responsible for including possible
oppositional opinions and suggests the required course of action for a chance in the evaluation
plan (D1).
•
What hidden agendas could become ethically problematic for the evaluator?
The hidden agenda of the evaluation is to declare certain domains of the HSC’s functioning
as “off-limits” in this evaluation. However, an adequate and professionally respectable
evaluation of the HSC cannot omit a consideration of the areas that the director wants to
avoid.
•
What might be some professional consequences for this evaluator if the ethical issues
remain unresolved?
10
•
The evaluator can lose professional license, memberships, and credibility
•
Be penalized for ignoring important aspects of evaluations
•
Can be fired from his/her job for not following their ethical principles and professional
ethics
•
Can face criticism by colleagues and other professionals
•
Funders can cease/her payments and disengage from evaluation
•
The impact of the evaluation can greatly impact those served by the evaluated
organization/program
Reflect and Discuss: Evaluator Sources for Ethical Thinking
Consider the five sources of ethical thinking described as follows and the examples provided.
Now, provide additional examples of instances when an evaluator should avoid or rely on one or
more of these sources while working in the field. Reflect on and discuss your responses with
others.
A program director can ask the evaluator to collect data from a particular school where “students’
academic program” was successfully implemented and school staff is directly related to the
program director for gaining some extra monetary benefit for implementations. In return, they
are expected to give their satisfaction and success stories of the program. An evaluator found
suspicion through his/her intuition. However, evaluators don’t rely completely on one source of
thinking for ethical issues and use multiple strategies. For example in the above-stated example,
an evaluator can not only use intuitions in foreseeing a potential risk but can use his/her vision
and experience in making decisions. He/she has to follow standard rules and procedures
11
beforehand and practice such rules along with using ones’ past experiences. Evaluators can’t rely
solely on one single source and evaluators who take advantage of their values and beliefs can be
shaky in many cases if they are working with different cultural groups. Similarly, their
experience can’t be adequate to guide them to the right decisions. For example, an instrument
used in a dominant setting does not make it automatically valid and reliable to apply in other
settings.
Case Study: Moving Beyond Past Experience
In this case study, it is given that items in the instruments that are acceptable in one culture are
not necessarily applicable to other cultures, For example, some respondents can be offended by
some of the items which are not greatly practiced in that culture and against the norms of cultural
values. To make such an instrument applicable to another setting, the evaluation team discusses
issues and revises protocol to make it feasible for the target population.
Reflect and Discuss: Self-Exploration
•
Consider diverse cultural experiences that have led you to further “self” exploration.
What did you learn about yourself that you were not consciously aware of before this
self-exploration?
Being culturally competent, one should know cultural values, norms, liking and disliking,
applicable and non-applicable things, biases, conflicting aspects, political and social issues,
religious aspects, and the role of men and women in society. After exploration of self, I
found that several aspects were not known to me about awareness of others’ culture.
Normally we consider that cultural competence means a general overview of cultural aspects
such as language, religion, foods, rituals, values, and common law. In the transition from
12
Saudi Arabia to the US, I became aware of several aspects which I didn’t know before.
Likewise being an evaluator, there are many things to be known to become a culturally
competent evaluator.
•
After reviewing the AEA’s (2011) Public Statement on Cultural Competence (available
at www.eval.org/ccstatement) in full, discuss how exploring the “self” can improve one’s
work as an evaluator.
•
After exploring the self through cultural perspective and evaluator can improve
his/her work significantly by
•
Use culturally appropriate approaches, methods, procedures, and research tools.
•
It increases trustworthy understating and increases validity and reliability.
•
It reduces errors induced in cultural biases, stereotypes, and lack of shared
worldview among stakeholders.
•
Can reflect accurately opinions of respondents and conclude accurately.
Activity: Ethical “Blind Spots” in Evaluation
Sezer and colleagues (2015) delineated various sources of ethical blind spots. After a review of
these sources of ethical blind spots, summarized as follows, organize into small groups. Then
discuss them as sources of evaluator blind spots. Give specific examples of how these sources
might affect evaluators’ sensitivities and course of action in practice.
Implicit biases can be seen as implicit attitudes and egocentric biases that can affect the
sensitivity of the evaluator. Evaluators may not be able to recognize their positive and negative
views of others which can be seen in evaluative practices. For example, a White evaluator may
13
have a preoccupation about African American children as cognitively slow learners. Similarly,
the evaluator may ignore unethical behavior of other people due to lack of knowledge, therefore
such action of the evaluator may produce negative or positive outcomes based on perceived
unethical behaviors.
Case Study: The Compromised Evaluator?
Presenting favorable results in the yearly report are issues of integrity and honesty of the
evaluators. The evaluator tried to gain the benefit of being involved in the project for the long
term.
Case Study: Revising the Evaluation Report
The ethical dilemma that occurred here is hiding or overlooking potential operational
factors that appeared to be negatively affecting the program. However, positively presenting the
report and informing stakeholders that the program has performed very well the drawbacks of the
program cannot be ignored. The positive tone of evaluator/s could bring stakeholders and funders
to the result that the program was successful and had not found any drawback. According to
AEA (2018) section C: integrity, it is the responsibility of an evaluator to behave with honesty
and transparency for the integrity of the evaluation.
1. What are the consequences of the evaluator’s choice? What would happen, for
example, if every evaluator made the same decision?
Initially, by presenting program outcomes in a positive tone to hide some important information
or decreasing the severity of the situation, evaluators will break the integrity of the evaluation
and violate honesty and transparency.
14
Secondly, presenting such information in a positive tone will bring a positive impact on the
program and potential problems can be ignored. This could lead to an inappropriate consequence
of the program implementation and misuse of funds. If the program has to be continued in the
future, more severe problems could occur such as failure to program and or the program
participants.
Thirdly the program director can’t influence the evaluators’ decision as the evaluator should
remain independent of bias and influence. It is the moral and ethical duty of the evaluator to stick
with procedures and present results in a professional way. The results are suggestive of program
changes that are ethically sound to the overall productivity of the program and the program
participants.
2. What duties and obligations do evaluators have to themselves, the funder, project
stakeholders, and society at large?
Evaluators, funders, society, and project stakeholders have an obligation of transparency and
integrity with public funds for the implementation of best strategies/practices of intervention. It
is within the ethical duty of the evaluator to use the best cost-efficient procedure to conduct an
evaluation, collect data, and present findings honestly without hiding any potentially damaging
piece of information (C5: AEA, 2018).
3. What would be just or fair in this situation?
In this case, the evaluator mentioned that the negative component of the evaluation was fairly
described in the report; therefore tone to describe negative aspects of functional aspects should
15
be formal and professional. The communication between the evaluator and audience should be
based on the facts and results, and not be a choice of a funder or program staff.
4. 4. What would be the caring response or course of action? Is that the ethical
response? Justify your position.
In this case, the evaluator should present the report as it was intended to, without changing the
tone of negative findings. The results should be presented in a manner as they appeared with
justification and explanations. However, the evaluator can quote the concerns of the program
director that the program has performed overall well regardless of its functional flaws (see C4 &
D1: AEA, 2018). Evaluators should disclose a conflict of interest in the evaluation report (C2).
Reflect and Discuss: You Didn’t Hear It From Me!
•
Which potential ethical dilemmas exist for the evaluator in the two situations described?
In these scenarios, confidentiality and the potential risk of harm are two dilemmas. Initially,
there is no evidence that the incident happened so the evaluator can’t believe the story as the
program member wants to keep her identity confidential and not to be disclosed to anyone. The
second scenario deals with falsified information that comes under the integrity and honesty of the
information to be disclosed for evaluation purposes.
•
Should the evaluator do anything? If so, what, and why?
In the first case, the evaluator should not disclose the name of the informer and consider the
incident while conducting the evaluation. In the second scenario, the evaluator can verify the
16
falsified information and bring it to the attention of stakeholders if there is no conflict of interest.
The evaluator should follow ethical principles of communicating results accurately and honestly
to make evaluation reliable and truthful.
Case Study: Application of the Evaluators’ Ethical Guiding Principles
Being an evaluator, I would do the following actions
•
Identify conflict of interest with any staff or board member before presenting a
proposal of evaluation.
•
I will engage professional and experienced research assistants, not graduate
students who are pursuing graduate studies yet.
•
I will keep information confidential and will not let anyone use it for personal
purposes. The information will solely be used for evaluation purposes.
•
The study could have been done objectively and fairly with all professional standards met
in implementing evaluation practice.
•
I would collect data anonymously and none of the HCC members involve in the data
collection process.
Principles
Issues raised
Systematic
Inquiry
The inquiry was not systematic and random procedures were used to collect
data without any data mapping techniques (violation of A1 through A6)
Competence
The evaluators were not competent in collecting information. A grad student
was engaged in data collection who was not trained and competent enough to
do so. (violation of principles B1 through B4)
17
Integrity
The evaluator did not disclose any conflict of interest being an ex-member of
HCC. The evaluator did not explicitly address the needs of stakeholders to
assess their needs. The evaluator used the findings of the evaluation for his
benefit and break the confidential agreement with respondents.
Respect for
People
The evaluator engaged HCC staff to collect data from the beneficiaries of the
program, which is against the ethics of the evaluation. The risks to clients were
not minimized and HCC staff had access to the data provided by clients. (E4:
Promote transparency and active sharing of data and findings with the goal of
equitable access to information in forms that respect people and honor
promises of confidentiality)
Common
Good
The evaluation was not for common good to assure public interest, instead, it
was done to publish research papers and get a student’s thesis done.
18
Chapter 3
Case Study: Example of Experimental Evaluation in the 1930s: The CambridgeSomerville Youth Study
Q1: What do you see as an important contribution of the Cambridge-Somerville Youth
Study to the field of evaluation?
The study is an important example of social program evaluation as a longitudinal study
that was initiated in the 1930s. It was one of the earliest experiments with randomized
sampling procedures designed to evaluate social interventions of youth delinquencies.
The study investigated youth delinquent behavior for many years (10 years, 30 years up
to 70 years) and evaluated the effect of the program on the delinquency of juveniles from
childhood to adulthood. The important contributions are summarized as,
1. One of the earliest experimental designs in social sciences to examine the effect
of interventions. It introduced experimental designs in evaluations of social
programs.
2. It initiated building evaluations into community-based treatment programs with
long follow-ups.
3. It used randomized groups to compare experimental with control groups.
4. It was one of the longitudinal studies which continued follow-up for 70 years.
Q2: What are the issues raised by the study…..
Some issues raised by the study were,
1. The duration of follow-up was too long and the continuation of study over 70
years often loses its effect.
2. 10 years follow-up resulted in higher negative results which recommend not
continuing with the program for a longer period.
3. The effects of the program were diminished after 30 years, therefore it was not
viable to continue the program even after 10 years of follow-up.
19
4. There was the risk of attrition and history on the validity of the study.
Reflect and Discuss: The Perry Preschool Project
A high-quality preschool program for children from disadvantaged backgrounds, the
Perry Preschool Project was conducted from 1962 to 1967 but led to a longitudinal study.
1. What social, political, economic, and racial issues were happening in the country
that might have influenced the conceptualization, design, and outcome of the
evaluation?
The sample of the study was 128 three and four years old African American
students living in poverty. During that period between 1962 to 1967, some
constitutional changes in the US could impact the conceptualization, design, and
outcomes of the evaluation.
a. Civil Rights Act of 1964 prohibited discrimination in public accommodations and
employment was passed which could affect the conceptualization and theoretical
framework of the study.
b. There was the movement of equal rights in the US and in 1965, the Elementary
and Secondary Education Act was passed to improve the educational equity of
students from lower-income families. This event can affect the design or
sampling procedure due to equal opportunities for African American children.
c. The events could lead to a change design of the study from longitudinal to crosssectional.
d. Other similar programs of that era could affect the results of the study for
rethinking the evaluation practices, such as methodologies, and nature of
problems.
e. Some of the evaluators of that period were tired of traditional approaches of
evaluation, therefore some of the new studies were trying to use randomized
experimental designs
2. If you were the evaluator of this project in 1969, what would be the top two
historical factors that you would consider during your work? Why?
I would have considered two topic historical factors in 1969
1. 1964 civil right act which gave equal rights to African Americans in the US to
provide equal rights in education, health, and other social matters. This law
20
opened up new and equal opportunities to all races, and ethnic groups regardless
of their country of origin.
2. In 1966 and 1967, sex and age discriminations laws were passed which could
have impacted the participation of people in social programs and evaluative
studies regardless of their age and sex differences.
Reflect and Discuss: Pre–Brown v. Board of Education (1954) Evaluation
Publications in the Journal of Negro Education
Reflect on the works of African American evaluators of color during the pre-Brown era
and the founding of the Journal of Negro Education discussed in the preceding sections.
Now, organize into pairs and discuss the potential impact of the omission of such early
works on our historical understanding of evaluation and evaluators’ role during the early
part of the 20th century.
During the pre-Brown era, the evaluators such as Boykin contributed their work towards
evaluations of financial resources, teachers’ salaries, and interpretation of quantitative
data for segregated schools. He interpreted data instead of findings to determine the
adequacy of Negro Education. The educational adequacy was examined in terms of statelevel activities for providing education to children, the educational benefits that the
average child of the nation receives, and the extent to which Negro education is
supported and maintained.
Caliver also made a significant contribution towards federally evaluative inquiry
of African American education. His work was mainly about national awareness about
racial disparities and inequalities in education in the rural South. Caliver’s most
influential work was related to national studies in the 1930s that opened up investigations
for educational inequalities in the 20th century.
The Journal of Negro Education was launched in 1932 to publish a variety of research
studies including evaluation studies conducted by African American scholars.
The potential impact of evaluators in the pre-Brown era is evaluations about racial
disparities, quality of education, and equality of students in schools. It also made some
impact on black-white wage differences and provided resources to schools to overcome
differences in education caused by inferior and superior schooling.
Activity: Research an Influential Person or Event
Organize into groups of two or three and work toward giving a 10- to 15-minute
presentation on a historical event or influential figure in evaluation during the 20th
century. First, identify an influential person or event relevant to the evolution of program
evaluation. Provide a biographical sketch of the person or an overview of the event,
21
identifying key contributions that were added to the evaluation field. Among the
contributions discussed, which does your group feel are the most significant, and why?
Evaluations of social programs were flourished in the 20th century and there have been
significant developments during this century. I would pick up a very important study that
was designed to monitor the educational progress of US students known as NAEP
(National Assessment of Educational Progress). The study was initiated in 1969 under
Tyler’s leadership. NAEP assessed the academic progress of students from the selected
grades, 4, 8, and 12 for reading, writing, mathematics, and science. The purpose of the
study was to assess students’ academic performance on grade-appropriate skills against
benchmarks. NAEP takes into account students’ demographics in interpreting their
performance and is conducted at the national level. It is more rigorous than state-level
assessments and considered a reliable assessment.
The key contribution of NAEP is assessing various skills that students should
know at a particular age. It also provides bases to compare students’ progress at national
and state levels taking into account the demographics of students. The most significant
contribution of the program in my opinion is providing a base to compare students in
various subjects across the states, school districts, and school levels. This practice
provides a greater understanding of the strengths and weaknesses of students associated
with curriculum, teaching, or physical resources.
Discuss what you learned during your inquiry. Why was this person or event important to
evaluation practice, scholarship, and/or society? Finally, consider the implications of
these contributions in today’s context.
In this activity, I learned the analysis and contribution of an educational program
evaluation. The program selected was started almost a half-century ago and still has its
importance and applicability in assessing students’ academic performance at the national
level. The program has significant implications as of today and it is contributing
significantly to improving students’ academic performance and helping policymakers to
develop new policies, assign funds to schools, and train teachers to improve the quality
of education.
22
Chapter 4
Reflect and Discuss: Paradigms Shaping Your Everyday Worldview
Reflect on your paradigm about one of the following broad areas: Paradigm is a
personal frame of reference, related to political, cultural, and religious affairs. Further
every evaluation, either explicitly or implicitly, is guided by a set of values, beliefs, and
assumptions that may also be underpinned by theory. These values, beliefs, and
assumptions, along with methodology, reflect a particular paradigm. I have my paradigm
of examining and observing phenomena through a particular lens. The set of beliefs and
values that I have is very much similar to what other people have, however, I might hold
a different perspective of phenomena.
Gender and gender roles
My thoughts about gender roles are strongly driven by my religious beliefs. I don’t
believe in the absolute equality of men and women, and find that in some cases men are
superior to women; however, in other perspectives, women are superior to men. For
example, men can’t give birth to a child and have no experience of motherhood. Baby has
been attached to mother since inception, and this attachment establishes a unique
relationship of a child with mother. On the other hand, men are made for hard labor work
and few women tend to do work competing with men in general notations.
Organize into small groups and discuss your paradigm in terms of your views and
assumptions about the topic and how you arrived at those views.
The topic I selected for discussing my worldview through the lens of my paradigm was
gender roles. The view under this paradigm comes from my understanding of my
religious beliefs which explicitly define the roles of men and women in society. The
assumptions and views about men and women are very evident in daily life and I
concluded these views from my personal experiences as well as observing other people
in different capacities. The discussion with other women and men also added useful
information to my knowledge and let me conclude such world views. However, the
situations are always dynamic and we keep observing various aspects to reframe or
rearrange the gender roles in society.
Activity: Applying Paradigms to Evaluation Study
Interventions: A gender equity intervention in science, technology, engineering, and
mathematics (STEM)
I would select two of the following paradigms to frame evaluations.
23
Positivism/post-positivism: These paradigms allow the researcher to collect numerical
data from a large sample size and express results in statistical form. Further, this
approach ensures objectivity through the quantification of phenomena by defining
variables of interest. In the given scenario following information will be obtained.
1. Number of female postgraduate students recruited by a male professor for a given
year
2. Number of citations for publication with a woman and first author
3. Number of female and male students in STEM and their success rate (GPAs and
percentages)
4. Number of graduated female and male students and their future career
This approach will help to gather several pieces of information to conclude biased for
female students in STEM majors through statistical analysis. The approach will provide
accurate and measurable data that could be used to project the findings numerically and
graphically.
The second approach I would use is constructivism which will allow me as an
evaluator to construct meaningful inference from personal experiences of male and
female students, professors who show any potential gender bias in STEM majors. This
paradigm will help to understand the underlying thoughts that cause such biases. This
paradigm uses qualitative methods given the need to construct meaning in context and
the evaluators’ role in seeking the subjective truth of each participant in evaluation. The
data will be collected in the form of words, and themes and collected through
observation, interviews of focus-group discussions. This paradigm will provide me an
insight into the problems, i.e. gender biases in STEM.
Activity: Interface Among Social Science Theory, Social Programming, and
Evaluation
Identify the social science theory that helped frame the conceptualization of the
intervention identified in the publication. Give specific examples of how the particular
social science theory and its accompanying research informed the intervention described.
Dierendonck, Dirk, Schaufeli, & Buunk, (1998) used equity theory to conceptualize the
interventions. According to the theory of equity, burnout results to an important extent from
perceived inequality, and the intervention programs designed to reduce burnout can be
performed by reducing perceptions of inequity. Equity theory is the theory of social justice in
which an individual perceives an inequitable situation and identifies the situation in which he/she
has to respond.
Following the theory of equity, the intervention program was designed to reduce inequity
perceptions in five weekly group sessions. The psychologist stimulated perceptions and cognitive
24
restructuring that could help to reduce negative perceptions of being inequitable. The
intervention was developed on the theoretical framework by designing weekly sessions for
cognitive restructuring. The burnout was measured in the form of emotional exhaustion,
depersonalization, and personal accomplishments.
Describe how the evaluator utilized this particular social science theory to design the
evaluation?
Evaluators used experimental design to examine the effect of the intervention on burnout
among mentally disabled individuals. Equity theory provided a rationale to develop
evaluation, to reduce perceived inequity in the relationships with the organization and
with the participants. There were one experimental and 2 control groups to measure the
effect of the intervention on the experimental group with comparison to others.
Activity: School Violence Reduction Program Theory of Change
Identify two assumptions you believe underlie the causal links in this program theory.
•
•
•
Youth who already have leadership skills will have the positive effect of
intervention by nurturing such skills.
Social phenomena such as violence can be treated with individualized treatments
by targeting individual behavior.
Interventions should be started when children are young to get maximum benefit.
What do you think are the most important assumptions for successful implementation
and outcomes inherent in this sample program theory?
I think a more important assumption is that individual behavior/violent behavior can be
treated and modified with intervention targeting individual behavior.
Identify three risks that threaten the accuracy of the theory of change. These can include
things (e.g., cultural, political, economic) beyond the organization’s control that can
derail program outputs and outcomes.
The three risks are
1. The theory of change takes into account contextual factors and failure to include
such factors could reduce the accuracy of the theory.
2. It should be based on at least one relevant social sciences theory and there should
be a linkage between theory and change practice
3. It should be based on good practice and practice can be drawn from prior research
and practices.
25
Chapter 5
Reflect and Discuss: Defining Social Justice
In small groups, discuss what you think should be included in a definition of social
justice and why you made the choices you did.
I selected the following definition of social justice as a reference to draw
discussion on it.
“Social justice is a concept of fair and just relations between the individual and society,
as measured by the distribution of wealth, opportunities for personal activity, and social
privileges” (Wikipedia, 2020a, para. 1).
Various aspects of our personal and social life are accounted for by social justice.
For example, when we talk about equality or justice, we look for equal participation in
our social life and don’t want to waive our rights to anyone for any reason. Injustice
system, we have some rights and obligations which we have to deliver in society. There
should be a distinction between individual and collective responsibilities, so we can’t
ignore our collective responsibilities for the sake of individual rights. Improver division
of power and wealth in the society creates imbalances, therefore it is important to make
sure that all privileges are equally distributed regardless of racial, gender, and ethnic
differences.
Activity: Finding My Privilege and Oppression
Look at Table 5.2 on the following page on privilege and oppression, and find
demographic characteristics that may cause you to have the privilege and other
characteristics that are more likely to cause you to be oppressed. Write down one or two
specific benefits you receive from being a member of a more privileged group and one or
two disadvantages or challenges you have as a result of your membership in a more
oppressed group.
Privilege
Able-bodied
European
Heritage
Gentile, Not
Jewish
Heterosexual
Demographic
Characteristics
Age and educated
Religious
Origin of country
Wealth
Gender
Oppression
A person with a
disability
Non-European
Heritage
Jewish
LGBTQ
26
Demographic
Characteristics
Age (old people)
Non-religious
Religious affiliation,
wealth/economic status
Gender, age, family
affiliation, poverty
Light Skinned
Native Eng.
Speaker
Traditional
gendered
Upper middle
class
White
Young
Country of origin,
nationality, economic
status
Country of Origin, Gender,
Age, and economic status
Age, education, Income,
Family orientation
Education, Income level
Nationality,
Education, Family
education and Gender
Dark Skinned
Gender, Education,
Economic status
Non-native
Eng. Speaker
Gender Fluid
Nationality, age, gender,
economic status
Family orientation, Income
level
Gender, Education, Income
level
Nationality
Education, Gender, Income,
and family orientation
Working Class,
Poor
Person of color
Old
The benefits of being a privileged group are access to education in highly expensive
educational institutions, living inexpensive and luxury houses, stoical status, and political
influence. There are some challenges as being part of an oppressed group such as
stereotypes, lack of social recognition, poverty, no or least access to wealth, and other
resources such as education.
Reflect and Discuss: Medical Condition or Culture or Both
As indicated in the following activity, there has been discussion as to whether or not
disability should be viewed primarily as a condition and whether or not people with
disabilities should be viewed primarily as members of minority groups.
To me, deafness is most likely a medical condition and in some conditions, it is a
treatable medical condition. However, for non-treatable conditions deaf people have
equal rights in a social justice system. Disability however is restriction or disadvantage
caused by a contemporary social organization due to physical impairments which
excludes them from participating in mainstream social activities (Oliver, 1996). It is
evident from the definition of disability that it limits some social activities of deaf people
or people with other physical impairments. However, culture should not promote the
notion of a minority group for people with disability.
Reflect and Discuss: What Would You Do?
The evaluator should be culturally competent and must be culturally responsive and use
one’s competence in performing evaluations. In the given scenario, a culturally
competent evaluator should honor the cultural context of people of color, there instead of
giving training to a white evaluator, they should hire an experienced evaluator of color. A
white evaluator who is not fully culturally competent can’t be trained at expense of
project money. This is also not a good idea to visit sites and learn from the participants
about their culture and community to develop trust. Technically an evaluator must have
a keen awareness of the context in which the project is taking place and a complete
understanding of how this context will influence the behavior of participants. The project
fund can’t be used to improve the cultural skills of the evaluators.
27
Reflect and Discuss: Applying a Transformational Paradigm
In small groups, discuss the ways you feel this work was or was not transformative.
The study conducted in the fall of 2015 followed a transformational paradigm because
the study involved a culturally diverse population and evaluators wanted to restore social
justice in diverse communities. The project was related to the results which did not
portray the real picture of community problems, and cultural norms, neighborhood
characteristics, and tears of the community were not addressed. The study has all
required factors necessary to apply transformational evaluation (presence of the social
problem, change in social demographics, increasing international cooperation) and social
consciousness.
CHAPTER 6
In small groups, discuss, from your observations or personal experiences (e.g., work,
community, or academic setting), a circumstance where a formative evaluation is needed
and could provide timely and useful information.
I am a teacher by profession and often use formative evaluation of my instructional
process and students’ academic progress. Formative evaluation of students’ progress
provides me immediate feedback to improve the instructional process, identify learning
problems of students, and need resources to improve teaching.
Now, using this same reflective process, identify a current situation where a summative
evaluation is needed and could provide timely and useful information.
Summative evaluation is also known as impact evaluation which is performed at the end
of a process and program. In my case, we engage in summative evaluation at every
academic year to examine the extent to which students have acquired skills in a given
content area. Students are evaluated for their academic, social and emotional skills at the
end of the school year as part of summative evaluation.
Give your rationale for the examples provided.
For both examples, formative and summative evaluations, two aspects primarily guide
the process. The first is to monitor the ongoing process and find discrepancies,
inconsistencies, and problems during the implementation of the instructional process.
However, there are some set goals to be evaluated at the end of a program delivery which
is dealt with in the summative evaluation process. The purpose of formative evaluation is
to improve students learning during the instructional process and assess their overall
performance according to set goals and pre-determined standards.
28
Reflect and Discuss: Six-Step Process for Conducting an Evaluability Assessment
What particular challenges do you see as possible at each stage of the evaluability
process?
The challenges I can see at each stage are,
At step 1, a potential challenge is the identification of key stakeholders and meeting with
them. Sometimes key stakeholders are not readily available or busy to not be able to
participate in the evaluation process.
In step 2, the program design can lead to some challenges such as poor planning,
inappropriate need assessment, improper approaches, poor data collection techniques,
and collecting data more than needed.
At step 3 challenges of exploring program reality include unavailability of concurrent
and up-to-date information about exiting the program. The evaluator can experience the
challenge of lack of skills to conducting interviews with people who are higher in
knowledge and experience. Similarly projects reports to substantiate the program reality
should be reliable and relevant which is sometimes not feasible to obtain directly from
available ventures.
At step 4, it is a challenge to foresee that the program will be delivered as intended. Their
evaluator develops interventions based on some assumptions and all assumptions are not
always realistically presumed by researchers. Moreover, due to time constraints, the
behavior of the recipient can be changed before the implementation of the program.
At step 5, challenges after implementation of program design is pre-agreement between
stakeholders and validating with program goals. It is to determine program rationale and
justification which can be understood by relevant stakeholders. It also includes the
readiness and knowledge level of an evaluator to execute the program.
At step 6, It is important to develop stakeholders’ agreement on one or more of the
evaluation options. Stakeholders have different background information and stakes in
their community. In the informative evaluation, we assess stakeholder attitudes such as
satisfaction level. There could be a conflict of interest with stakeholders for being the
difference in values, norms, need, and interests.
How might the evaluator overcome some or all of the identified challenges?
An evaluator may overcome these challenges with their experiences, knowledge, and
support from their colleagues. They talk to stakeholders, program staff, and other people
who are part of the program to figure out some appropriate ways to deal with such
challenges.
29
Activity: Planning a Formative Evaluation
Discuss different types of formative evaluation…
Need Assessment: It is done before designing a program and the evaluation can produce
information for a better understanding of the problem, available resources, the services
that are appropriate for recipients.
Evaluability Assessment is another form of formative assessment and in this evaluator
tries to find preliminary data about the execution and implementation of the program.
This is done before starting a formal program evaluation and is called a pre-evaluation
activity.
Process evaluation is also known as implementation evaluation is taken place when the
service delivery starts. In process evaluation, we include program implantation, context,
and research process.
Progress Evaluations: This type of formative evaluation assesses a program’s progress in
achieving its intended goals. The data is collected from multiple sources to examine to
what extent required benchmarks and intended goals are achieved.
What type of formative evaluation would you plan?
In this case, we will choose progress evaluation to monitor the ongoing progress of the
academic program at the university level. For example, we started an academic
enrichment program for women students from rural areas to improve their academic
skills for university education. Now we want to examine how this program is meeting its
intended goals and to what extent women students are benefiting from this program.
What would be the rationale for your selection?
The rationale of selecting process evaluation is to monitor progress by examining the
implementation process of the program. We will examine how the program is meeting its
goals, and how various components of the program are organized to deliver the intended
outcomes.
What kinds of recommendations would you hope to be able to provide to the unit’s
administrator(s) (department or division chief, department chairperson, dean of the
college, etc.)?
The projected recommendation to administrators would be,
1. The program is fulfilling almost 80% of its intended goals of providing
academic skills to women.
30
2. Almost 90% of women are completed academic enrichment programs
last year.
3. On average 90% of women who attended academic enrichment
programs before joining their university education showed aboveaverage grades in their first semester.
Reflect and Discuss: Outcome Evaluation of Teen Pregnancy and Parenting
Program
The case study on page 184 illustrated some of the issues a process evaluation of a teen
pregnancy and parenting program might examine:
Type, quality, and quantity of services delivered
A three-phase support system as training in functional areas: pregnancy prevention,
developmental stages anger management, discipline, communication, money
management, household management, health hygiene, employment preparedness, and
community resources.
Demographic characteristics of the beneficiaries of project services
Teenaged mothers in Maryland
Resources used to deliver the services, plus the barriers and facilitators to program
delivery
To deliver the services, it is required, trainers, healthcare providers, psychologists, and
social workers. In addition to people, training modules, videos, broachers, handouts, and
other reading material are also required. There could be barriers to identifying some
potential teen mothers and recruiting them as participants due to parental permission
problems. The baseline data might not available to compare the progress of the program
being implemented. The other barriers could be stigma, liability, confidentiality, and
legal issues. The facilitators were the administrators, supporting staff and evaluators, and
accompanying the team.
The case study did not provide significant information and all these questions are
answered hypothetically.
How implementation problems were resolved
Implementation problems were resolved by removing the conflict of interest, identifying
key components of the project, and removing barriers in the delivery of the program.
31
Recruitment and retention procedures, particularly sustaining the involvement of hardto-reach participants
Pregnant women are a vulnerable group especially when they get pregnant in a teenage.
Reflect on these process issues and then identify factors that an outcome evaluation of
this same program might be interested in pursuing.
Teen women were recruited through the involvement of schools and families
participating in social programs. The evaluator directly spoke to eligible teens for getting
their ascent and parental permission allowing them to participate in the study. All women
who participated in the program were given free medical exams, lab tests, and prenatal
care.
As an outcome evaluation of the same program, evaluators might be interested in
determining the effect of the program for promoting teem women as independent
individuals and having skills to survive as mothers and women. The following areas can
be seen individually for the progress that has been made over six months to one year.
Ares of concern: Pregnancy prevention, developmental stages, anger management,
discipline, communication, money management, household management, health,
hygiene, employment preparedness, and community resources
Compare and contrast the issues you identified with those identified by others.
In such cases, four types of issues are commonly observed, biomedical and sociological,
financial and psychological. Teenage women who experience pregnancy often
experience sociological and psychological distress. These challenges can be reduced by
giving the incentive to participate and ensuring confidentiality.
Reflect and Discuss: Importance of Detecting Cost Benefits
Question for Discussion
When considering the case from the perspectives of various key stakeholders (identified
as follows), what might be the major factors for determining the overall value or worth of
the new protocol?
Doctors (health and hygiene, detection of colon cancer)
Cancer patients and their families (Colon cancer symptoms, cancer detection/diagnosis,
and treatment process)
32
Funders (cost and financial issues to overcome color cancer vs. feeding poor families)
Policymakers (healthcare expense, healthcare facilities, and economic conditions of
people)
Broader societal members (healthcare facilities, expenditures to overcome cancer vs.
feeding people)
Provide a rationale for your answer.
Each stakeholder has their concern in the program such as doctors and policymakers are
more concerned about healthcare facilities whereas funders have concerns to spend
money appropriately on health and food.
Activity: Rapid Evaluation and You
Think of instances in your life (at school, at work, or in another professional or personal
setting) where a rapid evaluation could be useful.
In our institution, we introduced an academic enrichment program to equip women from
rural areas before starting their university education. A rapid evaluation of one
component of the program related to the use of technology can provide immediate
feedback to program developers and administrators about the program implementation
phase. To execute a rapid evaluation, short-term questions can be raised focusing on the
program’s outcomes.
Organize into small groups. Describe the situation to your group members, indicating the
rationale for a rapid evaluation; and the value of a rapid evaluation, in contrast to a
traditional evaluation, in the situation.
Rapid evaluation is like action research that can be conducted over a short period during
the middle and early stages of program implementation to evaluate the goals and
outcomes of the program. It takes less time and resources and provides immediate
feedback to remove implementation errors.
Activity: Putting It All Together: What Would You Do?
Scenario 1: The county executive is interested in obtaining information to understand the
concerns and problems residents have and to ultimately identify and prioritize issues for
action.
33
Need Assessment can address this issue.
Scenario 2: The project director of a teen pregnancy and parenting program for lowincome youth of color wants to know the extent to which the project is successfully
reaching its intended population.
Process or Implementation evaluation can provide the extent to which the program is
approaching its intended population.
Scenario 3: The funder of the science, technology, engineering, and mathematics
(STEM) project is interested in finding out the extent to which the project is resulting in
any significant changes in the participating girls’ attitudes toward pursuing STEM majors
in their postsecondary education.
Outcome or summative evaluation will be the appropriate procedure to find the impact
of the program on the attitude of girls towards STEM
Scenario 4: Funders of a community-based health program are unsure whether the funded
program, although in operation for 18 months, is ready for a formal evaluation; however,
they require some systematic data to help them determine the program’s readiness.
Rapid Evaluation can be done before starting a formal evaluation
Scenario 5: Organization X wants the evaluator to be a facilitator and learning coach who
brings evaluative thinking to the table that informs ongoing change and innovation that is
supportive of the organization’s goals.
Formative Evaluation (Evaluatibility assessment)
Scenario 6: A program director of a youth literacy program is interested in determining if
the program has affected the participating students.
Impact evaluation is an appropriate procedure
34
Chapter 7
Activity: Wicked Problems You See
Identify an issue within your (broader or immediate) context that you would characterize
as a wicked problem.
In my context, a potential wicked program could be diabetes control in youth and
teenagers in Saudi Arabia. Diabetes mellitus is a medical condition that can damage an
individual’s well-being by interfering with multiple body processes. Saudi Arabia from
1990 to 2007, the incidence rate of Type 1 Diabetes Mellitus (aged 0-14 years) among
children doubled in less than two decades, from 18.05 per 100,000 children between
1990 and 1998 to 36.99 per 100,000 children between 1999 and 2007, reflecting an
average annual rise in the incidence of 16.8 percent (Robert et al., 2018, p. 8).
Citation: Robert, A. A., Al-Dawish, A., Mujammami, M., & Dawish, M. A. A. (2018). Type
1 Diabetes Mellitus in Saudi Arabia: A Soaring Epidemic. International Journal of
Pediatrics, 2018, 1–9. https://doi.org/10.1155/2018/9408370
Identify the various key stakeholders, including people/groups/organizations who are
affected by and can affect the problem. For the stakeholders that you identified, provide a
brief written description of how each stakeholder group is likely to frame the problem
(e.g., causes, consequences) and potential solution(s).
Key stakeholders include children, youth and teenagers, parents, teachers, healthcare
providers, policymakers, government officials, and Funders and community members.
Primary stakeholders are children who are a victim of diabetes or potential target of
diabetes. Children who are obese are at great risk of diabetes; therefore controlling their
obesity is another challenge to control diabetes. Obesity is related to lifestyle, and change
in the lifestyle of children is a multi-task activity that needs the involvement of parents,
teachers, schools, and community members.
Teachers and parents are responsible to prevent obesity and diabetes in children by
engaging in various physical activities and promoting a healthy lifestyle.
Healthcare providers are people who will guide children and their parents to change their
lifestyles and improve general health conditions. They can help to suggest government
for increasing healthcare facilities.
35
Community members are concerned for the general well-being of children of the
community. They don’t want to seek sick children in society and want to improve their
quality of life.
Policymakers are concerned about good health, social and psychological health along
physical health. They form policies for schools, hospitals, and other community services.
Now, go back to Table 7.1 and delineate the characteristics of the problem that are
commensurate with Rittel and Webber’s (1973) 10 characteristics of wicked problems.
1. The problem of diabetes is a multi-directional and multi-problem issue. Diabetes
in children is due to their being less involved in physical activities and poor
lifestyle. The society dimensions are changed and people are relying on
technology and enjoying luxury in Saudi Arabia. There is no stopping rule.
2. The problem is not fully defined and there is no one-stopping rule. The problem
is related to many health and social issues therefore one single solution to these
problems.
3. The problem of diabetes is continuous and has limited availability of immediate
solutions, and only increased awareness among children and parents can’t
eradicate this problem
4. Diabetes is multi-dimensional and associated with other issues such as diabetes,
poor health conditions, lack of awareness, low physical activity, automation, etc.
5. Policymakers can’t be right and wrong in one solution, and an increase of one
option may cause a rise in others too, however, some aspects can’t be avoided
such as the government can’t stop automation or luxury life.
Reflect and Discuss: Major Social Problems in Your Community
•
What do you see as a major social problem in your community?
A major social problem in our society is drug abuse and people of different ages
consume illegal drugs and some other intoxication such as marijuana. Substance abuse
can cause social, physical, mental, and emotional problems.
Who is most negatively affected by this social problem, and how?
According to the Center for Disease Control (CDC) youth (school and college students)
are greatly affected by this social problem. According to CDC,
1. 15% of high school students reported having ever used select illicit or injection drugs.
2. 14% of students reported misusing prescription opioids
36
Students and youth engaged in substance abuse are involved in physical and sexual
violence. They are also at high risk of mental health and suicide attempts.
What actions, if any, do you see being undertaken to ameliorate these problems?
•
Parents and family engagement with complete support is required such as paying
attention to children’s issues with their colleagues, friends, and other people.
•
Parents should disapprove their children from substance abuse
•
School involvement is very necessary and students with behavioral problems or
other mental health issues must be treated appropriately.
•
Healthcare providers should follow up with parents to diagnose such issues at the
earliest.
Reflect and Discuss: Social Programming and Evaluation Through a Social Justice
and Transformative Lens
Imagine that you were asked to evaluate a housing program within one of the most
depressed areas of Detroit, Michigan.
From social justice and transformative perspectives, what might be some contextual
issues on which you would gather information to better understand the program’s
outcomes once received?
The depressed communities are constituted of poor-quality housing, few resources, and
unsafe conditions that impose stresses that cause anxiety and depression. The contextual
issues to evaluate a housing program in Detroit, most important is to consider
demographic characteristics of depressed people in Detroit. In a transformative lens, it is
important to consider the social, political, economic, and religious backgrounds of people
living in a depressed community. There could be negative biases and stereotypes
prevailing in the depressed community giving them a sense of inferiority.
What are some things that you, as the evaluator, might do? Might look for? Might ask
about it? Whom might you seek to talk with, and why?
I would consider several aspects as part of the transformative evaluation. I would look
for the demographics of people living in depressed conditions. I would consider the
common bases present against the community by other majority neighborhoods. In most
cases, it is propagated that people of color are involved in various criminal activities.
Further, we need employment status, economic condition, education level, and family
size to assess the overall socio-economic status of people. I would like to talk to people
of the community, school, social workers serving in the community, and other social
welfare program personnel to gain insight into the community dimensions.
37
Activity: Making Objectives SMART
What follows are several general and vague program objectives. Reformulate these
objectives, making them SMART, by using the criteria discussed in this chapter.
Increase student reading achievement
By the end of the academic year, almost 90% of students will attain higher grades (above
average) in reading.
Increase midlife adults’ engagement in physical exercise.
Almost 80% of midlife adults will be participating in physical exercise after completing
the intervention.
Enhance student motivation
By the end of the first quarter of intervention, almost 30% of students will demonstrate
motivation towards their academics.
Reduce bullying behavior
The bullying behavior of 80% of students will be reduced by 70% after participating in
the program.
Reduce school truancy
After receiving services from the health department, almost 90% of students’
truancy will be reduced significantly (more than 50%).
Activity: Design Your Project
Organize into small groups. Develop a hypothetical project in a selected area of your
choice (e.g., education, health, criminal justice). In your hypothetical program, you are to
do each of the following:
Give your hypothetical program a name
The name of the hypothetical project is, “ Academic Enrichment Program at University
Level”
Write a clear and concise mission statement
38
To develop academic and social skills and self-efficacy of college graduate students from
rural areas intend to seek admissions in various programs at the university level.
Write three program goals
1. To equip students with academic skills to increase students’ academic and research
skills required for programs at the university level.
2. To prepare students to meet emotional and social challenges at university.
3. To increase the self-efficacy level of fresh college graduate students planning to start
university education.
Write six SMART objectives (two per goal)
1. Almost 80% of students will show an appropriate level of academic and research
skills for university education.
2. Almost 90% will demonstrate required writing and computer skills for university
education.
3. Almost 80% of students will demonstrate appropriate emotional stability.
4. Almost 90% of students will develop good professional and working relationships
with their peers.
5. Almost 90% of students will show high self-efficacious beliefs for their academic
progress.
6. Almost 80% of students relate their academic performance to their abilities and skills.
Give a list of program activities
1.
2.
3.
4.
5.
6.
7.
Curricula that include basic academic, social, emotional, and motivational skills.
Hands-on practice
Printed and published material
Technology-based learning activities
Collaborative learning and leadership roles in activities
Filed trips to universities.
Seminars and workshops
Give a list of program resources
Computers, books, printed materials, transportation, and trainers (human resources)
Activity: Develop Your Draft Logic Model
Either using the project description that follows or the description of the hypothetical
project developed earlier in the Design Your Project activity (page 224), you are to
develop your project logic model. This activity should be completed in small groups so
39
that input can be gleaned from a variety of individuals. Imagine, as the evaluator, you
were asked to prepare and present this draft logic model to program staff for comment
and feedback.
40
Outputs
Outcomes — Impact
Inputs
Activities
Participation
Short
• Funds
• Trainers
• Classrooms
• Computer Labs
• Reading
Materials
• University
Partnership
• Volunteers
• Use of
technology-based
resources
• Academic writing
activities
• Research-oriented
activities that
required a review
of the literature.
• Group activities,
group projects
• Response to
emotional events
• Workshops and
training sessions
• Mentorships
• College
graduate
Students
Almost 80% of students will
show readiness towards
university education.
Almost 90% will
demonstrate required writing
and computer skills for
university education.
Almost 80% of students will
demonstrate appropriate
emotional stability.
Almost 90% of students will
develop good professional
and working relationships
with their peers.
Almost 90% of students will
show high self-efficacious
beliefs for their academic
progress.
Almost 80% of students
relate their academic
performance to their abilities
and skills.
41
Long
• Students will successfully
engage in university
education with an
increased level of
academic and research
skills.
• Students will show
balanced emotional and
social behavior during
university education.
• Students will likely
engage in social and
collaborative activities
• Students will demonstrate
high self-efficacy in
various subjects in
university education.
Chapter 8
Case Study: Classifying Stakeholders
Activities
Key Stakeholders: community people living in Ohio neighborhood, drug abusers (adults
and youth who do drug abuses), community workers, social workers, patients, families,
physicians, researchers, law enforcement and government regulators, media,
pharmaceutical companies, health department, and policymakers.
Underserved and/or underrepresented persons: There could be some people who are
underserved or underrepresented people are law enforcement agencies and
pharmaceutical companies who might not be directly considered as affected by the
program.
Second, go back and classify these stakeholders as primary vs. secondary, direct vs.
indirect, and internal vs. external. Specify what might be their primary interest in the
evaluation (build their evaluation capacity; use the evaluation results to improve service
delivery, etc.).
Primary stakeholders: Families, Patients/substance abusers, physicians, government, and
policymakers.
Secondary stakeholders: Community/society, police, and other law enforcement
agencies, pharmaceutical companies, social workers, religious organizations, youth
development programs, colleges and other educational institutions and media
Internal: Families, patients, and physicians
External: police, social workers, pharmaceutical companies, media, religious
organizations, youth development programs, colleges and other educational institutions,
and governmental agencies to control drug abuse.
Direct stakeholders are parents, families, patients, and physicians
Indirect stakeholders are those who are secondary and external stakeholders as listed
above.
Third, extend your thinking about Project End Zone to prioritize who you believe are the
“right stakeholders” for the evaluator to engage during the planning, implementation, and
dissemination of evaluation findings. Why did you select these particular stakeholders?
For the planning phase, we will consider primary and secondary stakeholders to target
however during the implementation phase only primary stakeholders were included to
42
deliver the intervention. For dissemination purposes, we will include all potentially
identified stakeholders.
Reflect and Discuss: Hidden Stakeholders Speak Out
Why do you believe that American Indian teens were “hidden stakeholders” in the teen
smoking intervention and research (and evaluation) efforts in North Carolina?
Because American Indians were less in numbers in the community and evaluators were
mostly whites and only concerned about ruling majority or other significant minorities
such as Hispanic and African Americans. For evaluators, it is important to have a
significant number of community members belonging to a particular sub-group to be
included in the research. There are two potential reasons that American Indians were
ignored; first that American Indians might not be smoking in the community and second,
the evaluators had biases for them to include in the study for some reasons.
What activities or action steps would you suggest for engaging this “hidden” population
in the design and evaluation of smoking intervention programs?
To include the hidden population, initially, demographic information is required targeting
vulnerable American Indians and their characteristics indicating their vulnerability in the
community. Secondly, the researchers should contact project administrators and funders
asking to include an important component that can significantly impact the project
evaluations. The researchers can identify the cause which excluded such groups to be
part of research before contacting ignored groups and or making decisions to include
them in the study.
Activity: Identify Potential Stakeholder Conflict
Who are the key stakeholders in the REACT intervention?
Patenting with coronary diseases of having the potential risk of heart attack, doctors,
paramedical staff, Emergency services, healthcare providers, pharmaceutical companies
and policymakers are key stakeholders.
What potential conflicts could emerge between/among the identified stakeholder groups?
There could be a conflict between patients, doctors, and emergency service providers. It
is presumed that delay can occur on behalf of a patent who couldn’t perceive heart attack
symptoms properly and response to medical treatment was delayed. The delay could be
fatal and patients can put the liability on doctors for providing emergency care.
43
What strategies could the evaluator use to minimize this tension?
Avoid direct communication between stakeholders
Try to find conflicting information and do not discuss it openly with everyone.
Develop a compromise between stakeholders and give equal weightage to the opinion of
both groups.
Questions
List of stakeholder groups
Who has deep cultural contextual
and?
Program recipients, their families, and
peers
Who are the individuals or groups
The program managers, and trainers
Who is affected directly and indirectly
The service providers and recipients
are affected directly, however
indirectly all those people are affected
who have vested interest in the
program such as people in proximity
of recipients.
Who is involved in program operation
Program manager, intervention
providers, supporting staff, etc
Who will use evaluation results?
Policymakers, and funders
What are hidden, but potentially
important
All those people that come under the
umbrella of the defined problem but
are not identified as the target
population
Who is providing support
Funders, government agencies, or
private organizations are responsible
for the execution of the program and
providing funds.
To determine which stakeholder does what?
1. Provide guidance: The policymakers, program managers, and funders
2. Enhance credulity of evaluation: Key stakeholders can increase the credibility of
evaluation by taking ownership of evaluative study. By including the perspective of
hidden stakeholders credibility can also be enhanced.
44
3. Influence how results are used: Funders can influence
4. Make decisions for initial funding: Funders and program managers, along with
policymakers can decide
Activity: Developing a Stakeholder Engagement Communications Plan
Stakeholder
s (who)
Message
(what)
Mediu
m
(How)
Schedule
(when)
Person
responsibl
e
Patients
Intervie
w
Inperson
Anytime
at
Evaluator
Families
Intervie
w
Inperson
Any time
scheduled
Evaluator
or staff
Doctors
Intervie
w
Inperson
Schedule
d time
Evaluator
45
Midterm Assignment “Chapter 1 – 8”
Chapter 1
Activity: Defining Evaluation
In small groups, discuss the characteristics you think should be included in a
definition of evaluation.
Egon Guba (1985) defined evaluation as “disciplined inquiry” with a target to improve a
program by determining its value.
The recent definitions include such as by Rossi et al. (2019) a reference to context and an explicit
goal to “inform social action to improve social conditions” (p. 6).
Based on the definition we should include in the definition,
An evaluand, the purpose of evaluation, process, or procedure and the context in which results
will be interpreted.
Activity: Applying Evaluative Thinking
“What are we assuming? Do we know that?”
From the text about money, we are assuming that money determines the socio-economic status
of race, ethnicity, and family education which we don’t know actually. We can’t determine race
or ethnicity-based on money.
“How do we know that?”
1
We know from prior studies which have established a relationship between money and
race/ethnicity and education of the family.
“What makes you say that?”
The evidence related to money as sources of power and access to resources and association with
some race and ethnic groups as available in the literature make us say that.
Seek Out Blind Spots
“What are we missing?”
We don’t know other financial resources, total assets, family sizes, number of people working in
the family, and types of jobs.
“Whose perspective isn’t represented?”
People from rich families are not presented here. In addition, the evaluator’s views are given but
not the respondent’s views. There should be the perspective of respondents who would-be
participants of the study.
“What other explanations could there be?”
The other explanations include the relationships of other demographics such as gender, family
size, and type of work. Further, it is required to add, the type of support poor families are
obtaining and what are their rights and obligations in participating in these surveys/studies.
Capture Musings & Learning Questions
“I wonder if …” people of poor families refused to participate in the study.
2
“I bet if we …” give right of voluntary participation, many could withdraw.
“If I knew __________, I could _________.” (IllumiLab, 2018b, “Asking Questions”)
If I knew people have fears, I could allow them to express their fears and participate voluntarily.
Reflect and Discuss: My Biases
In small groups, discuss some of the fairly superficial preferences and biases you might have.
Then, either speaking in general or personally, discuss preferences that might impact how
someone approaches a project.
From an evaluation perspective, I don’t have any specific superficial preferences, however, as the
text of the book reveals some group preferences such as the supremacy of whites over blacks. I
might prefer Caucasians or people of the Arabian Peninsula to others. My other potential
preferences include women over men and especially married student women who are taking care
of their families and meeting their academic requirements. Similarly, biases can be mostly
cultural, which could prevent reasonable, knowledge consideration of a question. The
international biases in my case would be the inclusion of international women students with
children, and sympathies for such families who are struggling with their education as well as
meeting their family needs.
The preferences could reduce development and reasonable consideration of a question.
The biases and preferences can result in the selection of inappropriate methods, sample (sample
biases), interpretation of results, and concluding. Most preferences and biases affect the quality
of evaluations in terms of validity and reliability of data and results.
3
Voices from the Field
In his 2019 Voices From the Field interview, as follows, Melvin E. Hall discusses objectivity,
bias, and why he became an evaluator.
He discussed built-in structural biases, which are little b-biases and big B biases. Melvin chose
the option of balancing biases with other perspectives out of two options. Melvin became an
evaluator to make other people realize and understand their own biases as well as the evaluator’s
biases. The biases are not only unavoidable but in many instances, these shouldn’t be avoided as
some biases are positive and others may be negative, For example, he expressed his positive bias
towards HBCUs being an affiliate of HBCU. The best thing he mentioned, is to know and
understand your own biases and entangle your own biases with other peoples’ biases.
Activity: Reflections on Working with White People and Antiracism
Read the following three statements and write a short paragraph about your response to them and
any impact they may have on your response to race and racism.
To three responses, I would endorse DiAngelo’s (2018) comments with an endorsement that
racial inequalities cannot be diminished from society just because of being nicer and having a
smile on the face for people of color. The internalized superiority of Whites in the system cannot
simply go away and effects of superiority can be seen in the society regardless of the positive
behavior of Whites with people of color. Similarly, following Tenney’s point of view, intentions
4
do not matter solely but what matters is the supremacy of ruling race and fight against racial
disparities.
Racial inequities are not contemporary and have a long history in the world. The racial
disparities were started in the US since the transportation of Negro slaves from Africa to Europe
and then to America. There is induced inferiority among people of color and superiority in
Whites since the inception of this society. The racial disparities can be seen in various areas of
public interests such as education, health, and justice. It is very quick and easy to understand
such behavior and therefore regardless of good behavior, the major concern is the fight against
racism.
Case Studies: Evaluation Results vs. Politicians
As the following case studies indicate, evaluation results can and at times do lose out to
politics.
In the first example of 21st Century Learning Centers, evaluation results were not politically
approved and the California governor got public approval in favor of the program regardless of
its failure to meet proposed objectives. The program was found to be not effective for students’
outcomes after evaluative study, however, general public opinion did not accept the evidence and
justified the existence of the program in the community.
Evaluations are performed under certain circumstances and public opinion is also
accounted for evaluations. A researcher should consider political policies and forces while
5
evaluating programs. Evaluation results lose the attention of political attention when evaluation
results don’t mitigate with political policies and general public opinion.
In the case of DARE which targeted school-aged youths was found to be ineffective through
multiple evaluations and meta-analysis; however Federal funding remained continued for the
program. The program continued to be used in some schools for a variety of reasons including
the belief that a short-term program can’t change students’ drug-taking behavior.
Sometimes government wants to fund some programs even with their short-term failures
as politicians have their vested interest in their communities. The community service programs
are not fully stopped based on evaluations, but can gradually be minimized with approval of
political opinion. Political disapproval appears when evaluation results do not negate public
opinion at large.
6
Chapter 2:
Reflect and Discuss: What Ethics Means to You
Most adults have a vision or image in their mind of an ethical business, ethical organization,
ethical government, or ethical society. On the individual level, ethics has a specific and
oftentimes unique meaning and source. Using the prompts that follow, reflect on and discuss
what ethics means to you.
•
What does ethics mean to you as a student, parent, spouse/partner, employee, and/or
other roles?
As a student my ethics are
Be fair and honest in my work, and don’t use unfair means to my academic work. I was a
student would practice my autonomy of work and follow institutional guidelines in exercising
my autonomy. I value my work and would like to not be harmful to anyone and work
independently within my limits. I believe in fair treatment, justice, and equality which is are
legitimate ethical considerations for any student, even beyond studentship.
•
What is the basis of your ethical decision-making and behaviors?
My ethical values come from my religious beliefs and cultural affiliations. The culture I
belong to is an ideal Muslim society that practices all good Islamic values and traditions,
therefore a source of my ethical decisions comes from my religion and culture. My family is
also values-oriented that inspires my ethical considerations.
7
Reflect and Discuss: The Tuskegee Timeline
Review the timeline of the Tuskegee Syphilis Study and then reflect on and discuss the questions
that follow.
•
Do you think the Tuskegee Syphilis Study has a current and/or lasting societal impact? If
so, what?
The study has a long-lasting effect of increased mistrust and mortality among people of color
due to violating ethical rules. They were never well informed about the purpose of the study
and were not treated for their disease even the treatment was available for Syphilis. In the
Tuskegee Syphilis Study, men were informed they were treated for their disease and given
other benefits such as a free exam, free meals, and burial insurance.
•
Might certain individuals or groups still be suspicious of evaluators and other researchers
because of experiments like the Tuskegee Syphilis Study? Elaborate on your response.
Other studies (e.g. The Radiation Studies, The HeLa Story, and others) revealed
vulnerable people who are potentially suspicious about evaluators and other researchers.
For example in the radiation study, most of the participants were not aware of the nature
and type of experiments that would be performed on them. The participants were hospital
patients, pregnant women, children with intellectual disabilities, and military personnel.
There could be any vulnerable pupation who could be recruited for research without
giving them informed consent and the right to withdraw.
8
Reflect and Discuss: Ethical Considerations and Authority Figures
In contrast to the 1940s and 1950s, in today’s society do you think everyday citizens are more
aware of ethical behavior related to social and behavioral sciences research—that is, right versus
wrong—or do you believe that an authority figure will always be able to sway people’s judgment
toward unethical behavior? Under what conditions do you believe this is less or more likely to be
the case? Provide examples.
Yes, at present people are aware of their rights and ethical behaviors; certain laws secure the
rights of participants and lower the risk of being misused or maltreated. For example, Belmont
Report established three basic ethical principles, respect for persons, beneficence, and justice.
People should be aware of their rights and it is the responsibility of researchers to inform
individuals of their safety and wellbeing beforehand. It is also practical to have an authority
figure to protect peoples’ judgment towards unethical behavior. For example, governmental
agencies and other non-profit organizations that look after public rights and safety issues could
be there to monitor public rights. Institutional review boards (IRBs) are defined authorities to
protect such rights and look for ethical behaviors.
Case Study: Identifying Hidden Agendas and Ethical Land Mines
•
What do you see as the major ethical issues that the evaluator is facing in this case?
•
The program director has perceptions that findings of evaluation could influence his post
and staff member can use it against the director.
•
The program director is not agreed with the survey questions and has apprehensions of
negative opinions of the angry students about HSC.
9
•
The director wants evaluators to modify the survey to minimize the negative responses of
students and other respondents.
•
The directors want to use items measuring negative aspects of these items that are used in
evaluations of other programs at the University. However, those programs are evaluated
by different evaluators.
All these factors created an ethical dilemma by affecting the evaluation procedure and
challenging evaluators’ integrity and honesty for evaluation (violations under sections C and E).
Evaluators should apprehend potential threats and influence of staff members in bringing
negative opinions of students, therefore it is recommended to use multiple ways of data
collections (Quantitative and Qualitative; Likert-type scales, date review, focus groups,
interviews, and or observations) and should not rely solely on the survey. Individual interviews
or focus groups can provide a better insight into the program and also can minimize the
apprehensions of the program director. An evaluator is responsible for including possible
oppositional opinions and suggests the required course of action for a chance in the evaluation
plan (D1).
•
What hidden agendas could become ethically problematic for the evaluator?
The hidden agenda of the evaluation is to declare certain domains of the HSC’s functioning
as “off-limits” in this evaluation. However, an adequate and professionally respectable
evaluation of the HSC cannot omit a consideration of the areas that the director wants to
avoid.
•
What might be some professional consequences for this evaluator if the ethical issues
remain unresolved?
10
•
The evaluator can lose professional license, memberships, and credibility
•
Be penalized for ignoring important aspects of evaluations
•
Can be fired from his/her job for not following their ethical principles and professional
ethics
•
Can face criticism by colleagues and other professionals
•
Funders can cease/her payments and disengage from evaluation
•
The impact of the evaluation can greatly impact those served by the evaluated
organization/program
Reflect and Discuss: Evaluator Sources for Ethical Thinking
Consider the five sources of ethical thinking described as follows and the examples provided.
Now, provide additional examples of instances when an evaluator should avoid or rely on one or
more of these sources while working in the field. Reflect on and discuss your responses with
others.
A program director can ask the evaluator to collect data from a particular school where “students’
academic program” was successfully implemented and school staff is directly related to the
program director for gaining some extra monetary benefit for implementations. In return, they
are expected to give their satisfaction and success stories of the program. An evaluator found
suspicion through his/her intuition. However, evaluators don’t rely completely on one source of
thinking for ethical issues and use multiple strategies. For example in the above-stated example,
an evaluator can not only use intuitions in foreseeing a potential risk but can use his/her vision
and experience in making decisions. He/she has to follow standard rules and procedures
11
beforehand and practice such rules along with using ones’ past experiences. Evaluators can’t rely
solely on one single source and evaluators who take advantage of their values and beliefs can be
shaky in many cases if they are working with different cultural groups. Similarly, their
experience can’t be adequate to guide them to the right decisions. For example, an instrument
used in a dominant setting does not make it automatically valid and reliable to apply in other
settings.
Case Study: Moving Beyond Past Experience
In this case study, it is given that items in the instruments that are acceptable in one culture are
not necessarily applicable to other cultures, For example, some respondents can be offended by
some of the items which are not greatly practiced in that culture and against the norms of cultural
values. To make such an instrument applicable to another setting, the evaluation team discusses
issues and revises protocol to make it feasible for the target population.
Reflect and Discuss: Self-Exploration
•
Consider diverse cultural experiences that have led you to further “self” exploration.
What did you learn about yourself that you were not consciously aware of before this
self-exploration?
Being culturally competent, one should know cultural values, norms, liking and disliking,
applicable and non-applicable things, biases, conflicting aspects, political and social issues,
religious aspects, and the role of men and women in society. After exploration of self, I
found that several aspects were not known to me about awareness of others’ culture.
Normally we consider that cultural competence means a general overview of cultural aspects
such as language, religion, foods, rituals, values, and common law. In the transition from
12
Saudi Arabia to the US, I became aware of several aspects which I didn’t know before.
Likewise being an evaluator, there are many things to be known to become a culturally
competent evaluator.
•
After reviewing the AEA’s (2011) Public Statement on Cultural Competence (available
at www.eval.org/ccstatement) in full, discuss how exploring the “self” can improve one’s
work as an evaluator.
•
After exploring the self through cultural perspective and evaluator can improve
his/her work significantly by
•
Use culturally appropriate approaches, methods, procedures, and research tools.
•
It increases trustworthy understating and increases validity and reliability.
•
It reduces errors induced in cultural biases, stereotypes, and lack of shared
worldview among stakeholders.
•
Can reflect accurately opinions of respondents and conclude accurately.
Activity: Ethical “Blind Spots” in Evaluation
Sezer and colleagues (2015) delineated various sources of ethical blind spots. After a review of
these sources of ethical blind spots, summarized as follows, organize into small groups. Then
discuss them as sources of evaluator blind spots. Give specific examples of how these sources
might affect evaluators’ sensitivities and course of action in practice.
Implicit biases can be seen as implicit attitudes and egocentric biases that can affect the
sensitivity of the evaluator. Evaluators may not be able to recognize their positive and negative
views of others which can be seen in evaluative practices. For example, a White evaluator may
13
have a preoccupation about African American children as cognitively slow learners. Similarly,
the evaluator may ignore unethical behavior of other people due to lack of knowledge, therefore
such action of the evaluator may produce negative or positive outcomes based on perceived
unethical behaviors.
Case Study: The Compromised Evaluator?
Presenting favorable results in the yearly report are issues of integrity and honesty of the
evaluators. The evaluator tried to gain the benefit of being involved in the project for the long
term.
Case Study: Revising the Evaluation Report
The ethical dilemma that occurred here is hiding or overlooking potential operational
factors that appeared to be negatively affecting the program. However, positively presenting the
report and informing stakeholders that the program has performed very well the drawbacks of the
program cannot be ignored. The positive tone of evaluator/s could bring stakeholders and funders
to the result that the program was successful and had not found any drawback. According to
AEA (2018) section C: integrity, it is the responsibility of an evaluator to behave with honesty
and transparency for th…