For your area of interest in the program, select a research article from the Journal of Management Information Systems journal and post the reference for that article in APA format. Summarize the article in terms of the research question and how it was researched.
A Wikipedia-based Method to Support
Creative Idea Generation: The Role of
Stimulus Relatedness
KAI WANG AND JEFFREY V. NICKERSON
KAI WANG (kaiwan@kean.edu) is an assistant professor of management in the
School of Management and Marketing at Kean University. His research domains
include using information systems to support creative work and crowdsourcing
innovation. His work appears in Creativity and Innovation Management and
Computers in Human Behavior.
JEFFREY NICKERSON (jnickerson@stevens.edu) is a Professor and Associate Dean of
Research in the School of Business, Stevens Institute of Technology. His research and
teaching interests include creativity support systems, autonomous tools, and collective
intelligence. He is currently the principal investigator of three NSF-funded projects
related to creativity and productivity in online communities. He has published in MIS
Quarterly and ACM Transactions on Computer-Human Interaction.
ABSTRACT: Providing stimuli may facilitate idea generation. Creativity theories ofte
n
suggest that stimuli unrelated to the problem task will improve creativity, but empirical
studies have yielded inconsistent results. We propose a Wikipedia-based approach that
is able to identify stimuli at different levels of relatedness. Specifically, we use
hypertext links in two sections of the Wikipedia article of a focal concept to identify
closely related concepts. Repeating this procedure leads to increasingly remote con-
cepts. Using this approach to obtain stimulus concepts, we examine the effect of
stimulus relatedness on idea generation. Our results show that stimulus relatedness is
positively related to idea quantity and idea usefulness. While creativity theories often
suggest using unrelated stimuli to promote idea novelty, results of this experimental
study indicate that remotely related stimuli, not unrelated stimuli, tend to improve idea
novelty. Because Wikipedia covers knowledge in almost all disciplines, our Wikipedia-
based approach can be used to discover appropriate stimuli and thereby support
creative work in most domains of knowledge.
KEY WORDS AND PHRASES: creativity, idea generation, creativity support systems,
creativity stimulation, association in creativity, Wikipedia.
Organizations develop new ideas, products, and services in order to survive and
succeed [4, 38, 40]. There is a research stream within the IS discipline that focuses
on creativity support systems (CSS), information technology-based tools that
Journal of Management Information Systems / 2019, Vol. 36, No. 4, pp. 1284–1312.
Copyright © Taylor & Francis Group, LLC
ISSN 0742–1222 (print) / ISSN 1557–928X (online)
DOI: https://doi.org/10.1080/07421222.2019.1661095
mailto:kaiwan@kean.edu
mailto:jnickerson@stevens.edu
https://crossmark.crossref.org/dialog/?doi=10.1080/07421222.2019.1661095&domain=pdf&date_stamp=2019-10-03
enhance the creative output of individuals or groups [3, 30, 51, 68]. A variety of
approaches have been developed in CSS research, including guiding people through
steps in a creative process [24, 46, 68], using mind maps [2, 47], facilitating
creative techniques [32], and supporting group collaboration [2, 23, 30, 39, 56].
One particularly interesting approach is to provide stimuli to inspire new ideas,
such as existing example solutions [3, 62], analogies [3, 39], concepts [42, 47, 64],
and pictures [37, 46, 65, 66]. Are external stimuli always beneficial for creativity?
Studies on idea generation indicate that showing concepts or design examples can
sometimes constrain thinking and reduce creativity by inducing fixation: designers
can be so attracted to examples that they find it difficult to fully explore the design
space [8, 10, 28, 41]. It is clear that creativity support systems need algorithms that
are cautious and selective in identifying stimuli for supporting creative work.
One important property of a stimulus is the degree to which it is related to
a creative task, that is, stimulus relatedness [58]. Some creativity theories suggest
that a stimulus that is less related to a creative task may lead to novel associations
and increase idea creativity [48, 52]. This notion has some empirical support, fo
r
example, in engineering design and group brainstorming [13, 14, 19, 39]. However,
there are also studies showing that the cognitive distance of stimuli from a domain
is not related to the creativity of resulting ideas or solutions [46, 63]. Two recent
studies even show that exposure to remote examples [3] or citing conceptually
distant solutions [12] is associated with lower creativity. In order to better under-
stand the effects of stimuli, and to build effective creativity support systems, there
is a need to find stimuli along a spectrum from highly related to unrelated, as well
as to understand the potential effects of such calibrated stimuli.
This study aims to develop an automatic way of providing stimuli for creative
ideation, and to improve the understanding of how stimulus relatedness influences
idea generation. The next section provides a review of past theoretical and empiri-
cal research that addresses the use of stimuli to promote creativity. Our hypotheses
on the effect of stimulus relatedness are then presented. Afterwards, we introduce
and validate a Wikipedia-based approach for finding stimuli that are related to an
initial concept along a spectrum of relatedness. Two experiments used this new
approach to find stimuli of a range of relatedness, and then tested their effects on
creative idea generation. The implications for future research in information sys-
tems and practice are discussed subsequently.
Creativity is typically defined as the generation of products or ideas that are novel
(or original) and appropriate (or useful) [2, 3, 28, 40]. Here a review is provided on
theoretical and empirical research on creativity and stimulus relatedness. We briefly
discuss cognitive theories on creativity, then we explain how stimuli may influence
idea generation, with a focus on the role of stimulus relatedness.
A WIKIPEDIA-BASED METHOD TO SUPPORT IDEATION 1285
Cognitive Theories of Creativity
In this section, we focus on those cognitive theories that have direct implications on
using stimuli in creative idea generation. Ideas are commonly considered as pro-
ducts of existing information in minds [52, 58]. Consequently, theories of idea
generation are often based on theories of memory processes. Two well-cited
theories on memory processes are the Search of Associative Memory theory
(SAM) [57] and the Adaptive Control of Thought (ACT) theory [5, 6]. Both
theories claim that long term memory is an associative network of memory units.
Short term memory, or working memory, has limited capacity and contains ele-
ments that can be thought of as search cues. That is, these elements are sources of
activation that probe long term memory. The probability of activating certain
memory units (chunks in ACT; images in SAM, no visual representation implied)
is based on the association strength between the search cue and the memory units.
SAM emphasizes the retrieval plan, which specifies a series of search and recovery
operations. A retrieval plan can be changed as search proceeds. The retrieval plan is
used to determine how to choose or combine cues and what cues are used at each
stage of search. Different search cues may be used at different stages of search.
ACT builds on the spreading-activation theory of semantic processing and assumes
that a stimulus will activate some concept node in the semantic network in mind,
and the activation automatically spreads to other concepts across the network based
on associations among memory units [16].
Consider the example of a particular creative task: coming up with new ideas to
promote a hotel. According to SAM, when a person is faced with this task, working
memory will necessarily contain the information related to this task. If a stimulus
word, for example cooking, is presented, the word may be added to the existing
information in working memory. A retrieval plan is generated in working memory
that uses both task information and cooking as search cues to probe long term
memory. SAM posits that search is focused on information that is strongly con-
nected to all the search cues, in this case, both the task and cooking. Therefore,
different features and associations about cooking may be activated, such as the
concepts of menu, recipe and diet. In addition, more idiosyncratic memories may be
triggered: for example, of cooking shows that are framed as athletic contests. The
activated knowledge is evaluated and if appropriate, will be used for ideation. For
example, an idea might be generated about a weekly menu with healthy choices; or,
perhaps, introducing a competitive walk on hotel grounds before or after meals as
part of an exercise regime. If this round of search does not turn out successful, the
newly activated information, combined with old information in working memory,
will be used to generate the next search cues, leading to a new round of search,
unless a decision to terminate search has been made.
According to ACT theory, knowledge in mind has a baseline level of activation,
which will be enhanced by external stimuli. Using the aforementioned example, the
word cooking will activate some chunk in declarative knowledge based on associa-
tion strength. For example, the features and associations of cooking are likely
1286 WANG AND NICKERSON
activated. The activated knowledge will then be used for further processing or
ideation. Such activation can automatically spread into other chunks based on
associations. Consequently, additional ideation based on further associations of
cooking is possible. Notwithstanding the differences, the SAM model and the
ACT theory depict a similar overall picture of the memory retrieval process
where knowledge is activated based on the association strength between search
cues and the knowledge units. Using concepts to stimulate ideation involves both
conscious and subconscious aspects. Deliberate attention to and use of concepts
takes a conscious effort. The activation or retrieval of knowledge in mind and its
spreading contain automatic and subconscious processes.
Based on SAM, a theory of idea generation was developed, called Search for Ideas in
Associative Memory (SIAM) [52]. According to the SIAM model, idea generation is
a repeated search process with two stages. First, a search cue, such as the problem
definition or a previous idea, is used as to activate certain knowledge in long term
memory. Such activation is probabilistic: the activation is dependent on the strength of
the association between the search cue and the knowledge. Then, in the second stage,
the activated knowledge is combined or processed by working memory to generate
ideas. A similar theory is called the cognitive network model of creativity (CNM) [58].
This theory also assumes the existence of long term memory as an associative network
of knowledge and the existence of working memory containing activated knowledge.
The CNM theory argues that, in problem solving, the diversity of external stimuli
increases the disparity among activated knowledge, which tends to increase the
creativity of solutions. However, the number of stimuli per unit time and the disparity
among activated knowledge also increase cognitive load, which can in turn inhibit
creative thinking. While the SIAM theory is quite detailed in elucidating all the steps in
the idea generation process, the CNM theory stresses the role of knowledge distance
and cognitive load in creativity.
Influence of Stimuli on Idea Generation
There are two major ways of using stimuli in idea generation: priming and
deliberate conscious use of stimuli. Priming is defined as presenting stimuli to
activate certain mental representations of concepts, attitudes, or beliefs that affect
the behavior on a later task [22, 55]. Priming typically affects people subcon-
sciously: people are unaware that the stimuli are activating mental representations
and do not know the intent of priming [22]. It has been shown that being exposed to
example uses of objects led people to associate the objects with certain functions,
making it difficult to come up with other functions for the objects [1, 34]. In one
study, the participants were primed with a computer game [22]. In this game, people
selected and arranged words into headlines that emphasize achievement, such as
“scholar aspires for honor.” The participants who experienced this achievement
priming generated more creative ideas in group electronic brainstorming, compared
to neutral priming. In another study, the researchers used a similar scrambled-
A WIKIPEDIA-BASED METHOD TO SUPPORT IDEATION 1287
sentence task to prime either prosocial or efficiency norm with related words [55].
Then, the participants generated ideas for an open-ended problem and the ideas
matched the prosocial or efficiency norm primed earlier. Priming using such games
may activate the semantic content related to the prime (such as achievement or
prosocial norm), which is used in later tasks [22]. However, it is also possible that
the priming introduces a subconscious goal (such as achievement) and affects the
motivation and effort [22].
Unlike priming, in the second approach to using stimuli people are consciously
aware of the intent of using stimuli. People deliberately consider stimuli as inputs in
the ideation process. The following studies, as well as our study, use this second
approach. Design examples can lead people to focus on familiar categories and
schemas and generate designs of limited originality [8, 28, 29, 61]. This narrow
focus can be attributed to the tendency towards taking the path of least cognitive
resistance [28, 61]. The fixation effect is especially salient when stimuli are
common rather than novel [54, 60, 71].
Some empirical studies on engineering design show the positive effect of remote
stimuli (examples or words) on creativity [13, 14]. Similarly, it is found that analogies
can transfer information or relational structures from distant stimuli and lead to more
creative outcomes [19, 35, 39]. In addition, it is found that when people are exposed to
novel or paradigm-modifying ideas (serving as remote stimuli), they tend to generate
such ideas [32, 46, 59, 60, 71]. However, the notion that distant stimuli promote creativity
is challenged by many studies. In an engineering design experiment, the patents that were
moderately dissimilar were more useful as analogies that stimulate ideas [31], which
means that example solutions that were too dissimilar were not very useful. In a study on
generating marketing campaign proposals for a beer company, people with access to
campaign proposals for dissimilar products generated less creative ideas, compared to
people with access to campaign proposals for similar products [3]. In another study, the
authors analyzed hundreds of design concepts on an online innovation platform that
tracks connections to sources of inspiration [12]. They found that conceptually closer
sources are more beneficial for design creativity, compared to conceptually far sources.
One disadvantage of using remote stimuli is that such stimuli are often not recognized as
relevant [31, 63]. Therefore, there might be an optimal range of stimulus relatedness,
within which stimuli are neither too close nor too far to be beneficial [2, 3, 19]. However,
to our best knowledge, this notion has not been empirically tested, perhaps due to the
difficulty in obtaining stimuli of a spectrum of relatedness.
In addition, there are studies showing that the cognitive distance of stimuli has
little impact on the creativity of resulting ideas or solutions [25, 46]. In an experi-
ment on the generation of new ice cream flavors, words and pictures that were
random or closely related to the task led to the same level of idea creativity [46]. In
studying cross-industry innovation, some researchers analyzed the impact of the
cognitive distance between the acquired knowledge and the problem to be solved
[25]. They found no direct correlation between the cognitive distance and the
radicalness of innovation.
1288 WANG AND NICKERSON
In summary, creativity-related theories often indicate the benefit of using stimuli
that are distant or unrelated to a creative task. But this notion is both supported and
refuted in empirical studies, so no definitive conclusion can be drawn. In addition,
previous studies typically have three limitations. First, since different studies often
have different ways of defining stimulus relatedness, remote stimuli in one study
could be considered moderately related or even closely related in another study
[31]. This inconsistency may contribute to the conflicting results. Second, the
literature rarely distinguishes unrelated stimuli (as may happen through random
selection of concepts from a wide range of knowledge domains) from remotely
related stimuli (that can be connected to the focal task through a couple of
associative steps). It is presupposed that people will consider and use unrelated
stimuli in a sensible way, even though this might not occur [12, 69]. Third, previous
experimental studies typically manually collect or generate stimuli (e.g., [10, 45,
46, 49, 58]). Such manual effort is often difficult to perform, and so can lead to
heuristics or short cuts that can potentially bias the selection of stimuli. There are
a few computational approaches for searching stimuli to support creativity [2, 3, 65,
66], but they typically focus on a specific domain and rely on a database that has
been built in advance. There is a lack of generalized and automatic approaches for
finding stimuli along a spectrum of relatedness. In this article, we propose a simple
and automatic way to find stimuli of various levels of relatedness and, based on it,
test our theoretical predictions of the effect of stimulus relatedness, which are
explained in the following sections.
A stimulus can be closely, moderately, remotely related, or unrelated to a focal
ideation topic. The stimulus relatedness can affect the number of ideas generated, as
well as idea novelty and usefulness. First, we contend that the number of ideas
generated is affected by stimulus relatedness. The responsiveness to a stimulus is
positively related to the similarity between a stimulus and the current cognitive
state [17, 48]. The less related a stimulus is, the less likely it is similar to the
currently considered concepts or categories. Therefore, it is less likely for people to
respond to an unrelated stimulus, internalize it, and use it to stimulate ideas.
Adopting the terms of SIAM, using unrelated stimuli as search cues may result in
cognitive failure which might terminate the idea generation process. In the perspec-
tive of the cognitive network model of creativity, considering a less related stimulus
would activate distant knowledge, adding to the disparity among active knowledge
and hence increases cognitive load [58]. The amount of cognitive resources for
ideation is reduced, which tends to lower the number of ideas. In other words, it is
difficult to jump among and connect remote areas in the cognitive network and use
the associations to generate ideas on the focal topic. Consistent with this set of
arguments, some studies show that original or irrelevant stimuli tend to reduce the
number of ideas [37, 69]. Typically ideation consists of the generation of multiple
A WIKIPEDIA-BASED METHOD TO SUPPORT IDEATION 1289
preliminary ideas and the development of a final idea [10]. In this case, we contend
that the number of preliminary ideas is positively related to stimulus relatedness.
Hypothesis 1: There is a negative relationship between stimulus relatedness
and the number of ideas generated.
Stimulus relatedness affects the usefulness of ideas generated. Relevant stimuli can
easily activate knowledge that is more applicable to the focal ideation topic [2, 3]. In
other words, related stimuli are connected to the target problem in meaningful ways,
thereby contributing to the production of useful ideas. Related stimuli tend to promote
the search within the current idea category [3] and thus possibly anchor the ideation on
existing useful ideas and improve them by adding relevant concepts, features and
mental frameworks [10]. Similarly, it is argued that familiar stimuli can improve idea
usefulness by infusing meaning, clarity, and legitimacy into ideas [10, 38]. In contrast,
we argue that unrelated stimuli reduce idea quantity (as argued in hypothesis 1) and
idea usefulness. There are no obvious connections between unrelated stimuli and
a focal topic. Consequently people find difficulty in applying the knowledge activated
by the stimuli and developing appropriate ideas [2, 3]. The resulted ideas might be
unique yet less useful. Indeed, in technology innovation, trying new components and
new combinations leads to less useful inventions on average [27]. An unrelated
stimulus might even point to an unproductive path that leads to meaningless ideas. In
summary, high stimulus relatedness is associated with both high number of ideas and
high level of idea usefulness. Combining the two effects, when generating preliminary
ideas, highly related stimuli lead to larger number of useful ideas. With a high number
of useful raw ideas, the usefulness of the final idea tends to be high as well.
Hypothesis 2: There is a positive relationship between stimulus relatedness and
idea usefulness.
As argued in the path of least resistance model [28, 70] and the cognitive network
model of creativity [58], when people solve problems or generate ideas, they tend to
use familiar examples or previous solutions as a starting point and include many
properties from these examples or solutions in ideas developed. Thus, without
external stimuli, people often come up with unoriginal ideas. If a stimulus is closely
related to the creative task, the stimulus tends to activate knowledge that is also
highly related to the task. This highly related knowledge would probably be
activated even when people consider the creative task without the stimulus.
Consequently, compared to using no stimuli, a closely related stimulus is unlikely
to introduce many new elements and improve idea novelty. Consistent with SIAM,
when a stimulus is remotely related to a task, the knowledge activated by it will be
less related to the task as well. The less relevant knowledge may lead to some
original ideas because it adds new cognitive elements and potentially brings in new
perspectives. Similarly, it is argued that exposure to new information can help
reduce design fixation and improve creativity [63]. Furthermore, the cognitive
network model of creativity argues that original solutions result from new
1290 WANG AND NICKERSON
connections among previously unconnected knowledge [58]. Because remotely
related stimuli can activate less related knowledge, they increase the possibility
of forming such new connections in knowledge. Therefore, remotely related stimuli
tend to result in higher novelty in final ideas.
Hypothesis 3: Using stimuli that are remotely related to a creative task leads to
higher idea novelty, as compared to using no stimuli.
We posit that totally unrelated stimuli are less effective in promoting idea novelty.
The SIAM model suggests that for a stimulus to be effective, it first needs to
activate knowledge in mind, and then the active knowledge needs to be processed
to produce ideas [52]. Failure in either step would make the stimulus ineffective.
When generating ideas on a creative task, people’s minds are necessarily oriented
towards elements and associations of the task. Therefore, people are less responsive
to a semantically unrelated stimulus [17, 48]. If the stimulus is indeed considered,
according to the CNM model, activating very distant knowledge adds to the
disparity among active knowledge and hence increases cognitive load [58].
Therefore, the amount of cognitive resources for ideation is reduced, which tends
to inhibit idea generation. For example, although people can try to connect unre-
lated stimuli to a creative task through analogies, the literature suggests that it is
difficult to make analogies from stimuli that are too far away [3, 31]. According to
the SIAM model, when people use a stimulus but fail to generate ideas, they may
stop using the stimulus or even terminate idea generation all together [52].
Furthermore, even if an unrelated stimulus leads to some preliminary ideas, such
ideas might be shallow or unusable [12, 69]. Consequently, these ideas might not be
selected for further development and hence have less impact on the ideation
process. In summary, when a stimulus is very distant to the point of being unrelated,
the activated knowledge is less likely to be used in meaningful ways and the
resulting preliminary idea, if any at all, is less likely to be used for further
development into a final idea. Therefore, it can be assumed that remotely related
stimuli are more effective in promoting idea novelty than both closely related
stimuli and unrelated stimuli. Consequently, there is an inverted U-shape relation-
ship between stimulus relatedness and the novelty of the final idea.
Hypothesis 4: There is an inverted U-shape relationship between stimulus
relatedness and idea novelty.
We propose an approach that generates different levels of stimuli by following
associative links among interconnected webpages in Wikipedia. Wikipedia is the
most popular collaborative resource of conceptual knowledge and is useful for
measuring different levels of stimulus relatedness [33, 71]. These relatedness
measures are all based on the assumption that hypertext links between Wikipedia
A WIKIPEDIA-BASED METHOD TO SUPPORT IDEATION 1291
pages indicate semantic relatedness. While other knowledge bases might also be
used as starting points (for example, WordNet [26] or CYC [43]), Wikipedia covers
essentially all disciplines in many different languages, and so can potentially be
used to generate stimuli for almost all problem domains in popular languages. For
example, Wikipedia is able to provide concepts related to highly specialized terms,
such as micro black hole and guanosine triphosphate. Furthermore, the entries in
Wikipedia are interconnected through hypertext links. This allows automatic asso-
ciation using computer programs.
We use the word link to denote a hypertext link that is in the Introduction section
(before the Contents list) or See also section in a Wikipedia page. We are making
two assumptions. First, links usually connect to concepts that are closely related to
the initial concept. For example, in the Wikipedia page called innovation, links
point to the Wikipedia pages called idea, product, and process, all highly related to
innovation. Second, closely related concepts probably appear as links in
a Wikipedia page. Using the example above, concepts that are highly related to
innovation (such as creativity) probably appear as links in the Wikipedia page of
innovation. Based on these two assumptions, we can use Wikipedia links to
automatically identify closely related concepts. By iterating this procedure, con-
cepts can be found at different levels of relatedness. For instance, starting with the
concept innovation, the Wikipedia page for innovation has a link to the Wikipedia
page for product. Now starting with the Wikipedia page product, there is a link to
the Wikipedia page for raw material, and so on. From innovation to product to raw
material, concepts are less and less related to the initial concept innovation. By
automating these steps using a computer program, stimuli of various levels of
relatedness can be found automatically. We assume that the more associative
steps taken by following the links, the less related the concepts are, on average.
The next section tests this assumption.
Testing the New Method
A computer program in Python was written to automatically identify links in
a Wikipedia page. The program is iterative so that it finds concepts that are
spreading out from an initial concept through hypertext linkages, for example,
from innovation to product, to raw material, then to lumber. These concepts are
labeled as 1st degree concepts, 2nd degree concepts, and 3rd degree concepts,
respectively. In our program, we remove 1st degree concepts from 2nd degree
concepts, and so on, such that there is no overlap between different degrees of
concepts. For any given initial concept, there are often hundreds of 2nd degree
concepts and thousands of 3rd degree concepts. To shorten the list of concepts, the
computer program selects the concepts whose Wikipedia pages have the largest
number of hypertext links. For example, if there are 300 2nd degree concepts, then
those 300 concepts are ranked in terms of the number of hypertext links in their
respective Wikipedia pages. Only the 30 top ranked concepts were selected. Our
1292 WANG AND NICKERSON
observation is that a Wikipedia page with many links tends to be about a well-
known concept (such as meat). In contrast, a Wikipedia page with few links is
probably about a concept that few people know about: for example there is an
article on tixel. Our program selects concepts with more hypertext links so that they
tend to be well known and thus useful in stimulating ideas. This selection method is
not the only one or necessarily the best one, but it has the virtue that it is
deterministic and will select the same stimuli each time we run the program.
Random selection of concepts would result in too much variability and make the
method difficult to replicate.
Additionally, a Python program was written to search random Wikipedia books.1
A Wikipedia book is a container for a collection of articles. For example, Nobel
laureates is a Wikipedia book corresponding to the webpage http://en.wikipedia.
org/wiki/Book:Nobel_laureates. The Python program uses a function provided by
Wikipedia that retrieves at random one Wikipedia book (http://en.wikipedia.org/
wiki/Special:Random/Book). The reason why random Wikipedia books are used as
random stimuli instead of just random Wikipedia articles is because Wikipedia
books are more likely to be well known than just random Wikipedia articles. For
example, there are many Wikipedia articles are about relatively obscure mountains
and rivers, and these are not likely to be useful for stimulating creativity.
In testing the effectiveness of the new method, we tested both concepts known to
the general public and concepts used only by certain professions. Six initial con-
cepts in total were used, two common objects (brick and kitchen), two concepts in
materials science (materials science and polymer engineering) and two information
systems concepts (management information systems and business intelligence).
From each initial concept, the computer program identified all the 1st degree
concepts, 30 2nd degree concepts and 30 3rd degree concepts (the method used to
select the 30 concepts was explained before). Also, another Python program
identified 30 random Wikipedia books as random concepts. All the relatedness
measurements were made on the scale of 1 to 7 (1 being totally unrelated, 7 being
highly related). As an example, every concept found for the concept “brick” was
evaluated with regard to its relatedness to “brick.” For the common object concepts,
because they are known to the general public, concept relatedness was judged by
ten workers employed through Amazon Mechanical Turk, an online work market-
place. For the concepts in materials science, the relatedness evaluation was done by
two Ph.D. students in that field. For the concepts in management information
systems, the relatedness evaluation was performed by two Master’s students in
the field. The ratings were averaged across different raters after checking the level
of inter-rater agreement.
The results are shown in Table 1. First degree concepts are closely related to brick
(Mean = 6.10, SD = 0.94) while 2nd, 3rd degree concepts and random concepts are less
and less related (in this order, Mean = 4.14, SD = 1.83; Mean = 3.04, SD = 1.48; Mean
= 1.36, SD = 0.41). Based on a one-way analysis of variance (ANOVA), the relatedness
is significantly different across different groups (F(3,102) = 49.8, p < 0.001). For all the
A WIKIPEDIA-BASED METHOD TO SUPPORT IDEATION 1293
http://en.wikipedia.org/wiki/Book:Nobel_laureates
http://en.wikipedia.org/wiki/Book:Nobel_laureates
http://en.wikipedia.org/wiki/Special:Random/Book
http://en.wikipedia.org/wiki/Special:Random/Book
six concepts tested, all the t-tests between adjacent groups of concepts show significant
difference in relatedness (p < 0.05). Because we ran three t-tests for each topic, we
further used the Benjamini-Hochberg procedure as the correction for multiple compar-
isons [9]. The three p-values for the three t-tests were ranked from the lowest to the
highest. Each of these three p-values then was compared to the corresponding
Benjamini-Hochberg critical value (0.05/3, 0.10/3, and 0.05, respectively). For all of
our six topics, our p-values are smaller than the corresponding Benjamini-Hochberg
critical values. Consequently, all the comparisons in Table 1 are significant using this
correction procedure. Therefore, the same pattern of decreasing relatedness is found for
all six initial concepts. These findings suggest that 1st degree concepts are closely
related to the initial concept, 2nd degree concepts are moderately related, 3rd degree
concepts are remotely related, while random concepts are unrelated.
Methods
Study Design and Participants
In this experiment, two hundred USA-based workers from Amazon Mechanical Turk
were employed to generate ideas for designing a mobile app for improving the physical
fitness of college students. The maximum time on task allowed was 30 minutes. Each
worker was offered and paid one US dollar for completing the task, and was told in the
task description that the best idea would be rewarded with a $50 bonus. On average,
these participants were 33.6 years old (SD = 10.8 years), spent 1003 seconds on the task
(SD = 404 seconds). A total of 61.7 percent of the participants were female.
The Wikipedia concept physical fitness was used to identify 1st, 2nd, and 3rd degree
concepts. The computer program collected seven 1st degree concepts, fifty 2nd degree
concepts, and fifty 3rd degree concepts (only the top 50 concepts with the highest
number of hypertext links in their Wikipedia pages were selected). Also collected were
50 random Wikipedia book concepts. This experiment adopted a between-subject
design where the workers were randomly assigned into 5 conditions, each condition
with 40 participants. In all conditions, each worker was asked to generate some
preliminary ideas about designing a fitness app for college students, then provide one
final idea. In the control condition, the workers saw no Wikipedia concepts. In the
unrelated condition, each worker contributed some preliminary ideas, then saw 3
random Wikipedia book concepts (randomly chosen from the 50 random concepts
collected) as stimuli. An example of using an unrelated stimulus was provided to the
participants (Online Supplemental Appendix 1 has the full instructions). After each
stimulus, the workers were asked to generate additional preliminary ideas based on that
specific stimulus. At the end, the workers were asked to provide a final idea for us to
consider. We chose this design to mimic design environments that often practice
a process of generating initial ideas and then developing a final idea [10]. The close,
moderate, and remote condition were the same as the unrelated condition, except that
1294 WANG AND NICKERSON
T
ab
le
1
.
C
o
n
ce
p
t
R
el
at
ed
n
es
s
In
it
ia
l
C
o
n
ce
p
t
(R
el
ia
b
il
it
y
o
f
re
la
te
d
n
es
s
d
at
a,
A
N
O
V
A
re
su
lt
fo
r
re
la
te
d
n
es
s)
C
o
n
d
it
io
n
M
ea
n
(S
D
)
T
-t
es
t
w
it
h
th
e
p
re
v
io
u
s
g
ro
u
p
E
x
am
p
le
C
o
n
ce
p
ts
B
ri
ck
(I
C
C
(1
,1
0
)
=
0
.9
5
.
A
N
O
V
A
:
F
(3
,1
0
2
)
=
4
9
.8
,
p
<
0
.
0
0
1
.)
1
s
t
d
e
g
re
e
6
.1
0
(0
.9
4
)
A
d
o
b
e
,
C
la
y
2
n
d
d
e
g
re
e
4
.1
4
(1
.8
3
)
t(
4
4
)
=
4
.8
,
p
<
0
.0
0
1
C
a
st
le
,
C
a
rp
e
n
tr
y
3
rd
d
e
g
re
e
3
.0
4
(1
.4
8
)
t(
5
6
)
=
2
.6
,
p
=
0
.0
1
3
U
n
iv
e
rs
ity
,
M
in
in
g
R
a
n
d
o
m
1
.3
6
(0
.4
1
)
t(
3
3
)
=
6
.0
,
p
<
0
.0
0
1
M
ic
ro
fa
ct
o
ry
,
T
h
o
m
a
s
Is
la
n
d
K
itc
h
e
n
(I
C
C
(1
,1
0
)
=
0
.9
7
.
A
N
O
V
A
:
F
(3
,1
0
7
)
=
5
7
.9
3
,
p
<
0
.0
0
1
.)
1
s
t
d
e
g
re
e
5
.9
4
(1
.1
9
)
S
in
k,
D
is
h
w
a
sh
e
r
2
n
d
d
e
g
re
e
3
.7
6
(2
.0
6
)
t(
4
9
)
=
4
.5
4
,
p
<
0
.0
0
1
S
p
o
ila
g
e
,
C
a
n
n
in
g
3
rd
d
e
g
re
e
2
.3
2
(1
.7
8
)
t(
4
2
)
=
3
.5
0
,
p
=
0
.0
0
1
T
e
ch
n
o
lo
g
y,
A
rt
m
o
ve
m
e
n
t
R
a
n
d
o
m
1
.2
2
(0
.4
4
)
t(
3
3
)
=
5
.9
1
,
p
<
0
.0
1
M
ic
ro
fa
ct
o
ry
,
T
h
o
m
a
s
Is
la
n
d
M
a
te
ri
a
ls
S
ci
e
n
ce
(I
C
C
(2
,2
)
=
0
.8
6
.
A
N
O
V
A
:
F
(3
,1
1
6
)
=
1
0
5
.8
8
,
p
<
0
.0
0
1
.)
1
s
t
d
e
g
re
e
6
.1
8
(0
.8
7
)
M
a
tt
e
r,
M
in
e
ra
lo
g
y
2
n
d
d
e
g
re
e
5
.2
5
(1
.4
1
)
t(
4
8
)
=
3
.0
9
,
p
=
0
.0
0
3
M
a
n
u
fa
ct
u
ri
n
g
,
M
e
d
ic
in
e
3
rd
d
e
g
re
e
3
.7
5
(1
.8
5
)
t(
5
4
)
=
3
.5
3
,
p
<
0
.0
0
1
F
o
o
d
,
B
o
ta
n
y
R
a
n
d
o
m
1
.8
8
(1
.1
9
)
t(
4
9
)
=
4
.6
6
,
p
<
0
.0
0
1
T
h
e
W
h
ite
V
ik
in
g
,
2
9
th
A
rm
y
P
o
ly
m
e
r
E
n
g
in
e
e
ri
n
g
(I
C
C
(2
,2
)
=
0
.7
8
.
A
N
O
V
A
:
F
(3
,9
4
)
=
4
1
.6
6
,
p
<
0
.0
0
1
.)
1
s
t
d
e
g
re
e
6
.3
1
(1
.0
0
)
P
o
ly
m
e
r
S
ci
e
n
ce
,
E
n
g
in
e
e
ri
n
g
2
n
d
d
e
g
re
e
4
.9
5
(1
.7
6
)
t(
2
0
)
=
2
.8
6
,
p
=
0
.0
1
A
ro
m
a
tic
s,
C
o
rn
3
rd
d
e
g
re
e
3
.4
3
(1
.5
9
)
t(
5
7
)
=
3
.5
0
,
p
<
0
.0
0
1
E
n
e
rg
y,
P
h
a
rm
a
ce
u
tic
a
ls
R
a
n
d
o
m
1
.6
0
(0
.4
6
)
t(
3
4
)
=
6
.0
6
,
p
<
0
.0
0
1
T
ru
ck
e
rs
,
W
ill
ia
m
G
re
in
e
r
M
a
n
a
g
e
m
e
n
t
In
fo
rm
a
tio
n
S
ys
te
m
s
(I
C
C
(2
,2
)
=
0
.7
4
.
A
N
O
V
A
:
F
(3
,1
1
1
)
=
3
8
.7
,
p
<
0
.0
0
1
.)
1
s
t
d
e
g
re
e
5
.5
2
(1
.1
6
)
A
cc
o
u
n
tin
g
,
B
u
si
n
e
ss
R
u
le
2
n
d
d
e
g
re
e
4
.7
8
(1
.2
8
)
t(
5
3
)
=
2
.2
6
,
p
=
0
.0
2
8
A
u
d
iti
n
g
,
C
lo
u
d
C
o
m
p
u
tin
g
3
rd
d
e
g
re
e
3
.2
1
(1
.2
4
)
t(
5
8
)
=
4
.8
1
,
p
<
0
.0
0
1
B
io
fu
e
l,
In
d
u
st
ri
a
l
R
e
vo
lu
tio
n
R
a
n
d
o
m
2
.5
7
(0
.9
2
)
t(
5
4
)
=
2
.2
8
,
p
=
0
.0
2
7
P
e
n
a
n
g
,
S
e
a
n
F
ra
se
r
B
u
si
n
e
ss
In
te
lli
g
e
n
ce
(I
C
C
(2
,2
)
=
0
.8
0
.
A
N
O
V
A
:
F
(3
,1
1
6
)
=
8
4
.1
,
p
<
0
.0
0
1
.)
1
s
t
d
e
g
re
e
6
.4
0
(0
.6
8
)
A
n
a
ly
tic
s,
B
u
si
n
e
ss
R
e
p
o
rt
in
g
2
n
d
d
e
g
re
e
5
.2
3
(1
.4
3
)
t(
4
2
)
=
4
.0
4
,
p
<
0
.0
0
1
H
a
d
o
o
p
,
In
fo
rm
a
tio
n
3
rd
d
e
g
re
e
3
.1
6
(1
.4
0
)
t(
5
8
)
=
5
.6
9
,
p
<
0
.0
0
1
L
o
g
ic
,
L
in
u
x
G
a
m
in
g
R
a
n
d
o
m
2
.3
2
(0
.7
3
)
t(
4
4
)
=
2
.8
9
,
p
=
0
.0
0
6
T
h
e
B
la
ck
P
a
n
th
e
r,
M
a
ry
R
o
b
in
so
n
A WIKIPEDIA-BASED METHOD TO SUPPORT IDEATION 1295
the stimuli were randomly chosen 1st, 2nd, and 3rd degree concepts, respectively.
Altogether, there are three types of ideas: initial preliminary ideas (before seeing any
stimuli), stimulated preliminary ideas (upon seeing stimuli), and final ideas. The
control condition does not have stimulated preliminary ideas because no stimuli were
given.
Dependent Variables
The experiment generated 200 final ideas. On average, each idea has 99 words (SD =
55). The final ideas were evaluated by two professional app developers with regard to
idea novelty and usefulness. Both app developers have college degrees and at least four
years’ experience in mobile app development. Therefore, they have experience on both
the customer side (as college students) and on the designer side (as app developers). We
notice that Dean et al. [20] advocate for evaluating ideas in four dimensions: novelty,
feasibility, relevance and specificity. In practice, when people do adopt this method,
they often omit specificity and instead focus on the former three dimensions [22]. In
our measurement, idea usefulness is similar to feasibility plus relevance in Dean et al.
[20]. Specifically, in our idea evaluation, novelty is defined as the degree to which an
idea is original and paradigm modifying [20]. Usefulness is the degree to which the
idea is feasible and effective in improving college students’ physical fitness. Novelty
and usefulness were rated on the scale of 1 to 7 (1 being not novel/useful at all, 7 being
highly novel/useful). The two raters first browsed the online Apple app store to become
familiar with existing fitness apps. Afterward, they independently rated all the final
ideas, which were randomly ordered. The intraclass correlation coefficients show
sufficient levels of agreement in their ratings, therefore their ratings were averaged
(ICC(2,2) = 0.79 for both novelty and usefulness). The average scores are called final
novelty and final usefulness, denoting the novelty and usefulness of the final ideas.
In addition, all the 1956 preliminary ideas were evaluated on novelty and useful-
ness by one of the raters. To ensure the quality of his ratings, the second rater also
evaluated 100 preliminary ideas. For those 100 preliminary ideas, two raters agree
on 87 ideas in novelty assessment and 86 ideas on usefulness assessment.
Agreement means the scores are no more than 1 point different. The two raters
discussed and reconciled their differences in the evaluation of those 100 ideas
before the first rater evaluated all the remaining preliminary ideas. The number
of preliminary ideas generated by each worker upon the exposure to three concepts
is called stimulated fluency. Stimulated novelty is the number of novel preliminary
ideas generated upon seeing three concepts (novelty score larger than 4, the scale
midpoint). Stimulated usefulness is defined in the same manner.
Independent Variables
The relatedness of all the concepts to physical fitness was evaluated by 20 workers
from Mechanical Turk (ICC(1,20) = 0.97). The scores from different raters were
1296 WANG AND NICKERSON
averaged. The average of the relatedness of the three concepts that each participant
saw was calculated and then centered for the regression purpose. This centered
variable is called concept relatedness.
Control Variables
The number of preliminary ideas generated at the beginning without seeing any
concepts is called initial fluency. Initial novelty is defined as the number of novel
preliminary ideas (novelty score being more than 4, the scale midpoint) generated
before seeing any concepts. Initial usefulness is defined similarly. At the end of the
survey, all the participants were given a set of questions measuring knowledge of
physical fitness, knowledge of mobile apps, and intrinsic motivation in this task. The
questions for knowledge of physical fitness and mobile apps were adapted from the
domain-specific consumer knowledge scale [44, 50]. We used the consumer knowl-
edge scale because the participants were from the general public and therefore more
similar to potential consumers, instead of developers, of the designed app. Knowledge
of mobile apps and physical fitness have a Cronbach’s alpha of 0.848 and 0.880,
respectively. Intrinsic motivation was measured with two items: “did you find the task
interesting” and “was it enjoyable to work on” [2]. For both the measures of knowledge
and intrinsic motivation, a Likert scale was used with 1 defined as strongly disagree and
5 defined as strongly agree. Having a high level of intrinsic motivation is considered
important for creative work [40]. In this experiment, the intrinsic motivation of the
participants was indeed high (M = 4.33, SD = 0.64).
Results
The relatedness of different degrees of concepts is shown in Table 2. A one-way
ANOVA shows that concept relatedness is different across different groups (F(3,153)
= 108.3, p < 0.001). Using the Benjamini–Hochberg procedure as the correction for
multiple comparisons, the three t-tests are still significant. Again, concept relatedness
decreases with the number of associative steps taken, as predicted.
The descriptive statistics and correlations between all the main variables are shown in
Table A in Online Supplemental Appendix 3. Concept relatedness is correlated with
stimulated fluency (r(158) = 0.239, p < 0.01). This is aligned with hypothesis 1 which
Table 2. Concept Relatedness for Study 1
Concepts Mean (SD) T-test with the previous group Example concepts
1st degree 6.76 (0.23) Nutrition, Bodybuilding
2nd degree 5.20 (1.10) t(48) = 8.77, p < 0.001 Fruit, Immune system
3rd degree 3.57 (1.33) t(95) = 6.67, p < 0.001 Surgery, Fungus
Random 1.71 (0.79) t(80) = 8.50, p < 0.001 George Washington, Gospel
A WIKIPEDIA-BASED METHOD TO SUPPORT IDEATION 1297
indicates a positive relationship between concept relatedness and idea quantity. Concept
relatedness is also correlated with stimulated usefulness (r(158) = 0.453, p < 0.01) and
final usefulness (r(158) = 0.158, p < 0.05). This is aligned with hypothesis 2 that proposes
a positive relationship between concept relatedness and final usefulness. These correla-
tions alone are not sufficient support for the hypotheses but they do provide information
that is consistent with the results of the regressions, reported below. Hypothesis 4
proposed an inverted U-shape between stimulus relatedness and final novelty. Our results
show that the correlation between concept relatedness and final novelty is not significant
(r(158) = −0.096, p = 0.227).
Idea novelty and idea usefulness in different conditions are presented in Table 3.
A one way ANOVA tests show that final novelty and final usefulness are both different
across conditions (F(4,195) = 3.452, p = 0.009 for novelty; F(4,195) = 2.668, p = 0.034
for usefulness). ATukey post hoc test shows that final usefulness in the close condition
(M = 4.988, SD = 0.755) is higher than the moderate condition (M = 4.275, SD = 1.311,
p = 0.015). A t-test shows that final novelty in the remote condition (M = 4.75, SD =
1.11) is significantly higher than that in the control condition (M = 3.85, SD = 1.06; t
(78) = 3.70, p < 0.001). This is consistent with hypothesis 3. The remote condition has
30 final ideas that score higher than 4 (the scale midpoint) in novelty. The control,
close, moderate and unrelated conditions have 15, 20, 25, and 27 such ideas.
Regression analysis is used to further test all the hypotheses while considering
control variables, including time on task, knowledge of mobile apps, knowledge of
physical fitness, and intrinsic motivation (Table 4). Four dummy variables (close,
moderate, remote, unrelated) represent the conditions in the experiment. For the
control condition, all these dummy variables are set to zero. In the analyses, we first
regressed a dependent variable on the control variables, such as time on task and
knowledge of mobile apps. These variables form the first block. The second block
contains either dummy variables for the experimental conditions or concept relat-
edness and its squared term (for testing inverted U-shape relationship). The change
in explained variance (ΔR2 in Table 4) indicates whether the experimental condi-
tions explain any variance in a dependent variable, over and above the variance
explained by the control variables.
Table 3. Idea Novelty, Usefulness, and Quantity in Different Conditions in Study 1
Condition
Final
Novelty
Mean (SD)
Final
Usefulness
Mean (SD)
Stimulated
Fluency
Mean (SD)
Stimulated
Novelty
Mean (SD)
Stimulated
Usefulness
Mean (SD)
Control 3.85 (1.06) 4.53 (0.98) NA NA NA
Close 4.35 (1.29) 4.99 (0.76) 8.70 (1.24) 3.45 (2.15) 4.53 (2.14)
Moderate 4.55 (1.34) 4.28 (1.31) 8.23 (1.85) 4.38 (2.20) 3.38 (2.19)
Remote 4.75 (1.11) 4.61 (0.91) 7.53 (2.29) 3.55 (2.74) 1.65 (1.64)
Unrelated 4.69 (1.34) 4.51 (0.97) 7.35 (3.02) 4.43 (3.00) 2.03 (2.01)
1298 WANG AND NICKERSON
T
ab
le
4
.
R
eg
re
ss
io
n
A
n
al
y
si
s
T
es
ti
n
g
th
e
H
y
p
o
th
es
es
in
S
tu
d
y
1
D
ep
en
d
en
t
V
ar
ia
b
le
S
ti
m
u
la
te
d
F
lu
en
cy
(H
1
)
F
in
al
U
se
fu
ln
es
s
(H
2
)
S
ti
m
u
la
te
d
U
se
fu
ln
es
s
F
in
al
N
o
v
el
ty
(H
3
)
F
in
al
N
o
v
el
ty
(H
4
)
S
ti
m
u
la
te
d
N
o
v
el
ty
In
te
rc
e
p
t
5
.2
9
3
**
*
3
.7
4
5
**
*
−
.0
2
4
2
.7
3
6
**
*
3
.5
3
7
**
*
3
.3
1
9
*
B
lo
c
k
1
:
C
o
n
tr
o
l
V
a
ri
a
b
le
s
T
im
e
o
n
ta
sk
.0
3
7
−
.0
0
2
.0
3
5
.0
2
9
*
.0
3
7
*
.0
4
3
K
n
o
w
le
d
g
e
o
f
m
o
b
ile
a
p
p
s
.5
2
1
†
.2
5
8
*
.3
6
1
−
.0
7
2
−
.1
1
4
.3
5
1
K
n
o
w
le
d
g
e
o
f
p
h
ys
ic
a
l
fit
n
e
ss
−
.2
1
4
−
.1
8
7
−
.1
3
7
.0
4
8
.0
5
0
−
.5
9
7
†
In
tr
in
si
c
m
o
tiv
a
tio
n
.2
0
9
.0
1
7
.1
2
5
.1
4
1
.1
1
2
.0
6
4
In
iti
a
l
flu
e
n
cy
−
.0
4
4
In
iti
a
l
u
se
fu
ln
e
ss
.2
9
6
**
*
.2
8
4
*
In
iti
a
l
n
o
ve
lty
.1
8
2
*
.2
6
6
**
.3
8
6
*
B
lo
c
k
2
C
o
n
ce
p
t
re
la
te
d
n
e
ss
.3
2
4
**
.1
0
2
*
.6
0
2
**
*
−
.0
7
0
−
.0
9
4
C
o
n
ce
p
t
re
la
te
d
n
e
ss
sq
u
a
re
d
.0
1
2
.0
3
2
.1
3
9
**
−
.0
2
5
.0
1
2
C
lo
se
co
n
d
iti
o
n
.3
9
6
M
o
d
e
ra
te
co
n
d
iti
o
n
.6
7
5
*
R
e
m
o
te
co
n
d
iti
o
n
.8
2
6
**
U
n
re
la
te
d
co
n
d
iti
o
n
.7
2
5
*
R
2
.1
0
6
.1
8
0
.2
9
4
.1
3
8
.1
2
1
.0
8
6
F
F
(7
,1
5
2
)
=
2
.5
7
9
*
F
(7
,1
5
2
)
=
4
.7
6
8
**
*
F
(7
,1
5
2
)
=
9
.0
2
4
**
*
F
(9
,1
9
0
)
=
3
.3
8
6
**
F
(7
,1
5
2
)
=
2
.9
9
0
**
F
(7
,1
5
2
)
=
2
.0
3
4
†
Δ
R
2
(Δ
F
)
.0
7
5
(6
.3
6
2
)*
*
.0
3
8
(3
.5
6
6
)*
.2
5
3
(2
7
.2
0
7
)*
**
.0
4
6
(2
.5
3
2
)*
.0
1
2
(1
.0
5
7
)
.0
0
5
(.
4
5
1
)
N
o
te
s:
N
=
2
0
0
.
V
al
u
es
re
p
re
se
n
t
u
n
st
an
d
ar
d
iz
ed
re
g
re
ss
io
n
co
ef
fi
ci
en
ts
,
w
it
h
st
an
d
ar
d
er
ro
rs
in
p
ar
en
th
es
es
.
†
p
<
.1
0
.
*
p
<
.0
5
.
*
*
p
<
.0
1
.
*
*
*
p
<
.0
0
1
;
tw
o
-t
ai
le
d
p
-v
al
u
es
.
A WIKIPEDIA-BASED METHOD TO SUPPORT IDEATION 1299
As Table 4 indicates, concept relatedness does explain significant variance over
and above the variance explained by the control variables, for stimulated fluency (B
= .324, t = 3.539, p = .001), stimulated usefulness (B = .602, t = 7.246, p < 0.001)
and final usefulness (B = .102, t = 2.534, p = .012). Therefore, concept relatedness
is indeed positively related to the number of ideas and idea usefulness. Hypotheses
1 and 2 are supported. Supporting Hypothesis 3, the regression shows that the
remote condition explains significant variance in final novelty beyond the variance
explained by the control variables (B = .826, t = 2.855, p = 0.005).
When final novelty and stimulated novelty are used as dependent variables, concept
relatedness and its quadratic term have nonsignificant coefficients and do not explain
significant variance beyond the control variables. The results provide no evidence for
Hypothesis 4 that predicts an inverted U-shape relationship. The remote condition has
the highest final novelty (M = 4.75, SD = 1.11), but it is not significantly higher than the
moderate condition (M = 4.55, SD = 1.34; t(78) = 0.728, p = 0.469) or the unrelated
condition (M = 4.69, SD = 1.34; t(78) = 0.227, p = 0.821).
We conducted a second study to test the generalizability of our results. Specifically,
we recruited college students, instead of Mechanical Turk workers, as participants
and used a different ideation task. In addition, the second study made some changes
to addresses three potential concerns about study 1. First, in study 1, the number of
candidate concepts is different across conditions. For example, in the close condi-
tion, the stimulus concepts were randomly selected from 7 first degree concepts
(there are only 7 first degree concepts). In the moderate condition, the stimulus
concepts were randomly selected from 50 second degree concepts. It is possible that
the number of candidate stimuli has an effect on idea diversity and potentially idea
novelty. In study 2, we randomly selected stimuli from the same number of
candidate stimuli across conditions. Second, in study 1, the participants in the
experimental conditions generated additional preliminary ideas based on stimulus
concepts, while the control condition did not have this step. The absence of this step
might have meant that those in the control condition exerted less effort. It is also
possible that they might have exerted equal or more effort until they became stuck
due to lack of external stimulation. To create more symmetry in the conditions, in
study two, the participants in the control condition were asked to generate three
extra sets of preliminary ideas without external stimuli. This step was parallel to the
step in the experimental conditions where the participants generated three sets of
ideas based on three stimuli. Third, in study 1 the stimuli are all regular Wikipedia
concepts except that the unrelated concepts are Wikipedia book concepts. It is
unclear whether Wikipedia book concepts are somehow different from regular
Wikipedia concepts, which could result in undesirable variability. In study 2, we
used the fifth degree concepts as unrelated stimuli to eliminate this concern.
1300 WANG AND NICKERSON
Methods
The design of study 2 is highly similar to study 1, with the following differences. In
this experiment, two hundred undergraduate students from a university on the east
coast of the United States were recruited as participants. They were instructed to
generate ideas on designing a mobile app for online shopping by college students.
The maximum time on task allowed was 60 minutes. The concept online shopping
was used to identify 1st, 2nd, 3rd, and 5th degree concepts. There are twenty 1st
degree concepts. The computer program also collected twenty 2nd, 3rd, and 5th
degree concepts by selecting the top twenty concepts with the highest number of
hypertext links in their Wikipedia pages.
One of the two idea evaluators is different from study 1. The intraclass correlation
coefficients for the ratings of the two raters are ICC(2,2) = 0.72 and 0.66 for
novelty and usefulness, respectively. These numbers indicate acceptable levels of
interrater agreement [15]. The relatively modest level of agreement is not uncom-
mon in creativity judgment (as shown in [7, 10]), perhaps because raters have
different experience bases and hence different associative networks. Out of the two
hundred final ideas generated, eight ideas were considered by both raters as
irrelevant. Therefore, only 192 participants’ data were used. On average, each
idea has 103 words (SD = 49). Online Supplemental Appendix 2 describes the
instructions of the survey. On average, the student participants were 21.82 years old
(SD = 1.41 years), spent 1,752 seconds on the task (SD = 718 seconds). Fifty-two
percent of the participants were female. The average score of intrinsic motivation
was 3.91 (SD = 0.98). We measured the participant’s knowledge of shopping apps
using the same scale mentioned earlier with a Cronbach’s alpha of 0.847.
Results
The relatedness of different degrees of concepts is shown in Table 5. A one-way
ANOVA shows that concept relatedness is different across different groups (F(3,76)
= 42.6, p < 0.001). Using the Benjamini–Hochberg procedure as the correction for
Table 5. Concept Relatedness for Study 2
Concepts
Mean
(SD)
T-test with the previous
group Example concepts
1st degree 5.86 (0.96) Credit card, Website
2nd degree 4.34 (1.37) t(34) = 3.97, p < 0.001 Internet Explorer, Shopping
streets
3rd degree 2.87 (1.34) t(38) = 3.36, p = 0.002 Concert hall, Emoji
5th degree 2.06 (0.66) t(28) = 2.36, p = 0.026 Shakespeare, Soviet scientists
A WIKIPEDIA-BASED METHOD TO SUPPORT IDEATION 1301
multiple comparisons, the three t-tests are still significant. The four groups corre-
spond closely to, moderately, remotely related concepts, and unrelated concepts.
The descriptive statistics and correlations between all the main variables are
shown in Table B in Online Supplemental Appendix 3. Concept relatedness is
correlated with stimulated fluency (r(150) = 0.195, p < 0.01). This is aligned with
hypothesis 1 which indicates a positive relationship between concept relatedness
and idea quantity. Concept relatedness is also correlated with stimulated usefulness
(r(150) = 0.272, p < 0.01), final usefulness (r(150) = 0.178, p < 0.05). This is
aligned with Hypothesis 2 that proposes a positive relationship between concept
relatedness and final usefulness. These correlations alone are not sufficient support
for the hypotheses but they do provide information that is consistent with the results
of the regressions, reported in the following section. Hypothesis 4 proposes an
inverted U-shape between stimulus relatedness and final novelty. Our results show
that the correlation between concept relatedness and final novelty is not significant
(r(150) = −0.041, p = 0.612), again consistent with the regression results.
Idea novelty and idea usefulness in different conditions are presented in
Table 6. Final novelty does not appear different considering all the conditions
(: F(4,187) = 1.491, p = 0.207). The control, close, and unrelated conditions all
have ANOVA 18 final ideas scoring higher than 4 in novelty. The moderate
and remote conditions both have 21 such final ideas. A one way ANOVA
shows that final usefulness is significantly different across conditions (F
(4,187) = 2.638, p = 0.035). A Tukey post hoc test shows that final usefulness
is higher in the close condition (M = 3.946, SD = 0.949) than the unrelated
condition (M = 3.338, SD = 1.140, p = 0.048).
In the control condition, there are three steps generating additional preliminary ideas,
parallel to idea stimulation by three Wikipedia concepts in other conditions. Stimulated
novelty for the control condition refers to the number of novel preliminary ideas
generated in these steps. An ANOVA indicates that stimulated novelty is different
across conditions (F(4,187) = 3.805, p = 0.005). A t-test shows that stimulated novelty
is significantly higher in the remote condition (M = 2.89, SD = 2.54), compared to the
control condition (M = 1.83, SD = 1.66; t(59) = 2.136, p = 0.037).
Table 6. Idea Novelty, Usefulness, and Quantity in Different Conditions in Study 2
Condition
Final
Novelty
Mean (SD)
Final
Usefulness
Mean (SD)
Stimulated
Fluency
Mean (SD)
Stimulated
Novelty
Mean (SD)
Stimulated
Usefulness
Mean (SD)
Control (n = 40) 4.04 (1.17) 3.83 (1.02) 5.43 (1.62) 1.83 (1.66) 2.63 (1.60)
Close
(n = 37)
4.05 (1.21) 3.95 (0.95) 6.89 (1.85) 2.11 (1.90) 3.54 (1.61)
Moderate (n = 39) 4.40 (1.42) 3.58 (0.70) 5.97 (1.72) 2.51 (2.06) 2.62 (2.02)
Remote (n = 36) 4.60 (1.01) 3.46 (0.94) 6.11 (1.79) 2.89 (2.54) 2.75 (1.76)
Unrelated (n = 40) 4.14 (1.25) 3.34 (1.14) 5.60 (1.50) 3.45(2.10) 1.95(1.54)
1302 WANG AND NICKERSON
Using the same method as study 1, regression analysis is used to further test all
the hypotheses considering the control variables (shown in Table 7).2 As Table 7
indicates, concept relatedness does explain significant variance over and above
the variance explained by the control variables, for stimulated fluency (B = .180,
t = 2.026, p = .045), stimulated usefulness (B = .280, t = 3.152, p = 0.002) and
final usefulness (B = .099, t = 1.943, p = .054). Therefore, concept relatedness is
indeed positively related to the number of ideas and idea usefulness. Hypotheses 1
and 2 are supported.
The regression shows that the remote condition explains marginally significant
variance in final novelty (B = .511, t = 1.944, p = 0.053) and significant variance in
stimulated novelty (B = 0.970, t = 2.084, p = 0.039) beyond the variance explained
by the control variables. So Hypothesis 3 is supported in part. When final novelty
and stimulated novelty are used as dependent variables, the quadratic term of
concept relatedness has nonsignificant coefficients. The results provide no evidence
for Hypothesis 4 that predicts an inverted U-shape relationship. The remote condi-
tion has the highest final novelty (M = 4.60, SD = 1.01), but it is not significantly
higher than the moderate condition (M = 4.40, SD = 1.42; t(69) = 0.709, p = 0.481)
or the unrelated condition (M = 4.14, SD = 1.25; t(74) = 1.753, p = 0.084).
Influence of Stimuli on Idea Quantity and Usefulness
Our first hypothesis states that the number of preliminary ideas is positively related to
stimulus relatedness. For study 1 and 2, the correlation and regression analysis con-
sistently support this hypothesis. As argued earlier, it is difficult to respond to a less
related stimulus and generate many ideas based on it. This result is also consistent with
previous studies where irrelevant or original stimuli led to lower number of ideas [37,
69]. Hypothesis 2 states that final usefulness is positively related to stimulus related-
ness. In both study 1 and 2, correlation and regression analysis alike show this positive
relationship for both final usefulness and stimulated usefulness. The exception is that
the regression coefficient for concept relatedness is only marginally significant for final
usefulness in study 2. Overall, there is support for Hypothesis 2. Highly related stimuli
bring relevant, appropriate and meaningful associations into the thinking process,
leading to useful ideas. Our results are consistent with Berg [10] where familiar
concepts led to more useful ideas.
Influence of Stimuli on Idea Novelty
Hypothesis 3 contends that remotely related stimuli are effective in increasing idea
novelty, compared to using no stimuli. Regression results support the hypothesis in
both experiments. In study 2, although the regression coefficient for the remote
condition is only marginally significant for final novelty, the coefficient is
A WIKIPEDIA-BASED METHOD TO SUPPORT IDEATION 1303
T
ab
le
7
.
R
eg
re
ss
io
n
A
n
al
y
si
s
T
es
ti
n
g
th
e
H
y
p
o
th
es
es
in
S
tu
d
y
2
D
ep
en
d
en
t
V
ar
ia
b
le
S
ti
m
u
la
te
d
F
lu
en
cy
(H
1
)
F
in
al
U
se
fu
ln
es
s
(H
2
)
S
ti
m
u
la
te
d
U
se
fu
ln
es
s
F
in
al
N
o
v
el
ty
(H
3
)
S
ti
m
u
la
te
d
N
o
v
el
ty
F
in
al
N
o
v
el
ty
(H
4
)
S
ti
m
u
la
te
d
N
o
v
el
ty
In
te
rc
e
p
t
5
.7
5
2
**
*
3
.9
0
5
**
*
.6
5
6
3
.1
9
9
**
*
.9
3
7
3
.4
3
2
**
*
2
.3
9
7
*
B
lo
c
k
1
:
C
o
n
tr
o
l
V
a
ri
a
b
le
s
T
im
e
o
n
ta
sk
0
0
.0
0
1
**
0
0
0
0
K
n
o
w
le
d
g
e
o
f
sh
o
p
p
in
g
a
p
p
s
−
.2
9
5
†
−
.0
5
0
.0
3
2
−
.0
8
2
−
.0
3
6
−
.0
6
6
−
.0
4
2
In
tr
in
si
c
m
o
tiv
a
tio
n
−
.0
6
7
−
.0
8
4
.0
3
3
.2
5
8
**
.1
9
2
.1
9
7
*
.0
3
8
In
iti
a
l
flu
e
n
cy
.3
8
6
**
*
In
iti
a
l
u
se
fu
ln
e
ss
.2
0
7
**
.4
3
1
**
*
In
iti
a
l
n
o
ve
lty
.4
8
1
**
*
.6
9
1
**
*
.4
9
2
**
*
.7
6
6
**
*
B
lo
c
k
2
C
o
n
ce
p
t
re
la
te
d
n
e
ss
.1
8
0
*
.0
9
9
†
.2
8
0
**
−
.0
6
8
−
.3
5
7
**
C
o
n
ce
p
t
re
la
te
d
n
e
ss
sq
u
a
re
d
.1
0
6
†
.0
1
0
.0
4
7
−
.0
4
2
−
.0
0
1
C
lo
se
co
n
d
iti
o
n
−
.2
5
8
−
.1
2
9
M
o
d
e
ra
te
co
n
d
iti
o
n
.1
7
0
.4
2
1
R
e
m
o
te
co
n
d
iti
o
n
.5
1
1
†
.9
7
0
*
U
n
re
la
te
d
co
n
d
iti
o
n
.0
1
7
1
.4
8
2
**
R
2
.1
8
5
.1
0
4
.2
2
9
.2
2
1
.1
8
2
.2
1
1
.1
6
7
F
F
(6
,1
4
5
)
=
5
.4
7
8
**
*
F
(6
,1
4
5
)
=
2
.7
9
3
*
F
(6
,1
4
5
)
=
7
.1
9
4
**
*
F
(8
,1
8
3
)
=
6
.4
9
8
**
*
F
(8
,1
8
3
)
=
5
.1
0
3
**
*
F
(6
,1
4
5
)
=
6
.4
7
4
**
*
F
(6
,1
4
5
)
=
4
.8
4
9
**
*
Δ
R
2
(Δ
F
)
.0
5
8
(5
.1
7
1
)*
*
.0
2
9
(2
.3
6
7
)
†
.0
7
2
(6
.7
5
3
)*
*
.0
3
8
(2
.2
1
8
)
†
.0
8
1
(4
.5
4
1
)*
*
.0
1
7
(1
.5
7
5
)
.0
6
4
(5
.6
0
4
)*
*
N
o
te
s:
N
=
1
9
2
.
V
al
u
es
re
p
re
se
n
t
u
n
st
an
d
ar
d
iz
ed
re
g
re
ss
io
n
co
ef
fi
ci
en
ts
,
w
it
h
st
an
d
ar
d
er
ro
rs
in
p
ar
en
th
es
es
.
†
p
<
.1
0
.
*
p
<
.0
5
.
*
*
p
<
.0
1
.
*
*
*
p
<
.0
0
1
;
tw
o
-t
ai
le
d
p
-v
al
u
es
.
1304 WANG AND NICKERSON
significant for stimulated novelty. Overall, the results are in line with the common
notion that remotely related stimuli introduce new cognitive elements, reduce
fixation, and lead to high novelty.
Hypothesis 4 suggests that final novelty has an inverted-U shape relationship
with stimulus relatedness. The regression analyses in study 1 and 2 show no
evidence for this relationship, either for stimulated novelty or final novelty.
Stimulated novelty is the number of stimulated preliminary ideas that are novel
(scoring higher than 4). Therefore, stimulated novelty can be influenced by two
variables: the number of stimulated ideas and the average novelty of stimulated
ideas. We conducted a post-hoc analysis by calculating the average novelty of
stimulated ideas. In both experiments, the average novelty of stimulated ideas is
negatively correlated with stimulus relatedness (r(158) = −0.227, p < 0.01 for
study 1; r(150) = −0.206, p < 0.05 for study 2). Therefore, the less related the
stimuli, the higher the average novelty of the preliminary ideas. However, as
hypothesis 1 states, low stimulus relatedness also leads to a lower number of
preliminary ideas. Combining these two effects, the relationship between stimulus
relatedness and stimulated novelty becomes complicated. While this relationship
is negative in study 2 (r(150) = −0.200, p < 0.05), the correlation is non-
significant in study 1 (r(158) = −0.095, p = 0.233).
In both studies, stimulated novelty is highest in the unrelated condition while final
novelty is highest in the remote condition. The discrepancy suggests that people may
not integrate every preliminary idea in their final ideas. We noticed that unrelated
stimuli may lead to preliminary ideas that are superficial or less meaningful and
consequently not used to generate final ideas. For example, in study 1 about fitness
apps, after a participant saw the unrelated stimulus assault rifles, a preliminary idea was
include gun safety tips in the app. This idea was not included in the final idea,
presumably due to low relevance. Because unrelated stimuli might result in preliminary
ideas that are abandoned later, high stimulated novelty in the unrelated condition does
not necessarily translate to high final novelty.
Our data also show that the difference in final novelty across conditions is not
large enough to lead to a significant inverted-U shape relationship, even though the
trend seems to be consistent with this shape. There might be two reasons behind the
small differences in final novelty. First, people can still generate novel ideas using
1st and 2nd degree concepts. The semantic network in long term memory is very
richly connected. While a Wikipedia concept may have ten 1st degree concepts,
a concept in mind may be directly connected to hundreds of other concepts.
Therefore, one step of association in mind can still be a semantic leap. As
a result, it is still possible for people to start with a 1st degree concept and generate
novel ideas. Second, it is usually difficult to generate extremely novel ideas.
Consequently, most ideas have moderate levels of novelty. The literature also
provides little direct evidence of an inverted-U shape relationship, even though
the theoretical argument has been made before [2, 3, 19, 31].
A WIKIPEDIA-BASED METHOD TO SUPPORT IDEATION 1305
Comparing Our Research with Priming Studies
Priming is typically through subconscious mechanisms, therefore different from our
approach of deliberate conscious use of stimuli. Even though our study shows that the
effect of unrelated stimuli is limited in our context, priming with unrelated stimuli can
promote creativity. For example, people generate more creative ideas after they play an
unrelated scrambled sentence game that emphasizes achievement [22]. This outcome
might result from activated knowledge related to achievement in the semantic network,
or elevated expectancy of achievement through motivational mechanisms. While the
exact mechanism seems unclear [22], it is safe to assume that priming may work
through different mechanisms than deliberate conscious use of stimuli. In our study,
because we provided a clear example of using stimuli as cognitive stimulation, the
effects of stimuli are likely (at least in part) due to cognitive mechanisms. In general,
however, it is also possible that the effects of stimuli are through a mix of various
mechanisms. In future experimental studies, it might be possible to measure the
cognitive, emotional and motivational influence on the use of stimuli of different
degrees, which would further elucidate the mechanisms and provide in-depth under-
standing of various possibilities of using stimuli in ideation.
Implications for Practice
Our results suggest that remotely related stimuli are more likely to promote idea
novelty than unrelated stimuli. Therefore, creative professionals are well advised to
search remotely related stimuli for inspiration. Our approach for concept search is able
to find remote stimuli, as well as other levels of stimuli from Wikipedia. This approach
for finding stimuli can be included in creative work processes or can be explicitly built
into creativity support systems. Another implication for practice in creative work is
related to the development of preliminary ideas. Our study shows that people tend to
abandon highly novel preliminary ideas. Such preference against novelty exists both in
our participants and in highly educated scientific communities. A recent study shows
that highly novel scientific research proposals systematically got lower evaluation
scores [11], which would increase the chance of rejecting such proposals. While it is
certainly sensible to consider the relevance and appropriateness of a preliminary idea,
such a prevalent bias against novelty can hurt the effectiveness of using external stimuli
and harm creative work. Therefore, when using external stimuli in creative work,
people might be encouraged to give highly novel nascent ideas more consideration.
It may be worthwhile to nurture such ideas through a sequential process, encouraging
several rounds of improvement before making a final judgment.
Implications for Future Research
Existing theories on idea generation (such as SIAM and CNM) typically focus on
the cognitive aspect. As our discussion on priming studies shows, idea generation
1306 WANG AND NICKERSON
can also be influenced by a change in emotion, attitude, or motivational level. To
our knowledge, these additional aspects are not well-integrated into existing the-
ories of idea generation, which might hinder our understanding of stimuli use in
creative tasks. One source of theory may lie in cognitive science: ACT-R 5.0 is an
updated version of the ACT theory and aims at an integrated theory of mind [6]. In
this theory, there are different modules in mind, such as a declarative module
containing semantic memory, a production unit containing production rules, and
an intentional module containing goal information. It might be possible to build on
this model and further explain how an external stimulus might activate not only
semantic memory but also production rules and goal information, which further
influences ideation. It might be possible to add some explanation of how stimuli
influence emotions and in turn ideation. In short, an integrated theory of idea
generation considering different aspects might greatly improve our understanding
of creative work – and our ability to aid it.
In our hypothesis development, we argued that unrelated stimuli may increase
cognitive load, which in turn limits idea quantity and idea novelty if the increased
load reduces success in finding relevant ideas. Since we did not explicitly measure
cognitive load, we have not provided direct empirical evidence for this argument.
Psychologists have measured cognitive load through perceived mental effort [53]
and pupillary response [36]. In that we have proposed and tested a way of finding
stimuli of varying degrees of relatedness, it should be possible to apply this
technique to directly study the relationship between stimulus relatedness and
cognitive load in ideation tasks. Such studies may provide valuable insights into
the phenomenon of using stimuli to promote idea generation.
Some additional research directions are suggested by our results. In our experi-
ments, each participant saw exactly three stimulus concepts. In future research,
a different number of stimuli might be tested. It is an open question whether using
fewer stimuli would lead to the same results: it could be that even just one external
stimulus is enough to break someone out of a fixation. With regard to using more
than three stimuli, it may be that there are diminishing returns to extra stimuli.
Conversely, more stimuli might increase the chances of novel idea being generated,
and therefore might lead to larger differences across conditions. Such larger
differences might in turn cast more light on the proposed inverted U-shape relation-
ship between stimulus relatedness and idea novelty. It is possible that stimulus
relatedness affects idea novelty through a different non-linear function, or that
relatedness has different effects at different stages of ideation.
It may be useful to examine different strategies for using the stimuli generated
through our method. For example, it might be effective if the stimulus is used by
following a step-by-step process of analogical thinking [42] or other creativity
techniques [18, 62, 67]. In generating stimuli, we only used the associative links
in Wikipedia. It may also be fruitful to use the category structure in Wikipedia to
search stimuli to support creative work, because category structure may have an
embedded ontology which is useful in generating ideas [33].
A WIKIPEDIA-BASED METHOD TO SUPPORT IDEATION 1307
Given the proposed method allows control of stimuli distance, it becomes possi-
ble to use a sequence of stimuli at different distances. For example, much like
simulated annealing, stimuli might first be provided at far distances to shake
fixation and generate novelty. Subsequent stimuli might be provided at closer
distances in order to trigger an increase in idea usefulness. That is, instead of trying
to stimulate full-blown creative ideas in a single step, a system might seek to first
generate one dimension of creativity, novelty, followed by the other dimension of
creativity, usefulness.
Researchers might compare this Wikipedia-based method with other methods of
supporting creative work with stimuli, such as case-based reasoning [3] and online
methods based on metaphor, such as Yossarian Lives (https://yossarian.co/) and analo-
gical idea generation by crowds [72]. Case-based reasoning method depends on the
development of an information system that stores existing solutions. The rich details in
such existing solutions may affect creativity differently than the vaguer prompts
derived from Wikipedia. Yossarian Lives provides words and images that are meta-
phorically related to an initial concept; these might lead to associations and emotions
both. Yu et al. [72] suggested the use of online crowds to find analogies to aid ideation.
Comparing our method with these methods may lead to a better understanding of the
role of stimuli in creative work, which in turn may improve the design of creativity
support systems.
Providing stimuli is a common approach in creativity support systems. This article
proposes a new approach to generating stimuli that can be used to support creative
work. Specifically, starting with an initial Wikipedia concept, hypertext links in
Wikipedia pages are followed iteratively to find concepts of decreasing levels of
relatedness. Consistent with our predictions, stimulus relatedness is positively related
to idea quantity and idea usefulness. When people were exposed to remotely related
concepts, they generated ideas of higher novelty, compared with seeing no stimuli. By
contrast, unrelated concepts do not consistently improve idea novelty. Our research
suggests a systematic way of catalyzing creativity by generating stimuli that are not
random, but instead are related to the creative task to different degrees.
Supplemental data for this article can be accessed on the publisher’s website.
The authors want to acknowledge the support from the National Science Foundation under
grants 0968561, 1422066, 1442840, and 1717473.
1308 WANG AND NICKERSON
https://yossarian.co/
https://doi.org/10.1080/07421222.2019.1661095
NOTES
1. The Python programs that we used in the study are in the following repository:https://
github.com/bertramman/JMIS
2. Compared to Table 4 in study 1, Table 7 has an additional column representing the
regression of stimulated novelty on experimental conditions. This is to test hypothesis 3.
Unlike study 1, study 2’s control condition has three additional ideation steps parallel to
stimulus presentation in other conditions. Thus, study 2’s control condition has a counterpart
“stimulated novelty”, based on these additional ideation steps. There is no such “stimulated
novelty” in study 1’s control condition. Consequently, it is not feasible to do the same
regression in Table 4 in study 1.
REFERENCES
1. Adamson, R.E. Functional fixedness as related to problem solving: A repetition of
three experiments. Journal of Experimental Psychology, 44, 4 (1952), 288–291.
2. Althuizen, N.; and Reichel, A. The effects of IT-enabled cognitive stimulation tools on
creative problem solving: A dual pathway to creativity. Journal of Management Information
Systems, 33, 1 (2016), 11–44.
3. Althuizen, N.; and Wierenga, B. Supporting creative problem solving with a case-based
reasoning system. Journal of Management Information Systems, 31, 1 (2014), 309–340.
4. Amabile, T.M. Creativity in Context. Boulder, CO: Westview, 1996.
5. Anderson, J.R. The Architecture of Cognition. Cambridge, MA: Harvard University
Press, 1983.
6. Anderson, J.R.; Bothell, D.; Byrne, M.D.; Douglass, S.; Lebiere, C. and Qin; Y. An
integrated theory of the mind. Psychological Review, 111, 4 (2004), 1036–1060.
7. Baer, M.; Leenders, R.T.A.; Oldham, G.R.; and Vadera, A.K. Win or lose the battle for
creativity: The power and perils of intergroup competition. Academy of Management
Journal, 53, 4 (2010), 827–845.
8. Baughman, W.A.; and Mumford, M.D. Process-analytic models of creative capacities:
Operations influencing the combination-and-reorganization process. Creativity Research
Journal, 8, 1 (1995), 37–62.
9. Benjamini, Y.; and Hochberg, Y. Controlling the false discovery rate: a practical and
powerful approach to multiple testing. Journal of the Royal Statistical Society. Series
B (Methodological), 57, 1 (1995), 289–300.
10. Berg, J.M. The primal mark: How the beginning shapes the end in the development of
creative ideas. Organizational Behavior and Human Decision Processes, 125, 1 (2014), 1–17.
11. Boudreau, K.J.; Guinan, E.C.; Lakhani, K.R.; and Riedl, C. Looking across and
looking beyond the knowledge frontier: Intellectual distance, novelty, and resource allocation
in science. Management Science, 62, 10 (2016), 2765–2783.
12. Chan, J.; Dow, S.P.; and Schunn, C.D. Do the best design ideas (really) come from
conceptually distant sources of inspiration? Design Studies, 36, (2015), 31–58.
13. Chan, J.; Fu, K.; Schunn, C.D.; Cagan, J.; Wood, K.L.; and Kotovsky, K. On the
benefits and pitfalls of analogies for innovative design: Ideation performance based on
analogical distance, commonness, and modality of examples. Journal of Mechanical
Design, 133, 8 (2011), 081004.
14. Chiu, I.; and Shu, H. Investigating effects of oppositely related semantic stimuli on
design concept creativity. Journal of Engineering Design, 23, 4 (2012), 271–296.
15. Cicchetti, D.V. Guidelines, criteria, and rules of thumb for evaluating normed and standar-
dized assessment instruments in psychology. Psychological Assessment, 6, 4 (1994), 284–290.
16. Collins, A.M.; and Loftus E.F. A spreading-activation theory of semantic processing.
Psychological Review, 82, 6 (1975), 407–428.
17. Coskun, H.; Paulus, P.B.; Brown, V.; and Sherwood, J.J. Cognitive stimulation and
problem presentation in idea-generating groups. Group Dynamics: Theory, Research, and
Practice, 4, 4 (2000), 307–329.
A WIKIPEDIA-BASED METHOD TO SUPPORT IDEATION 1309
https://github.com/bertramman/JMIS
https://github.com/bertramman/JMIS
18. Couger, J.D.; Higgins, L.F.; and McIntyre, S.C. (Un)structured creativity in informa-
tion systems organizations. MIS Quarterly, 17, 4 (1993), 375–394.
19. Dahl, D.W.; and Moreau, P. The influence and value of analogical thinking during new
product ideation. Journal of Marketing Research, 39, 1 (2002), 47–60.
20. Dean, D.L.; Hender, J.M.; Rodgers, T.L.; and Santanen, E.L. Identifying quality, novel,
and creative ideas: Constructs and scales for idea evaluation. Journal of the Association for
Information Systems, 7, 10 (2006), 646–699.
21. Dennis, A.R.; and Valacich, J.S. Electronic brainstorming: Illusions and patterns of
productivity. Information Systems Research, 10, 4 (1999), 375–377.
22. Dennis, A.R.; Minas, R.K.; and Bhagwatwar, A.P. Sparking creativity: Improving
electronic brainstorming with individual cognitive priming. Journal of Management
Information Systems, 29, 4(2013),195–215.
23. Dennis, A.R.; Valacich, J.S.; Connolly, T.; and Wynne, B.E. Process structuring in
electronic brainstorming. Information Systems Research, 7, 2 (1996), 268–277.
24. Elam, J.J.; and Mead, M. Can software influence creativity? Information Systems
Research, 1, 1 (1990), 1–22.
25. Enkel, E.; and Gassmann, O. Creative imitation: exploring the case of cross industry
innovation. R and D Management, 40, 3 (2010), 256–270.
26. Fellbaum, C. WordNet: An Electronic Database. Cambridge, MA: MIT Press, 1998.
27. Fleming, L. Recombinant uncertainty in technological search. Management Science,
47, 1 (2001), 117–132.
28. Finke, R.A.; Ward, T.B.; and Smith, S.M. Creative Cognition: Theory, Research and
Application. Cambridge, MA: MIT Press, 1992.
29. Finke, R.A. Creative insight and preinventive forms. In R.J. Sternberg and J.E. Davidson
(eds.), The Nature of Insight. Cambridge, MA: MIT Press, 1995, pp. 255–280.
30. Fjermestad, J.; and Hiltz, S.R. An assessment of group support systems experimental
research: Methodology and results. Journal of Management Information Systems, 15, 3
(Winter 1998–99), 7–149.
31. Fu, K.; Chan, J.; Cagan, J.; Kotovsky, K.; Schunn, C.; and Wood, K. The meaning of
“near” and “far”: The impact of structuring design databases and the effect of distance of
analogy on design output. Journal of Mechanical Design, 135, 2 (2013), 021007.
32. Garfield, M.J.; Taylor, N.J.; Dennis, A.R.; and Satzinger, J.W. Research report:
Modifying paradigms – Individual differences, creativity techniques, and exposure to ideas
in group idea generation. Information Systems Research, 12, 3 (2001), 322–333.
33. Genc, Y.; Sakamoto, Y.; and Nickerson, J.V. Discovering context: Classifying tweets
through a semantic transform based on Wikipedia, In D. Schmorrow and C. Fidopiastis (eds).
Foundations of Augmented Cognition: Directing the Future of Adaptive Systems, Lecture
in Computer Science, Berlin: Springer, 2011, pp.484–492.
34. German, T.P.; and Barrett, H.C. Functional fixedness in a technologically sparse
culture. Psychological Science, 16, 1 (2005), 1–5.
35. Gick, M.L.; and Holyoak, K.J. Analogical problem solving. Cognitive Psychology, 12,
3 (1980), 306–355.
36. Granholm, E.; Asarnow, R.F.; Sarkin, A.J.; and Dykes, K.L. Pupillary responses index
cognitive resource limitations. Psychophysiology, 33, 4 (1996), 457–461.
37. Guo, J.; and McLeod P.L. The impact of semantic relevance and heterogeneity of
pictorial stimuli on individual brainstorming: An extension of the SIAM model. Creativity
Research Journal, 26, 3 (2014), 361–367.
38. Hargadon, A.B.; and Douglas, Y. When innovations meet institutions: Edison and the
design of the electric light. Administrative Science Quarterly, 46, 3 (2001), 476–501.
39. Hender, J.M.; Dean, D.L.; Rodgers, T.L.; and Nunamaker, J.F. An examination of the
impact of stimuli type and GSS structure on creativity: Brainstorming versus
non-brainstorming techniques in a GSS environment. Journal of Management Information
Systems, 18, 4 (2002), 59–85.
40. Hennessey, B.A.; and Amabile, T.M. Creativity. Annual Review of Psychology, 61,
(2010), 569–598.
41. Jansson, D.; and Smith, S. Design fixation. Design Studies; 12, 1 (1991), 3–11.
1310 WANG AND NICKERSON
42. Knoll, S.W.; and Horton, G. Changing the perspective: Using a cognitive model to improve
thinklets for ideation. Journal of Management Information Systems, 28, 1 (2011), 85–114.
43. Lenat, D.B. CYC: A large-scale investment in knowledge infrastructure.
Communications of the ACM, 38, 11 (1995), 33–38.
44. Luo, L.; and Toubia, O. Improving online idea generation platforms and customizing
the task structure on the basis of consumers’ domain-specific knowledge. Journal of
Marketing, 79, 5 (2015), 100–114.
45. MacCrimmon, K.R.; and Wagner, C. Stimulating ideas through creativity software.
Management Science, 40, 11 (1994), 1514–1532.
46. Malaga, R.A. The effect of stimulus modes and associative distance in individual
creativity support systems. Decision Support Systems, 29, 2 (2000), 125–141.
47. Massetti, B. An empirical examination of the value of creativity support systems on
idea generation. MIS Quarterly, 20, 1 (1996), 83–97.
48. Mednick, S.A. The associative basis of the creative process. Psychological Review, 69,
3 (1962), 220–232.
49. Miron-Spektor, E.; Gino, F.; and Argote, L. Paradoxical frames and creative sparks:
Enhancing individual creativity through conflict and integration. Organizational Behavior
and Human Decision Processes, 116, 2 (2011), 229–240.
50. Mitchell, A.; and Dacin, P.A. The assessment of alternative measures of consumer
expertise. Journal of Consumer Research, 23, 3 (1996), 219–239.
51. Mueller-Wienbergen, F.; Mueller, O.; Seidel, S.; and Becker, J. Leaving the beaten
tracks in creative work – A design theory for systems that support convergent and divergent
thinking. Journal of the Association for Information Systems, 12, 11 (2011), 714–740.
52. Nijstad, B.A.; and Stroebe, W. How the group affects the mind: A cognitive model of
ideation in groups. Personality and Social Psychology Review, 10, 3 (2006), 186–213.
53. Paas, F.; Tuovinen, J.E.; Tabbers, H.K.; and Van Gerven, P.W.M. Cognitive load
measurement as a means to advance cognitive load theory. Educational Psychologist, 38, 1
(2003), 63–71.
54. Perttula, M.K.; and Sipila, P.P. The idea exposure paradigm in design idea generation.
Journal of Engineering Design, 18, 1(2007),93–102.
55. Postmes, T.; Spears, R.; Sakhel, K.; and De Groot, D. Social influence in
computer-mediated communication: The effects of anonymity on group behavior.
Personality and Social Psychology Bulletin, 27, 10 (2001), 1243–1254.
56. Potter, R.E.; and Balthazard, P. The role of individual memory and attention processes
during electronic brainstorming. MIS Quarterly, 28, 4 (2004), 621–643.
57. Raaijmakers, J.G.W.; and Shiffrin R.M. Search of associative memory. Psychological
Review, 88, 2(1981),93–134.
58. Santanen, E.L.; Briggs, R.O.; and de Vreede, G. Causal relationships in creative
problem solving: Comparing facilitation interventions for ideation. Journal of Management
Information Systems, 20, 4 (2004), 167–197.
59. Satzinger, J.W.; Garfield, M.J.; and Nagasundaram, M. The creative process: The
effects of group memory on individual idea generation. Journal of Management
Information Systems, 15, 4(1999),143–160.
60. Sio, U.N.; Kotovsky, K.; and Cagan, J. Fixation or inspiration? A meta-analytic review
of the role of examples on design processes. Design Studies, 39, (2015), 70–99.
61. Smith, S.M.; Ward, T.B.; and Schumacher, J.S. Constraining effects of examples in
a creative generation task. Memory and Cognition, 21, 6(1993),837–845.
62. Stein, E.W.; and Zwass, V. Actualizing organizational memory with information
systems. Information Systems Research, 6, 2 (1995), 85–117.
63. Tseng, I.; Moss, J.; Cagan, J.; and Kotovsky, K. The role of timing and analogical
similarity in the stimulation of idea generation in design. Design Studies, 29, 3
(2008),203–221.
64. Viriyayudhakorn, K.; Kunifuji, S.; and Ogawa, M. A comparison of four association
engines in divergent thinking support systems on Wikipedia. In T. Theeramunkong,
S. Kunifuji, V. Sornlertlamvanich, and C. Nattee (eds.), Knowledge, Information, and
Creativity Support Systems. Heidelberg: Springer Berlin, 2011, pp. 226–237.
A WIKIPEDIA-BASED METHOD TO SUPPORT IDEATION 1311
65. Wang, H.-C.; Cosley, D.; and Fussell, S.R. Idea expander: Supporting group brain-
storming with conversationally triggered visual thinking stimuli. In K. Inkpen, and
C. Gutwin (eds.), Proceedings of the 2010 ACM Conference on Computer Supported
Cooperative Work, New York: ACM, 2010, pp. 103–106.
66. Wang, H.-C.; Fussell, S.R.; and Cosley, D. From diversity to creativity: Stimulating
group brainstorming with cultural differences and conversationally-retrieved pictures. In
P. Hinds, J. Tang, and J. Wang (eds.) Proceedings of the 2011 ACM Conference on
Computer Supported Cooperative Work, New York: ACM, 2011, pp. 265–274.
67. Wang, K. Factors influencing the adoption and effective use of creativity techniques in
business settings: An exploratory study. Engineering Management Journal, 26, 4 (2014), 29–37.
68. Wang, K.; and Nickerson, J.V. A literature review on individual creativity support
systems. Computers in Human Behavior, 74, (2017), 139–151.
69. Wang, K.; Nickerson, J.V.; and Sakamoto, Y. Crowdsourced idea generation: The
effect of exposure to an original idea. Creativity and Innovation Management, 27, 2
(2018), 196–208.
70. Ward, T.B. Structured imagination: The role of category structure in exemplar
generation. Cognitive Psychology, 27, 1 (1994), 1–40.
71. Witten, I.; and Milne, D. An effective, low-cost measure of semantic relatedness
obtained from Wikipedia links. In R. Bunescu, E. Gabrilovich, and R. Mihalcea (eds.),
Proceeding of AAAI Workshop on Wikipedia and Artificial Intelligence: An Evolving
Synergy, Chicago, IL: AAAI Press, 2008, pp. 25–30.
72. Yu, L.; Kraut, R.E.; and Kittur, A. Distributed analogical idea generation with multiple
constraints. In D. Gergle, and M.R. Morris (eds.), Proceedings of the 19th ACM Conference
on Computer-Supported Cooperative Work and Social Computing, New York: ACM, 2016,
pp. 1236–1245.
1312 WANG AND NICKERSON
Copyright of Journal of Management Information Systems is the property of Taylor & Francis
Ltd and its content may not be copied or emailed to multiple sites or posted to a listserv
without the copyright holder’s express written permission. However, users may print,
download, or email articles for individual use.
- Abstract
- References
Introduction
Background
Cognitive Theories of Creativity
Influence of Stimuli on Idea Generation
Hypotheses Development
New Method for Finding Stimuli of Different Levels of Relatedness
Testing the New Method
Study 1
Methods
Study Design and Participants
Dependent Variables
Independent Variables
Control Variables
Results
Study 2
Methods
Results
Discussion
Influence of Stimuli on Idea Quantity and Usefulness
Influence of Stimuli on Idea Novelty
Comparing Our Research with Priming Studies
Implications for Practice
Implications for Future Research
Conclusions
Supplemental Material
Funding
Notes
An Information Processing View on Joint
Vendor Performance in Multi-Sourcing:
The Role of the Guardian
ILAN OSHRI, JENS DIBBERN, JULIA KOTLARSKY, AND OLIVER
KRANCHER
ILAN OSHRI (ilan.oshri@auckland.ac.nz) is a professor at Graduate School of
Management, University of Auckland business school, University of Auckland,
Auckland, New Zealand. He is the author of “Offshoring Strategies: Evolving
Captive Centers Models”(MIT Press, 2011), and the co-author of “The Handbook
of Global Outsourcing and Offshoring” (Palgrave, 2015). He co-authored 20 book
s
and published numerous articles in academic and professional journals including
MIS Quarterly, European Journal of Information Systems, Journal of Information
Technology, and Strategic Journal of Information Systems. His work on outsourcing
was featured on BBC Radio 4, Wall Street Journal, Businessweek and Financial
Times. Ilan currently serves as associate editor of MIS Quarterly and senior editor
of Journal of Information Technology.
JENS DIBBERN (jens.dibbern@iwi.unibe.ch) is Professor of Information Systems at
the University of Bern, Department of Business Administration in Switzerland. His
research focuses on IT sourcing, platform ecosystems, system implementation/use,
and distributed collaboration. His publications appeared in Information Systems
Research (ISR), Management Information Systems Quarterly (MISQ), Journal of
Management Information Systems (JMIS), Journal of the Association of
Information Systems (JAIS), and others. He served as associate editor of MISQ
and as senior editor of JAIS and ACM Sigmis Database and currently serves as
senior editor of MISQ Executive and department editor of Business & Information
Systems Engineering (BISE).
JULIA KOTLARSKY (j.kotlarsky@auckland.ac.nz) is a Professor of Technology and
Global Sourcing at the University of Auckland Business School in New Zealand.
Her research interests revolve around technology sourcing and innovation in knowl-
edge-intensive business services, and more recently, studying interface between
artificial intelligence technologies and humans. Her work was published in numer-
ous journals including MIS Quarterly, European Journal of Information Systems,
Journal of Strategic Information Systems, Wall Street Journals and others. Her book
“The Handbook of Global Outsourcing and Offshoring” is widely used by aca-
demics and practitioners. She is co-founder of the annual Global Sourcing
Workshop (www.globalsourcing.org.uk). Julia serves as a Senior Editor for the
All authors contributed equally.
Journal of Management Information Systems / 2019, Vol. 36, No. 4, pp. 1248–1283.
Copyright © Taylor & Francis Group, LLC
ISSN 0742–1222 (print) / ISSN 1557–928X (online)
DOI: https://doi.org/10.1080/07421222.2019.1661091
mailto:ilan.oshri@auckland.ac.nz
mailto:jens.dibbern@iwi.unibe.ch
mailto:j.kotlarsky@auckland.ac.nz
http://www.globalsourcing.org.uk
https://crossmark.crossref.org/dialog/?doi=10.1080/07421222.2019.1661091&domain=pdf&date_stamp=2019-10-03
Journal of Information Technology and a former Associate Editor for MIS
Quarterly.
OLIVER KRANCHER (olik@itu.dk) is an associate professor in the Business IT depart-
ment of IT University of Copenhagen. He holds a Ph.D. from University of Bern.
His research interests revolve around knowledge processes in the development, use,
and management of information systems. He has published in outlets such as the
Journal of Management Information Systems, the Journal of the Association of
Information Systems, and the Proceedings of the International Conference on
Information Systems. Prior to his academic career, he served as a consultant in
enterprise software and outsourcing projects.
ABSTRACT: This paper examines joint vendor performance in multi-sourcing
arrangements. Using an Information Processing View, we argue that managing
interdependencies between multiple vendors imposes substantial information pro-
cessing (IP) requirements on clients. To achieve high joint performance, clients
therefore need to possess sufficient IP capacity. We examine how three sources of
IP capacity, two internal (i.e., the client’s inter-vendor governance and the client’s
architectural knowledge) and one external (i.e., the guardian vendor), work together
in realizing joint performance. Our results show that formal governance and
architectural knowledge contribute to joint performance. The guardian vendor
contributes to joint performance in settings where the client deploys strong govern-
ance but lacks architectural knowledge. This suggests that, contrary to common
views in the literature, guardian vendors should not be understood as mediators (or
single points of contact) who relieve clients from governance efforts. Instead,
guardian vendors are more fruitfully understood as architects, who complement
the client’s governance efforts by compensating for knowledge gaps. Put simply,
client firms should consider using a guardian vendor to compensate for weak
architectural knowledge while still maintaining strong formal and informal govern-
ance of all vendors.
KEY WORDS AND PHRASES: multi-sourcing, joint performance, guardian, governance,
architectural knowledge, information processing view.
In the information systems (IS) domain, multi-sourcing is viewed as the practice of
procuring interdependent information technology (IT) and business services from
external vendors to achieve optimal business goals [4]. Such a definition brings to
the fore the interdependencies between outsourced tasks delivered by various
vendors, thus implying the need for interactions between the vendors in order to
jointly deliver an overall service [4, 49]. In assessing the success of a multi-
sourcing arrangement, it is not the performance of the individual vendors that
matters most, but their joint performance, i.e., the degree to which the combined
services delivered by the vendors meet the client’s expectations. An example of
such a multi-sourcing arrangement is British Airways’ (BA’s) “Know Me
Programme,” which was initiated in 2013 and involves three vendors, Tata
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1249
mailto:olik@itu.dk
Consultancy Services (TCS), Opera Solutions, and e-Dialog (now Zeta
Interactive).1 Together, these three vendors form a new personalized customer
contact system. Although each vendor has its own responsibilities, that is, TCS
for collecting, integrating, and managing customer data, Opera Solutions for pro-
viding business analytics services, and e-Dialog for creating e-mail-based market-
ing services, the success of the project relies on all three services working together.
Accordingly, the vendors have to manage the interdependencies between their
services, which require them to cooperate and coordinate their actions. This exam-
ple resonates with Bapna et al.’s [4] claim that: “In contrast to dyadic client-vendor
relationships that have been the subject of extant global sourcing research, multi-
sourcing necessitates individual and collaborative efforts of multiple vendors at the
back-end to come together to create a seamless, integrated service at the front end
for the client” (p. 786). While facets associated with governance of dyadic relation-
ships, such as putting in place Service Level Agreements (SLAs) and using various
organizational controls to motivate vendors to achieve desirable results [47], are
also relevant, the client firm needs to put greater effort into governing the vendor
network in IT multi-sourcing [31], as well as incentivizing and monitoring both
individual and joint vendor performance [4]. On this account, the use of a guardian
vendor to assist the client firm in governing the vendor network (e.g., Bapna et al.
[4], Wiener and Saunders [49]) has been portrayed as one of the unique features of
the IS multi-sourcing setting.2
While a few studies have examined multi-sourcing in the IS context3 [4, 6, 12, 31,
42, 49], we still know little about interactions and collaboration between multiple
vendors and the effects on joint performance. In this regard, research has shed light on
the importance of appropriate task design (e.g., modularization) and task distribution
among vendors (e.g., choosing specialized vendors while ensuring sufficient knowl-
edge overlaps between them) [49]. However, little is known about how the client can
facilitate and support vendors to achieve successful joint performance. Moreover, it is
not clear how the client’s support role is affected if the client assigns one of the
vendors the position of guardian, that is, the responsibility for managing the other
vendors. Currently, the literature suggests that the guardian vendor acts as a mediator,
thus standing between the client and the other vendors [49]. This implies that the
guardian substitutes the client in facilitating and supporting coordination and coopera-
tion activities among the vendors [4, 49]. Alternatively, we propose that the guardian
may improve joint performance by providing capacities that complement those of the
client. It is within these areas of interest that this paper seeks to advance our under-
standing of multi-sourcing settings by addressing the following questions: (i) How
does the client facilitate joint vendor performance in a multi-sourcing arrangement?;
and (ii) What role does the guardian vendor play in achieving joint performance?
We frame the challenge of achieving joint vendor performance (hereafter, joint
performance) as an information processing (IP) issue. Hence, the challenge of achiev-
ing joint performance in a multi-sourcing arrangement is essentially one of effective IP
to manage interdependencies between the vendors and between the client and the
1250 OSHRI ET AL.
vendors, thus imposing considerable IP requirements. For instance, in the aforemen-
tioned example regarding British Airways, IP is needed to understand the functional
and technical system requirements of the client (BA), and also to understand the
interdependencies that exist between TCS’ customer data management systems and
processes, Opera Solutions’ data analytics processes, and E-dialog’s email platform.
While such IP requirements may vary between multi-sourcing arrangements, for
example, according to the degree of modularization [44], the involvement of numerous
vendors and the interdependencies between them will pose challenges to the client in
achieving joint performance if the client does not ensure sufficient and relevant IP
capacity. In this regard, governance (formal and informal) and architectural knowledge
have repeatedly been suggested as key factors affecting IP capacities [7, 15, 33].
Consequently, we examined how clients can ensure joint performance by assum-
ing sufficient IP capacities in multi-sourcing arrangements [15, 48], to support our
claim that such IP capacities may be brought in by the client (i.e., as an internal IP
capacity) or by the guardian vendor (i.e., as an external IP capacity) [4, 49]. We also
aimed to clarify whether the guardian vendor will have a substitutional or
a complementary effect on the client’s IP capacities.
Using an international data set of 189 IT multi-sourcing arrangements, we found that
two internal IP capacities complement each other. Indeed, the client’s formal inter-
vendor governance and the client’s architectural knowledge positively affect joint
performance, while informal inter-vendor governance has a significant effect on joint
performance only when interacting with high architectural knowledge. With regard to
the external source of IP capacity, we found that a guardian vendor complements the
client’s formal and informal inter-vendor governance while substituting the client’s
architectural knowledge. Thus, the guardian model is beneficial in settings where the
client provides the formal framework for the guardian vendor to interact with the other
vendors, where the client remains involved in this interaction, and where the client
lacks architectural knowledge. This implies that, contrary to what has been suggested in
the existing literature (i.e., Wiener and Saunders [49] and Bapna et al. [4]), the role of
the guardian vendor may be more fruitfully understood as one of an architect rather
than a mediator. The guardian compensates for the client’s knowledge gaps, while the
client still needs to engage in formal and informal governance of all vendors.
Next, we provide theoretical foundations and develop hypotheses. We then
explain the method and findings, followed by a discussion of the results and their
implications for research and practice.
Information Processing View and Multi-Sourcing
The Information Processing View (IPV) is a broad theoretical perspective that
views entities (e.g., people, teams, organizations, and inter-organizational relation-
ships) as IP systems and explains the structures and behaviors of these systems by
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1251
referring to their IP limitations [15, 25]. An important property of IP systems is
their IP capacity, broadly defined as their ability to interpret, integrate, store, and
transmit information [32 (p. 42)]. One prominent stream of IPV research [15, 32]
focuses on the IP capacity that is generated by governance mechanisms, namely
“mechanisms for coordination and control” [48 (p. 618)]. Governance mechanisms,
such as goal setting, planning, and direct interaction, generate IP capacity because
they provide the information infrastructure through which the constituent elements
of IP systems align actions (i.e., achieve coordination) and interests (i.e., achieve
cooperation) [2, 5, 37]. A second stream of IPV research focuses on IP capacity
generated by knowledge. It draws on a cognitive IP perspective to argue that IP
capacity depends on existing knowledge, because existing knowledge provides the
infrastructure that enables humans to assimilate and integrate new information [9,
13]. Building on these two streams, we seek to examine how IP capacity within the
multi-sourcing environment affects joint performance.
Indeed, the use of the IPV appears particularly suited to the context of multi-
sourcing in light of the following four gaps. First, multi-sourcing research lacks an
overarching theory that fits with the idiosyncrasies of multi-sourcing as opposed to
single-sourcing. In our view, what makes multi-sourcing unique is its inherent
complexity, which is based on interdependencies between vendors — as opposed
to the client-vendor interdependencies of dyadic outsourcing. While IPV has been
applied to studying dyadic relationships (e.g., Bensaou and Venkatraman [5], Mani
et al. [32]), where IP requirements may substantially vary from case to case, we
argue that triadic settings, such as multi-sourcing, add a layer of complexity that
warrants focus on the composition of IP capacities. The interdependencies that exist
between tasks allocated to multiple vendors pose significant IP requirements for the
client. In particular, in comparison to single-sourcing, the need to integrate sub-
services or tasks outsourced to different vendors into a coherent whole creates
additional IP requirements in multi-sourcing. Therefore, it is imperative to under-
stand joint performance by modeling and testing the effects of certain IP capacities
available within the multi-sourcing arrangement.
Second, the two streams of IPV research, one focusing on governance and the
other on knowledge as sources of IP capacity, have mostly been developed in
isolation. Consequently, understanding the relationship between architectural
knowledge and governance in multi-sourcing and how these two IP capacities
interact is imperative for both IS outsourcing and IPV research knowledge.
Third, the IS outsourcing literature [40] and the literature on multi-sourcing have,
thus far, mostly treated performance as an aggregate of the performances of the
individual vendors (e.g., Gopal et al. [20]). In this paper, however, we emphasize
that what makes multi-sourcing unique is that performance consists of more than
the sum of the contributions of individual vendors. As such, it is imperative to
develop an understanding of the combined or joint performance, rather than the
individual contributions of the vendors.
1252 OSHRI ET AL.
Last, but not least, the few references in the extant academic and professional4
literature regarding the role that the guardian vendor plays in multi-sourcing
settings [4, 49] raise questions about the contribution of this actor to joint perfor-
mance. We argue that the guardian brings its own unique set of IP capacities that
can either complement or substitute the IP capacities provided by the client.5
Client’s Challenge: With or Without a Guardian Vendor
IP capacity can be provided by either the client or the guardian vendor, if the client
has appointed one vendor to act as guardian (see Figure 1b6). In the direct model
(see Figure 1a), the client takes full responsibility for managing the vendors. In the
guardian model, the client transfers some responsibilities to the guardian vendor.
We argue that each model has important implications for the IP capacities needed to
achieve a high joint performance.
In the direct model, the client relies on two sources of IP capacity, namely
governance and architectural knowledge. Governance in dyadic outsourcing rela-
tionships often manifests as formal and informal governance between a client and
a vendor [36]. However, in multi-sourcing, informal and formal governance are
likely to be required to support the coordination of actions between multiple
vendors, thus suggesting a need for inter-vendor governance, that is, joint govern-
ance structures between the multiple vendors and the client firm. In line with the
psychological IPV research stream, we argue that information processing requires
appropriate knowledge to guide governance, in particular the client’s architectural
knowledge [24, 38, 43].
While the conditions for achieving joint performance by utilizing the client’s IP
capacities are clear, it is still unclear how the choice of a guardian model affects
ledomnaidrauG.bledomtceriD.a
Figure 1. Direct and Guardian Models
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1253
these conditions. Currently, the few IS outsourcing studies that have discussed the
guardian role suggest the guardian acts as a mediator, that is, as an actor standing
between the client and the rest of the vendors, thus relieving the client from
facilitating coordination and cooperation between vendors [4, 49]. To perform
such a role, the guardian brings in its own IP capacity. From the client’s perspec-
tive, therefore, the guardian acts as an external source of IP capacity, applying its
own inter-vendor governance as well as its own architectural knowledge.
However, the view of guardian vendor as a mediator can be challenged. As
reported in numerous sources,7 the client maintains an individual contractual
agreement with each vendor in the multi-sourcing setting, while the guardian
vendor does not have legally binding contractual agreements with any of the
vendors. Consequently, the guardian vendor’s ability to enforce inter-vendor gov-
ernance is in fact rather limited, particularly as the guardian vendor is restricted in
the range of penalties and incentives it can use when governing the other vendors.
Hence, it is unclear whether the guardian vendor does indeed assume a mediating
role, as proposed in the literature (e.g., Bapna et al. [4], Wiener and Saunders [49]).
Evidence from similar settings in manufacturing and construction predominantly
suggests that the guardian vendor brings in superior knowledge about integrating
the various contributions of individual vendors [7]. As such, an alternative view to
the role of the guardian as a mediator is the guardian as an architect. This describes
the guardian vendor as assisting the client in managing the multi-sourcing arrange-
ment by complementing the client’s IP capacities, rather than substituting them.
Thus, there are two views of the guardian vendor’s role in multi-sourcing. In one,
the guardian substitutes the client’s inter-vendor governance (guardian-as-a-media-
tor) and, in the other, the guardian vendor complements the client’s inter-vendor
governance (guardian-as-an-architect).
With this in mind, we now turn to theorizing the effect of internal and external
sources of IP capacity on joint performance.
In this section, we use the IPV lens to derive hypotheses aimed at examining the
effect of internal and external sources of IP capacity on joint performance. Figure 2
depicts our conceptual model.
Client’s Inter-Vendor Governance
According to the IPV,governance isconsidered an importantsource of IPcapacity [15].In
the context of multi-sourcing, it is manifested in inter-vendor governance efforts directed
at achieving coordination and cooperation among multiple vendors. The literature distin-
guishes between formal and informal governance mechanisms [26, 36]. Formal, or
1254 OSHRI ET AL.
mechanistic, governance relies on pre-specified plans (or programs, procedures, and
behaviors) and goals (or outcomes), and includes efforts toward specifying, monitoring,
and enforcing these plans and goals. Thus, formal inter-vendor governance includes
procedures that specify how vendors shall collaborate to achieve joint performance. As
an example of a joint procedural mechanism, Wiener and Saunders [49] described how
a client firm set up a support team made up of representatives from each vendor for the
duration of the contract. In this type of formal arrangement, vendors’ representatives are
able to communicate with each other in order to coordinate work on interdependent tasks,
while the client firm maintains communications with all vendors to ensure compliance
with the contract requirements.8
In contrast to such formal governance, informal or organic governance relies
on ad hoc communication between people [36]. The IPV literature refers to
informal governance as lateral relationships that allow for joint decision processes
across levels of authority [15]. In the context of multi-sourcing, this means com-
munication is facilitated across different hierarchical levels between the client and
all vendors. Hence, we conceptualize informal inter-vendor governance as more or
less frequently undertaken efforts for joint communication, that is, communication
involving the client and all vendors that cuts across different hierarchical levels. For
example, client representatives may meet with corresponding staff from all vendors
in order to resolve accountability issues [49]. In line with prior IPV studies, we
anticipate that both formal and informal inter-vendor governance generate IP
capacity and as a result help to improve joint performance [14, 21, 26, 32].
Accordingly, we hypothesize:
Figure 2. Research Model
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1255
Hypothesis 1: The stronger the client’s formal and informal inter-vendor
governance, the higher the joint performance.
Client’s Architectural Knowledge
While the client can generate IP capacity through governance efforts to support
coordination and cooperation between vendors, the cognitive stream of the IPV
literature suggests that effective IP also depends on underlying knowledge. In this
regard, in order to improve joint performance, it is imperative that the client brings in
relevant knowledge on how the different services outsourced to different vendors
should work together. Indeed, past research in the related domain of product devel-
opment has shown that firms engaging in multi-sourcing have invested in developing
abilities to integrate components delivered from various vendors [7, 24, 43].
Specifically, architectural knowledge is seen as a crucial resource that firms should
retain or develop if choosing to source from multiple vendors [7, 43]. For example, in
their analysis of specialization in knowledge production, Brusoni et al. [7] reported
that although one manufacturer had fully outsourced the development of aircraft
engine control systems to multiple vendors, the manufacturer still made significant
efforts to develop and retain its architectural knowledge, that is, “knowledge about
the ways in which the components are integrated and linked together into a coherent
whole” [23, p. 11]. Possessing such architectural knowledge improves the clients’
ability to ensure joint performance in multi-sourcing arrangements [7, p. 614].
Thus, we argue that in addition to the governance efforts discussed above, a major
factor determining a client’s IP capacity for managing a multi-sourcing arrangement
is the client’s architectural knowledge. With the benefit of architectural knowledge,
the client is then able to cope with the interdependencies between the outsourced
sub-tasks and manage interfaces between services delivered by individual vendors.
Therefore, we posit:
Hypothesis 2: The higher the degree of a client’s architectural knowledge, the
higher the joint performance.
The two sources of IP capacity previously discussed — the client’s inter-vendor
governance and architectural knowledge — are likely to have complementary effects
on joint performance. It is in inter-vendor governance efforts that the client can bring its
knowledge to bear to improve the management of interdependencies. Knowledgeable
clients are able to anticipate dependencies when they are specifying formal plans for
joint action [19, 36]. They may also have a greater ability to interpret information about
actual behaviors or outcomes than less knowledgeable clients [7]. For example, they
may be able to determine which vendor is accountable for a faulty delivery and
leverage this information during formal and informal governance to avoid finger
pointing [4]. Indeed Wiener and Saunders [49] illustrated such a case, arguing that
“consistent with the competitive paradigm, when vendors are part of a sourcing
arrangement involving multiple, interdependent vendors, they act in ways to make
1256 OSHRI ET AL.
their performance look better than their competitors’ and try to develop advantages
over them (e.g., a vendor may seek to blame the other vendors for project or service
delivery problems)” (p. 212). Resolving such conflict requires both governance and
architectural knowledge [28]. A knowledgeable client, who well understands the nature
of the interdependencies between the vendors, is likely to be able to apply appropriate
informal and formal inter-vendor governance mechanisms that address the core of such
conflict within the multi-sourcing arrangement. Clearly, lacking the required under-
standing of interdependencies would prevent the client firm from enacting appropriate
inter-vendor governance mechanisms to resolve the problem. In sum, both formal and
informal inter-vendor governing efforts are likely to be more effective for joint
performance when the client has a strong architectural knowledge. Therefore, we
hypothesize:
Hypothesis 3: A higher degree of architectural knowledge held by the client
strengthens the positive association between inter-vendor governance and joint
performance.
Guardian Vendor as a Source of External IP Capacity
Our earlier examination of the guardian vendor’s role suggests that the guardian
may serve alternative purposes as a mediator or as an architect. The guardian in
either role has differing implications for the client firm. For the guardian as
a mediator, it is expected that the client firm would retreat from governance efforts
now to be carried out by the guardian vendor. For the guardian as an architect, the
client firm would retain governance effort while benefiting from the guardian’s
architectural knowledge. As the literature has so far only considered the guardian’s
mediator role, here we propose a competing explanation and seek to theorize the
effect of each role on joint performance.
Guardian-as-a-Mediator Perspective
Viewing the guardian vendor as a mediator suggests that the guardian vendor is
positioned between the client and the other vendor(s) in the multi-sourcing arrange-
ment. Seen through an IPV lens, the guardian-as-a-mediator receives and interprets
information from the client (such as information about the overall service expected
from all the vendors working together), conveys the information to the other
vendors, and receives, interprets, and conveys information from the vendors back
to the client. In line with this perspective, Wiener and Saunders [49] argue that “the
guardian vendor […] coordinates the other vendors on the client’s behalf” (p. 213).
This assertion implies that the client retreats from inter-vendor governance, hand-
ing over this responsibility to the guardian vendor. The two internal sources of IP
capacity, inter-vendor governance and the client’s architectural knowledge, are then
likely to become less important or even detrimental for joint performance.
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1257
Regarding the first, high amounts of inter-vendor governance by the client could
even be detrimental to joint performance because confusion may arise if the
guardian vendor believes it is to exercise inter-vendor governance, but the client
continues to do so as well. A client who actively exercises inter-vendor governance
would be at odds with the “single point of contact” [49, p. 213] principle inherent to
the guardian-as-a-mediator perspective. Such parallel governance efforts are likely
to result in coordination failures and accountability challenges.
The client’s architectural knowledge is also likely to become less important with
a guardian model based on the guardian-as-a-mediator perspective. Since the client
retreats from inter-vendor governance, it is likely to have far fewer occasions to
bring to bear its own knowledge. The occasions in which the client does bring to
bear its own knowledge are then largely limited to interactions with the guardian
vendor. Therefore, although architectural knowledge may still be beneficial in
helping to govern the guardian vendor more effectively, it is likely to be less critical
than in the case of a direct model.
In sum, we argue that should the guardian assume the role of a mediator, it is
plausible to suggest that the IP capacities of the client will be substituted by the IP
capacity generated through the guardian model. We therefore assert that:
Hypothesis 4a/b: The choice of the guardian model weakens the positive
effects (a) of the client’s inter-vendor governance and (b) of the client’s
architectural knowledge on joint performance.
Guardian-as-an-Architect Perspective
An alternative perspective to the guardian-as-a-mediator is the guardian-as-an-
architect. This suggests that the guardian vendor contributes to joint performance
by bringing in architectural knowledge that supports the client’s governance efforts,
rather than relieving the client from engaging in inter-vendor governance. In this
perspective, the guardian vendor has a complementary relationship with the client
regarding inter-vendor governance and a substitutive relationship with the client
regarding architectural knowledge, as we will argue next.
According to the guardian-as-an-architect perspective, we expect a complementary
relationship with the client’s inter-vendor governance for two reasons. First, the
guardian vendor brings in valuable knowledge, such as knowledge of governance
structures effective for multi-sourcing relationships [31], and of the service architec-
ture that underlies the multi-sourcing arrangement. As we argued earlier, knowledge is
likely to make governance more effective [7, 28], as the client managers are able to
leverage this knowledge to improve their inter-vendor governance. Second, while the
guardian vendor may lack the formal authority and thus legitimacy to enact effective
governance, the client maintains a high level of involvement in this capacity. Indeed,
the client is the only party with legally binding contractual agreements with all the
vendors [4, 12]. High levels of inter-vendor governance by the client paired with
a guardian model allow multi-sourcing arrangements to leverage the client’s authority
1258 OSHRI ET AL.
and the guardian vendor’s knowledge at the same time. In sum, we expect that the
external IP capacity generated through the knowledge brought in by the guardian will
complement the internal IP capacity generated through the client’s inter-vendor
governance. These ideas echo Bapna et al.’s [4] view of the governance efforts of
the client in the presence of a guardian vendor, in that: “[…] not only does the client
still engage in multilateral contracts with multiple vendors but also has to consider the
guardian’s ability to ensure cooperation and coordination in determining its overall
relationship structure” (p. 794).
In the guardian-as-an-architect perspective, while the guardian vendor complements
the client’s inter-vendor governance, it substitutes the client’s architectural knowledge.
Without the presence of a guardian vendor, the client requires strong architectural
knowledge in order to exercise effective governance (e.g., to tackle accountability
problems and to design effective plans for coordination). Conversely, the client’s
architectural knowledge is likely to be less critical (although still beneficial) in the
presence of a guardian vendor. If a client lacks architectural knowledge, the guardian
vendor can compensate by providing guidance on how to set up and exercise effective
inter-vendor governance. Thus, the positive effect of the client’s architectural knowl-
edge on joint performance is likely to be weaker in the guardian model than in the
direct model. This corresponds to a substitutive relationship [46, p. 88], whereby the
benefits from the architectural knowledge held by the client decrease with the choice
of the guardian model. Seen through the IPV, the external IP capacity generated
through the guardian vendor’s knowledge partially substitutes the internal IP capacity
generated through the client’s architectural knowledge. In conclusion, the guardian-as-
an-architect perspective leads us to the following hypothesis:
Hypothesis 5a/b: The choice of the guardian model (a) strengthens the
positive effect of the client’s governance on joint performance, while (b)
weakening the positive effect of the client’s architectural knowledge on joint
performance.
Control Variables
While our research model focuses on sources of IP capacity and their interactions,
we have also controlled for a number of other relationships established in the
outsourcing and IPV literature. First, we controlled for modularity. Modularity
refers to the degree to which the outsourced sub-tasks can be easily combined
into a coherent whole [39, 44]. Outsourcing arrangements of high modularity rely
on well-defined, standardized interfaces that facilitate the integration of the sub-
tasks performed by the different parties [3, 6, 44]. From an IPV perspective, such
modularity is a key determinant of IP requirements [44]. Modular arrangements
may ease the composition and integration of sub-tasks outsourced to different
vendors and, thereby, lower IP requirements. Accordingly, modularity may increase
joint performance independent of the IP capacity available in a multi-sourcing
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1259
arrangement. Second, in line with the existing IPV research, we controlled for
interactions of IP requirements with sources of IP capacity [2, 32]. Specifically, it
can be argued that high modularity lowers the need for IP capacities to satisfy the
client’s expectations of joint vendor performance. We therefore controlled for
interactions between modularity and formal and informal governance, the client’s
architectural knowledge, and the choice of the guardian model. Moreover, we
controlled for concentration (i.e., the degree to which a large fraction of the project
work is allocated to a few vendors) [27], age of the arrangement (i.e., the number of
years since the creation of the multi-sourcing arrangement), client size (as indicated
by the number of employees), client country, client industry, and tasks included in
the arrangement (business process outsourcing, application development). We also
controlled for the interaction between concentration and choice of the guardian
model. Low concentration indicates that many vendors are involved in the multi-
sourcing arrangement. The lower the concentration, the more difficult it may be for
the guardian vendor to manage the large number of other vendors, suggesting
a possible interaction effect between concentration and the choice of the guardian
model.
Data
We empirically tested the theoretical framework (Figure 2) using a survey ques-
tionnaire and “key informants” methodology for data collection [35], in line with
past IS outsourcing studies (e.g., Goo et al. [17]). The data were collected in 2012
and 2013 with the help of a UK-based market research firm.
The questionnaire was administered to organizations across different countries,
including the United Kingdom, Germany, France, Italy, Spain, and the United
States, and spanning a variety of industries. For this purpose, the original English
version of the questionnaire was translated by the market research firm and checked
by native speakers (chosen by the authors) who were familiar with the study context
to ensure the correctness of the translation. Responses were collected using both
telephone interviews and an online survey.
The questionnaire was distributed among potential middle and top-level infor-
mants who were familiar with multi-sourcing arrangements within their firms. To
ensure the targeted individuals’ familiarity with multi-sourcing arrangements (so
qualifying them as a “key informant”), the respondents needed to answer a set of
screening questions and meet the following criteria:9
● Being employed by an organization with at least 250 employees,
● Having an outsourcing arrangement(s) in place where the organization had
consciously divided a task or project into particular sub-tasks or sub-projects
that were outsourced to different vendors, and
1260 OSHRI ET AL.
● Having familiarity with the management of such a multi-sourcing arrange-
ment in her or his company.
The respondent then had to select one particular multi-sourcing arrangement cur-
rently in place in their company and with which they were familiar. Within this
particular multi-sourcing arrangement, the respondent was asked to select the two
vendors contributing the most to the multi-sourcing arrangement (in terms of
amount of work). The questions relevant for testing our model pertained only to
this particular multi-sourcing arrangement with the two chosen vendors, subse-
quently called vendor A and B throughout the questionnaire. Our study and
empirical testing thus focused on one particular “triad” within the multi-sourcing
arrangement [10], each triad consisting of the client and two key vendors. Focusing
on triads ensured that the unit of analysis was the same for all respondents.
Before sending out the final questionnaire, the questionnaire items were pilot-
tested with 15 international organizations to ensure that all items could be under-
stood and answered by the intended group of respondents. Each block of questions
was followed by an open field for comments, where respondents were asked to note
down any thoughts they had on the questions asked in the preceding section. The
comments were considered in the refinement of the questionnaire and some amend-
ments were introduced to improve the clarity of questions. In addition, we tested
our model on the pilot data to assess the validity of the constructs. Items that loaded
very low were removed from the questionnaire.
The finalized questionnaire was sent out to 2000 organizations. Overall, 20
0
usable questionnaires were made available after several follow-ups with the sample
organizations. From these 200 cases, we excluded 10 after reviewing the descrip-
tions of outsourced tasks. We excluded cases when the sub-tasks assigned to
different vendors were not interdependent (e.g., outsourcing IT procurement to
vendor A and sales advice to vendor B), or when the outsourced tasks did not
match our target services, which comprised IT services and IT-supported business
processes. For example, in one case the services were “providing a camera crew”
(vendor A) and “providing special equipment for camera crew services” (vendor
B). We also excluded one outlier, which reported a joint performance of four
standard deviations below the sample mean although the same firm reported above-
average individual performance10 of the vendors, suggesting an erroneous measure-
ment. Our final sample size was n = 189. Table 1 shows the sample characteristics.
Measures
Each construct was measured with a block of indicators (questionnaire items). Where
possible, we used existing measures that we adapted to the study context [43]. All
items were measured on a five-point Likert scale, ranging from “strongly disagree”
(=1) to “strongly agree” (=5) with “neither agree nor disagree” (=3) as the mid-point.
An overview of the constructs and measurement items is provided in Table 2. Joint
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1261
performance was measured by six items (developed in IS outsourcing research) that
focused on the degree to which the joint performance of the two vendors met the
client’s expectations. Architectural knowledge was measured by three items that
focused on the client’s knowledge in relation to the integration of the services
delivered by the two vendors. Our measures of formal governance referred to the
use of two key formal governance strategies in the IPV, that is, procedures and goals
[15, p. 43-46]. The measures focused on the client’s efforts for specifying joint
procedures and goals and for evaluating the vendors’ adherence to the procedures
and goals. Our measures of informal governance focused on what IPV researchers call
lateral relations, that is, “direct contact between two people who share a problem” at
Table 1. Sample Characteristics
Characteristics of the Sample
[Min;
Max]
Mean (Std.
Dev.)
Respondent work
experience
Number of years working in organization [.5; 35] 8.6 (6.5)
Age of multi-sourcing
arrangement
Years that have passed since the start
of the multi-sourcing arrangement
[1; 9] 3.7 (2.4)
Number Percentage
(%)
Client size Up to 1,000 employees 70 37
1,001 to 5,000 employees 61 32
5,001 to 50,000 employees 46 24
More than 50,000 employees 12 6
Country United Kingdom 33 17
France 31 16
Germany 33 17
Italy 32 17
Spain 30 16
USA 30 16
Industry sector Financial services 34 18
Manufacturing 39 2
1
Retail, distribution and transport 25 13
Public sector 35 19
Other 56 30
Respondent’s area of
work within client firm
Owner/executive 22 12
Finance 18 10
IT 103 54
Facilities 5 3
Marketing 7 4
Customer services 15 8
Human resources 10 5
Logistics 9 5
1262 OSHRI ET AL.
the same hierarchical level [15, p. 53]. These measures, adapted from Takeishi [43],
assessed the amount of direct contact at three levels: IT staff, middle management, and
top management. To assess whether a guardian vendor model was chosen, we asked
whether one of the two vendors was responsible for managing other vendors. The
measures for our control variable modularity were taken from Tanriverdi et al. [44].
Table 3 provides an overview of the measures for control variables.
Instrument Validation
To validate our survey instrument, we assessed convergent and discriminant validity
through factor analysis procedures. To examine convergent validity, we first performed
an exploratory factor analysis in SPSS. This analysis reproduced the five latent factors
of our research model with eigenvalues greater than 1.6. Eigenvalues greater than 1
suggest convergent validity [16]. To further corroborate convergent validity, we calcu-
lated composite reliability (CR), average variance extracted (AVE), and standardized
factor loadings, using confirmatory factor analysis procedures in SmartPLS [16]. CR
was well above the threshold of .7 for all constructs (see Table 2). AVE was well above
the threshold of .5 for all constructs (see Table 2). The standardized factor loadings
were greater than .7 with the exceptions of FG1 (.66) and FG4 (.65), which were close
to the threshold. These two slightly lower values could be due to our attempt to capture
formal governance as broadly and comprehensively as possible. By and large, the
measurement evidence supports convergent validity.
We then examined discriminant validity. We tested whether each item loaded
higher on its construct than on any other construct [16]. For each item, the
difference between the loading of the item on its construct and the cross-loading
of the item on any other construct was above .2. Moreover, we examined whether
the square roots of the AVE values exceeded correlations between latent constructs
[16]. The square root of the lowest AVE value (.75 for formal governance) was well
above the highest correlation between two latent constructs (.60 for the correlation
between joint performance and formal governance). These results, and the fact that
exploratory factor analysis reproduced the five latent factors, strongly support
discriminant validity.
Regression Analysis
We used a regression approach to test our hypotheses. Given our focus on interac-
tion effects, we chose regression over alternative approaches, such as partial least
squares (PLS) and covariance-based structural equation modeling (SEM).
Regression offers higher statistical power for detecting interaction effects than
PLS or covariance-based SEM [18]. The advantage gained in statistical power is
particularly pronounced in models such as ours, in which many items are subject to
interaction effects [18, p. 222]. We relied on standardized mean scores to transform
sets of items into regression variables.
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1263
T
ab
le
2
.
Q
u
es
ti
o
n
n
ai
re
It
em
s
C
o
n
st
ru
ct
It
em
W
o
rd
in
g
R
ef
er
en
ce
Jo
in
t
p
e
rf
o
rm
a
n
ce
(C
R
=
.9
0
,
A
V
E
=
.6
1
)
W
ith
re
g
a
rd
to
co
m
b
in
e
d
p
e
rf
o
rm
a
n
ce
o
f
ve
n
d
o
r
A
a
n
d
ve
n
d
o
r
B
a
s
p
a
rt
o
f
th
e
m
u
lti
–
so
u
rc
in
g
a
rr
a
n
g
e
m
e
n
t
so
fa
r
…
JP
1
…
th
e
p
ro
d
u
ct
s/
se
rv
i
c
e
s
d
e
liv
e
re
d
m
e
e
t
o
u
r
e
xp
e
ct
a
tio
n
s.
G
ro
ve
r
e
t
a
l.
[2
1
]
JP
2
…
w
e
h
a
ve
m
e
t
o
u
r
g
o
a
ls
.
JP
3
…
w
e
h
a
ve
co
m
p
le
te
d
ke
y
m
ile
st
o
n
e
s
in
a
cc
o
rd
a
n
ce
w
ith
o
u
r
o
b
je
ct
iv
e
s.
JP
4
…
w
e
h
a
ve
a
ch
ie
ve
d
o
u
r
d
e
si
re
d
co
st
sa
vi
n
g
s.
JP
5
…
w
e
a
re
sa
tis
fie
d
w
ith
o
u
r
o
ve
ra
ll
b
e
n
e
fit
s
fr
o
m
o
u
ts
o
u
rc
in
g
.
L
e
e
a
n
d
K
im
[3
0
]
JP
6
…
w
e
h
a
ve
so
fa
r
m
e
t
p
ro
je
ct
/s
e
rv
ic
e
re
q
u
ir
e
m
e
n
ts
.
T
iw
a
n
a
[4
5
]
A
rc
h
ite
ct
u
ra
l
kn
o
w
le
d
g
e
(C
R
=
.8
8
,
A
V
E
=
.7
2
)
T
h
e
fo
llo
w
in
g
q
u
e
st
io
n
s
a
re
re
la
te
d
to
th
e
le
ve
l
o
f
kn
o
w
le
d
g
e
o
f
yo
u
a
n
d
yo
u
r
in
-h
o
u
se
co
lle
a
g
u
e
s.
W
e
h
a
ve
kn
o
w
le
d
g
e
a
b
o
u
t
…
A
K
1
…
th
e
d
e
si
g
n
o
f
th
e
o
ve
ra
ll
p
ro
d
u
ct
a
n
d
se
rv
ic
e
a
rc
h
ite
ct
u
re
to
w
h
ic
h
ve
n
d
o
rs
A
a
n
d
B
co
n
tr
ib
u
te
.
T
a
ke
is
h
i
[4
3
]
A
K
2
…
h
o
w
to
st
ru
ct
u
ra
lly
co
o
rd
in
a
te
th
e
p
ro
d
u
ct
s
a
n
d
se
rv
ic
e
s
d
e
liv
e
re
d
b
y
ve
n
d
o
rs
A
a
n
d
B
w
ith
a
ll
o
th
e
r
re
la
te
d
p
ro
d
u
ct
s
a
n
d
se
rv
ic
e
s
o
f
o
u
r
o
rg
a
n
iz
a
tio
n
.
A
K
3
…
th
e
w
a
ys
in
w
h
ic
h
th
e
p
ro
d
u
ct
s
a
n
d
se
rv
ic
e
s
d
e
liv
e
re
d
b
y
ve
n
d
o
rs
A
a
n
d
B
a
re
in
te
g
ra
te
d
a
n
d
lin
ke
d
to
g
e
th
e
r
in
a
co
h
e
re
n
t
w
h
o
le
.
H
e
n
d
e
rs
o
n
a
n
d
C
la
rk
[2
3
]
F
o
rm
a
l
g
o
ve
rn
a
n
ce
(C
R
=
.9
0
,
A
V
E
=
.6
3
)
T
o
e
n
su
re
th
a
t
it
is
n
o
t
th
e
in
d
iv
id
u
a
l
p
e
rf
o
rm
a
n
ce
o
f
ve
n
d
o
r
A
a
n
d
B
,
b
u
t
ra
th
e
r
th
e
ir
co
m
b
in
e
d
p
e
rf
o
rm
a
n
ce
(i
.e
.,
so
lu
tio
n
s
b
y
ve
n
d
o
r
A
a
n
d
B
in
co
m
b
in
a
tio
n
a
s
p
a
rt
o
f
th
e
m
u
lti
-s
o
u
rc
in
g
a
rr
a
n
g
e
m
e
n
t)
th
a
t
m
e
e
ts
o
u
r
o
b
je
ct
iv
e
s,
w
e
…
K
ir
sc
h
e
t
a
l.
[2
8
]
F
G
1
…
e
xp
e
ct
b
o
th
ve
n
d
o
rs
to
fo
llo
w
a
n
u
n
d
e
rs
ta
n
d
a
b
le
w
ri
tt
e
n
se
q
u
e
n
ce
o
f
st
e
p
s
th
a
t
d
e
fin
e
s
in
te
ra
ct
io
n
s
b
e
tw
e
e
n
th
e
se
tw
o
ve
n
d
o
rs
.
F
G
2
…
a
ss
e
ss
th
e
e
xt
e
n
t
to
w
h
ic
h
b
o
th
ve
n
d
o
rs
in
te
ra
ct
in
a
cc
o
rd
a
n
ce
to
e
xi
st
in
g
w
ri
tt
e
n
p
ro
ce
d
u
re
s
a
n
d
p
ra
ct
ic
e
s
w
h
e
n
d
e
liv
e
ri
n
g
th
e
o
u
ts
o
u
rc
e
d
se
rv
ic
e
.
F
G
3
…
e
va
lu
a
te
th
e
e
xt
e
n
t
to
w
h
ic
h
co
m
b
in
e
d
se
rv
ic
e
s
a
re
d
e
liv
e
re
d
a
s
d
e
fin
e
d
in
th
e
co
n
tr
a
ct
re
g
a
rd
le
ss
o
f
h
o
w
th
is
g
o
a
l
is
a
cc
o
m
p
lis
h
e
d
.
F
G
4
…
te
st
in
te
rm
e
d
ia
ry
a
n
d
/o
r
fin
a
l
jo
in
t
o
u
tc
o
m
e
s/
d
e
liv
e
ra
b
le
s
a
g
a
in
st
cr
ite
ri
a
d
e
fin
e
d
in
th
e
co
n
tr
a
ct
,
re
g
a
rd
le
ss
o
f
h
o
w
th
is
g
o
a
l
is
a
ch
ie
ve
d
.
F
G
5
…
h
a
ve
se
ve
ra
l
so
u
rc
e
s
o
f
o
b
je
ct
iv
e
d
a
ta
w
e
ca
n
re
ly
o
n
.
F
G
6
…
h
a
ve
d
e
fin
e
d
q
u
a
n
tif
ia
b
le
m
e
a
su
re
s
d
e
p
ic
tin
g
th
e
e
xt
e
n
t
to
w
h
ic
h
co
m
b
in
e
d
o
b
je
ct
iv
e
s
a
re
a
ch
ie
ve
d
.
F
G
7
…
h
a
ve
d
e
fin
e
d
a
cc
u
ra
te
a
n
d
re
lia
b
le
m
e
a
su
re
s
th
a
t
in
d
ic
a
te
th
e
e
xt
e
n
t
to
w
h
ic
h
th
e
d
e
liv
e
re
d
se
rv
ic
e
s
jo
in
tly
m
e
e
t
o
u
r
o
b
je
ct
iv
e
s.
1264 OSHRI ET AL.
In
fo
rm
a
l
g
o
ve
rn
a
n
ce
(C
R
=
.8
6
,
A
V
E
=
.6
6
)
IG
1
O
u
r
IT
st
a
ff
in
te
ra
ct
jo
in
tly
w
ith
b
o
th
ve
n
d
o
rs
’
IT
p
e
rs
o
n
n
e
l.
T
a
ke
is
h
i
[4
3
]
IG
2
O
u
r
m
id
d
le
m
a
n
a
g
e
rs
in
te
ra
ct
jo
in
tly
w
ith
b
o
th
ve
n
d
o
rs
’
m
id
d
le
m
a
n
a
g
e
rs
.
IG
3
O
u
r
to
p
m
a
n
a
g
e
rs
/
e
xe
cu
tiv
e
s
in
te
ra
ct
jo
in
tly
w
ith
b
o
th
ve
n
d
o
rs
’
to
p
m
a
n
a
g
e
rs
/
e
xe
cu
tiv
e
s.
G
u
a
rd
ia
n
ve
rs
u
s
D
ir
e
ct
(s
in
g
le
ite
m
)
G
U
A
re
e
ith
e
r
o
f
th
e
tw
o
ve
n
d
o
rs
re
sp
o
n
si
b
le
fo
r
m
a
n
a
g
in
g
a
ll
o
th
e
r
ve
n
d
o
rs
in
th
e
m
u
lti
–
so
u
rc
in
g
a
rr
a
n
g
e
m
e
n
t?
●
Y
e
s,
ve
n
d
o
r
A
→
G
u
a
rd
ia
n
●
Y
e
s,
ve
n
d
o
r
B
→
G
u
a
rd
ia
n
●
N
o
,
th
is
is
o
u
r
re
sp
o
n
si
b
ili
ty
→
N
o
n
-g
u
a
rd
ia
n
●
O
th
e
r
(p
le
a
se
e
xp
la
in
)
→
M
a
n
u
a
lly
co
d
e
d
1
1
S
e
lf-
d
e
ve
lo
p
e
d
M
o
d
u
la
ri
ty
(C
R
=
.8
1
,
A
V
E
=
.6
8
)
R
e
g
a
rd
in
g
th
e
tw
o
ta
sk
s/
p
ro
je
ct
s
o
u
ts
o
u
rc
e
d
to
ve
n
d
o
r
A
a
n
d
B
,
…
M
O
1
…
it
is
ve
ry
e
a
sy
to
co
m
b
in
e
th
e
ir
p
a
rt
ic
u
la
r
o
u
tc
o
m
e
s
in
to
a
co
h
e
re
n
t
w
h
o
le
.
T
a
n
ri
ve
rd
i
e
t
a
l.
[4
4
]
M
O
2
…
th
e
y
h
a
ve
w
e
ll-
d
e
fin
e
d
in
te
rf
a
ce
s
w
ith
e
a
ch
o
th
e
r.
N
o
te
s:
C
R
=
C
o
m
p
o
si
te
R
el
ia
b
il
it
y
;
A
V
E
=
av
er
ag
e
v
ar
ia
n
ce
ex
tr
ac
te
d
.
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1265
We used a four-step hierarchical regression strategy. In the first step (Model 1),
we included the main effects of control variables. In the second step (Model 2), we
added the main effects of the hypothesized predictors. In the third step (Model 3),
we added the interactions of IP requirements (i.e., modularity) with sources of IP
capacity (i.e., formal and informal governance, architectural knowledge, guardian
model) to control for interaction effects established in the IPV research. In the
fourth step (Model 4), we added the hypothesized interaction effects.
We examined whether the assumptions of regression analyses were met [50, pp.
104-105]. The histograms and q-q plots showed that the residuals followed normal
distributions, indicating that the assumption of normally distributed error terms was
met. Variance inflation factors were below 3, suggesting that multicollinearity
problems were not salient in the data. Plotting residuals and joint performance in
a scatter plot diagram showed no departure from the assumption of homoscedastic
error terms.
Although our study focused on interaction effects which cannot be artifacts of
common-method variance (e.g., Siemsen et al. [41]), we performed Harman’s single
factor test to appreciate whether item responses varied due to one single factor. We
found that a single factor was able to explain 26% of the variance and that five
factors were needed to explain half of the variance. Given these results and our
Table 3. Control Variables
Variable Measurement
Concentration The fraction of the overall budget of the multi-sourcing
arrangement that is assigned to vendors A and B;
measured through a single-item question
Modularity Measured through two questionnaire items (see Table 2)
Age of the multi-sourcing
arrangement
The number of years since the start of the multi-sourcing
arrangement; measured through a single-item question
Client size The client’s number of employees; measured through
a single-item question (transformation: natural logarithm)
Country Single-item question on the client’s country (United Kingdom,
Germany, Italy, Spain, USA, France); incorporated through
five dichotomous dummy variables (reference category:
France)
Industry Single-item question on the client’s sector (financial services,
manufacturing, retail, public sector, other); incorporated
through four dichotomous dummy variables (reference
category: Other)
Business Process
Outsourcing (BPO)
Indicates whether business processes (other than IT) were
part of the outsourced tasks; coded based on task
descriptions
Application Development Indicates whether the development of application software
was part of the outsourced tasks; coded based on task
descriptions
1266 OSHRI ET AL.
focus on interaction effects, it is unlikely that the findings reported in this study are
artifacts of common-method variance.
To assess potential endogeneity threats in our analysis, we estimated alternative
models based on Heckman correction for self-selection. Specifically, it is possible
that clients self-select the guardian model based on factors that also correlate with
joint performance (e.g., vendor capability). This could result in biased, inconsistent
estimates [22]. Following prior research on governance [32], we performed the
following steps to correct for the potentially endogenous choice of the guardian
model. First, we estimated a first-stage probit selection model that regressed the
choice of the guardian model on all main-effect predictors of the second-stage model
and on BPO. BPO served as an exclusion restriction, that is, as a variable that helps
predict the selection variable (i.e., choice of the guardian model) but does not
correlate with the dependent variable (i.e., joint performance) [11]. We chose BPO
as our exclusion restriction because the guardian model has only recently gained
popularity in IS outsourcing. Hence, we expected that BPO arrangements would
make greater use of the guardian model than IS outsourcing arrangements, while we
had no reason to expect that BPO arrangements would differ from IS outsourcing
arrangements in their level of joint performance. In line with these ideas, BPO
correlated strongly with the choice of the guardian model (ß = .82; p < .01) but not
with joint performance (p > .05). Second, we calculated the inverse Mills ratio for
each observation as follows:
λ1i ¼ φ β
0Xið Þ
ϕ β0Xið Þ for arrangements that chose a guardian model
λ0i ¼ � φ β
0Xið Þ
1�ϕ β0Xið Þð Þ for arrangements that chose a direct model
where λji is the inverse Mills ratio, φ the standard normal probability density
function, β0 is the vector of regression coefficients estimated by the probit selection
model, and ϕ the cumulated standard normal probability function. Third, we
included the inverse Mills ratio as a control variable in the second-stage model
predicting joint performance. The Heckman correction approach helps control for
the client’s propensity to choose a guardian model. Moreover, the regression
coefficient related to the inverse Mills ratio serves as an indicator for the presence
of endogeneity. If the coefficient is significantly different from 0, this indicates that
endogeneity is present and, hence, should be corrected for by including the inverse
Mills ratio as a control variable. We estimated alternative models based on
Heckman correction (Model 2b, 3b, 4b) for each model that contained the guardian
model as a predictor (i.e., Model 2a, 3a, 4a).
To examine nonresponse bias, we compared the means of eight key variables
(joint performance, modularity, concentration, age of the arrangement, formal
governance, informal governance, architectural knowledge, guardian) between
multi-sourcing arrangements that were in our sample and multi-sourcing arrange-
ments not included in the sample (most frequently due to the respondents’ lack of
willingness to provide descriptions of the outsourced tasks). Comparisons revealed
no significant differences with the exception of formal governance, which was
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1267
somewhat higher in the arrangements included in the final sample than in those
excluded (3.99 vs. 3.82; t test; n = 369; p < .05). With only one of eight
comparisons yielding a significant difference, we inferred that nonresponse bias
was unlikely to be a serious threat to the validity of our analysis.
Table 4 shows descriptive statistics, separated by guardian versus direct subsam-
ples. The only significant differences referred to business process outsourcing,
which was more frequent in the guardian sample, and informal governance,
which was stronger in the guardian sample. Table 5 presents bivariate correlations.
The results of our four-step regression are presented in Table 6. The first column
(Model 1) presents results related to our control variables. Modularity (ß = .44; p < .001) had significant positive associations with joint performance while the other control variables shown in Table 6 were insignificant. The second column (Model 2a) shows the main effects of our four predictors (i.e.,
the sources of IP capacity), allowing us to test H1 and H2. H1 predicts positive main
effects for formal and informal governance on joint performance. We found
a significant positive effect for formal governance (ß = .31; p < .001), but not for
informal governance (ß = .00; p > .05). Hence, H1 is partially supported. H2 predicts
a positive main effect of architectural knowledge on joint performance. We found
a significant positive effect (ß = .29; p < .001), supporting H2. Although we did not
hypothesize a main effect of the presence of the guardian vendor on joint perfor-
mance, Model 2a showed a significant negative main effect (ß = −.31; p < .05).
Before adding the hypothesized interaction effects, we controlled for possible
interactions of our hypothesized sources of IP capacity with modularity (in order to
reflect IP requirements), and for the interaction of concentration with the choice of
the guardian model. As the third column (Model 3a) shows, none of interactions of
sources of IP capacity with modularity were significant. Conversely, we found
a significant negative interaction effect of concentration with the choice of the
guardian model (ß = −.42; p < .01).
The fourth column (Model 4a) presents the results of our full model, which
includes the interaction effects hypothesized in H3 to H5. H3 predicts positive
interaction effects of formal/informal inter-vendor governance and the client’s
architectural knowledge. As can be seen, only the interaction between informal
inter-vendor governance and the client’s architectural knowledge was significant
and positive (ß = .13; p < .05), thus partially supporting H3. Following the
guardian-as-a-mediator perspective, H4 predicts negative interaction effects
between the choice of the guardian model and the client’s formal/informal inter-
vendor governance (H4a), and with the client’s architectural knowledge (H4b). As
Model 4a shows, we found positive rather than negative interaction effects of the
guardian model with formal (ß = .42; p < .05) and informal (ß = .38; p < .01) inter-
vendor governance. H4a is thus rejected. In line with H4b, the interaction effects
1268 OSHRI ET AL.
between the guardian model and architectural knowledge were significant (ß =
−.52; p < .01). While the results do not fully align with the predictions derived from
the guardian-as-a-mediator perspective, they do align with the predictions derived
from the guardian-as-an-architect perspective. In line with H5a, we found signifi-
cant positive interaction effects of the guardian model with formal (ß = .42; p < .05)
and informal (ß = .38; p < .01) inter-vendor governance. Moreover and in line with
H5b, we found significant negative interaction effects of the guardian model with
architectural knowledge (ß = −.52; p < .01).
Overall, our full model (Model 4a) showed the strongest explanatory power
(adjusted R2 = .51) of all the tested models (see the bottom of Table 6). The
hypothesized interaction effects between sources of IP capacity (from Model 3 to
Model 4) added statistically significant amounts of explained variance (ΔR2 = .05;
ΔF = 3.63; p < .01), supporting the relevancy of interaction effects expressed in H3
and H5.
To examine threats of endogeneity, alternative model specifications based on
Heckman correction were undertaken and are reported in Table 7. Hypothesis
testing based on these alternative models yielded coefficient signs and significance
levels that were identical to those resulting from models 2a and 4a. Moreover, the
Inverse Mills Ratio was insignificant (p > .05) in all models. This suggests that the
support found for our hypotheses is unlikely to be a statistical artefact of the
potentially endogenous choice of the guardian model.12
The purpose of this study was to examine joint performance in multi-sourcing
arrangements in light of the interdependencies between multiple vendors. Indeed,
Table 4. Descriptive Statistics and Sample Comparison
Direct sample:
Mean (standard
deviation)
Guardian sample:
Mean (standard
deviation)
Difference
statistically
significant
Joint performance 4.06 (.65) 4.02 (.75) No
Concentration 52.17 (31.49) 55.63 (29.28) No
Modularity 3.72 (.83) 3.96 (.88) No
Age of arrangement 3.76 (2.44) 3.44 (2.29) No
Client size 7.88 (1.80) 8.03 (1.56) No
Business process
outsourcing
.56 (50) .82 (.38) Yes (p < .05)
Application development .20 (.40) .11 (.31) No
Formal governance 3.93 (.74) 4.12 (.56) No
Informal governance 2.73 (1.06) 3.12 (.96) Yes (p < . 05)
Architectural knowledge 4.03 (.76) 4.19 (.66) No
Sub-sample size (n) 132 57 -
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1269
T
ab
le
5
.
B
iv
ar
ia
te
C
o
rr
el
at
io
n
s
(*
p
<
.0
5
)
Jo
in
t
p
er
f.
C
o
n
.
M
o
.
A
g
e
C
l.
si
ze
B
P
O
A
p
p
.
D
ev
.
G
rd
.
F
o
rm
.
g
o
v.
In
f.
g
o
v.
A
rc
h
.
k
n
o
w
l.
Jo
in
t
p
e
rf
o
rm
a
n
ce
1
C
o
n
ce
n
tr
a
tio
n
.0
3
1
M
o
d
u
la
ri
ty
.4
4
*
.1
3
*
1
A
g
e
o
f
a
rr
a
n
g
e
m
e
n
t
.0
7
−
.0
9
.0
7
1
C
lie
n
t
si
ze
−
.0
5
.0
8
.0
5
.2
0
*
1
B
P
O
.1
2
.1
0
.1
5
*
.1
4
*
−
.1
0
1
A
p
p
l.
D
e
ve
lo
p
m
e
n
t
−
.1
2
−
.1
0
−
.1
2
−
.0
3
.0
5
−
.3
8
*
1
G
u
a
rd
ia
n
−
.0
2
.0
5
.1
3
−
.0
6
.0
4
.2
5
*
−
.1
2
1
F
o
rm
a
l
g
o
ve
rn
a
n
ce
.5
8
*
.0
3
.4
0
*
.0
1
−
.0
7
.1
3
−
.1
4
.1
2
1
In
fo
rm
a
l
g
o
ve
rn
a
n
ce
.1
2
.0
9
.0
9
.1
1
.0
4
.1
2
.0
0
.1
7
*
.1
5
*
1
A
rc
h
ite
ct
u
ra
l
kn
o
w
le
d
g
e
.5
2
*
−
.1
0
.2
9
*
.0
4
.0
7
.0
3
−
.0
1
.1
0
.5
7
*
.1
6
*
1
1270 OSHRI ET AL.
T
ab
le
6
.
R
eg
re
ss
io
n
R
es
u
lt
s
M
o
d
el
1
M
o
d
el
2
a
M
o
d
el
3
a
M
o
d
el
4
a
(C
o
n
st
a
n
t)
−
.2
9
(.
2
3
)
−
.1
3
(.
1
9
)
−
.0
3
(.
1
9
)
−
.1
3
(.
1
9
)
C
o
n
ce
n
tr
a
tio
n
−
.0
3
(.
0
7
)
.0
2
(.
0
6
)
.1
5
*
(.
0
7
)
.1
7
*
(.
0
7
)
M
o
d
u
la
ri
ty
.4
3
**
*
(.
0
7
)
.2
3
**
*
(.
0
6
)
.2
5
**
(.
0
8
)
.2
4
**
(.
0
8
)
A
g
e
o
f
a
rr
a
n
g
e
m
e
n
t
−
.0
2
(.
0
7
)
−
.0
1
(.
0
6
)
.0
1
(.
0
6
)
.0
1
(.
0
6
)
C
lie
n
t
S
iz
e
−
.0
6
(.
0
7
)
−
.0
4
(.
0
6
)
−
.0
6
(.
0
6
)
−
.0
4
(.
0
6
)
B
u
si
n
e
ss
P
ro
ce
ss
O
u
ts
o
u
rc
in
g
.0
6
(.
1
6
)
.1
0
(.
1
4
)
.0
8
(.
1
4
)
.0
6
(.
1
3
)
A
p
p
lic
a
tio
n
D
e
ve
lo
p
m
e
n
t
−
.1
1
(.
1
9
)
−
.0
8
(.
1
6
)
−
.0
8
(.
1
6
)
−
.0
7
(.
1
5
)
In
ve
rs
e
M
ill
s
R
a
tio
−
−
−
−
G
u
a
rd
ia
n
−
−
.3
1
*
(.
1
3
)
−
.3
2
*
(.
1
3
)
−
.3
7
**
(.
1
3
)
F
o
rm
a
l
g
o
ve
rn
a
n
ce
−
.3
1
**
*
(.
0
7
)
.2
6
**
*
(.
0
7
)
.1
8
*
(.
0
8
)
In
fo
rm
a
l
g
o
ve
rn
a
n
ce
−
.0
0
(.
0
6
)
.0
2
(.
0
6
)
−
.0
7
(.
0
6
)
A
rc
h
ite
ct
u
ra
l
kn
o
w
le
d
g
e
−
.2
9
**
*
(.
0
7
)
.3
4
**
*
(.
0
7
)
.4
6
**
*
(.
0
8
)
M
o
d
u
la
ri
ty
�
G
u
a
rd
ia
n
−
−
.0
0
(.
1
3
)
−
.0
4
(.
1
4
)
M
o
d
u
la
ri
ty
�
F
o
rm
a
l
g
o
ve
rn
a
n
ce
−
−
−
.1
1
(.
0
7
)
−
.1
2
(.
0
7
)
M
o
d
u
la
ri
ty
�
In
fo
rm
a
l
g
o
ve
rn
a
n
ce
−
−
.0
6
(.
0
5
)
−
.0
2
(.
0
6
)
M
o
d
u
la
ri
ty
�
C
lie
n
t’s
a
rc
h
ite
ct
u
ra
l
kn
o
w
le
d
g
e
−
−
.0
2
(.
0
7
)
.0
3
(.
0
7
)
C
o
n
ce
n
tr
a
tio
n
�
G
u
a
rd
ia
n
−
−
−
.4
2
**
(.
1
3
)
−
.4
6
**
*
(.
1
3
)
F
o
rm
a
l
g
o
ve
rn
a
n
ce
�
C
lie
n
t’s
a
rc
h
ite
ct
u
ra
l
kn
o
w
le
d
g
e
−
−
−
−
.0
1
(.
0
6
)
In
fo
rm
a
l
g
o
ve
rn
a
n
ce
�
C
lie
n
t’s
a
rc
h
ite
ct
u
ra
l
kn
o
w
le
d
g
e
−
−
−
.1
3
*
(.
0
6
)
G
u
a
rd
ia
n
�
F
o
rm
a
l
g
o
ve
rn
a
n
ce
−
−
−
.4
2
*
(.
1
9
)
G
u
a
rd
ia
n
�
In
fo
rm
a
l
g
o
ve
rn
a
n
ce
−
−
−
.3
8
**
(.
1
4
)
G
u
a
rd
ia
n
�
C
lie
n
t’s
a
rc
h
ite
ct
u
ra
l
kn
o
w
le
d
g
e
−
−
−
−
.5
2
**
(.
1
7
)
A
d
ju
st
e
d
R
2
.2
1
.4
4
.4
7
.5
1
R
2
.2
7
.4
9
.0
5
.5
9
Δ
R
2
.2
7
.2
2
.5
4
.0
5
F
C
h
a
n
g
e
(d
.f
.)
4
.3
4
**
*
(1
5
,
1
7
3
)
1
8
.3
3
**
*
(4
,1
6
9
)
3
.2
0
**
*
(5
,
1
6
4
)
3
.6
3
**
(5
,
1
5
9
)
N
o
te
s:
*
p
<
.0
5
.
*
*
p
<
.0
1
.
*
*
*
p
<
.0
0
1
,
re
su
lt
s
fo
r
d
u
m
m
y
v
ar
ia
b
le
s
re
la
te
d
to
co
u
n
tr
y
an
d
in
d
u
st
ry
o
m
it
te
d
.
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1271
T
ab
le
7
.
R
es
u
lt
s
o
f
A
lt
er
n
at
iv
e
M
o
d
el
s
b
as
ed
o
n
H
ec
k
m
an
C
o
rr
ec
ti
o
n
M
o
d
el
2
b
M
o
d
el
3
b
M
o
d
el
4
b
(C
o
n
st
a
n
t)
−
.2
1
(.
2
3
)
−
.1
2
(.
2
3
)
−
.1
5
(.
2
2
)
C
o
n
ce
n
tr
a
tio
n
.0
3
(.
0
6
)
.1
5
*
(.
0
7
)
.1
7
(.
0
7
)
M
o
d
u
la
ri
ty
.2
2
**
(.
0
6
)
.2
4
**
(.
0
8
)
.2
4
**
(.
0
8
)
A
g
e
o
f
a
rr
a
n
g
e
m
e
n
t
.0
2
(.
0
7
)
.0
4
(.
0
6
)
.0
2
(.
0
6
)
C
lie
n
t
S
iz
e
−
.0
6
(.
0
6
)
−
.0
8
(.
0
6
)
−
.0
5
(.
0
6
)
A
p
p
lic
a
tio
n
D
e
ve
lo
p
m
e
n
t
−
.0
7
(.
1
6
)
−
.0
6
(.
1
6
)
−
.0
7
(.
1
5
)
In
ve
rs
e
M
ill
s
R
a
tio
−
.3
1
(.
3
2
)
−
.2
9
(.
3
1
)
−
.1
3
(.
3
1
)
G
u
a
rd
ia
n
−
.2
0
(.
5
2
)
−
.1
7
(.
1
7
)
−
.1
4
(.
5
1
)
F
o
rm
a
l
g
o
ve
rn
a
n
ce
.3
0
**
*
(.
0
7
)
.2
5
**
*
(.
0
7
)
.1
8
*
(.
0
8
)
In
fo
rm
a
l
g
o
ve
rn
a
n
ce
−
.0
3
(.
0
7
)
.0
0
(.
0
7
)
−
.0
8
(.
0
7
)
A
rc
h
ite
ct
u
ra
l
kn
o
w
le
d
g
e
.2
7
**
*
(.
0
7
)
.3
3
**
*
(.
0
7
)
.4
6
**
*
(.
0
8
)
M
o
d
u
la
ri
ty
�
G
u
a
rd
ia
n
−
−
.0
0
(.
1
3
)
−
.0
5
(.
1
4
)
M
o
d
u
la
ri
ty
�
F
o
rm
a
l
g
o
ve
rn
a
n
ce
−
−
.1
2
(.
0
7
)
−
.1
2
(.
0
7
)
M
o
d
u
la
ri
ty
�
In
fo
rm
a
l
g
o
ve
rn
a
n
ce
−
.0
6
(.
0
5
)
−
.0
2
(.
0
6
)
M
o
d
u
la
ri
ty
�
C
lie
n
t’s
a
rc
h
ite
ct
u
ra
l
kn
o
w
le
d
g
e
−
.0
2
(.
0
7
)
.0
3
(.
0
7
)
C
o
n
ce
n
tr
a
tio
n
�
G
u
a
rd
ia
n
−
−
.4
2
**
(.
1
3
)
−
.4
6
**
*
(.
1
3
)
F
o
rm
a
l
g
o
ve
rn
a
n
ce
�
C
lie
n
t’s
a
rc
h
ite
ct
u
ra
l
kn
o
w
le
d
g
e
−
−
−
.0
1
(.
0
6
)
In
fo
rm
a
l
g
o
ve
rn
a
n
ce
�
C
lie
n
t’s
a
rc
h
ite
ct
u
ra
l
kn
o
w
le
d
g
e
−
−
.1
3
*
(.
0
7
)
G
u
a
rd
ia
n
�
F
o
rm
a
l
g
o
ve
rn
a
n
ce
−
−
.4
2
*
(.
1
9
)
G
u
a
rd
ia
n
�
In
fo
rm
a
l
g
o
ve
rn
a
n
ce
−
−
.3
7
**
(.
1
4
)
G
u
a
rd
ia
n
�
C
lie
n
t’s
a
rc
h
ite
ct
u
ra
l
kn
o
w
le
d
g
e
−
−
−
.5
2
**
(.
1
7
)
A
d
ju
st
e
d
R
2
.4
4
.4
7
.5
1
R
2
.4
9
.5
4
.5
9
N
o
te
s:
*
p
<
.0
5
.
*
*
p
<
.0
1
.
*
*
*
p
<
.0
0
1
,
re
su
lt
s
fo
r
d
u
m
m
y
v
ar
ia
b
le
s
re
la
te
d
to
co
u
n
tr
y
an
d
in
d
u
st
ry
o
m
it
te
d
.
1272 OSHRI ET AL.
multi-sourcing has become a dominant sourcing model attracting growing attention
in the IS community [6, 31, 49]. While multi-sourcing offers client firms numerous
advantages through a competitive and yet cooperative regime, it also poses new
challenges, mainly in the form of interdependencies that require the client firm to
increase its efforts to achieve coordination and cooperation. Building on key IPV
concepts, we framed these efforts as greater IP requirements. Given these IP
requirements, a critical challenge in multi-sourcing arrangements is to generate
sufficient IP capacity. In this regard, we proposed two possible sources of IP
capacity. The first, the direct model (see Figure 1a), relies on internal sources of
IP capacity only, seeking to leverage the client’s formal and informal governance
and architectural knowledge. The second, the guardian model (see Figure 1b), relies
both on internal sources and on the use of a guardian vendor as a means of
providing additional IP capacity.
Direct Model
Our results regarding the direct model (i.e., the client alone managing the multi-
sourcing arrangement) suggest that both formal inter-vendor governance and archi-
tectural knowledge lead to higher joint performance. In particular, each of these two
sources of IP capacity (as captured in H1 and H2) individually equip the client firm
with the IP capacity needed to manage interdependencies in the multi-sourcing
arrangement. The results for the direct model highlight the importance of formal inter-
vendor governance based on written procedures and the contractual agreement struc-
ture (e.g., objective and quantifiable measures) as a means of coping with coordination
and integration efforts between vendors, manifested here as an IP challenge.
Interestingly, formal inter-vendor governance seems to be an effective strategy irre-
spective of the client’s level of architectural knowledge (see the insignificant interac-
tion effect of formal inter-vendor governance and client’s architectural knowledge).
On the other hand, informal inter-vendor governance seems to be effective only when
the client has strong architectural knowledge (see the insignificant main effect of informal
inter-vendor governance and the significant positive interaction effect of informal inter-
vendor governance and client’s architectural knowledge). The interaction plot depicted in
Figure 3 below further illustrates the relationships. When a client possesses strong
architectural knowledge (i.e., one standard deviation above the mean), greater informal
inter-vendor governance is associated with higher joint performance. Conversely, when
a client possesses weak architectural knowledge (i.e., one standard deviation below the
mean), greater informal inter-vendor governance is associated with lower joint perfor-
mance. The lack of a positive main effect of informal inter-vendor governance is rather
surprising, given that the IS outsourcing literature has persistently found a positive effect
of informal governance (often viewed as relational governance) on outsourcing perfor-
mance in dyadic settings (e.g., Poppo and Zenger [36]). One possible explanation for the
surprising result in our study is that informal inter-vendor governance in triadic relation-
shipsisdifferentfrominformalgovernance in dyadic relationships.Havingmore thantwo
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1273
parties involved may erode the sense of being “informal” and make all parties involved
feel part of a “formal” relationship. The sense of competition [49] between the vendors is
also likely to contribute to such a “formal” attitude. The relational benefits seen in dyadic
settings, therefore, may be less pronounced in multi-sourcing settings.
Guardian Model
We found joint performance in arrangements using a guardian model to be very similar to
arrangements using a direct model (4.06 vs. 4.02 in a 5-point scale, see the first row in
Table 4). Nonetheless, we also found two significant interaction effects, suggesting that
the effectiveness of the guardian model is contingent on the two internal sources of IP
capacity.
Our results support the perspective that the guardian can best be utilized as an architect
rather than as a mediator. Indeed, we found a complementary effect between the guardian
vendor and the client’s governance and a substitutive effect between the guardian vendor
and the client’s architectural knowledge, supporting our hypotheses derived from the
guardian-as-an-architect perspective.
The interaction plot depicted in Figure 4 illustrates the complementary effect of the
guardian model and inter-vendorgovernance. A guardian model is likely to diminish joint
performance when the client firm exercises weak formal and informal inter-vendor
governance (see Figure 4). Yet, the negative effect of the guardian model is reversed to
positive when the client firm has strong formal and informal inter-vendor governance
mechanisms in place. Therefore, in multi-sourcing settings, the internal IP capacity of
governance (formal and informal) is required by the client in order to gain the benefits of
the external IP capacity of architectural knowledge brought by the guardian vendor. This
demonstrates the complementary effect of these two sources of IP capacity.
Our results also suggest a substitutive effect between the guardian’s and the client’s
architectural knowledge, as illustrated by the interaction plot depicted in Figure 5. The
-1
–
0,5
0
0,5
1
Weak informal
governance (- 1 SD)
Strong informal
governance (+ 1 SD)
J
o
in
t
p
e
rf
o
rm
a
n
c
e Client’s strong
architectural knowledge
(+ 1 SD)
Client’s weak
architectural knowledge
(- 1 SD)
Figure 3. Informal Governance Affecting Joint Performance Under Strong versus Weak
Client’ss Architectural Knowledge
1274 OSHRI ET AL.
guardian model improves joint performance when clients have weak architectural knowl-
edge, while it worsens performance when clients have strong architectural knowledge.
Indeed, these results suggest a substitutive effect between the IP capacity brought forward
by the guardian vendor and the client’s architectural knowledge. In this regard, the
guardian model compensates for the client’s weak in-house architectural knowledge
and, therefore, a client with weak architectural knowledge may benefit from the guar-
dian’s ability to integrate interdependent sub-tasks in multi-sourcing arrangements.
Conversely, clients who possess strong architectural knowledge may benefit to a much
lesser extent from the guardian’s integration ability.
We found additional support for the perspective of “guardian-as-an-architect.” As
depicted in Figure 6, the joint performance of the multi-sourcing arrangement will be
higher should a client with weak architectural knowledge choose a guardian model and
exercise strong formal and informal governance. Conversely, a client with strong
–
1,5
-1
-0,5
0
0,5
1
Direct model
Guardian model
J
o
in
t
p
e
rf
o
rm
a
n
c
e Strong formal and
informal governance
(+ 1 SD)
Weak formal and
informal governance
(- 1 SD)
Figure 4. Guardian Model (versus Direct Model) Affecting Joint Performance Under
Strong Versus Weak Governance
-1
-0,5
0
0,5
1
Direct model Guardian model
J
o
in
t
p
e
rf
o
rm
a
n
c
e Client’s strong
architectural
knowledge (+ 1 SD)
Client’s weak
architectural
knowledge (- 1 SD)
Figure 5. Guardian Model (versus Direct Model) Affecting Joint Performance Under
Strong versus Weak Client’s Architectural Knowledge
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1275
architectural knowledge will benefit from using the direct model, as Figure 7 depicts.
Interestingly, both Figure 6 and Figure 7 show that joint performance is at its lowest
when the client chooses a guardian model and exercises weak inter-vendor governance.
This is precisely the configuration prescribed by the guardian-as-a-mediator perspec-
tive. Irrespective of whether the client’s architectural knowledge is high or low,
employing a guardian-as-a-mediator model is likely to result in low levels of joint
performance. These findings bear important implications for theory and practice, on
which we elaborate next.
-1,5
-1
-0,5
0
0,5
1
1,5
Weak formal and informal
governance (- 1 SD)
Strong formal and informal
governance (+ 1 SD)
J
o
in
t
p
e
rf
o
rm
a
n
c
e
Client’s weak architectural knowledge
Direct model
Guardian model
Figure 6. Governance Affecting Joint Performance Under Direct versus Guardian Model
and Under Client’s Weak Architectural Knowledge
-1,5
-1
-0,5
0
0,5
1
1,5
Weak formal and informal
governance (- 1 SD)
Strong formal and informal
governance (+ 1 SD)
J
o
in
t
p
e
rf
o
rm
a
n
c
e
Client’s strong architectural knowledge
Direct model
Guardian model
Figure 7. Governance Affecting Joint Performance Under Direct versus Guardian Model
and Under Client’s Strong Architectural Knowledge
1276 OSHRI ET AL.
Theoretical Contributions
This paper offers two main contributions to the IS outsourcing literature. First, to our
best knowledge, this is the first study to model and test determinants of joint perfor-
mance in IS multi-sourcing arrangements. While the IS outsourcing literature has, by
and large, examined dyadic relationships as a basis for understanding the determinants
of outsourcing success [14], our study assumed interdependencies between multiple
vendors, thus requiring an examination of triadic relationships at the minimum. As
interdependencies may affect the likelihood of multi-sourcing success, formal and
informal inter-vendor governance and architectural knowledge were examined as two
key antecedents. Our study shows that while formal inter-vendor governance and the
client’s architectural knowledge are likely to improve multi-sourcing success, informal
governance, often referred to as relational governance in the IS literature and consid-
ered key in achieving dyadic IS outsourcing success, shows no direct effect on joint
performance. Our results show a positive effect of informal governance on joint
performance only in conjunction with high levels of client’s architectural knowledge,
or with the choice of a guardian model. These results suggest that multi-sourcing does
not simply mimic dyadic outsourcing at a larger scale. Its inherent independencies
require unique governance mechanisms and associated abilities (i.e., architectural
knowledge) directed toward the interface between vendors.
The second contribution of this study is in offering insights into the role that
a guardian vendor plays in a multi-sourcing arrangement. Bapna et al. [4] noted that,
although the choice for or against a guardian model is one of the key design choices in
multi-sourcing arrangements, “[t]here is little in the academic literature on the guardian
vendor model” (p. 794). They called for research to examine “[w]hat aspects of the
engagement should be handled by the guardian vendor and the client” (p. 794). Wiener
and Saunders [49] argue that the guardian model can be regarded as a “mediated
model”, wherein the guardian vendor acts as a “single point of contact” (p. 213),
mediating the interaction between the client and the remaining vendors. This would
imply that the only actor responsible for governing interdependencies between vendors
is the guardian vendor. Our results, however, show that it can be perilous for the client
to withdraw from governance efforts and mandate these to the guardian vendor. In fact,
the least successful multi-sourcing arrangements in our sample were those where the
client appointed a guardian vendor and then exercised weak joint formal and informal
governance (see Figure 6 and Figure 7). In other words, the clients who practiced the
guardian-as-a-mediator model were the least successful. We therefore theorized an
alternative role in which the guardian acts as an architect.
Our results do indeed suggest that a guardian vendor may be better understood as
an architect than as a mediator. Much like the architect of a building contributes
knowledge of how the elements of a building fit with each other; the guardian-as-an
-architect contributes knowledge as to how the sub-tasks of a multi-sourcing
arrangement can be integrated. Two findings support the guardian-as-an-architect
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1277
view. First, clients who lack architectural knowledge are particularly likely to
benefit from the inclusion of a guardian vendor, suggesting that the guardian vendor
compensates for the client’s knowledge gaps. Second, just like the client’s archi-
tectural knowledge enables more effective informal governance, so does utilizing
the guardian vendor as an architect too. Informal inter-vendor governance involves
complex ad hoc communication and decisions by the client, requiring considerable
architectural knowledge in order to be exercised effectively. This knowledge may
come either from the client, or from a guardian-as-an-architect to support the client
in informal governance efforts. Thus, a guardian vendor complements the client’s
formal and informal inter-vendor governance while substituting the client’s archi-
tectural knowledge. As such, the guardian model does not relieve clients from
governance (as assumed in the guardian-as-mediator model), but it does help them
compensate for knowledge gaps.
Another contribution of this study revolves around the body of research that
explains the choice and effectiveness of governance mechanisms through an IPV
lens. Indeed, the IPV-based literature stream on governance mostly argues that the
choice of governance mechanisms determines the IP capacity of an organization, and
that such IP capacity should fit IP requirements [2, 27, 32]. Although another
literature stream implicitly argues that architectural knowledge is an important source
of IP capacity in inter-organizational relationships [7, 24, 38, 43], these two literature
streams have mostly developed in isolation. As a consequence, interactions between
governance and knowledge have rarely been considered in IPV research. Conversely,
a key argument of our study is that governance mechanisms, such as goal setting,
planning, and direct interaction, enable effective IP to the extent that these mechan-
isms are enacted or assisted by a knowledgeable party.
Practical Implications
Our study offers specific recommendations for practice. Clients in multi-sourcing
arrangements should consider their architectural knowledge when deciding for or against
the guardian model. Clients with strong architectural knowledge (i.e., clients who under-
stand well how the various sub-tasks outsourced to different vendors relate to each other)
are advised to choose a direct model, whereas clients with weak architectural knowledge
are better off with a guardian model. Although clients may believe that having a guardian
model means they can economize on or relinquish governance efforts, this is not the case.
Instead, clients are well advised to engage in extensive formal and informal governance
efforts that involve all vendors. Specifically, clients should define and monitor the joint
outcomes to be achieved and the joint procedures to be followed, and they should also put
emphasis on informally interacting with all involved vendors at various levels.
Importantly, extensive governance efforts are essential, both in a direct model, where
the clients can leverage their own knowledge during informal governance in particular,
and in a guardian model, where the guardian vendor should bring in additional knowl-
edge to enable effective governance by the client.
1278 OSHRI ET AL.
Limitations and Future Research
There are several limitations to this study that may encourage future research. First,
while this is one of the first studies to examine the effect of the guardian on a multi-
sourcing arrangement, our study sheds little light on what exactly the guardian vendor
does and what information capacities the guardian vendor brings to the multi-sourcing
arrangement. Consequently, following on our guardian-as-an-architect perspective, our
study provides a number of fruitful directions for future research. Future studies could
take a practice-view and explicitly examine and document the IP requirements that
multi-sourcing settings face. Consequently, future research could study the activities
performed by the guardian vendor vis-à-vis the IP requirements, as well as in steering
the relationships with the client and with other vendors in multi-sourcing arrangements.
Building on this, future studies could also explore the relationship between the nature
of the task (simple or complex) and the implications for the architectural knowledge
and governance efforts that the guardian vendor contributes to multi-sourcing arrange-
ments. Our study calls for a more in-depth examination of the practices and the
knowledge contributions of the guardian vendor.
Second, while our measures of formal and informal inter-vendor governance were
closely linked to the IPV, they did not include some mechanisms of contractual
governance, such as contract duration and contract type. Future research could
integrate these mechanisms into IPV conceptualizations. Moreover, although we
focused on the client’s inter-vendor governance (i.e., governance involving all
vendors at the same time), we did not contrast inter-vendor governance efforts
with governance efforts that involve only one vendor at a time (such as an SLA
applicable for a single vendor only).
We also see an opportunity for further research around the role of the client in
a guardian vendor model. For example, drawing on our finding that informal
governance — with the involvement of the client — complements the role of the
guardian in achieving high levels of joint performance, a future study could zoom
into such informal meetings and explore the activities performed by the client and
the knowledge needed. Such a study could, in fact, explore the evolution of triadic
relationships between client, guardian and other vendors and how their actions and
knowledge evolve over time [8].
Ultimately, such zooming into the client and guardian vendor roles would also
further address our call for a more in-depth understanding of the interactions
between IP capacities, by studying interactions not only between capacities, for
example, informal and formal governance, but also between the underlying knowl-
edge and the practices needed to bring such capacities to fruition.
The main objective in this paper was to examine joint-vendor performance in multi-
sourcing. In particular, we took interest in understanding joint-vendor performance in two
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1279
common multi-sourcing settings, namely, the direct model and the guardian model. Using
the logic of the Information Processing View, we theoretically developed the idea that
information processing capacity in multi-sourcing can be internal (i.e., the client’s inter-
vendor governance and the client’s architectural knowledge) and external (i.e., the
guardian vendor). To discover how these three sources of IP capacities affect joint-
vendor performance as well as interact with each other, we tested our model using an
international data set of 189 IT multi-sourcing arrangements. We found that in the direct
model, the client’s formal inter-vendor governance and the client’s architectural knowl-
edge positively affect joint performance. We also found that a guardian vendor comple-
ments the client’s formal and informal inter-vendor governance while substituting the
client’s architectural knowledge. These results suggest that the guardian’s role is best
understood as an architect (i.e., beneficial in terms of architectural knowledge) rather than
as a mediator (i.e., beneficial in terms of inter-vendor governance). Put simply, client
firms should consider using a guardian vendor to compensate for weak architectural
knowledge while still maintaining strong formal and informal governance of all vendors.
NOTES
1. e-Dialog was part of GSI Commerce (which was acquired by eBay and renamed eBay
Enterprise in 2013), and sold to Zeta Interactive in 2015 (http://zetaglobal.com/clients).
2. It is important to note that the IS outsourcing literature has so far conceptually
discussed the role of the guardian and suggested that it corresponds with the notion of
a mediator. More specifically, two key studies have explored the guardian role: Bapna et al.
[4] is a research commentary and largely conceptual; second, while Wiener and Saunders
[49] report a case study that follows a direct rather than a guardian model, with some
suggestions made regarding the guardian.
3. Multi-vendor settings have been broadly studied in the supply chain literature [e.g., 1]
in the context of production, logistics and procurement of physical goods (e.g., automotive
and manufacturing industries), where clients use multiple suppliers to procure similar/
identical physical parts. In the case of IT-enabled business processes and services, each
vendor is delivering a unique yet interdependent service (as illustrated in the British Airways
example in the Introduction). Thus the nature of the interdependencies and joint performance
in IT multi-sourcing that are the focus of this paper is different to the interdependencies in
triadic relationships between suppliers of physical parts discussed in the literature [e.g., 10].
4. http://www.computerweekly.com/blog/Investigating-Outsourcing/IT-sourcing-models-
are-shifting-A-Deloitte-perspective.
5. As put by Tiwana [46], “Two things are complements if more of one increases the
benefits of using the other. They are substitutes if more of one diminishes the benefits of
using the other” (p.88).
6. This is different from situations where the prime contractor is used, because in such
a scenario the prime contractor is the only vendor contracted by the client and thus
responsible for delivering the service. In the academic and professional literature, the
prime contractor model “consists of a network with several vendors that operate under the
control of the head contractor. The head contractor is accountable for the delivery of the
service and liable for this under the terms of the contract” [34, p.134]. For example, Koo
et al. [29] refer to the prime contractor outsourcing configuration as the “single-vendor-
dominant model” where “a client directly contracts with one dominant vendor and indirectly
contracts with other vendors through the dominant vendor” (p. 3). Such contracting should
not be confused with a true multi-sourcing scenario, where each vendor is contracted directly
by the client firm, as depicted in Figure 1a.
1280 OSHRI ET AL.
http://zetaglobal.com/clients
http://www.computerweekly.com/blog/Investigating-Outsourcing/IT-sourcing-models-are-shifting-A-Deloitte-perspective
http://www.computerweekly.com/blog/Investigating-Outsourcing/IT-sourcing-models-are-shifting-A-Deloitte-perspective
7. E.g. https://www.slaughterandmay.com/media/2535633/multi-sourcing-a-different-way
-of-contracting .
8. https://www.information-age.com/how-to-make-multi-sourcing-work-123457348/
9. The market research firm used these criteria to select key informants from a panel of
individuals that had agreed to participate in surveys.
10. The survey included three items measuring individual performance (composite relia-
bility .87), which were not used for this study.
11. Only one respondent selected the “Other” category. The comment suggested than
a third vendor (not vendor A or B) was responsible for managing the other vendors. We
therefore coded this response as a guardian model.
12. We performed two further analyses to examine threats of endogeneity. First, to
examine whether clients deliberately chose highly capable vendors as their guardian vendors,
we compared the clients’ assessment of the vendors’ individual (rather than joint) perfor-
mance (measured through three items not used in this study, composite reliability .87).
Individual performance was very similar for guardian vendors and for non-guardian vendors,
with average standardized scores of -.02 for guardian vendors and .00 for non-guardian
vendors (difference not statistically significant). This suggests that clients did not select
highly capable vendors as their guardian vendors. Second, we estimated a switching regres-
sion model, using the movestay command in Stata [11]. The switching regression model
produced results that were highly consistent with the results from OLS regression.
Specifically, the differences between coefficients in sub-sample with guardian model and
the coefficients in sub-sample with direct model were highly similar to interaction coeffi-
cients obtained from OLS regression (architectural knowledge: difference between coeffi-
cients in switching regression of -.55 compared to an OLS interaction effect of -.52; formal
governance: difference between coefficients in switching regression of .37 compared to OLS
interaction effect of .42; informal governance: difference between coefficients in switching
regression of .35 versus OLS interaction effect .38).
REFERENCES
1. Agrawal, N.; Smith, S.A.; and Tsay, A.A. Multi-vendor sourcing in a retail supply
chain. Production and Operations Management, 11, 2 (2002), 157–182.
2. Andres, H.P.; and Zmud, R.W. A contingency approach to software project
coordination. Journal of Management Information Systems, 18, 3 (2001/2002), 41–70.
3. Baldwin, C.Y.; and Clark, K.B. The architecture of participation: Does code architec-
ture mitigate free riding in the open source development model? Management Science, 52, 7
(2006), 1116–1127.
4. Bapna, R.; Barua, A.; Mani, D.; and Mehra, A. Research commentary: cooperation,
coordination, and governance in multisourcing: An agenda for analytical and empirical
research. Information Systems Research, 21, 4 (2010), 785–795.
5. Bensaou, M.; and Venkatraman, N. Configurations of interorganizational relationships:
A comparison between U.S. and Japanese automakers Management Science, 41, 9 (1995),
1471–1492.
6. Bhattacharya, S.; Gupta, A.; and Hasija, S. Single-sourcing versus multisourcing: The
roles of output verifiability on task modularity. MIS Quarterly, 42, 4 (2018), 1171–1186.
7. Brusoni, S.; Prencipe, A.; and Pavitt, K. Knowledge specialization, organizational
cloupling, and the boundaries of the firm: Why do firms know more than they make?
Administrative Science Quarterly, 46, 4 (2001), 597–621.
8. Cao, L.; Mohan, K.; Ramesh, B.; and Sarkar, S. Evolution of governance: Achieving
ambidexterity in IT outsourcing. Journal of Management Information Systems, 30, 3 (2014),
115–140.
9. Chase, W.G.; and Simon, H.A. Perception in chess. Cognitive Psychology, 4, 1 (1973),
55–81.
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1281
https://www.slaughterandmay.com/media/2535633/multi-sourcing-a-different-way-of-contracting
https://www.slaughterandmay.com/media/2535633/multi-sourcing-a-different-way-of-contracting
https://www.information-age.com/how-to-make-multi-sourcing-work-123457348/
10. Choi, T.Y.; and Wu, Z. Taking the leap from dyads to triads: Buyer–supplier relation-
ships in supply networks. Journal of Purchasing and Supply Management, 15, 4 (2009),
263–266.
11. Clougherty, J.A.; Duso, T.; and Muck, J. Correcting for self-selection based endogene-
ity in management research: Review, recommendations and simulations. Organizational
Research Methods, 19, 2 (2016), 286–347.
12. Cohen, L.; and Young, A. Multisourcing: Moving beyond outsourcing to achieve
growth and agility. Boston, MA: Harvard Business School Press, 2006.
13. Cohen, W.; and Levinthal, D. Absorptive capacity: A new perspective on learning and
innovation. Administrative Science Quarterly, 35(1990), 128–152.
14. Dibbern, J.; Goles, T.; Hirschheim, R.; and Jayatilaka, B. Information systems out-
sourcing: A survey and analysis of the literature. DATA BASE, 35, 4 (2004), 6–102.
15. Galbraith, J.R. Organization Design. Reading, MA: Addison-Wesley, 1977.
16. Gefen, D.; and Straub, D. A practical guide to factorial validity using PLS-graph:
Tutorial and annotated example. Communications of the Association for Information Systems
16, 1 (2005), 91–109.
17. Goo, J.; Huang, C.D.; and Hart, P. A path to successful IT outsourcing: Interaction
between service-level agreements and commitment. Decision Sciences, 39, 3 (2008),
469–506.
18. Goodhue, D.; Lewis, W.; and Thompson, R. Research note-statistical power in analyz-
ing interaction effects: Questioning the advantage of PLS with product indicators.
Information Systems Research, 18, 2 (2007), 211–227.
19. Gopal, A.; and Gosain, S. The role of organizational controls and boundary spanning
in software development outsourcing: Implications for project performance. Information
Systems Research, 21, 4 (2010), 960–982.
20. Gopal, A.; Mukhopadhyay, T.; and Krishnan, M. The role of software processes and
communication in offshore software development. Communications of the ACM, 45, 4
(2002), 193–200.
21. Grover, V.; Cheon, M.J.; and Teng, J.T.C. The effect of service quality and partnership
on the outsourcing of information systems functions. Journal of Management Information
Systems, 12, 4 (1996), 89–116.
22. Heckman, J. Sample selection bias as a specification error. Econometrica, 47, 1 (1979),
153–161.
23. Henderson, R.M.; and Clark, K.B. Architectural innovation: The reconfiguration of
existing product technologies and the failure of established firms. Administrative Science
Quarterly, 35, 1 (1990), 9–30.
24. Hobday, M.; Davies, A.; and Prencipe, A. Systems integration: A core capability of the
modern corporation. Industrial and Corporate Change, 14, 6 (2005), 1109–1143.
25. Huber, G.P. Organizational learning: The contributing processes and the literature.
Organization Science, 2, 1 (1991), 88–115.
26. Huber, T.L.; Ficher, T.A.; Dibbern, J.; and Hirschhem, R. A process model of com-
plementarity and substitution of contractual and relational governance in IS outsourcing.
Journal of Management Information Systems, 30, 3 (2013), 81–114.
27. Hui, P.P.; Davis-Blake, A.; and Broschak, J.P. Managing interdependence: The effects
of outsourcing structure on the performance of complex projects. Decision Sciences, 39, 1
(2008), 5–31.
28. Kirsch, L.J.; Sambamurthy, V.; Ko, D.G.; and Purvis, R.L. Controlling information
systems development projects: The view from the client. Management Science, 48, 4 (2002),
484–498.
29. Koo, Y.; Lee, J.-N.; Heng, C.S.; and Park, J. Effect of multivendor outsourcing
on organizational learning: A social relation perspective. Information & Management, 54, 3
(2017), 396–413.
30. Lee, J.N.; and Kim, Y.G. Effect of partnership quality on IS outsourcing success:
conceptual framework and empirical validation. Journal of Management Information
Systems, 15, 4 (1999), 29–61.
1282 OSHRI ET AL.
31. Levina, N.; and Su, N. Global multisourcing strategy: The emergence of a supplier
portfolio in services offshoring. Decision Sciences, 39, 3 (2008), 541–570.
32. Mani, D.; Barua, A.; and Whinston, A. An empirical analysis of the of information
capabilities design on business process outsourcing performance. MIS Quarterly, 34, 1
(2010), 39–62.
33. Mani, D.; Barua, A.; and Whinston, A. An empirical analysis of the contractual and
information structures of business process outsourcing relationships. Information Systems
Research, 23, 3 (2011), 618–634.
34. Oshri, I.; Kotlarsky, J.; and Willcocks, L.P. The Handbook of Global Outsourcing and
Offshoring. 3rd ed. London: Macmillan, 2015.
35. Pinsonneault, A.; and Kraemer, K.L. Survey research methodology in management
information systems: An assessment. Journal of Management Information Systems, 10, 2
(1993), 75–105.
36. Poppo, L.; and Zenger, T. Do formal contracts and relational governance function as
substitutes or complements? Strategic Management Journal, 23, 8 (2002), 707–725.
37. Premkumar, G.; Ramamurthy, K.; and Saunders, C.S. Information Processing view of
organizations: An exploratory examination of fit in the context of interorganizational
relationships. Journal of Management Information Systems, 22, 1 (2005), 257–294.
38. Prencipe, A.; Corporate strategy and systems integration capabilities: Managing net-
works in complex systems industries. In A. Prencipe, A. Davies, and M. Hobday (eds.), The
Business of Systems Integration. Oxford, UK: Oxford University Press, 2005, pp. 114–132.
39. Sanchez, R.; and Mahoney, J.T. Modularity, flexibility, and knowledge management
in product and organization design. Strategic Management Journal, 17, Winter (1996),
77–91.
40. Schwarz, C. Toward an understanding of the nature and conceptualization of out-
sourcing success. Information & Management, 51, 1 (2014), 152–164.
41. Siemsen, E.; Roth, A.; and Oliveira, P. Common method bias in regression models
with linear, quadratic, and interaction effects. Organizational Research Methods, 13, 3
(2010), 456–476.
42. Su, N.; and Levina, N. Global multisourcing strategy: Integrating learning from
manufacturing into IT service outsourcing. IEEE Transactions on Engineering
Management, 58, 4 (2011), 717–729.
43. Takeishi, A. Knowledge partitioning in the interfirm division of labor: The case of
automotive product development. Organization Science, 13, 3 (2002), 321–338.
44. Tanriverdi, H.; Konana, P.; and Ge, L. The choice of sourcing mechanisms for business
processes. Information Systems Research, 18, 3 (2007), 280–299.
45. Tiwana, A. Does technological modularity substitute for control? A study of alliance
performance in software outsourcing. Strategic Management Journal, 29, 7 (2008), 769–780.
46. Tiwana, A. Systems development ambidexterity: Explaining the complementary and
substitutive roles of formal and informal controls. Journal of Management Information
Systems, 27, 2 (2010), 87–126.
47. Tiwana, A.; and Keil, M. Control in internal and outsourced software projects. Journal
of Management Information Systems, 26, 3 (2009), 9–44.
48. Tushman, M.L.; and Nadler, D.A. Information processing as an integrating concept in
organizational design. Academy of Management Review, 3, 2 (1978), 613–624.
49. Wiener, M.; and Saunders, C. Forced coopetition in IT multi-sourcing. Journal of
Strategic Information Systems, 23, (2014), 210–225.
50. Wooldridge, J.M., Introductory Econometrics – a Modern Approach. 4th ed. (Mason,
OH: South-Western Cengage Learning, 2009.
JOINT VENDOR PERFORMANCE IN MULTI-SOURCING 1283
Copyright of Journal of Management Information Systems is the property of Taylor & Francis
Ltd and its content may not be copied or emailed to multiple sites or posted to a listserv
without the copyright holder’s express written permission. However, users may print,
download, or email articles for individual use.
- Abstract
- Guardian Vendor as aSource of External IP Capacity
- Notes
- References
Introduction
Theoretical Background
Information Processing View and Multi-Sourcing
Client’s Challenge: With or Without aGuardian Vendor
Hypotheses
Client’s Sources of Internal IP Capacity
Client’s Inter-Vendor Governance
Client’s Architectural Knowledge
Guardian-as-a-Mediator Perspective
Guardian-as-an-Architect Perspective
Control Variables
Methods
Data
Measures
Instrument Validation
Regression Analysis
Results
Discussion
Direct Model
Guardian Model
Implications
Theoretical Contributions
Practical Implications
Limitations and Future Research
Conclusions