Divergent Thinking in a Neurodynamical Model of Ideation

Mei Mei and Ali A. Minai, Senior Member, IEEE

Abstract—Divergent thinking refers to a style of thinking that be able to make more unusual conceptual combinations than ranges across a broad range of concepts, and is considered to the former. be a core enabler of creativity. Thinking is often modeled as a Conceptual associations – or, more accurately, word asso- process of conceptual combination, and creative ideas are seen as those using unconventional combinations of concepts. Since ciations – have been studied extensively through the exper- conceptual combination is fundamentally an associative process, imental determination of association norms [7] or analysis it has been proposed that creative thinking reflects stronger than of associations using dictionaries and thesauri [8], which expected associations between normally remote associates in the provide valuable information on objective patterns of associ- mind of the thinker. We recently proposed a neurodynamical ation in a reference populatio, e.g., modern English speakers. model called Itinerant Dynamics with Emergent Attractors (IDEA) to explore the process by which ideas can emerge in Associative recall in individual subjects has been studied an associative memory system through conceptual combination. through priming and cued recall experiments [9], [10], [11], In this paper we study how the functional dynamics of this [7], and recently through experiments designed to explicitly model changes when its associative weights are changed to make map the patterns of association in individuals [12], [13]. remote associations stronger. We apply the model to data from Mednick’s hypothesis has also been evaluated experimentally four sets of papers from previous IJCNN meetings. In addition to identifying the differences in the dynamics due to changes in with mixed results [14]. However, none of these approaches the association pattern, we consider whether divergent thinking completely captures the functional strength of associations can potentially predict future ideas or reconstruct past ideas. in thinking, which is fundamentally a dynamical and fluid process with one thought generating the next. Concepts occur I.INTRODUCTION in this stream based not on their pairwise associations with other concepts but on their joint participation in complex The process by which ideas – especially novel ideas – ideas involving multiple concepts. We recently introduced a are generated within the mind has long been a topic of neurodynamical model called itinerant dynamics with emer- interest to philosophers and scientists. Within the framework gent attractors (IDEA) to model this process [15], [16], [17], of modern psychology and cognitive science, a widely held [18]. In this model, a recurrent semantic neural network assumption is that ideas emerge through the combination is built from empirically determined associations between of existing concepts, and that novel ideas represent new, concepts, and the dynamics of the network then generates an unexpected combinations [1], [2], [3], [4], [5], [6]. The key itinerant sequence [19] of metastable attractors representing factor underlying this process of conceptual combination is combinations of mutually associated concepts. While the association. The mind’s conceptual space can be seen as associations used to instantiate the model could come from an associative memory encoding complex patterns of useful any valid source, in most of our work they have been derived associations between concepts from multiple modalities and from corpora of continuous text or speech produced in at multiple levels of representation. The process of generating natural settings. The assumption is that the correlations found thoughts or ideas can then be considered a dynamical recall through such data represent actual associations underlying process that unmasks conceptual groups with sufficiently fluent thought, and are thus a more accurate reflection of as- strong mutual associations. In a seminal paper, Mednick [2] sociations in the minds of the authors or speakers. The IDEA proposed that an individual’s capacity for creative thinking model has been applied to identifying significant conceptual depended strongly on the structure of their conceptual asso- combinations in texts such as conference proceedings [20], ciations. He argued that, for any given concept, most people and to identifying semantically salient words in several types have only a few strongly associated concepts, but creative of text [21]. individuals have relatively strong associations with a larger In our previous work, the weights of the network were set of concepts that would generally be considered unusual. based purely on measures of association such as co- For example, most people may asscociate the concept “table” occurrence, correlation or pointwise mutual information be- with “chair”, “top”, “dining” and “coffee”, but someone with tween words in the reference corpus [22]. In this paper, we more grounding in science and mathematics would also think consider how a nonlinear transformation of these weights of the periodic table from chemistry, the water table from to model divergent thinking influences the dynamics of the geology, truth tables from logic, etc. The latter person would system, and whether it allows the generation of more wide- ranging – possibly even predictive – ideas. Mei Mei (Email: [email protected]) and Ali Minai (Email: [email protected]) are all with the Department of Electrical Engineering II.BACKGROUNDAND MOTIVATION and Computing Systems, University of Cincinnati, Cincinnati OH 45221 Acknowledgement: This work was supported in part by a National Science The cognitive mechanisms and neural processes underly- Foundation INSPIRE grant to Ali Minai (BCS-1247971). ing creative thinking have been an object of fascination and study for a long time. The modern tradition of such studies corpus of a poet or single books on a specific topic as sources goes back of Guilford [23] and Hebb [24], who proposed of data [18], [22]. An especially interesting and useful that new ideas emerged through the combinatorial activity of source of data is corpora of papers published in technical neuronal assemblies. Since then, there have been numerous conferences or journals. While these typically involve many behavioral studies [25], [26], [27], [28], [29], [30], [31], authors, they can be seen as representing the “collective [32] as well as neurobiological studies using brain imaging mind” of an epistemic community, and application of a methods [33], [34], [35], [36], [37], [38], [39]. They indi- model such as IDEA to this data can be used to identify the cate that creative thinking arises through a combination of core ideas within this community. We have previously used semantic associations and cognitive control [37], [40], [14], abstracts from a set of IJCNN meetings for this purpose [20], [38]. Many of the influential theoretical models of creative [21]. In this paper, we extend this to much larger corpora thinking have postulated that associative recombination of built from full papers across a number of IJCNN meetings. concepts is the main mechanism for generating novel ideas This paper focuses on two main issues: 1) How does [1], [2], [3], [5], [4], [32], [6], though processes such as modifying the connectivity of the epistemic network in the generalization, off-label use and analogy have also been IDEA model in ways consistent with Mednick’s proposal implicated [41], [42], [43]. Various computational models [2] change the dynamics of conceptual combination? 2) Can have also been proposed to study, understand and possibly the IDEA model – especially in divergent mode – predict enhance creative thinking [44], [3], [41], [45], [42], [43], future ideas in an epistemic community based on associations [46], [47]. derived from past ideas. Clearly, the latter issue is highly A key element in many theoretical models of creativity speculative, but any progress on it can potentially be of great is the idea that the pattern of conceptual associations in value. the thinker’s mind is an important determinant of creativity [2], [42], [12], [13]. The use of associative neural networks III.DATASETS to model creativity has been a central feature of models We used proceedings from several years of the Interna- developed by our research group [48], [49], [15], [46], [50], tional Joint Conference on Neural Networks (IJCNN) to [51], and others [47]. The IDEA model [16], [17], [18], [20], create the datasets. The datasets were created by taking only [21], [22] forms the core element of our overall model for the main body of papers from the proceedings, removing creative thinking, and is based on the following postulates: figures, equations, citations, etc., and representing each paper 1) Semantic knowledge is represented in the brain through as a sequence of sentences. Stop words were removed using associations between conceptual elements represented a standard list, and the other words were stemmed to group as neural units, forming an epistemic network. together variants of the same root, e.g., “test”, “tested”, 2) Ideas arise in the epistemic network as emergent attrac- “testing”, etc. The data were then filtered further to remove tors – metastable activity patterns – through itinerant non-salient words using methods described in our previous neural activity dynamics [19]. The activity of the work [20]. network lingers around an emergent attractor for some In all, four datasets were generated, as follows: time before moving on through a series of transient 1) The 98-99-01 Dataset: This comprised papers from patterns until it encounters another emergent attractor. the 1998, 1999, and 2001 meetings of IJCNN. The If an emergent attractor persists sufficiently long, it is attributes were as follows: Number of papers = 1,468; recognized consciously as a thought or idea. Number of sentences = 181,175; Number of word 3) The itinerant dynamics of the epistemic network is tokens = 1,591,515; Vocabulary size (after filtering) = shaped by the pattern of association between concep- 7,317. tual elements [2], [12], [52]. 2) The 11-13-14 Dataset: This comprised papers from While the initial work on the IDEA model focused purely the 2011, 2013, and 2014 meetings of IJCNN. The on the effect of network connectivity [16], [17], subsequent attributes were as follows: Number of papers = 1,236; studies sought to instantiate it with meaningful real-world Number of sentences = 140,508; Number of word data. Obtaining data on semantic associations relevant to tokens = 1,284,149; Vocabulary size (after filtering) = fluid thinking is problematic for several reasons. As discussed 9,063. earlier, associations elicited through word cues do not meet 3) The Pre2006 Dataset: This comprised papers from the the need because they are not obtained under conditions of 1998, 1999, 2001, 2003 and 2004 meetings of IJCNN. natural thought. An alternative we have previously proposed The attributes were as follows: Number of papers is to use what an individual (or group) says or writes to = 2,476; Number of sentences = 302,596; Number extract associations. This may work better but, if done care- of word tokens = 2,687,029; Vocabulary size (after lessly, still encounters the problem of context: People modify filtering) = 8,180. their mental associations in different epistemic domains [53]. 4) The Post2006 Dataset: This comprised papers from Thus, it is important to use natural text or speech that, the 2007, 2009, 2011, 2013 and 2014 meetings of in some concrete sense, belongs to a meaningful epistemic IJCNN. The attributes were as follows: Number of equivalence class. In previous work, we have used the literary papers = 2,140; Number of sentences = 240,578; Number of word tokens = 2,208,520; Vocabulary size (after filtering) = 8,644.

IV. BUILDING EPISTEMIC NETWORKS A. Estimating Associative Strength To construct epistemic networks from text, we assume that associations can be estimated from the co-occurrence of words in the same sentence. Following the methods used in our previous work [18], [20], [21], [22], the probability q pij of words vi and vj co-occurring in the same thought is calculated as the fraction of sentences in which they co- q q occur within the corpus G . The occurrence probability pi of the individual word vi is also calculated as the fraction of sentences in which it occurs. Associative weights for the q epistemic network are then obtained from pij – which equals q q pji – and pi . We have previously used and compared several association metrics between words, including co-occurrence probability, condition probability of co-occurrence, correlation coefficient of co-occurrence and pointwise mutual information [18], [20], [22], [21]. Based on the results of the comparison, we use the correlation coefficient of co-occurrence in the present work. The association strength between two words i and j, this is defined as:

pq − pqpq aq = ij i j (1) ij p q q q q q pi (1 − pi ) pj (1 − pj ) with only positive values retained. Once the association weights have been computed, the epistemic network is formed with each node representing a word and the weight of the edge between each pair of nodes given by the corresponding association weight, normalized by the weight of the largest association. The resulting asso- q ciation matrix of connections is denoted as A = [aij]. The IDEA model instatiates this epistemic network in a neural network as described in the next section.

B. Modeling Divergent Thinking As discussed earlier, our motivating hypothesis [2], [42] is that creative thinking is supported by minds with stronger than usual associations betwen unusual concepts. In the context of our epistemic network model, there are many possible ways to represent this. In this paper, we designate Fig. 1: Recurrence plots for normal (left column) and diver- the network with weights specified by Eq. (1) as the non- gent (right column) simulations. Top Row: 98-99-01 Corpus. divergent case (ND) and transform these weights as follows Second Row: 11-13-14 Corpus. Third Row: Pre2006 Corpus. to get the weights for the divergent case (D): Bottom Row: Post2006 Corpus. Each x-y pixel in a plot shows the pairwise normalized hamming distance between q q the ideas generated in the same run. Black indicates perfect aˆ = 1 − (a − 1)2µ (2) ij ij match between the ideas on the x and y axes, and lighter where µ ∈ {1, 2, 3, ...} is called the divergence factor. The colors progressively indicate lower match. The ideas on each divergent case simulations in this paper use µ = 4. The axis are ordered identically in the sequence generated. weights are renormalized after the transformation, and the association matrix is denoted as Aˆ. Network Total Unique than a fixed threshold level, ymin. The output of unit i is Corpus Type Ideas Ideas Efficiency given by: 98-99-01 ND 169 / 159 73 / 73 0.435 / 0.459 98-99-01 D 78 / 86 47 / 59 0.603 / 0.686 11-13-14 ND 120 / 131 56 / 74 0.467 / 0.564 ( 11-13-14 D 77 / 75 57 / 47 0.740 / 0.627 1, if yi(t) ∈ {K most excited units} zi(t) = f(yi(t)) = Pre 2006 ND 154 / 160 65 / 71 0.422 / 0.444 0, otherwise Pre 2006 D 80 / 94 47 / 58 0.587 / 0.617 Post 2006 ND 146 / 143 51 / 54 0.349 / 0.378 (5) Post 2006 D 95 / 82 53 / 49 0.558 / 0.598 The competitive rule is flexible so that a unit i with yi(t) within 1% of the Kth most excited unit can also fire, which TABLE I: Productivity and efficiency of idea generation. The means that the number of active units can vary around the two items in each entry correspond to two independent runs nominal K value. of the simulation. The network is subject to two types of modulation: Refractoriness: The ability of a unit i to fire is determined Source Target Network Max Max by a variable resource, r (t), which depletes when the unit Corpus Corpus Type Γ Γ Pre Post i P re P ost fires and is replenished when it is refractory: 98-99-01 11-13-14 ND 0.68 0.32 0.54 0.29 98-99-01 11-13-14 D 0.74 0.26 0.52 0.31 11-13-14 98-99-01 ND 0.35 0.65 0.41 0.35 ( (1 − λ−)r (t − 1), if active 11-13-14 98-99-01 D 0.31 0.69 0.41 0.35 r (t) = i (6) Pre 2006 Post 2006 ND 0.60 0.40 0.48 0.34 i + ri(t − 1) + λ (1 − ri(t − 1)), if inactive Pre 2006 Post 2006 D 0.52 0.48 0.50 0.33 Post 2006 Pre 2006 ND 0.24 0.76 0.54 0.35 where λ− is the resource depletion rate, and λ+ the resource Post 2006 Pre 2006 D 0.37 0.63 0.46 0.34 recovery rate. Unit i becomes refractory if ri(t) falls below − TABLE II: Relative and maximal prevalence of generated a threshold θr , and cannot fire again until ri(t) recovers to + ideas in Pre and Post corpora for each experiment. Ideas a level θr . The depletion and recovery rates for individual from both runs of each simulation were pooled. units are similar with a small random variation. Synaptic Modulation: Synaptic modulation is known to be important for short-term memory[55], [56]. Synapses acti- V. NEURAL NETWORK MODEL vated repeatedly over short periods can temporarily become depressed [57], and then recover gradually during periods A. Model Description of inactivity. This short-term change, which is independent The network model used is similar to the one described of long-term synaptic learning, allows the activity dynamics in our previous papers [16], [17], [20], [22], and is briefly in recurrent neural networks to integrate long-term learning described here. The network has N neural units, one for each with recent context, and to explore their state space more word. The neural units can be seen as representing coherent fully. This modulation is implemented as follows: cell assemblies [54]. The connection weight matrix for the network is given by W = [w ], with w denoting the weight ( ij ij (1 − ψ−)w (t − 1), if active from unit j to unit i. In simulations, W is set equal to the ij wij(t) = + (7) appropriate association matrx, A or Aˆ. All negative values wij(t − 1) + ψ [w ¯ij − wij] if inactive are set to 0. where ψ- and ψ+ are synaptic decay and recovery rates, i t The input of unit and time is: respectively, and w¯ij is the nominal (pre-modulation) weight of the synapse. N The depletion and recovery rate parameters for both types 1 X x (t) = w (t)x (t−1)+γ ξ (t) of modulation are set such that several cycles of synaptic i PN ij j noise i j=1 xj(t − 1) +  j=1 modulation are nested within each non-refractory duration for (3) a neural unit, allowing the dynamics to linger in a semantic where xj is the output of unit j, ξi(t) is uniform white neighborhood before transitioning away. noise, and γnoise, is a fixed gain parameter. The division by total activity smooths out the effect of fluctuation in overall B. Model Dynamics activity level through shunting inhibition, and  is a small The interaction between the competitive activity rule and constant to avoid division by zero. To simulate continuous- the two types of modulation ensures that the network’s time dynamics, the activation of unit i at time t is given activity dynamics wanders itinerantly between metastable by: attractors [19], as postulated by the IDEA model. If a group of active units have strong mutual connections to each other, they can remain active as a group through this mutual support yi(t) = αyi(t − 1) + (1 − α)xi(t) (4) like an attractor in standard asociative memory models [58]. where α is set to 0.95 in all simulations. However, this gradually depletes the activity resources of At each time-step t, the K most highly activated non- the units in the group and weakens the synapses involved, refractory units are allowed to fire if they have yi(t) greater leading to the destabilization of the attractor. The network then goes through a period of transient activity until another metastable attractor arises. The metastable attractors that persist for a period longer than an awareness threshold are perceived as conscious ideas , whereas the transient dynamics remains subconscious. Thus, the sequence of metastable attractors in the epistemic network constitute a “train of thought” [46], [16], [17], and transitions between successive attractors can be seen as ‘mental saccades” as proposed by Starzyk [59]. In the current set of simulations, we set the awareness threshold to 20 time-steps based on the modulation parameters. An important aspect of the dynamics described above is that the attractors which emerge during the dynamics are not embedded explicitly in the network, as is the case in standard models of recurrent associative memoy networks [60], [58]. Rather, they become embedded implicitly as the system is configured from experience (in this case, the texts). Thus, it is possible for the system to generate ideas that it was not “trained” on, but represent recombinations of parts of several explicit ideas. From a cognitive perspective, these purely emergent ideas can also be seen as spurious memories [58], which leads to the intriguing implication that novel or creative ideas represent a kind of useful confabulation – making up better new stories from old ones.

VI.SIMULATIONS AND RESULTS

A. Setup and Metrics

Networks with N = 1, 000 neural units corresponding to the 1,000 most frequent words in each corpus were simulated for all four corpora and both weight types (ND and D) using the same parameter settings. Each run was 20,000 time-steps long divided into four episodes of 5,000 time-steps each: The network’s state – but not its adaptive parameters – was randomly re-initialized after 5,000 steps to see if the system moved to different regions of semantic space or fell into the same pattern. The activity level K was set to 6. We ran two sets of experiments, each using a specific pair Fig. 2: Mean prevalence of ideas for normal (left column) Pre of datasets, with the temporally prior one designated and and divergent (right column) simulations. Row 1: Source = Post the temporally posterior one . These were: 98-99-01 , Target = 11-13-14. Row 2: Source = 11-13-14, 1) Experiment 1: Pre = 98-99-01 dataset and Post = 11- Target = 98-99-01. Row 3: Source = Pre2006 , Target = 13-14 dataset. Post2006. Row 4: Source = Post2006 , Target = Pre2006. 2) Experiment 2: Pre = Pre2006 dataset and Post = Post2006 dataset. found in the sentence. The prevalence of idea Ck in paper Each experiment consisted of two parts. In each part, Dq was then quantified as: one of the two datasets was used as the source corpus i and the other as the target corpus. The source set was mq 1 Xi used to instantiate an IDEA network from which ideas were πq = V (C ,Sq ) (8) ki mq k ij generated as described above, thus creating an idea set, where i j=1 each idea comprises a set of words. Each generated idea, Ck, q q q q was then compared with each sentence, Sij, of each paper Di where mi is the number of sentences in Di . Thus, while q q in each dataset G , obtaining an overlap value V (Ck,Sij) ideas were generated only from the source corpus, their in each case as the fraction of words in the idea that are also prevalence was calculated in papers of both corpora. q The mean prevalence of Ck in corpus G was measured as: N q 1 X hπqi = πq (9) k N q ki j=1 where N q is the number of papers in corpus Gq. To assess the relative prevalence of ideas in the two corpora, we calculated the differential mean prevalence as:

P ostP re P ost P re ∆k = hπk i − hπk i (10) where P re and P ost indicate the temporally prior corpus and the temporally posterior, respectively, regardless of which one was used as source and which one as the target. The degree to which the generated set of ideas correspond pre- dominantly to one or the other corpus, the following two quantities are calculated:

ni 1 X P ostP re ΓP ost = H(∆k ) (11) nI k=1

ni 1 X P ostP re ΓP re = H(−∆k ) (12) nI k=1 where ( 1, if u > 0 H(u) = (13) 0, otherwise

Thus, ΓP ost quatifies the degree to with the Post corpus contributed more to the generated ideas than the Pre corpus, whereas ΓP re quantifies the converse. Taken together, these metrics allow assessment of how prevalent ideas generated from the source corpus are within both the source and target corpora. B. Results and Discussion We ran two independent trials of 20,000 time-steps for each case of Experiments 1 and 2. One reason for doing so was to reduce the possibility of using an anomalous trial for inferring results. Table I shows the total number of ideas generated by each trial (column 3) and the number of unique ideas in each trial (column 4). The efficiency of ideation is measured as the fraction of all generated ideas that were Fig. 3: Maximum prevalence of ideas for normal (left col- unique (column 5). As can be observed, the ND cases all umn) and divergent (right column) simulations. generated a larger number of ideas by 60-80%, but their efficiency was significantly lower than the corresponding D cases. Also, all the statistics were consistent across both trials corresponding D cases. There is a clear difference between of each case, indicating that the trials were likely to be typical the two columns. In all the normal (ND) cases, the dynamics rather than anomalous. repeats significant sub-sequences of ideas throughout the Figure 1 shows the mutual similarity between all ideas trial, which show up as tracks of black pixels parallel to the generated in the first trial of each experiment. Each image diagonal. Furthermore, such tracks occur not just close to the – called a recurrence plot – has the generated ideas in diagonal but over the entire image, indicating that even with order of generation on the x and y axes, and the color of periodic resets, the network often returns to the same idea the (i, j)th position indicates the similarity between the ith sequences. This is a classic instance of convergent thinking, and the jth idea. Black indicates complete similarity and where the thinker has difficulty escaping conventional trains white complete dissimilarity, with colors ranging from deep of thought, and converges to them repeatedly. In extreme red to ligh yellow with increasing dissimilarity. The left cases, this can lead to fixation. In contrast, the plots in the column shows all the ND cases and the right column the right column, corresponding to the D case, show far less repetition, though there are occasional short tracks. Thus, the range of maximum prevalence is always larger with while the total number of ideas generated is smaller, the rspect to the Pre corpus than the Post corpus. Even though search here moves preferentially towards new ideas rather ideas generated from the Post corpus generally have larger than reprising old ones. Even more noticeably, when an old differential prevalence in their source corpus, they still tend idea is encountered again, it does not trap the search process to have larger maximum prevalence in the Pre corpus. This into a previously seen sequence of old ideas, but allows it is summarized in Table II, which shows the mean predom- to move on to new ones. This in contrast to the ND case, inance of the generated ideas with respect to each corpus and can be seen as an instance of thinking that moves along – ΓP re and ΓP ost – as well as the maximum prevalence diverging paths rather than convergent ones. Of course, a of any idea in the generated set in both corpora. Based on small amount of repetition is to be expected since thinking is anecdotal observations, we speculate that this phenomenon a structured process, and completely random jumps between arises because the Post corpora tend to be richer and more ideas would be pathological. Thus, in a qualitative sense, it diverse in terms of ideas, so that no single idea is likely can be concluded that the model simulates the phenomenolgy to be strongly prevalent in a single paper, whereas earlier of convergent and divergent thinking, at least at a simplistic corpora may have a higher incidence of some core ideas, level. but further study is needed to verify this. The observation The second issue of interest in this study was to see how could also reflect a change in writing styles or in the relative well the generated ideas fit with the actual material of the frequency with which certain standard terms such as “feed- source and target corpora. Figure 2 plots the mean prevalence forward neural networks” or “principal components analysis” P ost are used. The approach presented in this study is potentially of each idea in the Post corpus (hπk i), against its mean P re an exciting tool for studying such issues. prevalence in the Pre corpus (hπk i). A striking feature of the plot is that ideas maintain their mean prevalence VII.CONCLUSION levels remarkably across the source and target corpora: Ideas that are unusual in one are unusual in the other, and those In this study, we have compared the dynamics of a common in one are common in the other. Though there neurodynamical model of conceptual combination with two were some ideas that deviated from the diagonal significantly, types of association weights, representing conventional and indicating differences in their relative mean prevalence, we creative patterns of association between concepts. The results saw no ideas that showed wide disparity in mean prevalence show promise for modeling convergent and divergent think- across corpora. It was also noticeable that the ideas showing ing, and provide preliminary evidence for several intertesting the most disparity were those with high prevalence in both hypotheses regarding the evolution of ideas within an epis- corpora, whereas unusual ideas (i.e., those plotted closer temic community such as neural network researchers. Future to the origin) were tightly clustered on the diagonal. This studies will focus on the latter issue in more detail. was somewhat disappointing because it suggested that, on Future studies will also look at other mathematical models average, even in its more divergent state, the model is for modifying conceptual associations to represent patterns not predicting ideas that actually emerged in the future, or consistent with creative thinking. Analyzing corpora of cre- recalling past ideas that had disappeared. ative texts such as poetic work [18], [22] are potentially a rich source for such models. This situation changes somewhat when we look at the maxmum prevalence of each idea in the Pre and Post corpora, REFERENCES which is shown in Figure 3. Maximum prevalence is defined [1] D. T. Campbell, “Blind variation and selective retention in creative paper-wise, so a high maximum prevalence of idea Ck in thought as in other knowledge processes,” Psychol. Rev., vol. 67, pp. corpus Gq indicates that at least one paper in the corpus had 380–400, 1960. [2] S. Mednick, “The associative basis of the creative process,” Psycho- a high prevalence of the idea. Two things are quite noticeable logical Review, vol. 69(3), pp. 220–232, 1962. in Figure 3. First, when the Pre corpus is the source (rows [3] V. Brown, M. Tumeo, T. Larey, and P. Paulus, “Modeling cognitive in- 1 and 3), maximum pevalence of generated ideas skews teractions during group brainstorming,” Small Group Research, vol. 29, pp. 495–526, 1998. strongly towards the Pre corpus, but when the Post corpus [4] D. K. Simonton, “Scientific creativity as constrained stochastic be- is the source (rows 2 and 4), maximal prevalence has only a havior: the integration of product, person, and process perspectives,” slight bias towards Post. Based on informal sampling of the Psychol. Bull., vol. 129, pp. 475–494, 2003. [5] G. Fauconnier and M. Turner, The Way We Think: Conceptual Blending generated ideas, this phenomenon reflects the fact that, in a And The Mind’s Hidden Complexities. Basic Books, 2003. field of scientific study such as neural networks, ideas tend [6] D. K. Simonton, “Creative thought as blind-variation and selective- to accumulate over time, so that, while new ideas are added retention: Combinatorial models of exceptional creativity,” Physics of Life Reviews, vol. 7, pp. 156–179, 2010. with time, old ones do not disappear. Generating ideas from [7] D. L. Nelson, V. M. McKinney, N. R. Gee, and G. A. Janczura, an earlier corpus will thus tend to miss ideas that are yet to “Interpreting the influence of implicitly activated memories on recall develop, but ideas generated from a later corpus are likely to and recognition,” Psychological Review, vol. 105, pp. 299–324, 1998. [8] M. Steyvers and J. Tenenbaum, “The large scale structure of semantic include most of those prevalent at an earlier time. Verifying networks: Statistical analyses and a model of semantic growth,” this and obtaining other insights will require a much more Cognitive Science, vol. 29, pp. 41–78, 2005. detailed study of the generated ideas. [9] D. L. Nelson, T. A. Schreiber, and C. L. McEvoy, “Processing implicit and explicit representations,” Psychological Review, vol. 99, pp. 322– The second noticeable feature of the plotted data is that 348, 1992. [10] R. Ratcliff and G. McKoon, “Retrieving information from memory: people solve verbal problems with insight,” PLoS Biology, vol. 2, pp. Spreading-activation theories versus compound-cue theories,” Psycho- 0510–0510, 2004. logical Review, vol. 101, pp. 177–184, 1994. [35] E. Bowden, M. Jung-Beeman, J. Fleck, and J. Kounios, “New ap- [11] D. L. Nelson and J. Xu, “Effects of implicit memory on explicit recall: proaches to demystifying insight,” Trends in Cognitive Sciences, vol. 9, Set size and word frequency effects,” Psychological Research, vol. 57, pp. 322–328, 2005. pp. 203–214, 1995. [36] O. M. Razumnikova, “Creativity related cortex activity in the remote [12] Y. Kenett, D. Kenett, E. Ben-Jacob, and M. Faust, “Global and local associates task,” Brain Research Bulletin, vol. 73, pp. 96–102, 2007. features of semantic networks: Evidence from the Hebrew mental [37] S. G. Shamay-Tsoory, N. Adler, J. Aharon-Peretz, D. Perry, and lexicon,” PLoS ONE, vol. 6, p. e23912, 2011. N. Mayseless, “The origins of originality: The neural bases of creative [13] Y. Kenett, D. Anaki, and M. Faust, “Investigating the structure of thinking and originality,” Neuropsychologia, vol. 49, pp. 178–185, semantic networks in low and high creative persons,” Frontiers in 2011. Human Intelligence, vol. 8, p. Article 407, 2014. [38] M. Benedek, E. Jauk, A. Fink, K. Koschutnig, G. Reishofer, F. Ebner, [14] M. Benedek and A. C. Neubauer, “Revisiting Mednicks model on and A. C. Neubauer, “To create or to recall? neural mechanisms creativity-related differences in associative hierarchies. evidence for a underlying the generation of creative new ideas,” NeuroImage, vol. 88, common path to uncommon thought,” Journal of Creative Behavior, pp. 125–133, 2014. vol. 47, pp. 273–281, 2012. [39] A. Fink and M. Benedek, “Eeg alpha power and creative ideation,” [15] L. R. Iyer, A. A. Minai, S. Doboli, V. R. Brown, and P. B. Paulus, Neuroscience and Biobehavioral Reviews, vol. 44, pp. 111–123, 2014. “Effects of relevant and irrelevant primes on idea generation: A [40] M. Benedek, T. Konen,¨ and A. C. Neubauer, “Associative abilities computational model,” in Proceedings of IJCNN 2009, 2009, pp. 1380– underlying creativity,” Psychology of Aesthetics, Creativity, and the 1387. Arts, vol. 6, pp. 273–289, 2013. [16] N. Marupaka and A. A. Minai, “Connectivity and creativity in semantic [41] M. Boden, The Creative Mind: Myths and Mechanisms. Routledge, neural networks,” in Proceedings of IJCNN 2011, 2011, pp. 3127– 2004. 3133. [42] M. A. Schilling, “A small-world network model of cognitive insight,” [17] N. Marupaka, L. R. Iyer, and A. A. Minai, “Connectivity and thought: Creativity Res. J., vol. 17, pp. 131–154, 2005. The influence of semantic network structure in a neurodynamical [43] W. Duch, “Intuition, insight, imagination and creativity,” IEEE Com- model of thinking,” Neural Networks, vol. 32, pp. 147–158, 2012. put. Intell., pp. 40–52, 2007. [18] S. Doumit, N. Marupaka, and A. A. Minai, “Thinking in prose and [44] V. Brown and P. B. Paulus, “A simple dynamic model of social factors poetry: A semantic neural model,” in Proceedings of IJCNN 2013, in group brainstorming,” Small Group Research, vol. 27, pp. 91–114, 2013. 1996. [45] S. L. Thaler, “Neural nets that create and discover,” PC AI, vol. [19] I. Tsuda, “Towards an interpretation of dynamic neural activity in May/June, pp. 16–21, 1996. terms of chaotic dynamical systems,” Behavioral and Brain Sciences, [46] L. R. Iyer, S. Doboli, A. A. Minai, V. R. Brown, D. S. Levine, and vol. 24, pp. 793–847, 2001. P. B. Paulus, “Neural dynamics of idea generation and the effects of [20] M. Mei, A. Vanarase, and A. A. Minai, “Chunks of thought: Finding priming,” Neural Networks, vol. 22, pp. 674–686, 2009. salient semantic structures in texts,” in Proceedings of IJCNN 2014, [47] P. Thagard and T. C. Stewart, “The aha! experience: Creativity through 2014. emergent binding in neural networks,” Cognitive Science, vol. 35, pp. [21] M. Candadai, A. Vanarase, M. Mei, and A. A. Minai, “ANSWER: An 1–33, 2011. unsupervised attractor network method for detecting salient words in [48] A. A. Minai, L. R. Iyer, D. Padur, and S. Doboli, “A dynamic Proceedings of IJCNN 2015 text corpora,” in , 2015. connectionist model of idea generation,” in Proceedings of IJCNN [22] S. Doumit and A. A. Minai, “Effect of associative rules on the 2009, 2009, pp. 2109–2116. dynamics of conceptual combination in a neurodynamical model,” in [49] S. Doboli, V. R. Brown, and A. A. Minai, “A conceptual neural model Proceedings of IJCNN 2015, 2015. of idea generation,” in Proceedings of IJCNN 2009, 2009, pp. 723– [23] J. P. Guilford, “Creativity,” American Psychologist, vol. 5, pp. 444– 729. 454, 1950. [50] L. R. Iyer, V. Venkatesan, and A. A. Minai, “Neurcognitive spot- [24] D. O. Hebb, Essay on Mind. Lawrence Erlbaum, 1980. lights:configuring domains for ideation,” in Proceedings of WCCI [25] M. Diehl and W. Stroebe, “Productivity loss in brainstorming groups: 2010, 2010, pp. 3026–3033. Toward the solution of a riddle,” Journal of Personality and Social [51] P. B. Paulus, D. Levine, V. R. Brown, A. A. Minai, and S. Doboli, Psychology, vol. 53, pp. 497–509, 1987. “Modeling ideational creativity in groups: Connecting cognitive, neural [26] M. D. Mumford and S. B. Gustafson, “Creativity syndrome: Integra- and computational approaches,” Small Group Research, vol. 41, pp. tion, application, and innovation,” Psychological Bulletin, vol. 103, pp. 688–724, 2010. 27–43, 1988. [52] I. Lerner and O. Shriki, “Internally- and externally-driven network [27] M. I. Mobley, L. M. Doares, and M. D. Mumford, “Process analytic transitions as a basis for automatic and strategic processes in semantic models of creative capacities: Evidence for the combination and priming: theory and experimental validation,” Frontiers in Psychology, reorganization process,” Creativity Research Journal, vol. 5, pp. 125– vol. 5, p. Article 314, 2014. 155, 1992. [53] K. A. Ericsson and W. Kintsch, “Long-term working memory,” Psy- [28] P. B. Paulus and H. Yang, “Idea generation in groups: A basis chological Review, vol. 102, pp. 211–245, 1995. for creativity in organizations,” Organizational Behavior and Human [54] G. Buzsaki,´ “Neural syntax: Cell assemblies, synapsembles, and Decision Processes, vol. 82, pp. 76–87, 2000. readers,” Neuron, vol. 68, pp. 363–385, 2010. [29] T. B. Ward, “Creative cognition, conceptual combination, and the [55] R. S. Zucker and W. G. Regehr, “Short-term synaptic plasticity,” creative writing of stephen r. donaldson,” American Psychologist, Annual Review of Physiology, vol. 64, pp. 355–405, 2002. vol. 56, pp. 350–354, 2001. [56] L. F. Abbott and W. G. Regehr, “Synaptic computation,” Nature, vol. [30] V. Brown and P. Paulus, “Making group brainstorming more effective: 431, pp. 796–803, 2004. Recommendations from an associative memory perspective,” Current [57] M. V. Tsodyks and H. Markram, “The neural code between neocortical Directions in Psychological Science, vol. 11, pp. 208–212, 2002. pyramidal neurons depends on neurotransmitter release probability,” [31] K. L. Dugosh and P. B. Paulus, “Cognitive and social comparison pro- Proceedings of the National Academy of Sciences USA, vol. 94, pp. cesses in brainstorming,” Journal of Experimental Social Psychology, 719–723, 1997. vol. 41, pp. 313–320, 2005. [58] D. Amit, Modelling Brain Function. Cambridge, UK: Cambridge [32] B. A. Nijstad and W. Stroebe, “How the group affects the mind: A University Press, 1989. cognitive model of idea generation in groups,” Personality and Social [59] J. Starzyk, “Mental saccades in the cognitive process,” in Proceedings Psychology Review, vol. 3, pp. 186–213, 2006. of the 2011 International Joint Conference on Neural Networks, San [33] M. Molle,¨ L. Marshall, W. Lutzenberger, R. Pietrowsky, H. L. Fehm, Jose, CA, 2011, pp. 495–502. and J. Born, “Enhanced dynamic complexity in the human eeg during [60] J. J. Hopfield, “Neural networks and physical systems with emer- creative thinking,” Neuroscience Letters, vol. 208, pp. 61–64, 1996. gent collective computational abilities,” Proceedings of the National [34] M. Jung-Beeman, E. Bowden, J. Haberman, J. Frymiare, S. Arambel- Academy of Sciences USA, vol. 79, pp. 2554–2558, 1982. Liu, R. Greenblatt, P. Reber, and J. Kounios, “Neural activity when