The 4Th International Seville Conference on Future-Oriented Technology Analysis (FTA)
Total Page：16
File Type：pdf, Size：1020Kb
The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011 RETHINKING THE EXPERTISE OF EXPERTS IN ASSESSING
DISRUPTIVELY TRANSFORMATIVE FUTURES
Sofi Kurki and Sirkka Heinonen
Sofi Kurki, Finland Futures Research Centre, University of Turku, 768 Mission Oaks Lane, Santa Barbara, 93105 California, USA.
Sirkka Heinonen, Finland Futures Research Centre, University of Turku, Korkeavuorenkatu 25 A 2, 00130 Helsinki, Finland, [email protected]
Abstract
The Delphi technique leans on three basic principles, equally emphasized in the traditional version of the Delphi (Linstone & Turoff 1975; 2010) as well as in the more recent updates of the technique (Tapio 2002). The core elements of the method include anonymous experts responding to other experts´ insights in an iterative manner over multiple rounds. The third assumption is that the quality of the panelists´ expertise transfers to the quality of the results, usually interpreted as a correlation between highest possible expertise and the best possible quality of the insights, and often leading to very quantitative criteria of a desirable expert panel (measured as years of experience from a given field or other quantifiable merits). Recently research from behavioural sciences (Tetlock 2005) has accumulated evidence that seems to question and even contradict common interpretations about the latter assumption. In this study we suggest a method to combine these insights to the Delphi method. An experimental pilot study is presented and its results are assessed. The results suggest the method could provide a promising new way to approach expert foresight methods, especially when assessing disruptively transformative futures. It could be developed into a powerful method bridging, for example, experts and political decision makers. The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011 Introduction
The question of how to define relevant expertise in Delphi studies is not new. Already Olaf Helmer in his article Analysis of the Future – the Delphi method (1967) took up the issue of the problems of defining expertise for Delphi studies:
“...it is far from obvious what we mean – or should mean – when we say that somebody is an expert; and even given reasonable criteria of expertness it may not be easy to obtain adequate data for determining a person´s degree of expertise.” Helmer 1967).
The question of relevant expertise, however, becomes especially topical when trying to understand change processes that by nature are disruptive for much of the existing expertise in the given field. The experts still retain the key understanding of the relevant drivers of change, but their assessments of their implicatins or outcomes in the future may contain a seed of bias due to the possibly detrimental effect of the change to their position in the field. (On effects of disruptive innovation, see e.g. Christensen 1997). As expertise on any field requires decades of training, this is a true challenge for conducting Delphi studies in a time of discontinuous change.
Another relevant finding for assessing expertise in the Delphi studies comes from the field of behavioural sciences, where it was found that not only the depth of expertise has an effect on the ability to assess the future outcomes of current trends, but also the cognitive style of the experts directs the way they are able to integrate new elements into their mental models about the world. (Tetlock 2005). A longitudinal study over 20 years on political experts´ insights about the development of various indicators revealed that cognitive traits of individual experts have significant effect on the forecasting skills, suggesting even that on occasion dilettantes with a high degree of general knowledge can provide more insightful views on the future than top experts of a highly specialized field.
Our topic, the future of prosumerism in news media, is directly linked to a field experiencing disruptive change. Journalism as a field and business is in transition. Rapid technological, social and institutional changes affect the field, and as a result, the search for new plausible business models concerns all involved parties.
As within the scope of this research it was not possible1 to pick out experts for Delphi panels by assessing both the depth of their expertise and their cognitive styles, or directly assess the
1 Although Gordon (2009) does suggest an elegant solution that might solve the cognitive style issue: he suggests asking as part of the background information the question “Are you a generalist or a specialist?”. However, even if this question could on some level differentiate between Tetlock´s “foxes and hedgehogs”, we feel more research should be conducted for understanding the effect of cognitive style in Delphi studies. The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011 possibility of other bias in the experts´ estimates of the future, we took to seeking out other measures that might be utilised for evening out the bias in the expert´s answers. In this tentative pilot study we assembled two Delphi panels, one based on the traditional criteria for an expert panel, the other composed of a sample of professional futurists, here representing a group with high general knowledge but a lesser amount of task specific expertise of the topic.
Separate panels have been used before in situations where the type of relevant expertise has been difficult to define. In the seminal study by Bell Canada (Day 2002) regarding the future of communications services in the residence market, the main question was whether housewives or researchers and planners would be the best experts on the future adoption of communications services in the home. The question was resolved by forming two separate panels to assess the future in this area. One panel consisted of housewives (experts through experience) and the other of formal experts, who were either researchers or who had professional planning expertise on the topic. The Bell Canada study with two “competing” panels who answered the same set of questions remains a landmark in ways to solve the problem of expert definition in Delphi. Even though the aims and theoretical background differ for our study and the Bell Canada (mostly in that the housewifes were true experts on the future of adoption of communications services in the home, whereas in our study we deliberately sought for panellists, whose primary expertise lay outside of the study area), the Bell study has served as an inspiration in developing the method suggested in this article.
Methodology
For studying the highly contested topic of the future of prosumerism in journalism through the year 2020, we formulated a study in two phases. In Phase I, the experts participated in a two round Argumentative Policy Delphi about the future of journalism. The results from the first round were written into four images of the future. On the second round, experts were asked to review the four images of the future in terms of perceived subjective likelihood and desirability. In phase II, the four images of the future along with a selection of background information about media usage trends were subjected to the panel of futurists. The futurists´ task was to review the four images of the future in terms of perceived subjective likelihood and desirability, and to provide comments on all four images of the future.
The panels
A central phase in conducting any Delphi study is the selection of participants. One of the leading principles of Delphi study is the anonymity of the respondents, which allows the focus to be on the The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011 arguments, not on the panellists´ external role or background organization. In argumentative policy Delphi the panels tend to be smaller (a recommended panel size is from 15 to 30), and as a consequence, the selection of participants is even more important. The panel was formed based on recommendations from identified experts (a variation of snowball sampling). An expertise matrix (Table 1.) was used to make sure all the relevant aspects of expertise were covered. In addition to ensuring the needed expertise, also age and sex were controlled as background variables (Table 2.). The composition of the expert panel had some changes in the second round (some of the experts did not complete both rounds). The second round panel is presented as part of the Tables 1. and 2.
Table 1. Number of participants with specific expertise (the expert matrix). On the left side are the background organizations of the experts, which are cross-tabulated with the different expertise areas in the study. A refers to the first Delphi round, B to the second of the Phase I. The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011 Journalistic The Interdependencies Technology Values practice production & consumption of media contents
A B A B A B A B A B
Alternative 2 2 2 2 2 2 media (political press)
Developers2 4 4 4 4
Editors 1 1 1 1
Futurists 2 1 2 1 2 1
Journalists 6 4
Regional 2 2 2 2 papers
Magazines 2 1 2 1
Media 4 3 4 3 researchers3
National 2 2 papers
New media 2 2 2 2 (prosumers, bloggers etc.)
Radio & 1 0 Television
2 The developers refer to leaders or participants of development projects within media companies,
3 The media researchers´ expertise covered the following sub-fields: brand research, cultural industries and creative economy, advertising, media economy, leadership in media corporations, social media and its developer communities, games and gaming communities. The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011
Table 2. General background variables (number of cases) by Delphi rounds (A signifies the first round, B the second).
A B
Age 20-29 1 1
Age 30-45 10 8
Age 46- 6 4
Women 8 5
Men 9 8
The futurist panel was composed by placing a description of the study and a call to participate to professional futurists´ bulletin boards, internet resources and e-mailing lists. Such included, among others, the Millennium Project mailing list, Association of Professional Futurists mailing list, Shaping Tomorrow – a social media site for futurists, Foresight.fi –portal for Finnish foresight professionals, and the internet page for the research group (Future of Media and Communications).
Altogether 13 futurists responded the questionnaire, matching the 13 respondents on the second round of the expert panel. We controlled the same general background variables as on the expert panel (summarized in table 4.). The panellists represented in total 10 nationalities (Brazil, Croatia, Dubai, Finland, Germany, Israel, Mexico, Russia, United Kingdom and United States). In addition to general background information, we asked the panelists to provide a self assessment of their expertise. On a scale of 1 to 7, where 1 is a novice and 7 expert (with approximately ten years of experience), the panelists rated their experience as follows (table 3.):
Table 3. Self assessment of expertise by the futurist panel.
1= novice 2 3 4 5 6 7= expert
Futures 1 0 0 2 5 1 4 research
Journalism 1 2 0 3 2 0 5
Media 0 2 2 3 2 1 3 business The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011
Table 4. General background variables (number of cases) in the futurist panel.
Number of respondents 12
Age 20-29 1
Age 30-45 5
Age 46- 7
Women 6
Men 7
Phase I: Creating the four images of the future
The images of the future were created based on the answers by the expert panel on the first round of the expert Delphi. The I Phase of the study took place 26.3.-1.5.2009. The Delphi was conducted as a questionnaire executed on an internet based Webropol-platform, consisting of 18 questions, of which nine were qualitative, open-ended questions and nine quantitative, grouped under five larger thematic areas (journalistic practice, the production & consumption of media contents, interdependencies, technology and values).The quantitative questions were typically of the form: “evaluate the direction of development of the following media types through the year 2020”. The answers to the quantitative questions were given on a seven point Likert-scale. All the quantitative questions included a possibility to comment them or add a missing dimension. The qualititative questions concerned more holistic issues, such as the effect of new technology on the audience relations or the values directing the journalistic profession in the year 2020.
The images of the future were constructed by grouping the answers of the respondents (cases) from the first round by hierarchical cluster analysis using the SPSS Statistics (version 17.0) program. Cluster analysis is an unsupervised machine learning algorithm where a set of observations in a high-dimensional space are organized into clusters on the basis of mutual similarity. The Furthest Neighbor method for the grouping and the normal Euclidean distance as the measure of dissimilarity (see Everitt et al. 2001) was used. Four clusters were chosen based on the hierarchical tree output of the software (Figure 1.). The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011 The groups formed by clustering acted as skeletons4 to which demonstrative details from the open- ended questions were added based on a qualitative analysis. The quotations from the answers were not marked separately, but were written as parts of the narrative. All the depictions of the future were taken directly from the answers. Researchers are responsible for qualitatively combining the quotations to the groups formed by quantitative analysis.
Figure 1. Dendrogram using Average Linkage (Between Groups) showing the formation of the clusters.
The four images of the future The following four futures images were produced. They were written in a narrative form to be used in subsequent rounds for voting and commenting. Here, the images of the future are presented by their core elements.
I: THE INFORMATION DIVIDE
Blogs, social media and other means of journalism outside the traditional media have developed to form the core of information transfer in a society.
4 The centroids by which the images of the future were formed along with the full set of questions are available from the first author. The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011 Professional journalists deliver information found in networks to their paying customers.
There is a divide between institutions able to pay for information, media literate elite and those who cannot differentiate between quality information.
All media aims mostly at forming social networks, not so much at objective knowledge transfer.
II: THE RETURN OF THE PRINT
The traditional professional information transfer system has gained from the technological advances and its position is stronger than ever.
Mediahouses operate both as global megacompanies and very local papers. The business model is close to that of the current, with subscriptions and advertising revenues forming the basis of business.
The change towards digital environments has significantly increased the interaction between journalists and their audiences. Print refers to the organizational model, not the actual printed medium. The line between the subject matter of amateur journalists and the work of professionals is rather thin, but important.
III: STRENGTH FROM THE COMMUNITY
The building of communities and the maintaining of a national viewpoint in the global operational environment are considered important tasks of media. Also regionality and locality are emphasized.
Alongside newspapers, the role of the Internet and social media as sources of information and entertainment has remained important.
The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011 Journalism is considered the watchdog for those in power, and as such watchdog journalism it aims to bring forth a multitude of viewpoints, which can then be assessed and analyzed as fairly as possible.
IV: NEW ATLANTIS
Spontaneous action, the rise in the importance of one’s own social contacts and communities, the search for meaningfulness, and the creation and sharing of subject matter related to one’s own interests are the cultural drivers of change in the 2010s.
Journalists are the maintainers of systems of collective intelligence.
The ethos of journalism is to understand the logic behind the knowledge creation that takes place in new networks, and to offer the citizens better access to the best possible information.
Phase II
The panelists were asked to pick out one of the four images of the future based on their a) preferability. b) probability, and c) avoidability. We also asked the futurist panel to comment on the four images. The expert panel completed Phase two as the second round of the Delphi, the futurist panel completed the Phase II during the time between 16.3.-30.4.2011.
Results
Both panels were asked to choose their most preferred of the four images of the future. Additionally, the respondents chose an image of the future they assessed as most probable, and an image they considered most avoidable. The results are presented in Figure 2.
Of the respondents, 8 out of 135 preferred the “Return of the print” over the others.
5 The amount of participants declined from the first round by four respondents. The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011
10
8 6 Preferred 4 Probable 2 Avoidable 0 I: THE II: THE RETURN III: STRENGTH IV: NEW INFORMATION OF THE PRINT FROM THE ATLANTIS DIVIDE COMMUNITY
Figure 2: The preferability, probability and avoidability of the four images of the future according to the expert panel.
5
4 3 Preferred 2 Probable 1 Avoidable 0 I: THE II: THE RETURN III: STRENGTH IV: NEW INFORMATION OF THE PRINT FROM THE ATLANTIS DIVIDE COMMUNITY
Figure 3: The preferability, probability and avoidability of the four images of the future according to the futurist panel (notice the different scale in comparison to the Figure 2.).
In the expert panel, the image judged most avoidable but also the most improbable was the Information Divide. In contrast, the futurist panel anticipated the Information Divide to be most probable of all the images. Despite being ranked as highly probable, nobody in the futurist panel preferred the Information Divide image over the other images as the most preferred future. The futurist panel was split in assessing the New Atlantis –image, deeming it to be at the same time the most avoidable and the most preferred of all images. In the expert panel no-one judged the New Atlantis to be the most avoidable image. The Strength from the Unity received two to three votes in all categories, in both panels. The most notable differences between the two panels were the more pronounced avoidability of the Information Divide in the expert panel, but also that the Return of the Print was judged to be the most preferred and the most probable image of the future in the expert panel. No futurist saw it as the most probable image of the future and it was also the least preferred of all images. The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011 When asked if there were elements in the images that the futurists found particularly interesting, or if they would change (add, delete or modify) something in order to make them more preferable, the futurists offered, among others, the following comments.
I: THE INFORMATION DIVIDE
- The most likely future scenario is: There is a divide between institutions able to pay for information, media literate elite and those who cannot differentiate between quality information.
- I find this future a bit scary. The fact people will rely on social networks for sources of information in particular. (...) This implies those with access to money and resources have the potential to be much better informed and potentially mislead others. (...) I also very much disagree we need more of contextual information, we are overloaded with this now. I am of the opinion we need more objective sources of journalism, not less, which is what this future brings.
- Interesting that traditional skills of inquiry and information gathering retreat into scarcity, being paid for on a consulting basis, and for certain specified clients. I've always believed that the increasing flow of information would require an adaptation of the role of information broker and interpreter.
- I agree with the point about the conditions of a journalist’s occupation. But journalists will be not only consultants and researchers. To an even greater degree their aim will be to fascinate, to entertain the audience. And this entertainment will have in part the social significance. (...) – because it is the socially significant content that is be able to attract the serious audience.
- ... I would add journalists become trends researchers, developing news that put readers ahead of time to prepare for change.
II: THE RETURN OF THE PRINT
- Survival for the global fittest and minority groups, but not mainstream. Paper will die as a dominating mass media, but slower than expected.
- I find this model almost entirely credible, but depressing. The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011 - I don´t think objectivity will have the same importance as today. Readers will require analysis and solutions for the critical and objective productions of the journalists. Objectivity will be replaced by roadpmaps.
- Journalism will not be the public institution and will become the tools for free publicators – private persons, well known persons, brands. At the same time the socially significant functions of journalism will be carried out by brand’s journalism or journalism which will be subsidized by society (including donation and “public taxes”).
- I find it interesting the human journalists need to do the summaries. I have a hunch that in 10- 20 years from now, the computers would search the relevant information and sum it for us.
III: STRENGTH FROM THE COMMUNITY
- Unfortunately many journalists more and more try to put their ideology into the news instead of facts. Intelligent readers understand this, but the huge crowd not.
- The 'community' of today and the future is no longer based on nationality (although it may be for some specific purposes).
- “The most important task of journalism is to create a sense of unity”. I find this comment very interesting. This unity is about finding common truth in perspectives I am assuming. I like the reporting paradigm of global, national, state and local. Also, with electronic publishing / internet what is the role of print news?
- This scenario is not differentiated enough from the first two. The sutlety of promoting unity is not enough to make it different, the model within it, and the role is too similar to the previous two.
- National boundaries in the new informational environment will lose their value. Language and cultural traditions will play the role of boundaries.
- Specialized journalist always loses to an expert who can write. So, there will be no need in too narrow specialization of journalists.
- Journalists will be forced to refuse a textcentrism and to operate the full range of special codes transferring the sence: video, animated infographic. Prevalent means of expression will be not a text but a synthetic content design.
IV: NEW ATLANTIS The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011 - Social networks are creating new forms of community and although some of that information transmission apes journalism, it does not replace it. News gathering and dissemination by a professionally trained individual is the hallmark of journalism irrespective of how it is delivered.
- This is the most troubling 2020 model for me. Journalists should never be attached to an organisation (NGO or otherwise). They should always be fiercely independent so that their loyalty does not lie with one individual.
- A journalist does do research, and should apply his/her skills intelligently, but this scenario of research based intellectuals would dangerously limit information gathering. Also, very few people have the skill to take difficult subject matter and make it easily understandable. This would be more so if the journalist was always surrounded by an intellectual/academic atmosphere.
- The statement about there being no division between producing and consuming. This is a game changing element and monitoring the quality of information is really key.
- The social, environmental, economic and political impacts of what this might mean and how society works is missing. The media today is a huge part of creating how we live, culture, community, etc. If the role changes what does that do to where/how we live?
- This seems to be a grass roots driven scenario, with the flow of information emanating naturally and journalists being there to shape and direct it. Information appears to be related to "doing." I don't really believe in the development of systems of collective intelligence. Wikipedia works, but how much room is there for similar collections? And how much opportunity to work on them? Wikipedia editors, I would guess, are not journalists, but like collecting knowledge for knowledge's sake, a different function altogether.
- Scenarios like this depend on journalists graduating from "broadcast" to personalized & interactive journalism. We'll get a lot of that, but there are still good reasons to broadcast.
- This image of the future is the closest to my opinion. But instead of a journalist who filters the output of collective intelligence, the collective intelligence on its own will be the filter.
Conclusions
The Delphi method was originally developed to “force consensus” among panellists about complex topics concerning the future. This view has then shifted into an understanding that more promising The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011 results about the future can be obtained when instead of seeking a final result in a form of consensus, the attention is placed on the process of argumentation and highlighting differences in opinion (e.g. Glenn 2009). Recent research from the behavioural sciences has once again opened up the question of futures expertise for discussion and further research. The study at hand offers some support to the hypothesis that expert foresight work can be enriched by adding an “enlightened outsider” –view to the problem. The combined insights from field experts and futurists regarding the development of journalism, press and media paint, albeit in part contradicting, paint an image of a future where both groups agree on the avoidability of the highly differentiated and unequal media landscape of the Information Divide -image. Interestingly, however, their opinions on the probability of this image differ dramatically: the futurist panel chose this image to be the most probable image of the future of journalism until 2020, whereas in the expert group no-one anticipated this to be the most probable future. It should be noted, though, that the votes are very evenly distributed in the futurist panel, and due to the small size of the panel no real conclusions can be drawn from these results. On the other hand, the result regarding the image Return of the Print seems symptomatic of a more general difference in opinion between the two groups. The expert panel voted this image to be both the most preferred and the most probable, whereas no- one in the futurist group assessed it to be most probable, and only one respondent thought it to be most preferable, making it the least preferred image of the future (although only one respondent in the futurist group judged it to be directly most avoidable).
Many in the futurist panel offered actively comments and improvements to the existing images. Criticism was offered mainly on the ground that the four scenarios did not differ enough from one another. Also some members of the futurist panel would have wanted to see a more comprehensive depiction of the year 2020, taking into account the political, social, economic, cultural and environmental implications the changes in journalism might bring about and / or be rooted in. Critique was also pointed by some futurist panellists at the images depicting a displeasing image of the future. Here, the researchers clearly should put more emphasis on pointing out that the images were merely depictions of possible futures, not particularly aimed at being preferable or probable. Even though the researchers paid careful attention to framing the questions as being about the journalistic function in societies in general, not in any specific country or in any specific medium, the fact that the expert panel consisted mainly of Finnish experts and the futurist panel was an international one, most likely affects the results. Also, the original images had the tendency to emphasize written media, despite not being encouraged to do so. The time difference between the two panels in this rapidly developing field may have an impact on the assessments the panellists provided.
The contradicting results in the two panels lend themselves to many readings, also to interpretations that defend the traditional way of using strictly experts as informants in the Delphi The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011 panels. However, the comments from the futurist panel clearly provided interesting angles to the images of the future. Therefore, we aim to continue exploring this approach by refining the images with the futurists´ comments for a final round, where the two panels will be mixed. For the third round we will rewrite the images of the future, and enrich them with the futurist panel´s views. We will conduct the third round as an eDelphi, in which the different panellist will be able to exchange views in real time. The task for the third round will be to elaborate on the societal impacts of each image.
Development ideas and policy impact/implications
This study at hand suggests promising new ways to approach expert foresight methods. In the future, it could be developed into a method bridging, for example, experts and political decision makers. One possible avenue for future experimenting could be for example to include a separate panel of political decision makers in Delphi studies with policy relevance. This way the Delphi could function in its original purpose as facilitating group decision making, and also stimulate conversation on the common futures. As shown in this study, separate panels create possibilities to highlight difference in opinion, and it can be argued that they create a competition setting between the two panels, which may enhance both motivation for argumentation and real discourse between opposing opinions. On the other hand, there were several cases present in this study, where the two panels genuinely agreed on the importance of key trends and drivers, making them all the more convincing results for policy concerns.
The experience from this pilot study suggests that the approach presented here could look especially promising if applied to sc. wicked problems, where problem solving often depends on the way the problem is framed, and where stakeholders often have radically different perspectives to the issue. The research points out that also topics where discontinuous change is imminent can be a motivation for such a Delphi setting. The setting exposes contrasting viewpoints and forces argumentation for one´s own position in an even more pronounced way than the one-panel Delphi. For future development of the method we suggest that a fertile development path could be to aggregate the method presented here with Integrated Assessment approach. The method presented here could feed into the mix of other relevant data and thus contribute to a more comprehensive and multi-perspective understanding of the change process.
References
Christensen, C.M. (1997): The Innovator's Dilemma: When New Technologies Cause Great Firms to Fail. Harvard Business School Press. The 4th International Seville Conference on Future-Oriented Technology Analysis (FTA)
12 & 13 May 2011 Day, L. H. (2002): Delphi Research in the Corporate Environment. In Linstone, H. A. & Turoff, M. (Eds.) The Delphi Method: Techniques and Applications. http://is.njit.edu/pubs/delphibook/index.html
Everitt, B. S.; Landau, S. & Leese, M. (2001) Cluster Analysis, 4th ed. London: Arnold.
Gordon, T. (2009): The Delphi Method. In Glenn, J. C. & Gordon, T. (eds.) Futures Research Methodology Version 3.0. Millennium Project.
Helmer, O (1967): Analysis of the Future: the Delphi Method. P-3558. Santa Monica : RAND Corp. http://www.rand.org/content/dam/rand/pubs/papers/2008/P3558.pdf
Linstone, H.A. & Turoff, M. (1975): The Delphi Method: Techniques and Applications. London: Addison-Wesley.
Linstone, M. H.A. & Turoff, M. (2010): Delphi: A brief look backward and forward, Technol. Forecast. Soc. Change (2010), http://ec.europa.eu/yourvoice/ipm/forms/dispatch1 / 3 22.11.2010 17:44doi:10.1016/j.techfore.2010.09.011
Tapio, P. (2003) Disaggregative policy Delphi. Using cluster analysis as a tool for systematic scenario formation,
Technological Forecasting and Social Change 70(1), s. 83–101.
Tapio, P. (2002): The Limits to Traffic Volume Growth. Acta Futura Fennica, No 8.Helsinki: Finnish Society for Futures Studies.
Tetlock, P. (2005): Expert Political Judgment: How Good is it? How Can we Know? Princeton University Press.