DOCSLIB.ORG

Probing Neural Language Models for Human Tacit Assumptions

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Probing Neural Language Models for Human Tacit Assumptions Probing Neural Language Models for Human Tacit Assumptions Nathaniel Weir, Adam Poliak, and Benjamin Van Durme Department of Computer Science, Johns Hopkins University fnweir,azpoliak,[email protected] Abstract Prompt Model Predictions A has fur. dog, cat, fox, ... Humans carry stereotypic tacit assumptions (STAs) (Prince, 1978), or propositional beliefs about generic concepts. Such A has fur, is big, and has claws. cat, bear, lion, ... associations are crucial for understanding natural language. A has fur, is big, has claws, has bear, wolf, cat, ... We construct a diagnostic set of word prediction prompts to teeth, is an animal, eats, is brown, evaluate whether recent neural contextualized language mod- and lives in woods. els trained on large text corpora capture STAs. Our prompts are based on human responses in a psychological study of con- Figure 1: The concept bear as a target emerging as the highest ceptual associations. We find models to be profoundly effec- ranked predictions of the neural LM ROBERTA-L (Liu et al., 2019) tive at retrieving concepts given associated properties. Our re- when prompted with conjunctions of the concept’s human-produced sults demonstrate empirical evidence that stereotypic concep- properties. tual representations are captured in neural models derived from semi-supervised linguistic exposure. Keywords: language models; deep neural networks; concept representations; norms; semantics defining properties across multiple metrics; this perfor- mance discrepancy is consistent with many other lan- Introduction guage understanding tasks (Wang et al., 2018). We also find that models associate concepts with perceptual cat- Recognizing generally accepted properties about con- egories of properties (e.g. visual) worse than with non- cepts is key to understanding natural language (Prince, perceptual ones (e.g. encyclopaedic or functional). 1978). For example, if one mentions a bear, one does We further examine whether STAs can be extracted not have to explicitly describe the animal as having teeth from the models by designing prompts akin to those or claws, or as being a predator or a threat. This phe- shown to humans in psychological studies (McRae et nomenon reflects one’s held stereotypic tacit assump- al., 2005; Devereux et al., 2014). We find significant tions (STAs), i.e. propositions commonly attributed to overlap between model and human responses, but with “classes of entities” (Prince, 1978). STAs, a form of notable differences. We provide qualitative examples in common knowledge (Walker, 1991), are salient to cog- which the models’ predictive associations differ from nitive scientists concerned with how human representa- humans’, yet are still sensible given the prompt. Such tions of knowledge and meaning manifest. results highlight the difficulty of constructing word pre- As “studies in norming responses are prone to re- diction prompts that elicit particular forms of reasoning peated responses across subjects” (Poliak et al., 2018), from models optimized purely to predict co-occurrence. cognitive scientists demonstrate empirically that hu- Unlike other work analyzing linguistic meaning cap- mans share assumptions about properties associated tured in sentence representations derived from language with concepts (McRae et al., 2005). We take these con- models (Conneau et al., 2018; Tenney et al., 2019), we ceptual assumptions as one instance of STAs and ask do not fine-tune the models to perform any task; we in- whether recent contextualized language models trained stead find that the targeted tacit assumptions “fall out” on large text corpora capture them. In other words, do purely from semi-supervised masked language model- models correctly distinguish concepts associated with a ing. Our results demonstrate that exposure to large cor- given set of properties? To answer this question, we de- pora alone, without multi-modal perceptual signals or sign fill-in-the-blank diagnostic tests (Figure 1) based task-specific training cues, may enable a model to suffi- on existing data of concepts with corresponding sets of ciently capture STAs. human-elicited properties. By tracking conceptual recall from prompts of itera- Background tively concatenated conceptual properties, we find that the popular neural language models, BERT (Devlin et Contextualized Language Models Language models al., 2019) and ROBERTA (Liu et al., 2019), capture (LMs) assign probabilities to sequences of text. They STAs. We observe that ROBERTA consistently outper- are trained on large text corpora to predict the probabil- forms BERT in correctly associating concepts with their ity of a new word based on its surrounding context. Uni- 377 ©2020 The Author(s). This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY). directional models approximate for any text sequence she does not need to say “I am a person, and people w = [w1;w2;:::wN] the factorized left-context probabil- have dogs, and dogs need to be walked, so I have to N ity p(w) = ∏i=1 p(wi j w1 :::wi−1). Recent neural bi- walk my dog!” Comprehending STAs allows for gen- directional language models estimate the probability of eralized recognition of new categorical instances, and an intermediate ‘masked out’ token given both left and facilitates learning new categories (Lupyan et al., 2007), right context; this task is colloquially “masked language as shown in early word learning by children (Hills et al., modelling” (MLM). Training in this way produces a 2009). STAs are not explicitly facts.2 Rather, they are probability model that, given input sequence w and an sufficiently probable assumptions to be associated with arbitrary vocabulary word v, predicts the distribution concepts by a majority of people. A partial inspiration p(wi = v j w1;:::wi−1;wi+1;:::wn). When neural bi- for this work was the observation by Van Durme (2010) directional LMs trained for MLM are subsequently used that the concept attributes most supported by peoples’ as contextual encoders,1 performance across a wide search engine query logs (Pasca & Van Durme, 2007) range of language understanding tasks greatly improves. were strikingly similar to examples of STAs listed by We investigate two recent neural LMs: Bi- Prince. That is, there is strong evidence that the beliefs directional Encoder Representations from Transformers people hold about particular conceptual attributes (e.g. (BERT) (Devlin et al., 2019) and Robustly optimized “countries have kings”), are reflected in the aggregation BERT approach (ROBERTA) (Liu et al., 2019). In ad- of their most frequent search terms (“what is the name dition to the MLM objective, BERT is trained with an of the king of France?”). auxiliary objective of next-sentence prediction. BERT Our goal is to determine whether contextualized lan- is trained on a book corpus and English Wikipedia. guage models exposed to large corpora encode associa- Using an identical neural architecture, ROBERTA is tions between concepts and their tacitly assumed prop- trained for purely MLM (no next-sentence prediction) erties. We develop probes that specifically test a model’s on a much larger dataset with words masked out of ability to recognize STAs. Previous works (Rubinstein larger input sequences. Performance increases ubiqui- et al., 2015; Sommerauer & Fokkens, 2018; Da & Ka- tously on standard NLU tasks when BERT is replaced sai, 2019) have tested for similar types of stereotypic with ROBERTA as an off-the-shelf contexual encoder. beliefs; they use supervised training of probing classi- fiers (Conneau et al., 2018) to identify concept/attribute Probing Language Models via Word Prediction Re- pairs. In contrast, our word prediction diagnostics find cent research employs word prediction tests to ex- that these associations fall out of semi-supervised LM plore whether contextualized language models cap- pretraining. In other words, the neural LM inducts STAs ture a range of linguistic phenomena, e.g. syn- as a byproduct of learning co-occurrence without receiv- tax (Goldberg, 2019), pragmatics, semantic roles, and ing explicit cues to do so. negation (Ettinger, 2020). These diagnostics have psy- cholinguistic origins; they draw an analogy between the Probing for Stereotypic Tacit Assumptions “fill-in-the-blank” word predictions of a pre-trained lan- Despite introducing the notion of STAs, Prince (1978) guage model and distribution of aggregated human re- provides only a few examples. We therefore draw sponses in cloze tests designed to target specific sen- from other literature to create diagnostics that evalu- tence processing phenomena. Similar tests have been ate how well a contexualized language model captures used to evaluate how well these models capture sym- the phenomenon. Semantic feature production norms, bolic reasoning (Talmor et al., 2019) and relational i.e. properties elicited from human subjects regarding facts (Petroni et al., 2019). generic concepts, fall under the category of STAs. Inter- Stereotypic Tacit Assumptions Recognizing associa- ested in determining “what people know about different tions between concepts and their defining properties is things in the world,”3 McRae et al. (2005) had human key to natural language understanding and plays “a crit- subjects list properties that they associated with individ- ical role in language both for the conventional meaning ual concepts. When many people individually attribute of utterances, and in conversational inference” (Walker, the same properties to a specific concept, collectively 1991). Tacit
Load more

Recommended publications
  • The Nature of Student Thinking and Its Implications for the Use of Learning Progressions to Inform Classroom Instruction
    The Nature of Student Thinking and Its Implications for the Use of Learning Progressions to Inform Classroom Instruction Alicia Alonzo, Michigan State University, [email protected] Andrew Elby, University of Maryland, [email protected] Abstract: Underlying much of the work on learning progressions (LPs) is a strong though often tacit assumption that student thinking is theory-like and context-independent. In this work-in-progress, we use both theoretical perspectives on the nature of novices’ knowledge and empirical evidence of the context-dependent variability of students’ reasoning in physics to question this assumption and to argue that characterizing students in terms of LP “levels” inadequately captures their understanding of force and motion. We then analyze one teacher’s use of LP-based data to reason about student thinking and instructional responses. While the teacher reasoned fluidly using LP levels, he more frequently used finer-grained knowledge elements and contextual factors to interpret student thinking; and these finer-grained interpretations led to more actionable instructional implications. Thus, while recognizing LPs as models (imperfect representations) of student thinking, we argue that their assumption of “levels” of student understanding may limit their utility for classroom decision-making. Introduction Learning progressions (LPs)—“descriptions of the successively more sophisticated ways of thinking about a topic” (National Research Council [NRC], 2007, p. 219)—have been touted as having the potential to influence educational policies and practices including standards, curriculum, instruction, and assessment (e.g., Black, Wilson, & Yao, 2011; National Assessment Governing Board, 2008; NRC, 2007, 2012). Although the NRC (2007, 2012) has tended to emphasize large-scale applications and LPs that span many years of instruction, LPs have also been promoted as tools to support teachers’ instructional decision-making (Alonzo, 2011; Black et al., 2011; Furtak, 2012).
    [Show full text]
  • Hindu Fundamentalism and Christian Response in India
    HINDU FUNDAMENTALISM AND CHRISTIAN RESPONSE IN INDIA Rev. Shadakshari T.K. (Bangalore, India and Pastoring Divyajyothi Church of the Nazarene) Introduction One of the purposes of religion, humanly speaking, is to enable people to live a responsible life. One desire is that religious people may not disturb the harmonious life; rather, they may contribute towards it. Today, religions have become a source of conflict and violence in many Asian societies. This is very evident in India where the inter-relationship among religions is breaking up. The contemporary problem in India is the question of nationalism and the issue of marginalized identities. Christians are caught between two: participation in the nationalism in the one hand and commitment to the cause of the marginalized on the other. There is an awakening of nationalism, which bears strong religious stamp, which is strongly promoted by the Hindutva ideology. At the same time there is a strong awakening of the Tribals and Dalits. In this context, the question comes to our mind: how do Christians in India serve both nationalism and marginal groups when both of them are opposing each other? I am not promising the absolute answer for the question raised. However, this article provides some clues by analyzing the historical development of religious fundamentalism and suggesting an appropriate response for Christians. Though there are several religious fundamental groups in the history of India (including Hinduism, Islam, Sikhism, and others), this article limits its study to the religious fundamentalism of Hinduism. The importance of Hindu fundamentalism lies in its very contemporary and nationalistic scope, compared to other, more regional expressions.
    [Show full text]
  • The Misconceived Assumption About Constitutional Assumptions
    Georgetown University Law Center Scholarship @ GEORGETOWN LAW 2009 The Misconceived Assumption About Constitutional Assumptions Randy E. Barnett Georgetown University Law Center, [email protected] This paper can be downloaded free of charge from: https://scholarship.law.georgetown.edu/facpub/852 http://ssrn.com/abstract=1285485 103 Nw. U. L. Rev. 615-661 (2009) This open-access article is brought to you by the Georgetown Law Library. Posted with permission of the author. Follow this and additional works at: https://scholarship.law.georgetown.edu/facpub Part of the Constitutional Law Commons THE MISCONCEIVED ASSUMPTION ABOUT CONSTITUTIONAL ASSUMPTIONS Randy E. Barnett* INTRODUCTION Whether or not they are originalists, most constitutional scholars and observers assume that the basic assumptions held at the time the Constitution was enacted or amended are relevant to ascertaining its original meaning. Some originalists use constitutional assumptions to constrain judges in their interpretations of the more abstract passages of the text. Others reject originalism precisely because they object to the outmoded assumptions that prevailed at the time of enactment, and they assume that these assumptions must shape the original meaning of the text. For example, some claim that because of widespread concern about the continued vitality of the militia, the original meaning of the Second Amendment’s right to keep and bear arms is limited to those who are in service to the militia. Others think that because many in the Thirty-Ninth Congress and elsewhere assumed the continued existence of segregated government schools or the inferiority of women, the original meaning of the Fourteenth Amendment must be consistent with segregated schools and the common law rules of coverture.
    [Show full text]
  • Personal Knowledge: Towards a Post-Critical Philosophy
    PERSONAL KNOWLEDGE PERSONAL KNOWLEDGE Towards a Post-Critical Philosophy by MICHAEL POLANYI London First published 1958 corrected edition 1962 by Routledge & Kegan Paul Ltd © 1958, 1962 Michael Polanyi Routledge is an imprint of the Taylor & Francis Group This edition published in the Taylor & Francis e-Library, 2005. “To purchase your own copy of this or any of Taylor & Francis or Routledge's collection of thousands of eBooks please go to www.eBookstore.tandf.co.uk.” All rights reserved. No part of this book may be reprinted or reproduced or utilized in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers. British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library. ISBN 0-203-44215-6 Master e-book ISBN ISBN 0-203-75039-X (Adobe e-Reader Format) ISBN 0-415-15149-X (Print Edition) To SIR THOMAS AND LADY TAYLOR PREFACE THIS is primarily an enquiry into the nature and justification of scientific knowledge. But my reconsideration of scientific knowledge leads on to a wide range of questions outside science. I start by rejecting the ideal of scientific detachment. In the exact sciences, this false ideal is perhaps harmless, for it is in fact disregarded there by scientists. But we shall see that it exercises a destructive influence in biology, psychology and sociology, and falsifies our whole outlook far beyond the domain of science. I want to establish an alternative ideal of knowledge, quite generally.
    [Show full text]
  • Sabotaging Progress: the Cultural Economy of Resentment in Late Neoliberalism William Davies
    Sabotaging Progress: The Cultural Economy of Resentment in Late Neoliberalism William Davies The contemporary populist moment appears perplexing, even irrational, from the perspective of most orthodox traditions of social science, with their tacit assumption of utilitarian psychology and government. Whether allied to the Left or the Right, rational-choice paradigms seek to interpret individual and collective activity in terms of the pursuit and maximization of tangible interests. It’s not clear how Brexit, nativism, and “Trumpism” advance anyone’s interests, other than the financial interests and media careers of those who are personally invested in the new political formations. Centrist technocrats, such as those left over from the “Third Way” era of the late 1990s, make similar allegations about the populists of the Left, such as Corbyn, Mélenchon, and Podemos, pointing to Syriza’s capitulation to the Troika as evidence of the naiveté of these new political forces. There are two riddles in particular that need to be solved, but which defeat a rational-choice perspective. First, how to understand the apparent absence of any macroeconomic logic of the new political forces, which seem not only to have broken free of any model of accumulation, growth, or progress, but often to be sabotaging those very notions. If events such as Brexit do the harm that many economists expect, then surely people must be laboring under some kind of false consciousness or have been victims of propaganda (with or without the intervention of the Kremlin). Is it possible to consciously and reasonably seek economic regress? Secondly, how to understand the power and popularity of the super-rich within nativist movements, combined with extreme antipathy towards “liberal elites” such as the “mainstream media,” “mainstream politicians,” and universities.
    [Show full text]
  • Assumptions to Unravel the Tangle of Complexity, Values, and Cultural Responsiveness
    Working with Assumptions to Unravel the Tangle of Complexity, Values, and Cultural Responsiveness Session 1: How can assumptions be recognized? Apollo Nkwake Katrina Bledsoe Jonny Morell Presenters Jonathan Morell, Ph.D. Katrina L. Bledsoe, Ph.D. Principal, 4.669 Evaluation and Apollo M. Nkwake, CE. Ph.D. Research Scientist, Planning/Editor, Evaluation International Technical Education Development and Program Planning Advisor, Monitoring and Center/Principal Phone: (734) 646-8622 Evaluation, Education Consultant Katrina Email: [email protected] Development Center Bledsoe Consulting Website www.jamorell.com Email: [email protected] Email: [email protected] Blog:https://evaluationuncertai nty.com/ YouTubeTinyurl.com/morell- YouTube Session 1: How can assumptions be • Facilitators walk through scenarios that illustrate applications recognized? 1/14/19 and use tools and processes that are designed to help reveal and monitor the use of assumptions. We will help participants learn to gauge the tools' and processes' appropriateness, their strengths and limitations, and when to use which tools. • This session will cover assumptions about how programs work Session 2: The role of assumptions in the and relationships among outcomes; assumptions about program design and conduct of evaluation. 1/28/19 logic; assumptions about evaluation and assumptions about program purpose. • Discuss the intersection of cultural responsivity and values Session 3: Assumptions, Values and Cultural assumptions in program design and evaluations. Reflect on Responsiveness. 2/11/19 case studies and scenarios. Session 1 Outline • Propositions to frame the discussion • The tangle of complexity, assumptions, values and responsiveness • Tools and processes for examining assumptions • Diagnostic assumptions focused tools/processes • Prescriptive Assumptions focused tools/processes • Causal Assumptions focused tools/processes • Findings from an Assumptions survey • Principles • Next steps One program’s overelaboration is Carol Weiss – evaluator another program’s theory.
    [Show full text]
  • Three Principles Underlying the Formulation 6F Questionnaires By
    Survey Research Center February, I9h9 University of Hichigan THE ART OF ASKING WHY Three Principles Underlying The Formulation 6f Questionnaires By Paul F..Lazarsfeld Reprinted From National Marketing rteview Summer 193$ - Vol, I - He. 1. I. Ascertaining What A Question Means; The Principle of Specification,. Asking for reasons and giving answers are commonplace habits of everyday life. V-'e have all had the experience of acting under certain impulses and certain influences so many times that we are sure that our fellow men have had the same experiences and reasons for their own actions. And we are seldom disappointed if we inquire. Our respondent not only has had reasons for his actions; he usually knows, also, in which reason we might be especially interested, and it is upon this assumption that he bases his answer. If a friend explains why he had come to see me, he does not start to tell ne that he was once born, and that he moved to this city two years ago, although these, too, are reasons for his being here today. He is aware that most of these rea• sons are known as well to me as to him, and he picks out the reason which he hopes will contribute especially to a mutual understanding of the present situation. In market research, the question-and-answer business is not so simple, and the ease of furnishing answers in everyday life may in• volve dangerous pitfalls. In social intercourse, it is most likely that what is important 'for our respondent is important also for us who have made the inquiry.
    [Show full text]
  • Mistake and Disclosure in a Model of Two-Sided Informational Inputs
    Missouri Law Review Volume 73 Issue 3 Summer 2008 Article 2 Summer 2008 Mistake and Disclosure in a Model of Two-Sided Informational Inputs Michael J. Borden Follow this and additional works at: https://scholarship.law.missouri.edu/mlr Part of the Law Commons Recommended Citation Michael J. Borden, Mistake and Disclosure in a Model of Two-Sided Informational Inputs, 73 MO. L. REV. (2008) Available at: https://scholarship.law.missouri.edu/mlr/vol73/iss3/2 This Article is brought to you for free and open access by the Law Journals at University of Missouri School of Law Scholarship Repository. It has been accepted for inclusion in Missouri Law Review by an authorized editor of University of Missouri School of Law Scholarship Repository. For more information, please contact [email protected]. Borden: Borden: Mistake and Disclosure in a Model MISSOURI LAW REVIEW VOLUME 73 SUMMER 2008 NUMBER 3 Mistake and Disclosure in a Model of Two- Sided Informational Inputs Michael J Borden* I. INTRODUCTION ............................................................................................ 667 II. CURRENT LAW AND COMMENTARY ............................................................ 671 A . The D octrine........................................................................................ 671 B. The Comm entary ................................................................................. 675 1. K ronm an's A pproach .................................................................... 676 2. Levm ore's Contribution ...............................................................
    [Show full text]
  • Jung, Christianity, and Buddhism
    Jung, Christianity, and Buddhism James W. Heisig An abbreviated version of this paper was presented at a conference on ‘“Buddhism and Depth Psychology: Japan and America,” held in Kyoto in May 1999. The fuller paper was prepared for presentation in Japanese at the 1999 annual meeting of the Japan Society for Buddhist-Christian Studies. Throughout the past generation of Christianity’s encounter with Buddhism, the role played by Jungian psychology has been ancillary at best. True, Jung’s ideas are cited with a certain regularity, but for some reason the systematic body of thought he left behind has not attracted Christians or Buddhists as a common forum for mutual criticism and enrichment. In this essay I would like to draw attention to what I see as an unnecessary closure in Jung’s idea of the psyche and to suggest how its opening could nudge Christianity and Buddhism into closer contact with each other and with the shifts that have taken place in the general spirituality of our age. The body of writings Jung left behind is at once forbidding and fascinating. The sheer volume of his output, which continues to grow as notes from his seminars and other unpublished material are added, makes it more and more unlikely that any but a small coterie of devoted specialists will take the time to become familiar with the whole. Even so, the academic background the specialist needs to read with understanding all but ensures that large blocks of his work will simply be skimmed over uncritically. For the same reason, general readers, in which group one must include the greater part of Jungian analysts, commentators, and critics, tend to dip into his writings randomly according to their needs and to rely on secondary sources for general outlines of his thought.
    [Show full text]
  • Incredible Utility 1
    Incredible utility 1 Incredible utility: The lost causes and causal debris of psychological science John E. Richters Rockville, Maryland Direct correspondence to: John E. Richters 13200 Valley Drive Rockville, MD 20850 [email protected] Incredible utility 2 Abstract .......................................................................................................................................................................................... 3 The incredible utility of perpetual motion ................................................................................................................................ 4 Incredible utility thesis ................................................................................................................................................................ 4 The plumbing and wiring of individual differences methodology ........................................................................................ 6 Table 1. .............................................................................................................................................................................. 7 Individual-level theoretical models ................................................................................................................................. 11 Aggregate-level empirical models .................................................................................................................................. 12 The inferential passage from aggregates to individuals ..............................................................................................
    [Show full text]
  • Insiders and Outsiders in the Sociology of Science, Or How Can We Foster
    INSIDERS& OUTSIDERSIN THE SOCIOLOGYO F SCIENCEO; R, HOW CAN WE FOSTER AGNOSTICISM? Bruno Latour Volumesh ave been written evaluatingt he prospectsf or developmento f social scientificm ethodologya s rigorousa s that of the natural sciences. They ask, should social scientists attempt to experiencet he "lived world" of the peoplet hey are studying?O r, in the pursuito f objectivity, shoulds ocial scientistsa ttempt to distancet hemselvesa s much as possi- ble from their subjects?C an sociologistsc onfidentlye xpectt he eventual appearanceo f a Newton of the social world? Or should they ignore the canon of the hard sciencesa nd formulate new forms of measurement, deficientb y the standardso f natural scientifici nquiry but suited to the distinctivec haracteristicso f the socialw orld? However interestingt hese Knowledgea nd Society:S tudiesi n the Sociologyo f Culture Past and Present,v olume 3, pages1 9-216. Copyright @ f98f by JAI PressI nc. All rights of reproduction in any forrn reserved. ISBN: 0-89232-161-X 2OO BRUNOL ATOL. questionsm ay be from the standpointo f most social scientists,t hey arc peculiarly inappropriatef or those sociologistsw ho are concernedt o er- plain the conduct of sciencei tself. All methodologicala rgumentsi n the social sciencesa re basedo r onc' tacit assumption:t hat scientifica ctivity is distinct from all other forms oi activity.t All methodologicadl isputesc an be viewed as argumentsa bour the place that social scientific method occupies on the objective- subjectivec ontinuum.2M ost of the debatesc oncern the direction the continuums ocialr esearchs houldt ake. But whethert he conclusioni s that more objectivity or more subjectivityi s neededi n socials cientifici nquiry.
    [Show full text]
  • How Evaluative Informetrics Relates to Scientific, Socio-Historical, Political, Ethical and Personal Values
    Moed, H. F. (2020). How Evaluative Informetrics Relates to Scientific, Socio-Historical, Political, Ethical and Personal Values. Scholarly Assessment Reports, 2(1): 9. DOI: https://doi.org/10.29024/sar.18 RESEARCH How Evaluative Informetrics Relates to Scientific, Socio-Historical, Political, Ethical and Personal Values Henk F. Moed Sapienza University of Rome, IT [email protected] Policy highlights: • Ifevaluation isdefinedas“asystematicdeterminationofasubject’smerit,worthandsignificance, usingcriteriagovernedbyasetofstandards”,thereis noevaluationwithoutanevaluativeframework specifyingthesecriteriaandstandards. • On the other hand, evaluative informetrics itself, defined as the study of evaluative aspects of scienceandscholarshipusingcitationanalysis,altmetricsandotherindicators,doesnot evaluate. • However,informetricindicatorsareoftenusedinresearchassessmentprocesses.Toobtainabetter understandingoftheirrole,thelinksbetweenevaluativeinformetricsand‘values’areinvestigated, andaseriesofpracticalguidelinesareproposed. • Informetriciansshouldmaintainintheirevaluativeinformetricstudiesaneutralpositiontowardthe policyissuesaddressedandthecriteriaspecifiedinanevaluativeframework. • Asprofessionalexperts,informetricians’competenceliesprimarilyinthedevelopmentandapplication ofanalyticalmodelswithinthecontextofagivenevaluativeframework. • Informetric researchers could propose that evaluators and policy makers incorporate fundamental scientificvaluessuchasopennessandadoptingacriticalattitudeinassessmentprocesses. • Informetricians could also
    [Show full text]