DECEMBER:2018

research matters

Four decades of social change By Alison Park, director, CLOSER, UCL Institute of Education he SRA began 40 years ago, in 1978. The UK’s population TJames Callaghan was Prime Minister of a Labour government entering its final year The UK’s population has grown by some ten million since 1978, in power. That year, the first IVF baby, Louise from 56 million to 66 million. We’re getting older too – the Brown, was born and Viv Anderson became proportion of the population aged 65 and over has increased England’s first black international footballer. from 14% in 1976 to 18% in 2016 (ONS, 2017). We are also The year before had seen the launch of the more ethnically diverse. In the 1991 census (the first one to Apple 2, the first easy-to-use ‘microcomputer’, include a question about ethnicity), 94% identified themselves selling for $1,298 (a mere $4,650 in today’s money). as ‘white’, compared with 86% in 2011 (ONS, 2012). These examples give a sense of some of the many ways in Employment which the UK has changed over the 40 years since the SRA The decline of manufacturing, and the rise of the service sector, began. But what can we learn from more systematic sources began a long time before 1978, but the last 40 years have seen of data? This is the challenge tackled by this article, which this seemingly inexorable trend continue. In 1981, 23% of the uses as evidence some of the key surveys and other datasets labour force in England and Wales worked in the manufacturing that so many SRA members have helped produce and analyse sector; now it’s 9%. The service sector was strong even then over the years. (60%) but now accounts for 81% of jobs (ONS, 2016).

INSIDE: Research ethics in the SRA after 40 years • Standing on the shoulders of giants • From the archives: changing technology and social research • Qualitative research: a magpie discipline • Evaluation: past, present and future • The changing landscape of social surveys: 1978-2018… • Data science, big data and ONS • Research and government: 40 years on • How much do you know about the SRA? • Gains and losses in social research • Plus news, reviews and briefings Number of degrees awarded in UK 1980-2011 FOUR DECADES OF SOCIAL CHANGE 400,000

300,000

200,000 These changes partly explain the changing composition of the labour market, with a 15-percentage-point increase 100,000 since 1978 in the proportion of adult women in employment, 0 reaching 71% in 2018 (ONS, 2018). 1980 1990 2000 2005 2010 2011

Family life and relationships First degrees Higher degrees There have been huge changes in family formation over the decades, most notably in family size and the proportion of people having no children. Just over half of children (53% in Politics 2013) are born to married parents, with nearly a third (31%) born Just over three quarters of the electorate (76%) voted in the to cohabiting couples, up threefold since 1986 (ONS, 2014). 1979 general election. Since then, considerable concern has been expressed about falling levels of turnout, with just 59% Our views about marriage have changed too. In 1983, the first of the electorate voting in 2001, the lowest rate in the post- of NatCen’s British Social Attitudes surveys found that 42% war period. However, turnout has gradually risen again since, of people thought sex before marriage was ‘not wrong at all’, reaching 69% in 2017. compared with 75% in 2016. But the magnitude of this shift is dwarfed by changing views about same-sex relationships. Levels of political interest are higher now than they were in In 1983, 17% thought that sex between two adults of the 1986, according to the British Social Attitudes survey. Then, same sex was ‘not wrong at all’, increasing to 64% by 2016 29% described themselves as having a ‘great deal’ or ‘quite a (Swales and Attar Taylor, 2017). lot’ of interest in politics, compared with 42% in 2016. But we are less likely now to identify with a political party (although Gender roles most of us, if pushed, still do); in 1986, 8% did not identify with any party, rising to 14% by 2016. Increasingly liberal views are also apparent about gender roles. In 1984, 37% disagreed with the view that ‘a man’s job is to earn money; a woman’s job is to look after the home and Europe family’; now 72% do (Attar Taylor and Scott, 2018). Studies like the British Social Attitudes survey have shown that Euroscepticism has been the mood of the majority for Religion over two decades. Until 1996, no more than 40% wanted either to see the UK leave the European Union or for its Increasingly liberal attitudes partly reflect changing religious powers to be reduced; but from that point onwards (with a beliefs. In 1983, when the British Social Attitudes survey few exceptions) the balance tipped in favour of a Eurosceptic began, 67% described themselves as Christian, and 31% majority (Curtice and Evans, 2015). As we know, the said they did not see themselves as belonging to any religion. consequences became apparent in June 2016, when The proportion of non-believers has gradually increased over 52% voted in support of the UK leaving the EU. the decades, standing at 53% in 2016, while the proportion of Christians has fallen to 41%. Inequality Health The 1980s saw sharp increases in inequality as measured by the Gini coefficient, followed by more modest increases. Since We are living longer than we were in 1978, but increases in the ‘great recession’, inequality measured in this way has fallen, life-expectancy are now slowing, reflecting new challenges and remains at around the level it was in the early 1990s. But a to the nation’s health. These include an increase in the different pattern is apparent if we focus on income inequality at proportions of people who are overweight or obese. Recent the top of the income distribution. The income share of the top research using the UK’s birth cohort studies has found that 1% increased sharply during the 1980s, and continued to rise children born since 1990 are up to three times more likely until it peaked at 8% in 2000 (Cribb et al, 2018). than older generations to be overweight or obese by the age of ten (Johnson et al, 2015). Views about government policy Education Our views about taxation and spending have fluctuated considerably over the last four decades. In 1983, 54% There have been dramatic increases in the numbers of wanted to see taxation and spending stay as they were students going into higher education. In 1980, 68,000 first (then the top rate of income tax was 83%); in 2017 this view degrees were awarded; by 2011 this had increased fivefold to was taken by 32%, with 60%, instead, wanting to see more 351,000 (while the number of higher degrees had increased taxation and spending. But people’s top priorities for more tenfold) (Bolton, 2012). Over this period, student composition spending remain as they have always been – health and changed markedly, with the proportion of women leaving education (Morgan and Harding, 2018). university with a first degree increasing from 37% of all graduates in 1980 to 56% in 2011. Overall, while 13% of For a full list of references for the research and data sources 25- to 29-year-olds had an undergraduate or higher degree cited in this article see http://the-sra.org.uk/whats-new/ in 1993, this had tripled to 41% by 2015 (Blundell et al, 2016). research-matters/ once the December 2018 issue is published online.

2 : SOCIAL RESEARCH ASSOCIATION EDITORIAL The SRA at 40 SRA chair, David Johnson, on the importance of understanding where a profession has come from in thinking about the challenges of the future.

elcome to this her lead article, Alison Park picks up it is important to understand where a Wspecial edition the theme of social change using a profession has come from in thinking of SRA Matters variety of familiar data sources. There about the challenges of the future. celebrating 40 years of are also articles on the changes and Part of the future will be our new logo, the SRA. Just before development in research ethics, as pictured below. The trustees have he became Prime technology, government’s use of felt for a while that our website and Minister, Harold Wilson research, social surveys, data science digital presence needed an overhaul. is supposed to have and evaluation. Part of the work we are taking forward coined the phrase ‘a week is a SRA chief executive, Graham Farrant, in this space is to refresh our brand and long time in politics’, and so it is hardly reflects on the SRA’s achievements, visual identify. The new logo has been surprising that 40 years have seen and Dan Clay, in mid-life crisis, looks the subject of extensive discussion by huge social and political change. In the at qualitative research as a discipline. the trustees, and I am grateful for the year the SRA was founded, the ‘winter Elsewhere in this issue, look out for time, effort and thought people have of discontent’ was just around the people’s reflections on what they miss/ put in to this area of our work. We hope corner, the UK had not yet had its first don’t miss about how social research you like it and you can expect to see female Prime Minister, the Falklands has changed and see how much you it in various guises over the coming War hadn’t happened, the Berlin Wall remember about the SRA by doing the months. still stood, the term New Labour, first quiz on page 12. used as a conference slogan in 1994, Anyway, that’s all from me for this issue. was still 16 years away, and the internet For some of you, I guess this issue will Happy reading and I hope to see many hadn’t been invented. Makes you think stir feelings of nostalgia, and for others of you at the December conference. doesn’t it? it will all seem a bit like ancient history. I Don’t forget about the AGM directly think though, as I was reflecting on in a beforehand when we’ll present highlights This issue of SRA Matters takes a talk I gave recently to HMRC colleagues from the annual report and seek your look back over the last 40 years about the history of social research, approval to appoint new trustees. from a variety of perspectives. In

New colours for 40th year Social Research Practice The current SRA logo has been in use for many Issue 7 is due in January 2019 years, although exactly when it started is lost in The overall aim of the journal is to encourage and the mists of time. We hear sporadically from SRA promote high standards of social research for public members that the logo has been looking a bit benefit. It promotes openness and discussion of ‘tired’, and the SRA trustees felt the same. The 40th problems. It is free to download at: www.the-sra.org.uk/ anniversary was too good an opportunity to miss, journal-social-research-practice and after due process the graphics coop has come up with a new logo. Here it is. We hope you like it! We welcome offers of articles and research notes for future issues. Read the guidelines for authors and download the article template at the link above. If you have an idea for an article or research note but are not sure if it’s suitable, please email Richard Bartholomew, the editor: rabartholomew@ btinternet.com

RESEARCH MATTERS: DECEMBER 2018: 3 ETHICS Research ethics in the SRA after 40 years By Jane Evans, SRA trustee (incoming); Ron Iphofen, independent consultant; and Helen Kara, SRA trustee (outgoing)

t should be of no surprise that ethics Feedback was constructive, and Iand professional integrity formed part resulted in a 2003 publication which of the SRA’s mission from its foundation. has been commended and shared practice across all sectors of social In the 1980s, political developments widely throughout the social research research. This has become increasingly were seen as a threat to independent community. During this period, the SRA linked to accurate, evidenced-based social science, so researchers became a key partner in a European policymaking and impact evaluation. from government, academic and Commission project on socio-economic Developments in research methods, independent sectors joined together research ethics (RESPECT); advised such as research using social media; to ensure an ethos of transparent, the UK Research Integrity Office on arts-based research; participatory collaborative and collegial relationships incorporating social science as part methods; and the use of video, bring that could transcend sectoral location of its brief; and participated in the new ethical challenges. Society is and methodological inclinations. This Association for becoming more diverse, diversity has been part of the SRA’s Research Ethics with more awareness of strength but it can often be tested when (AREC) activities. Debate continues issues such as multiple conflicts of interest and outlook arise. Following about the right ways to categories of gender The SRA’s work the updated ensure compliance with and family. Researchers guidelines there good ethical practice are realising that on ethics were intermittent research ethics does across all sectors of The original SRA ethics guidelines were requests for not exist in isolation, drawn up by Roger Jowell, a founding guidance and social research but is inextricably linked member of the SRA, in the early years. advice on ethical with individual, social, They were always intended to be matters. So, we professional, institutional educational rather than rules-based. established the Ethics Advisory Forum and political ethics. This produces They were drafted with new researchers which Ron wrote about in the last issue a complex picture for the future of in mind and to pre-empt some of the of Research Matters. This is a free, research ethics, with not only dilemmas dilemmas they would inevitably confront. confidential service for members to but also trilemmas and quadrilemmas As such, the guide was a distillation of put their ethical dilemmas to a group becoming commonplace. We are the better part of current practice, and of knowledgeable volunteers. The increasingly aware that research ethics both learned from, and informed, similar guidelines are now being updated involves far more questions than guidelines and codes being adopted by again to consider methodological answers. Yet it is essential to continue other professional associations. developments such as the expanding raising those questions and having the role of technology in research; the new debates they provoke. By 2000 there was clear need for version should be published in 2019. an update, and a working group While there may be innovative methods of the more experienced members Looking ahead and new topics of research, many of accomplished this, again drawing the fundamentals of research ethics Ongoing legislative changes, such as from different research sectors. The remain the same as 40 years ago: the GDPR and concerns about litigation, group spent 18 months restructuring confidentiality, consent and comfort of have increased concern about ethical the guidelines, focusing on how research participants remain the key social research practice among researchers addressed emerging concerns. practitioners, commissioners and funders. concerns in society. A draft of the new guidelines was circulated across the Debate continues about the right ways UK and internationally for comment. to ensure compliance with good ethical

4 : SOCIAL RESEARCH ASSOCIATION PEOPLE Standing on the shoulders of giants By Graham Farrant, SRA chief executive

significant The SRA’s representation of These channels of communication may Aanniversary allows researchers is seen in responses look effortless from the outside, but a us to reflect on the to consultations on key issues lot of work goes on behind the screen fundamentals. The affecting research; and less visibly, to identify, gather and collate material. purpose of the SRA in committee work by volunteers. Again, there are dozens of individuals is described on our For example, a longstanding group who have contributed over the years, website: ‘promoting focuses on improving the way exemplified by Gillian Smith, a volunteer high-quality standards research is commissioned. They who has tirelessly and single-handedly of social research, we seek to have written guidance for clients on produced the SRA e-newsletter every represent, support, connect and inform procuring research to get the best from fortnight for the past seven years. our members and the wider social suppliers, and are currently involved in Finally, the SRA continues to benefit research community’. providing robust feedback on the new from trustees of exceptional calibre. On government-wide electronic framework Training has a major role both a personal note, I must thank Ceridwen system. Deborah Harding of MRS in upholding high standards and Roberts, a founder member active in continues to be a source of support to supporting researchers. We’re many spheres, and Graham Hughes, the group; among many volunteers, SRA very fortunate that the essential treasurer for six years from 2011. founder member Janet Lewis deserves introductions to qualitative, quantitative When I joined the SRA in 2012 they a special mention for guiding the work and evaluation methods are taught were effectively running the SRA office of this committee over many years. by a cohort of experts, committed to between them, as volunteers, and since passing on their skills and standards Face-to-face events are a good way then they have never ceased to be a to new generations. It is possibly unfair to connect people, and most SRA source of support and wise advice. to single out Liz Spencer and Pam seminars, talks and events include If the SRA can continue to attract such Campanelli here, but what a beneficial time to chat and network. Events outstanding people, it should have a influence their courses have been, for arranged by the branch committees very healthy future! hundreds, maybe thousands of social in Wales, Scotland and Ireland are researchers! ideal for encouraging connections at a local level, but of course they need Providing guidance is a broader way volunteers to step forward. A good to enable researchers to maintain high recent example is SRA North, where standards. As Ron Iphofen describes members in the north of England elsewhere in this issue, the SRA has have come together to run events always been known for offering sound with local speakers. The trustees practical advice on research ethics, are keen to encourage this kind starting with Roger Jowell’s ground- of local activity, so do come breaking work, and then the 2003 forward if you have a plan. guide which is still widely used. Thanks are due to Ron for his dedicated The SRA e-newsletter, support to the SRA over 40 years, and magazine, journal and to the teams of volunteers with whose Twitter account all do their help the guidelines are written. bit to inform members and others about what’s As well as ethics, there are guides happening in our covering data protection, research profession. commissioning, and safety for researchers in the field.

RESEARCH MATTERS: DECEMBER 2018: 5 SRA ARCHIVE From the archives: changing technology and social research By Annika Coughlin, PhD student at UCL Institute of Education

he SRA archive have such luxuries. An advert in the Do you remember the Social Science Tis a modest box SRA newsletter from October 1982 Information Gateway (SOSIG)? It was containing paper stated that for £485+VAT educational a service promoted in SRA newsletters copies of Research institutions could buy in the 1990s that aimed to provide Matters (formerly a videotape course called trustworthy social science SRA News) from the ‘introducing SPSS’. The information for students, 1970s, 80s, and 90s. course consisted of seven, academics and practitioners in Later issues are online at 25-minute videotapes and social sciences, business and http://the-sra.org.uk/whats- accompanying materials law. What a huge task that would new/research-matters/. I was looking produced by the British be today with billions of websites for recurring themes and discovered Universities Film Council. and resources to sift through. that technology has been an important It is difficult to imagine The growth of web-based social topic across the decades (as in this how researchers did their surveys in the 2000s prompted anniversary issue), particularly in the work without a personal the question ‘are web-based following areas: computer, since today we use these surveys the survey method of the ◗◗ Collecting and analysing data for almost every aspect of future?’ to be asked at the Cathie the research process. In Marsh Memorial Lecture in November ◗◗ Disseminating 1984, the SRA organised 2008. By 2013, 83% of UK households research six one-day events on ‘the had internet access but this did not ◗◗ As a research topic use of micro computers mean there was an easily accessible in social research’. These population waiting to be surveyed. ◗◗ For collaborating and events were ‘designed to be Issues about ethics of online research finding resources consumed in one-day bites prompted a conference in 2014 where ◗◗ For archiving data (bytes!) or even better, in the key message coming from the conference report by Helen Kara was ◗◗ Research ethics totality’, said Geoff Payne, chair of the SRA committee (May that ‘every researcher working online is Conference notices, advertisements 1984). The course covered: sampling, on a steep learning curve’ (June 2014). and articles published in the past 40 questionnaire construction, fieldwork, Seeing how researchers used to do years provide a glimpse into life as a administration, coding, editing, data their day-to-day work makes me realise social researcher and how technology management, analysis and report that each generation of researchers has has changed how we work. production. experienced technological change that In 1979, researchers new to recording Today, researchers can download, has transformed their research lives. interviews could get together at a immediately and for free, digitised So, I leave you with two questions: one-day SRA conference simply quantitative and qualitative archives and called ‘tape recorded interviews’. Its datasets via the UKDataArchive http:// ◗◗ What aspect of technological aim was ‘to provide an introduction data-archive.ac.uk/. In an advert development are you most grateful to the research method, rather than in SRA News in July 1986, the Data for as a researcher? a sophisticated discussion for those Archive was letting new owners of a ◗◗ What technology do you wish already experienced in their use’ microcomputer know about its floppy someone would invent to help (SRA News, March 1979). disks containing data files from the you as a researcher? Today when we get confused or archives. Some researchers today may Let us know on: frustrated using equipment or software, never have even seen a floppy disk, http://twitter.com/thesraorg we search YouTube for immediate and except as the ‘save’ icon. free advice from around the world. The late 1990s was when the internet How much do you know about the Researchers in the 1980s did not and email came into most of our lives. SRA? Take the quiz on page 12

6 : SOCIAL RESEARCH ASSOCIATION QUALITATIVE METHODS Qualitative research: a magpie discipline By Dan Clay, head of qualitative, Kantar Public

ith the SRA ten years, mobile apps have enabled impactful, research. One of these Whitting 40 there us to engage people ‘in the moment’. trends has been the tightening of is no time like the This helps reduce recall bias or policy-research procurement processes present to have a post-rationalisation, and enables us allowing for less dialogue between symbolic mid-life crisis to follow-up with participants almost clients and research agencies. and take stock of how instantly: we can now be far more Another has been the trend towards qualitative research present in people’s lives than ever covering a greater range of complex has evolved over the before. Deborah Mattinson highlighted issues in depth within tighter resource past four decades, as well as the value that online communities constraints. This latter trend presents a speculate where things might be going can bring in creatively supporting real challenge for researchers – needing in future. It would be impossible for conversations and engaging with to provide clients with the reassurance me to write this without engaging with dispersed communities – like young that objectives will be covered, but also peers across the industry so a massive men with prostate cancer – in an embracing the qualitative principles of thank you to everyone who contributed influential and cost-effective way. openness and flexibility that lead to the including Sarah Castell (Ipsos MORI), most insightful outputs. Alongside this increasing proximity to Debbie Lee Chan (Which?), Lucy Joyce those being researched, there has been (Kantar Public), Deborah Mattinson an important shift in how we see our Looking forward (Britain Thinks), Karen O’Reilly relationship with research participants. Looking forward, the focus is on (Goldsmiths, University of London), Both Sarah Castell and Karen O’Reilly the role that technology will play in and Caroline Turley (NatCen). noted the progression from ‘subject’ helping researchers enter ever more or ‘respondent’ to ‘participant’, deeply into people’s lives and, when Looking back signalling a change from a more top- appropriate, bringing disparate groups Looking back over the past 40 years, down, extractive engagement to one together to collaborate within virtual one clear trend has been the continuing where we are increasingly reflexive, environments. As the magpies of the expansion of the portfolio of tools consciously immersing ourselves in research world, it is inevitable that we and methods available to qualitative the lived experience of individuals. This will continue to innovate and look to research – aptly described as a aligns with a trend towards creating leverage technology to deliver better ‘magpie discipline’ by Sarah Castell. inclusive research environments that quality evidence and insight more We now have more opportunities than actively engage and inform participants effectively. However, the consensus ever to draw from across psychology, about the issues that affect them. This view is that artificial intelligence, for all anthropology, sociology, data sciences includes a growth in co-creative and its promise, is not going to replace the and more, using everything from deliberative approaches, empowering intuitive, empathetic and artistic role ethnography, semiotics and social participants with tools and information, of the researcher in listening to, and media analytics through to role-play, respecting their ability to step outside interpreting, what people tell us. art and poetry as means to understand their own circumstances in shaping From my own perspective, I’d say views and behaviours. policies, services and interventions. the past 40 years have seen a step- One clear facilitator of change has Of course, not all changes have been change not just in the range of ways we been the development of mobile for the better. While the portfolio of collect data but in how we collaborate technologies that allow us to options open to qualitative researchers across diverse disciplines to develop gain more real-time insights into is now broader than ever, there were insight in the movement towards more behaviours. Debbie Lee Chan reflected interesting reflections from Lucy impactful dissemination of research that, in the past, researchers have Joyce and Deborah Mattinson on findings. In the future, I can see this used paper diaries to help research countervailing pressures resulting trend continuing with tools, teams and participants record thoughts, feelings in more cautious, and arguably less outputs increasingly tailored to meet the and behaviours but, in the past five to specific needs of clients.

RESEARCH MATTERS: DECEMBER 2018: 7 EVALUATION METHODS Evaluation: past, present and future By Kelly Beaver, managing director, Social Research Institute, Ipsos MORI (and professional evaluator)

he Great Society paper1 in 2012; the NAO review of and sustained, and, more generally, TExperiments of evaluation in government in 20132; enhance the understanding of ‘whom the 1960s in America and the late Jeremy Heywood’s strong interventions work for’. With the recent marked the start advocation for evidence in policymaking launch of the five Office for National of the evaluation and his launch of the What Works Statistics data centres3 for areas of profession but it was Network in 2013. significant policy interest, this element not until the 1980s of the evaluator’s toolkit is likely to More recently, there has been further that the UK actively develop further, as will the ethics and exploration of evaluation approaches followed suit. Back then, the protocols for the use of such data. that could provide robust and reliable social policy evaluator’s approaches truths for policymakers, when RCTs So, what does this mean for the future and techniques were limited to and quasi-experimental approaches of evaluation and the profession itself? literature reviews, a little management are not feasible. For example, the While government is demanding ever information and a lot of primary data aptly named Centre for Evaluation greater levels of sophistication and collection. That was that. Fast forward Complexity across the Nexus (CECAN) quality in its evaluation studies (both 30-odd years and evaluation has was established by several government experimental and non-experimental diversified greatly, and the industry departments and agencies in 2016 to approaches), there has been no has professionalised all the while. that end. In contrast to the more widely tangible increase in willingness to I would argue that there are three established RCT or quasi-experimental pay for the greater skill requirements noteworthy trends that have emerged approaches, the Centre explores lesser- this entails. In future we may find it from the UK evaluation scene over the known and largely unproven evaluation increasingly challenging to recruit past 20 years: approaches: contribution analysis and talent into the profession, and retain realist approaches, for example. The it. And with the ongoing increase ◗◗ The rise in experimental approaches use of non-experimental yet robustly in What Works activity, access to to assess impact applied evaluation approaches has improving administrative data and the ◗◗ The Government’s increasing increased throughout government, and like, the role of evaluators is solidifying recognition of non-experimental the long-awaited revised edition of the as independent arbitrators of a approaches Magenta Book is likely to include a far progressively complex patchwork wider range of evaluation approaches, of curated evidence. ◗◗ The increased availability of ‘other research methods and analytical data’ with which evaluators may So, while an independent perspective, approaches than ever before. work good stakeholder management and Arguably, some of the greatest recent effective problem-solving skills are Over the last ten years, the evaluation advancements in evaluation have been requirements for any evaluator, the community has participated in an enabled by improved access to, and evaluator’s role in the production of increasingly polarising debate – to quality of, administrative datasets. evidence will be further refined and randomise or not to randomise. The Evaluators now make extensive use limited to populating gaps in existing push for Randomised Controlled Trials of administrative data, social media evidence, and there will be yet more (RCTs) and quasi-experimental evidence data and other contextual information on the quality of thought underpinning resulted from Cabinet Office and to support the construction of any evaluation framework, and on the National Audit Office (NAO) interjections: counterfactuals, understand the extent analytical skills to assess and arbitrate the spin out of the Behavioural Insights to which outcomes are achieved the evidence. Team in 2010; the Test Learn Adapt

1 https://www.gov.uk/government/publications/test-learn-adapt-developing-public-policy-with-randomised-controlled-trials 2 https://www.nao.org.uk/wp-content/uploads/2013/12/10331-001-Evaluation-in-government_NEW.pdf 3 https://blog.ons.gov.uk/author/liz-mckeown/

8 : SOCIAL RESEARCH ASSOCIATION SURVEY METHODS The changing landscape of social surveys: 1978-2018… By Patten Smith, director of research methods, Ipsos MORI and Peter Cornick, research director, NatCen Social Research

he SRA was founded in 1978 to Unfortunately, recent rises in Tprovide a membership body for the mobile phone use, in call-screening growing discipline of social research. At technologies and falling landline use the time, there was a rapid increase in have rendered them largely infeasible high-quality social surveys – examples again. Telephone methods are now include the British Cohort Study 1970, rarely used for single-mode high-quality This ‘mobile-first’ approach is making the General Household Survey (1971- general population surveys. researchers re-think the conventions 2007), the National Travel Survey of questionnaire design. There was also a rapid rise in internet (since 1965) and the Crime Survey surveys, often for general population for England and Wales (since 1982). surveys. However in the UK we lack Developments in Typically, these surveys collected data good sample frames with online contact methodology to inform government policy and social details, so most online surveys used science, and were largely delivered by Methodological innovation over the volunteer panels instead of high-quality specialist survey organisations in the past 40 years has been considerable. random probability samples. These public, charitable and private sectors. In The Total Survey Error (TSE) framework were cheap but unusable for high- this, little has changed over the last 40 now attunes researchers to errors at all quality social research because they years, but in other respects the social stages of survey workflows. Collecting could not reliably deliver accurate data. survey landscape has changed hugely. process paradata allows us to make Recently there have been efforts to real-time adaptations to data-collection CAI and mode changes collect online data from genuine random during fieldwork, using responsive design principles. During the first half of the SRA’s probability samples, using ‘web-push’ existence, the most high-quality social methods. These involve recruiting Research on non-response bias surveys were conducted face-to-face by random probability samples using offline indicates that response rate is often professional interviewers, although some methods (for example mail, or face-to- weakly associated with estimate used postal administration. In the 1990s face) followed by online data collection. quality, allowing consideration of new traditional surveys became harder to Surveys such as the Sport England methodologies for high-quality surveys. administer: response rates began to Active Lives Survey send letters to fall, with greater effort (and expenditure) large random address samples to The future persuade recipients to complete needed to resist the decline. Social surveys have evolved significantly online questionnaires. The NatCen since 1978, reflecting social, This was a spur to methodological online panel repeatedly contacts a technological and methodological innovation at a time when new panel recruited through a pre-existing developments. In the ‘big data’ era, opportunities were emerging. face-to-face survey. Good web-push there is a growing aspiration for greater surveys mix modes: following up non- Computer-assisted interviewing (CAI) integration of administrative and survey responders using a second, offline, began to replace paper questionnaires data. Researchers are increasingly mode to reduce well-documented for both face-to-face and telephone looking at how administrative data biases found in online-only samples. surveys. This led to improved data can answer some of the questions quality because it eliminated routing Web-push surveys deliver considerably previously answered by surveys, and errors, and allowed edit anomalies to lower response rates than the face- this trend will undoubtedly continue. be resolved during interviews, and the to-face surveys they replace, but There will, however, always be a need data-entry stage to be omitted. lower levels of error than non-random for data that can be obtained only by alternatives. In the early 2000s, effective random asking people direct questions, and probability telephone surveys Questionnaires are being radically high-quality surveys will, undoubtedly, methods using random digit dialling simplified and shortened to facilitate remain in demand. (RDD) became technically feasible. completion on mobile phones.

RESEARCH MATTERS: DECEMBER 2018: 9 BIG DATA Data science, big data and ONS By Dr Tom Smith, managing director, ONS Data Science Campus he raw material Image data for analysis Alongside novel big data sources, Tavailable to it is important to remind ourselves researchers interested Comprehensive high-resolution global of the huge potential of administrative in the economy and satellite imagery (such as that on data. Securely linking microdata from society is changing. Google Earth) is now available to individual or business administrative Alongside survey complement the longstanding openly records is a major ONS goal, to data, ‘administrative’ available Landsat data. This ‘earth enable more detailed, timely and sources of data observation’ data is being used in a local analysis, over time, of social and generated by public services variety of ways by statistics offices and economic issues for research purposes. such as the taxes and benefits systems social researchers. Statistics Canada Internationally, statistics offices such as are increasingly available to and used is using Landsat satellite images to those in Denmark and the Netherlands by researchers around the world. The identify crop types at local level, have long experience of working with growth of novel sources of data such and to produce crop statistics that linked microdata, allowing them to use as text, images and movement traces were previously dependent on surveys. additional survey and big data sources – often lumped together under the ‘big Survey organisations are starting to to target gaps in knowledge. Recent data’ umbrella – can also be highly use features identified from satellite progress by ONS has incorporated valuable. Below, I show three examples data in small area estimation models, business VAT data into our GDP of big data that the UK’s ONS and other with a similar approach taken by estimates, part of the GDP improvement statistics offices are using to better the Flowminder team to estimating programme that has enabled the move understand the economy and society. incomes at small area. to monthly estimates. Other image data sources are also These are just a few examples of Better use of text useful. The ONS Data Science the novel sources of data that are Text responses have long been a Campus ‘Urban Forest’ project is increasingly in use by researchers, staple tool for social researchers. ONS producing local environment indicators and being assessed for use by national work to automatically code text fields of greenery and vegetation in urban statistics offices. As with all data in surveys is making use of natural areas by using machine learning sources, understanding the quality of language processing (NLP) techniques techniques to classify Google Street the source is critical, as is responsible to improve survey processing. View images. and ethical use of the data for research However, NLP tools also enable very Movement data and statistics. different sources of data to be used at As our understanding of these data Movement of people and goods is scale. The Data Science Campus has sources matures, we should expect clearly a major interest for national been developing automated tools to to see more use in research and statistics offices – for understanding classify the raw manifest descriptions statistics outputs. I look forward to commuter flows, tourism, migration of what is being carried by ships and more exciting changes over the next and trade. Mobile phone data is one lorries, potentially opening-up a huge 40 years of the SRA. source of information on trade in goods potential source of information, and at national and regional level. These analysis by the ONS Big Data team has @_datasmith (https://twitter.com/_ tools can be applied in many different shown that this data correlates well with datasmith) travel-to-work data from the census. settings, with another campus project For more on the ONS Data Science A separate source of movement data analysing 90 million patent applications Campus work programme see https:// is the global Automatic Identification to identify emerging trends in new datasciencecampus.ons.gov.uk/ technologies and the ONS Big Data System (AIS) generated by shipping team using company records and GPS signals, and the campus is Links to more information on the websites to classify companies into exploring whether AIS data can different projects mentioned above will the most appropriate sector. provide early indicators for trade. be available in the online version of SRA Research Matters (December issue).

10: SOCIAL RESEARCH ASSOCIATION SOCIAL RESEARCH AND GOVERNMENT Research and Government: 40 years on By Richard Bartholomew, previously chief research officer at the Department for Education, and currently editor of SRA journal ‘Social Research Practice’

he late 1970s Also, stricter financial discipline move towards ‘framework agreements’ Twas a time introduced by the Treasury through the for commissioning. This may bring a of expansion in regular ‘spending reviews’ meant that more systematic approach but this social research with departments had to make better cases can be overly bureaucratic and create researchers working for funds. It forced departments to take an unhelpful distance between those in many (but not all) of evaluation of the effectiveness (and who plan and use research and social the ‘big-hitter’ social cost-effectiveness) of their policies researchers outside government who policy departments of and programmes more seriously. conduct empirical studies. It may look central government as well like a fairer and more open system, The demands of evaluation have as in Scotland and Northern Ireland. less prone to over-cosy relationships, become ever more sophisticated and Partly, this reflected the political but the barriers it erects can make it precise, involving vast expenditure on climate. There was a willingness to more difficult to achieve creative and research. The research community address the big social issues of poverty, innovative research designs. has benefited from this spending. But intergenerational deprivation, race and its scale has sometimes distorted the gender equality, health inequalities, social research market in unhelpful The government’s use and chronic unemployment. This led to ways, leading, for example, to ‘feast of research many innovative studies (for example followed by famine’ when policies or The Workplace Industrial Relations The $64,000-dollar question is ‘has governments change. There have Survey (now Workplace Employment government use of social research been valid criticisms that ministers and Relations); Women and Employment; findings changed?’ Yes and no. policy officials have been too impatient the British Crime Survey). Partly because of the discipline of to wait for the full results of complex spending reviews but also, because But it was growth largely driven by evaluations before embarking on we live in a more data-rich and data- individual departments rather than further initiatives. accessible world, it is now much by any cross-governmental decision. The focus on evaluating policies – and harder for ministers to launch new Government social research was a a narrow interpretation of ‘policy need’ policies based on ideology alone decentralised entity without a clear – has sometimes led to a neglect of (though the (mis)use of evidence in corporate identity and only limited important issues which are not on the the Brexit ‘debate’ gives pause for central co-ordination. Today Government political agenda but may be in future. thought on this!). Policy officials have Social Research (GSR) is a cohesive This has tended to inhibit exploratory a much better understanding of how professional body providing common studies of underlying social issues research and analysis can help, with standards for recruitment and training, and emerging trends. The neglect greater trust between analysts and ethics and publication protocols. of research on inequality and social policymakers. That does not mean mobility in the 1980s and early 1990s that policies are always determined by What’s changed? is one example. a coherent evidence case. Too many Perhaps the most significant change are evidence-informed rather than has been the huge expansion in policy Commissioning evidence-based, with research playing evaluation such that evaluation is now a selective and confirmatory role rather 40 years ago, the processes the dominant form of social research than one of challenge. Is it naïve to of research commissioning by commissioned by government. believe that, in a political environment, government departments were often Several factors have driven this. this could be otherwise? Perhaps, decentralised and sometimes ad The introduction of more directly but I remain optimistic. hoc. The biggest change, especially interventionist programmes to tackle in the last ten to 15 years, has been social problems including programmes the emergence of procurement as a to reduce mass unemployment and lack distinct activity (with its own cadre of training; to improve youth justice and of procurement specialists) and the reduce re-offending; and many others.

RESEARCH MATTERS: DECEMBER 2018: 11 SRA QUIZ How much do you know about the SRA? Just how much do you know about the SRA? Take the quiz set by Annika Coughlin. Answers on page 14.

1. What was the hourly wage of the 5. When did the SRA launch 8. When did the SRA set up a jobs one-day-a-week SRA administrator its website? board for social research vacancies (working from home) in 1985? https://jobs.the-sra.org.uk/? ■ a. 1995 ■ a. £1.50 per hour ■ a. 2006 ■ b. 1998 ■ b. £2.90 per hour ■ b. 2013 ■ c. 2000 ■ c. £4 per hour ■ c. 2016 6. What was becoming an increasing 2. In 1988 what new addition greatly area of interest that led to the SRA 9. What percentage of members of improved the functioning of the to put together a series of evening the SRA in 1978 were women? SRA office? seminars and workshops during ■ a. 34% 1982? ■ a. Pager ■ b. 55% ■ a. The government cuts ■ b. Computer to social research ■ c. 68% ■ c. Rolodex ■ b. Qualitative methods 10. How many times has the SRA 3. When was Sue Duncan, a founder moved office in the past seven ■ c. Questions about ethnicity member of the SRA, appointed as years? in the census the first UK government chief social ■ a. 1 researcher? 7. How many training courses have been available on the SRA website ■ b. 3 ■ a. 1997 during 2018? ■ c. 5 ■ b. 2002 ■ a. 44 ■ c. 2006 ■ b. 77 4. What three items were mentioned ■ c. 111 as being on the menu of the SRA annual conference December 1984 – ‘The Best Conference Yet’? ■ a. Tri-coloured pasta salad, chicken kiev, stuffed peppers ■ b. Avocado, salmon en croute, chocolate mousse ■ c. Chicken, smoked mackerel and quiche

12: SOCIAL RESEARCH ASSOCIATION REVIEWS

Questionnaire Evan Kindley Bloomsbury, 2016 Reviewed by Diarmid Campbell-Jack, Ecorys

Social researchers often have a love- used for more nefarious purposes with ability to crunch huge amounts of data hate relationship with the questionnaire. Kindley highlighting the eugenic views and provide answers to a variety of Sure, there are feelings of intellectual behind many questionnaires in the important and less important questions. achievement when cognitively testing Victorian era (and later), the worrying While ‘Questionnaire’ ends, quite rightly, questions, finding that elusive validated cod-psychology of Scientology’s by pointing to the possibility that this scale or uncovering a genuinely ‘Oxford Capacity Analysis’ and the freely provided personal data can be new way of approaching an issue. misogynist assumptions behind tools used for malicious purposes as well However, there are also late nights used by the American Institute of as more positive ones, it raises other stuck on the ninth iteration of a Family Relations. questions relevant to social research. questionnaire, wondering whether While social researchers rightly focus on What, if anything, can we learn from the minuscule differences in phrasing evidence that response rates on many the successes of Buzzfeed or OK we are considering actually make any national surveys are falling, this book Cupid in getting people to enjoy taking difference to the person answering. reminds us that internet questionnaires part in questionnaires? Are people Evan Kindley’s ‘Questionnaire’ is remain popular. Buzzfeed made its more concerned about providing an ideal pick-me-up for the jaded name off the back of questionnaires data to private companies (who pass researcher in those moments, taking us varying from ‘How stereotypically this who knows where) than to social rapidly through an entertaining history white are you?’ to ‘How well do you researchers and public organisations? If of the questionnaire and reminding know your Muppets?’ Millions provide so, why? Kindley’s book won’t provide us not just that questionnaires are answers to a variety of weird and any direct answers but will prove a very important but also that they can be fun personal questions on dating websites entertaining and thought-provoking and genuinely interesting. This is not to in the search for that elusive partner, distraction when the next set of say that questionnaires haven’t been providing these organisations with the questionnaire revisions are required.

Critical thinking Tom Chatfield SAGE Publishing, 2018 Reviewed by Imogen Birch, senior researcher, Citizens Advice

‘Critical thinking’ is written in a text- through literature or using social media. Definitions are highlighted in yellow, book style, and primarily targeted He covers some quantitative research with explanations in the side margins at students who need to read or to concepts such as representativeness, and within the text, so everything is well produce lengthy documents. As a statistical significance, correlation and explained. The text is broken up with researcher, I think it will be useful causation and reversion to the mean. brightly coloured pictures and quotes. for thinking about written and verbal Because this is aimed at those with little There are exercises to complete. At the arguments: helping readers make or no knowledge of these topics, some end of each chapter is a summary of and recognise an argument, and researchers may find this information key learning points and links to a short understand when a valid argument a bit basic. YouTube video. Each of the ten chapters isn’t being made. includes one of the author’s ten ‘Critical thinking’ is easy to read with an commandments: also listed at the end The author describes common pitfalls engaging format. The author, by writing of the book. The first commandment including faulty reasoning, empty in blue biro, encourages the reader to is to slow down and consider what is rhetoric and unjustified assumptions. He write notes in the book. He also directs in front of you: does it need deeper also discusses priorities and techniques readers to social media and to engage thought, or a strategy? If not, move on. for using your time well when reading with the hashtag #TalkCriticalthinking. See these and other new reviews at: http://the-sra.org.uk/sra_resources/publications/book-reviews/ We are always looking for reviewers. Write a short review for us and you get to keep the book. All books up for review are listed on the website; http://the-sra.org.uk/sra_resources/publications/book-reviews/ If you are interested, please email [email protected] and we’ll send you guidelines.

RESEARCH MATTERS: DECEMBER 2018: 13 REPORTS

SRA Scotland SRA Cymru SRA North By Karen Kerr By Faye Gracey By Leanne Dew incoming trustee Thank you to Building on the SRA Scotland is everyone who joined success of the busy planning for an us in October to SRA North early event that will take celebrate the SRA’s careers event in place in 2019 to celebrate 40th birthday in Wales. September, we returned the SRA’s 40th anniversary. We It was great to see almost a to Manchester to celebrate the are delighted to welcome Georgina hundred researchers gather to hear SRA’s 40th anniversary with a fantastic Southern (Skills Development our future generations commissioner event with Stephen Morris exploring Scotland) to the committee. If you for Wales, Sophie Howe, highlight the RCTs past, present and future. We are would like to get in touch with the importance of research in improving now starting to look forward to next branch and be more involved with the all aspects of our social, economic, year, and planning interesting events. organising committee, please contact environmental and cultural wellbeing. But before then, SRA North members the SRA office. The commissioner and Dr Jamie Smith will be heading down south for the SRA (director of research and Innovation annual conference in December – hope SRA Ireland at Hafod) as the event chair, really to see you there! As ever, do come challenged us to consider how we as and say hello or drop us a message if See the SRA website for further details researchers should think, talk about, you have any questions or want to get of events. Email us on SRAIreland@ and plan for the future. Please follow involved (@SRANorth and sranorth@ the-sra.org.uk or follow us on Twitter us @SRACymru to hear our latest gmail.com). @SRAIreland. news. Do visit the-sra.org.uk/events/ for details of events. If you want to get more involved with our organising committee don’t hesitate to get in touch: [email protected] T: 03000 257459.

ANSWERS TO QUIZ Q1: (c) £4 per hour Q2: (b) a computer Q3: (b) 2002 Q4: (c) Chicken, smoked mackerel and quiche Q5: (b) 1998 Q6: (b) Qualitative methods Q7: (c) 111 training courses Q8: (c) 2016 Q9: (a) 34% (2018 membership is approximately 30% male and 70% female) Q10: (c) 5 times

research matters Views expressed by individual contributors do not necessarily reflect those of the SRA. Publication dates 2019 Editorial team The Social Research SRA Research Matters will be Sarah Butt, City, University of Association (SRA) published in March, June, London (commissioning editor) Tel: 0207 998 0304 September and December 2019. [email protected] Email: [email protected] Diarmid Campbell-Jack, Ecorys Copy deadlines: 6 February www.the-sra.org.uk (March issue); 3 May (June issue); Peter Cornick, NatCen 22 July (September issue); Patten Smith, Ipsos MORI Keep up to date with 7 October (December issue). Martina Vojtkova, NatCen latest news online at Graham Farrant, SRA www.the-sra.org.uk

14: SOCIAL RESEARCH ASSOCIATION REFLECTIONS Gains and losses in social research Natalie Low, Nick Moon, I don’t miss: the old tape recorders because it was all too easy if you were National Audit Office formerly GfK NOP tired to record a second interview over I miss: old questionnaire drafts. Without I miss: how much easier it was to get the top of an earlier one, something computers, the process of questionnaire people to take part. I remember one that, mercifully, can’t happen with digital drafting involved photocopying past postal survey of people on a particular recorders. Turning out on cold winter versions, physically cutting and pasting benefit that got a 96% response rate. nights to run focus groups in drab and new and old questions together, and Although it may have helped that the draughty community halls! scrawling on amendments. By the end, questionnaire was sent out with their the final draft was about two-foot-thick, benefit cheques … Ivana Le Valle, and it felt like a real achievement. I don’t miss: how primitive the research consultant I don’t miss: when people complain technology was. I spent the whole of my I miss: face-to-face interaction. nowadays about computers being too first month in the office copying numbers When I first started, this was at the slow, I remind them about running (even in my best handwriting from some very heart of the job. Research designs pretty basic) analysis in the 90s. You non-user-friendly computer printout on were brainstormed and refined in had to write the syntax, submit it to the to lined paper that could then be given team meetings; qualitative research server, go home for the evening, come to typists to form part of the report. interviews were usually face-to-face; back in the morning … to find an ‘error developing analytical frameworks and in line 4’. And repeat. Jane Barret, Department interpreting findings involved endless of Health and Social Care face-to-face discussions. Nowadays, Bob Erens, London many methodological and analytical School of Hygiene and I miss: the time we used to have. discussions happen by email; many Working at a faster pace these meetings are virtual; qualitative telephone Tropical Medicine days means there is less time, as interviews and online data-collection a commissioner, to be as close to I miss: high response rates, for example methods have become the norm. the data-gathering stage. I miss 81% on the first Scottish Health Survey the opportunities to accompany a I don’t miss: the 5am starts to do an in 1995. researcher in the field and feeling part interview or attend a meeting on the I don’t miss: hole counts (researchers of the whole process rather than just other side of the country. under 45 will have to ask their older the beginning and the end. colleagues what these are!) Clarissa White, I don’t miss: some of the old Sue Body, technonology. IT has improved independent researcher even though I still grumble about it! I miss: the research budgets and NatCen (retired) I remember having to walk across timescales; having time to collaborate I don’t miss: old methods of Parliament Square with a projector and effectively; having fun on fieldwork trips. screen to rig up a presentation for a correcting errors. One benefit from I don’t miss: having to explain what minister in the House of Commons – the introduction of CAPI (computer- qualitative research is and why it is very Laurel and Hardy. Now I can simply assisted personal interviewing) needed; the slow procedures and share slides using Skype – much slicker. was the ease with which changes systems; the consequences of the could be made to the questionnaire. machinery failing. Previously we sometimes had to pay Liz Spencer, The interviewers to amend errors in their Qualitative Workshop Sarah Cheesbrough, paper questionnaires. Even worse, on I miss: proper recognition of what one survey, the error totally altered the Kantar Public qualitative research does best – meaning of the question, and it was I miss: working on the first ever CAPI mapping nuance, range, depth, context, felt it needed to be changed prior to surveys – making a lot of it up as we stories and processes, discovering distribution. It therefore fell to the field went along! the ‘why’, ‘when’ and ‘how’ of social staff to manually change a couple of interaction. In my view, some new ‘semi- I don’t miss: (from the early 1990s) sentences on 2,500 questionnaires. quantitative’ analysis techniques and an leaving an SPSS program that you had Needless to say, that researcher overreliance on analysis software, with worked on all day to run overnight on the wasn’t very popular for a while. its seductive automatic functions like mainframe; only to discover the next day Word Cloud, have begun to erode the that it had fallen over in the first few lines. ‘qualitative’ in qualitative research.

RESEARCH MATTERS: DECEMBER 2018: 15 TRAINING 2019

BRISTOL 27 February Qualitative interviewing Professor Karen O’Reilly 28 February Conducting focus groups Professor Karen O’Reilly 1 March Qualitative data analysis: approaches and techniques Professor Karen O’Reilly

CARDIFF 28 March Introduction to data visualisation Lulu Pinney

EDINBURGH 28 February Introduction to data visualisation Lulu Pinney 1 March Narratives and storytelling in qualitative research Dr Karen Lumsden 24 April Designing a qualitative study Professor Karen O’Reilly 25 April Qualitative Interviewing Professor Karen O’Reilly 26 April Conducting focus groups Professor Karen O’Reilly 29 April Qualitative data analysis: approaches and techniques Professor Karen O’Reilly 30 April Interpreting and writing up your qualitative findings Professor Karen O’Reilly

LONDON 18 January Narratives and storytelling in qualitative research Dr Karen Lumsden 31 January Introduction to data visualisation and infographic design Lulu Pinney 1 February Ethnographic methods Professor Karen O’Reilly 7 February Qualitative interviewing Professor Karen O’Reilly 8 February Conducting focus groups Professor Karen O’Reilly 11 February Qualitative data analysis: approaches and techniques Professor Karen O’Reilly 12 February Interpreting and writing up your qualitative findings Professor Karen O’Reilly 25 February Writing effective research reports Dr Simon Haslam 7 March Research with children and young people Dr Louca-Mai Brady & Berni Graham 15 March Introduction to participatory action research Dr Karen Lumsden 21 & 22 March Depth interviews NatCen Learning (two-day course) 1 & 2 April Analysis of qualitative data NatCen Learning (two-day course) 5 April Introduction to qualitative research NatCen Learning 3 May Narratives and storytelling in qualitative research Dr Karen Lumsden

20 May Reporting qualitative data NatCen Learning Designed by www.graphics.coop Edited by Shirley Henderson www.shirleyhenderson.co.uk

We have increased course prices by £10 a day. From 2019 We regularly add courses and course locations to our these will be £270 instead of £260: SRA members will continue programme. Keep up to date at: www.the-sra.org.uk/training to receive a 25% discount, paying £202.50 a day rather You can also join our mailing list at www.the-sra.org.uk. If you than £195. It is always a difficult decision to raise prices, but have any queries contact Lindsay Adams on 0207 998 0304 or our costs have increased noticeably since the last time we [email protected]. increased course prices (more than seven years ago). This modest increase will help us to cover our increasing costs. To get your SRA member discount, make sure to use your Full details of all SRA courses and booking at: promo code. www.the-sra.org.uk/training