APPLYING SOCIAL NETWORK ANALYSIS TO EXAMINE PROGRAM OUTPUT AND IMPACT Ying Zhang, Ph.D. Borjan Zic, Ph.D. January 2021 Manhattan Strategy Group 4340 East-West Highway, Suite 1100 Bethesda, MD 20814 [email protected] TABLE OF CONTENTS Introduction ............................................................................................................................. 1 APPLICATION 1: Tracking Interactions in a Community of Practice .........................................2 APPLICATION 2: Understanding Interdisciplinary Collaboration ...........................................4 Conclusion ................................................................................................................................6 References ................................................................................................................................ 7 Manhattan Strategy Group | i INTRODUCTION he COVID-19 pandemic has changed social In this paper, Manhattan Strategy Group (MSG) lives across the United States. Not only have presents two applications of SNA to illustrate how T the individuals we interact with changed, but social network data can help us understand the impact so has the manner and frequency of our interactions of federal programs. The first study focused on an with others. The changing social dynamics remind education program intended to enhance services that us how our lives are dependent on the people and promote adult literacy. The second project focused communities around us as well as the relevance of on a federal program designed to promote cross- community connections to social policy design and disciplinary collaboration among scientists to address implementation. environmental challenges. MSG contracted with these programs to deliver technical assistance and conduct Research on social connections and communal program evaluation. The social network analyses in interactions plays a key role in improving social the two studies documented program performance policies and public services. Knowing that social as measured by program participation and grantee networks come in different sizes, durations, and collaboration. strengths, researchers have used social network analysis (SNA) to examine the systems of relationships We conclude the paper with a discussion of SNA’s among individuals and the social and economic potential to evaluate the performance of social impact of such systems. With advances in technology programs and suggest potential ways to use social and the increasing use of social media platforms, networks as measures for intangible program researchers now have access to richer data on human outcomes. interactions than ever before and have demonstrated how to collect and analyze these data. Manhattan Strategy Group | 1 APPLICATION 1: Tracking Interactions in a Community of Practice n a Community of Practice (CoP), community community-building activities over 3 contract years. members provide communal support to each The intensified interaction and the shifting patterns of Iother through regular interactions to facilitate communication among group members illustrate the learning and improve practices in a particular topical impact of LINCS on building a learning community area. Since 2016, MSG has supported a CoP for users among its stakeholders. of the Literacy Information and Communications In 2020, the project team conducted a follow-up System (LINCS), a program funded by the U.S. SNA study and used back-end data from the LINCS Department of Education. This CoP supports adult platform between 2016 and 2019 to examine how educators’ professional learning and provides a place users connected with each other through online for adult educators to collaborate, network, and share discussions. These data included posts and comments knowledge and resources. The virtual community associated with those discussions as well as Likes consists of 16 groups and 7 micro groups, with 11,754 and other opportunities for users to connect. The registered users (members) in 2016. Moderators in project team analyzed user activity levels, including this community help members access information and participation in groups and micro groups, access engage in productive, ongoing peer interactions. to resources, and utilization of additional platform To identify patterns of interaction among CoP features such as Likes and Bookmarks. The project members, the project team conducted an SNA study to team performed aggregate analyses to gauge changes examine changes in user activities between 2013 and in levels of participation and engagement and overall 2015. In Figure 1, each dot represents a person, and platform growth. The follow-up analysis showed that each line represents a connection. The larger a dot, the more CoP members became connected through LINCS more connections this person has. The connectivity and the average number of connections per user of community members intensified through increased. Figure 1. Changes in CoP Engagement among LINCS Users between 2013 and 2015 2013 201 2015 Manhattan Strategy Group | 2 Further analysis showed that connecting to other to best practices for online education and technology users has helped LINCS members gain access to new integration in such an uncertain time. ideas, solutions, and experts that they would not The LINCS project illustrates how SNA can yield keen have been able to access otherwise. Public Groups insights about educational outcomes with network and Courses are the most successful features for data. An even more extensive application of SNA in facilitating connection and collaboration, and it is this instance can link changed behaviors with the more important to users to connect with experts learning experience and results of adult educators. In rather than other users, although users prioritize particular, a future study could explore CoP impacts connecting to others sharing similar issues, from on program outcomes of interest by linking individual- similar types of organizations or sectors, or holding level outcome data to the network data presented a similar organizational role. During the COVID-19 above. pandemic, the CoP has been helpful to users, many of whom have reported the importance of having access Manhattan Strategy Group | 3 APPLICATION 2: Understanding Interdisciplinary Collaboration ne particular type of social network is a They show the networks of a PI in chemistry and the professional network. It plays a crucial role in PI’s collaborators in geology (CO-I_1), engineering, Oone’s career development, allowing individuals computer science (CO-I_2), and social science to access information about career opportunities, (CO-I_3) before, during, and after the funding period. be exposed to different perspectives, and create The four researchers had disconnected research innovative solutions to career challenges. However, networks before the funding period, as shown in some professional networks, particularly in academia, Figure 2(a). CO-I_2 had a more extended network are more tightly bounded and siloed to specific bodies than the other three. of knowledge and practice, leading to a disciplinary Figure 2(a). Professional Network of a PI before a divide. The disciplinary divide of a professional Funding Period network not only limits our understanding of complex T0: Pre-Funding Period issues but also results in duplicated efforts to solve MATH OTHER_3 similar or identical problems. Recognizing the need PHYSICS CHEM OTHER_2 for a diverse workforce to solve the increasingly OTHER_1 OTHER_5 OTHER_4 complex issues we face today, many organizations PI 1 7 GEO 2 and governmental agencies have designed and 10 6 CS CO-I_2 implemented programs to foster interdisciplinary CO-I_1 3 5 collaboration. SOC 4 ENG 8 Between 2014 and 2018, MSG staff studied the CO-I_3 OTHER_6 impact of a federal grant program to promote 9 multidisciplinary collaboration for scientists to address environmental challenges. The project team Through the federal program, their networks were surveyed principal investigators (PIs) who received connected and expanded to include new collaborators grants from this federal program and asked them in engineering (CO-I_4) and social science (CO-I_5). to list their five most important collaborators. The Figure 2(b). Professional Network of a PI during a project team also collected information about PIs’ Funding Period funding history from the agency’s administrative T0 + T1: Funding Period record and extracted PIs’ professional connections 9 MATH GEO from public sources, such as publications and contacts CO-I_3 8 on LinkedIn. The compiled PIs’ networks provided SOC CO-I_1 OTHER_2 tracking records of collaborative networks before the OTHER_3 CO-I_5 funded project started, during the funding period, and 1 7 Program CS OTHER_4 PHYSICS after the funded project was complete. 6 PI CO-I_4 CO-I_2 OTHER_1 With these network data, the project team conducted CHEM 5 10 ENG 3 two types of social network analyses, one focusing 4 OTHER_5 2 on the PI’s entire network (egocentric analysis) OTHER_6 and the other on the strength of the connection of all collaborators on an awarded project (whole After the funding period, not only did these expanded network analysis). The social network diagrams in networks stay active, the connectivity of these Figure 3 are an example of the egocentric analysis. networks also intensified. Manhattan Strategy Group | 4 Figure 2(c). Professional Network of a PI after the Figure 3. Professional Network Structure of Funding