National Research University Higher School of Economics Institute for Statistical Studies and Economics of Knowledge Big Data Augmented Horizon Scanning: Combination of Quantitative and Qualitative Methods for Strategic and Marketing Analytics [email protected] [email protected] XIX April International Academic Conference on Economic and Social Development Moscow, 11 April 2018 Outline - Role of artificial intelligence and big data in modern analytics - System of Intelligent Foresight Analytics iFORA - Combined quantitative and qualitative analysis methodology and software solutions - Use cases - Conclusion and discussion 2 Growing interest in Artificial Intelligence, Big Data and Machine Learning International analytical reports & news feed 12000 10000 8000 Artificial Intelligence 6000 Big Data Machine Learning 4000 2000 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 Russian analytical reports & news feed 800 700 600 500 Artificial Intelligence 400 Big Data 300 Machine Learning 200 100 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 3 Source: System of Intelligent Foresight Analytics iFORA™ (ISSEK HSE) Role of Big Data Augmented Horizon Scanning in Strategic and Marketing Analytics AI-related tasks Tracking latest and challenges trends, technologies, drivers, barriers Market forecasting Trend analysis Understanding S&T modern skills and Instruments for Customers Market Intelligence competences analysis feedback knowledge discovery HR policy Vacancy Feedback mining Monitoring mining company’s Strategic CV Education reputation Sentiment processing and analysis Analytics adaptation of staff Knowledge Email extraction Process mining mining Security Document policy Majority of the Monitoring of Portfolio clustering analyzed data is information management technologies BPM text Document workflow Implementation of modern optimization technological solutions Main aspects of strategic Source: ISSEK HSE development System of Intelligent Foresight Analytics iFORA™ (ISSEK HSE): quantitative and qualitative methods for Foresight and Strategic Analytics Articles Research Research Patents Education Employment Analytical reports Regulatory Materials of Media, social from top grants reports programs websites of international framework international networks and scientific organizations and conferences professional journals consultancies blogs iFORA System of quantitative and qualitative methods Big data, AI & machine Ontologies and expert Computer linguistics learning knowledge 푢푛푞푢푒 푐표푚푏푛푎푡표푛 표푓 푡ℎ푟푒푒 푡푒푥푡 푚푛푛푔 푎푝푝푟표푎푐ℎ푒푠 User-friendly interface • Millions of documents • Transparent, reproducible, • Full texts validated methodology • A variety of data formats • Human factors risks are • Selection by unified objective minimized criteria • Rapid acquisition of analytical • Constant replenishment results Source: ISSEK HSE International validation: OECD (Paris, 2018) 5 iFORA software decisions, strengths and challenges Open sources of data iFORA Strengths + open source libraries Python NLP algorithms + open source data (Machine learning + Computer Linguistics) + flexible in-house development + high- performance local PostgreSQL servers Database Dgraph (NoSQL) + state-of-art (RDBMS) algorithms linguistic documents’ layer metadata ontologies Challenges ! hardware costs ! high quality R programming libraries human resources topic network entity clustering modelling analysis linking costs Data Data Data Data Data Data mart mart mart mart mart mart Analytical output Source: ISSEK HSE 6 Combined quantitative and qualitative analysis methodology sentence 1 Text segmentation sentence 2 iFORA sentence 3 quantitative word 1 Sentence tokenization layer sentence word 2 word 3 Lemmatization, stemming, morphological lemma analysis word stem part of speech … Syntactic analysis terms Word and document embeddings, topic modeling term / vector(x1, x2, …, xn) document Entity linking term (vector) entity (vector) Statistical analysis, network analysis, clustering qualitative Expert analysis of In-depth and Expert panels and Delphi surveys layer literature structured interviews focus groups Source: ISSEK HSE 7 Big data augmented horizon scanning: semantic map of “NLP” sphere (2013-2017) Terms derived from open analytical reports, news feed, etc Location, size, and color of each item have a strict mathematical sense Terms are grouped into clusters Source: System of Intelligent Foresight Analytics iFORA™ (ISSEK HSE) based on their semantic closeness 8 Big data augmented horizon scanning: deep cluster view of “NLP” sphere (2013-2017) Linkages are shown only for terms of the cluster Source: System of Intelligent Foresight Analytics iFORA™ (ISSEK HSE) 9 Big data augmented horizon scanning: cluster dynamics of “NLP” sphere (2013-2017) Percentage shows frequency of terms in clusters for each year compared to all term frequencies of “NLP” topic 10 Source: System of Intelligent Foresight Analytics iFORA™ (ISSEK HSE) Big data augmented horizon scanning: trend analysis in the sphere of “NLP” (2013-2017) Terms and colors match values Terms in this quadrant (Q1, Q2) High from semantic map represent promising trends Term frequency Term Low Average Annual Term Frequency Growth Rate Low Medium High Source: System of Intelligent Foresight Analytics iFORA™ (ISSEK HSE) 11 Big data augmented horizon scanning: centers of competence in “NLP” sphere (2013-2017) Terms derived from open analytical reports, news feed, etc Location, size, and color of each item have a strict mathematical sense Terms are grouped into clusters Source: System of Intelligent Foresight Analytics iFORA™ (ISSEK HSE) based on their semantic closeness 12 Use case: target orientation to the solution of socio- economic, managerial and other tasks – case of medicine 13 Source: ISSEK HSE Use case: integration of quantitative and qualitative market estimates Database as a Service Area for science and Area for project finance technology support , % CCS in Power Battery Energy Generation Storage Systems Carbon Nanotube annual growthrate annual Heat Transfer Fluids Market compoundMarket Aquaculture Market Hearing Aids Industrial Robotics Area for stimulation of Niche areas demand Result of aggregation of Term frequency in scientific articles market reviews Source: ISSEK HSE 14 Источник: Система интеллектуального анализа больших данных iFORA (правообладатель - ИСИЭЗ НИУ ВШЭ) Use case: identifying promising market - startups in aviation Startups Business models Bid Aero ICON Aircraft Biz Airlines Advanced Vehicle and ImmaculateFlight Qualified Technologies Engine Innovation Book With Biggles Reliance Air Charters Surf Air is the All-You-Can-Fly private air Aero Centro Aircraft Boom InnerVision Robotics CaptainOps Rotaradar Aero Glass Innovative Binaries travel membership, providing frequent JetSuite’s vision to provide the CARIBLift Aero Infinity RPAS Solutions regional business and leisure travelers with a freedom and exhilaration of private ClimaCell International Aviation Aeropoint Academy sophisticated and hassle-free air travel Coavmi.com Group air travel to more people than ever is Shen Zhen Manly Tech Aerostarter Correyvuela Jet Genius experience that saves valuable time and realized DARTdrones Flight Jet Rebellion Sia Aerospace money AI Intellectual Property Academy JetHunt Sibelloptics Aimfill International Drone Industry JetInsight Skycatch Reputation analysis Air Somaliand Insights Jetonair Skycorp Airbus BizLab dronedeploy JetSuite Skyfront Negativity of Number of Positivity of Number of AirDB DroneRush Jetway Startup Skytech mentions negative mentions mentions positive mentions Airdog Easy Fly JETWISE Sparewings AirHelp EasyFBO Juniper Unmanned Surf Air -26 7 185 32 EHang Airlines Technology Kespry Inc. Stratos Group JetSuite -14 7 98 31 Airly Surf Air Encel Space Systems KleanKraft, L.L.C. Airware -11 6 35 17 AirMod Consulting Encore Jets Lake Zone Charter Surfreps Airscape enhanceaviation Services Swift Navigation ICON Aircraft -9 5 44 15 Airtime Aircraft ExpertJet Charter Lilium Aviation SynapseMX Aero Glass -3 3 20 8 Airware FBO Sales and Mileways SYNERJETS Alba Orbital Advisory Services Skytech -6 5 10 6 Take Air Alpha Red Solutions Ferris Aviation MRO Exchange Tallamond SYNERJETS -2 2 16 5 Sdn Bhd Flight CenterOne LLC/MROmarketplace Flighttime Talos Aviation AMIntegrated Aerial MySkyForce SynapseMX -4 1 9 3 FlyEasy An1ken Aviation NetSky Tekinno Flyezee.com EHang -6 1 3 2 Astranis NextGreatTrip TestU ATP Flight School Flying Software Labs Nimble Aircraft This Will Fly Flighttime -4 1 3 1 Australian Corporate FlyPilot Vector Space International Jet Centres Galaxy Unmanned North Bridge Aviation Venus Aircraft AvAir Aviation Group -1 1 2 1 Systems OFFCRAFT AviAsia Aviation Vertical AI Globalpundits OpenJet EasyFBO -1 1 3 1 Solutions Vibes Air Green Tech Aircraft OpJets Airlines VIRES Aeronautics Paperclip Design Aviation Fueling Service Heading 370 Clothing Technology -1 1 ParaZero VUELOJET Aviation Infinity HelloJet Partsbase Waxwing Avionics Aviation Fueling Aviation Safety Horizon Airport Pilot AI Labs Wingly Service 7 2 Advancements Services PilotNeeded AviationStore.in Hubworkair Wrong Bros. Flight OpenJet 1 1 PlaneLogiX Training Wing Avion Media Group IBS Flightcases Predictive Aviation Partsbase 2 1 Avionix YAMJETS Analytics AvSKY Zee.Aero JetHunt 2 1 BestAVIA PRENAV ZyenaLABS Source: ISSEK HSE 15 Use case: evaluation of the quality of technologies, products, services - identification of products in the sphere of aircraft