National Research University Higher School of Economics Institute for Statistical Studies and Economics of Knowledge
Big Data Augmented Horizon Scanning: Combination of Quantitative and Qualitative Methods for Strategic and Marketing Analytics
[email protected] [email protected]
XIX April International Academic Conference on Economic and Social Development
Moscow, 11 April 2018 Outline
- Role of artificial intelligence and big data in modern analytics
- System of Intelligent Foresight Analytics iFORA
- Combined quantitative and qualitative analysis methodology and software solutions
- Use cases
- Conclusion and discussion
2 Growing interest in Artificial Intelligence, Big Data and Machine Learning International analytical reports & news feed 12000
10000
8000
Artificial Intelligence 6000 Big Data Machine Learning 4000
2000
0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Russian analytical reports & news feed 800
700
600
500 Artificial Intelligence 400 Big Data 300 Machine Learning
200
100
0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 3 Source: System of Intelligent Foresight Analytics iFORA™ (ISSEK HSE) Role of Big Data Augmented Horizon Scanning in Strategic and Marketing Analytics
AI-related tasks Tracking latest and challenges trends, technologies, drivers, barriers
Market forecasting Trend analysis Understanding S&T modern skills and Instruments for Customers Market Intelligence competences analysis feedback knowledge discovery HR policy Vacancy Feedback mining Monitoring mining company’s Strategic CV Education reputation Sentiment processing and analysis Analytics adaptation of staff Knowledge Email extraction Process mining mining Security Document policy Majority of the Monitoring of Portfolio clustering analyzed data is information management technologies BPM text Document workflow Implementation of modern optimization technological solutions
Main aspects of strategic Source: ISSEK HSE development System of Intelligent Foresight Analytics iFORA™ (ISSEK HSE): quantitative and qualitative methods for Foresight and Strategic Analytics
Articles Research Research Patents Education Employment Analytical reports Regulatory Materials of Media, social from top grants reports programs websites of international framework international networks and scientific organizations and conferences professional journals consultancies blogs
iFORA System of quantitative and qualitative methods
Big data, AI & machine Ontologies and expert Computer linguistics learning knowledge
푢푛𝑖푞푢푒 푐표푚푏𝑖푛푎푡𝑖표푛 표푓 푡ℎ푟푒푒 푡푒푥푡 푚𝑖푛𝑖푛푔 푎푝푝푟표푎푐ℎ푒푠
User-friendly interface
• Millions of documents • Transparent, reproducible, • Full texts validated methodology • A variety of data formats • Human factors risks are • Selection by unified objective minimized criteria • Rapid acquisition of analytical • Constant replenishment results
Source: ISSEK HSE International validation: OECD (Paris, 2018) 5 iFORA software decisions, strengths and challenges Open sources of data iFORA Strengths + open source libraries Python NLP algorithms + open source data (Machine learning + Computer Linguistics) + flexible in-house development + high- performance local PostgreSQL servers Database Dgraph (NoSQL) + state-of-art (RDBMS) algorithms
linguistic documents’ layer metadata ontologies Challenges ! hardware costs ! high quality R programming libraries human resources topic network entity clustering modelling analysis linking costs
Data Data Data Data Data Data mart mart mart mart mart mart
Analytical output Source: ISSEK HSE 6 Combined quantitative and qualitative analysis methodology sentence 1 Text segmentation sentence 2 iFORA sentence 3 quantitative word 1 Sentence tokenization layer sentence word 2 word 3 Lemmatization, stemming, morphological lemma analysis word stem part of speech … Syntactic analysis terms
Word and document embeddings, topic modeling term / vector(x1, x2, …, xn) document
Entity linking term (vector) entity (vector)
Statistical analysis, network analysis, clustering
qualitative Expert analysis of In-depth and Expert panels and Delphi surveys layer literature structured interviews focus groups
Source: ISSEK HSE 7 Big data augmented horizon scanning: semantic map of “NLP” sphere (2013-2017)
Terms derived from open analytical reports, news feed, etc
Location, size, and color of each item have a strict mathematical sense Terms are grouped into clusters Source: System of Intelligent Foresight Analytics iFORA™ (ISSEK HSE) based on their semantic closeness 8 Big data augmented horizon scanning: deep cluster view of “NLP” sphere (2013-2017)
Linkages are shown only for terms of the cluster
Source: System of Intelligent Foresight Analytics iFORA™ (ISSEK HSE) 9 Big data augmented horizon scanning: cluster dynamics of “NLP” sphere (2013-2017)
Percentage shows frequency of terms in clusters for each year compared to all term frequencies of “NLP” topic
10 Source: System of Intelligent Foresight Analytics iFORA™ (ISSEK HSE) Big data augmented horizon scanning: trend analysis in the sphere of “NLP” (2013-2017)
Terms and colors match values Terms in this quadrant (Q1, Q2)
High from semantic map represent promising trends Term frequency Term
Low Average Annual Term Frequency Growth Rate
Low Medium High Source: System of Intelligent Foresight Analytics iFORA™ (ISSEK HSE) 11 Big data augmented horizon scanning: centers of competence in “NLP” sphere (2013-2017) Terms derived from open analytical reports, news feed, etc
Location, size, and color of each item have a strict mathematical sense Terms are grouped into clusters Source: System of Intelligent Foresight Analytics iFORA™ (ISSEK HSE) based on their semantic closeness 12 Use case: target orientation to the solution of socio- economic, managerial and other tasks – case of medicine
13 Source: ISSEK HSE Use case: integration of quantitative and qualitative market estimates
Database as a Service Area for science and Area for project finance
technology support , % ,
CCS in Power Battery Energy Generation Storage Systems
Carbon Nanotube annual growthrate annual
Heat Transfer
Fluids Market compoundMarket Aquaculture Market Hearing Aids Industrial Robotics
Area for stimulation of Niche areas demand
Result of aggregation of Term frequency in scientific articles market reviews
Source: ISSEK HSE 14 Источник: Система интеллектуального анализа больших данных iFORA (правообладатель - ИСИЭЗ НИУ ВШЭ) Use case: identifying promising market - startups in aviation
Startups Business models Bid Aero ICON Aircraft Biz Airlines Advanced Vehicle and ImmaculateFlight Qualified Technologies Engine Innovation Book With Biggles Reliance Air Charters Surf Air is the All-You-Can-Fly private air Aero Centro Aircraft Boom InnerVision Robotics CaptainOps Rotaradar Aero Glass Innovative Binaries travel membership, providing frequent JetSuite’s vision to provide the CARIBLift Aero Infinity RPAS Solutions regional business and leisure travelers with a freedom and exhilaration of private ClimaCell International Aviation Aeropoint Academy sophisticated and hassle-free air travel Coavmi.com Group air travel to more people than ever is Shen Zhen Manly Tech Aerostarter Correyvuela Jet Genius experience that saves valuable time and realized DARTdrones Flight Jet Rebellion Sia Aerospace money AI Intellectual Property Academy JetHunt Sibelloptics Aimfill International Drone Industry JetInsight Skycatch Reputation analysis Air Somaliand Insights Jetonair Skycorp Airbus BizLab dronedeploy JetSuite Skyfront Negativity of Number of Positivity of Number of AirDB DroneRush Jetway Startup Skytech mentions negative mentions mentions positive mentions Airdog Easy Fly JETWISE Sparewings AirHelp EasyFBO Juniper Unmanned Surf Air -26 7 185 32 EHang Airlines Technology Kespry Inc. Stratos Group JetSuite -14 7 98 31 Airly Surf Air Encel Space Systems KleanKraft, L.L.C. Airware -11 6 35 17 AirMod Consulting Encore Jets Lake Zone Charter Surfreps Airscape enhanceaviation Services Swift Navigation ICON Aircraft -9 5 44 15 Airtime Aircraft ExpertJet Charter Lilium Aviation SynapseMX Aero Glass -3 3 20 8 Airware FBO Sales and Mileways SYNERJETS Alba Orbital Advisory Services Skytech -6 5 10 6 Take Air Alpha Red Solutions Ferris Aviation MRO Exchange Tallamond SYNERJETS -2 2 16 5 Sdn Bhd Flight CenterOne LLC/MROmarketplace Flighttime Talos Aviation AMIntegrated Aerial MySkyForce SynapseMX -4 1 9 3 FlyEasy An1ken Aviation NetSky Tekinno Flyezee.com EHang -6 1 3 2 Astranis NextGreatTrip TestU ATP Flight School Flying Software Labs Nimble Aircraft This Will Fly Flighttime -4 1 3 1 Australian Corporate FlyPilot Vector Space International Jet Centres Galaxy Unmanned North Bridge Aviation Venus Aircraft AvAir Aviation Group -1 1 2 1 Systems OFFCRAFT AviAsia Aviation Vertical AI Globalpundits OpenJet EasyFBO -1 1 3 1 Solutions Vibes Air Green Tech Aircraft OpJets Airlines VIRES Aeronautics Paperclip Design Aviation Fueling Service Heading 370 Clothing Technology -1 1 ParaZero VUELOJET Aviation Infinity HelloJet Partsbase Waxwing Avionics Aviation Fueling Aviation Safety Horizon Airport Pilot AI Labs Wingly Service 7 2 Advancements Services PilotNeeded AviationStore.in Hubworkair Wrong Bros. Flight OpenJet 1 1 PlaneLogiX Training Wing Avion Media Group IBS Flightcases Predictive Aviation Partsbase 2 1 Avionix YAMJETS Analytics AvSKY Zee.Aero JetHunt 2 1 BestAVIA PRENAV ZyenaLABS Source: ISSEK HSE 15 Use case: evaluation of the quality of technologies, products, services - identification of products in the sphere of aircraft engines
№ Name of engine Alternative naming options № п/п Name of engine Alternative naming options № п/п Name of engine Alternative naming options
1 CFM International CFM56-7 CFM56-7 34 HW HTF7000 Honeywell HTF7000 68 R-R Tay 2 CFM LEAP 35 HW TFE731 Honeywell TFE731 69 Rolls-Royce RB183 Tay R-R Tay CFM LEAP CFM International LEAP 36 IAE V2500 3 IAE V2500 70 RR RB183 Tay 4 LEAP-X 37 V2500 71 Rolls Royce RB183 Tay 5 CFM56 38 LTV XC-142 Ling-Temco-Vought XC-142 72 R-R Trent 1000 CFM56 6 CFM International CFM56 39 P&W PW1000G 73 RR Trent 1000 P&W PW1000G R-R Trent 1000 7 EA GP7000 40 Pratt & Whitney PW1000G 74 Rolls-Royce Trent 1000 EA GP7000 8 Engine Alliance GP7000 41 P&W PW4000 P&W PW4000 75 Rolls Royce Trent 1000 9 EA GP7270 GP7270 42 Pratt & Whitney PW4000 76 R-R Trent 500 10 Engine Alliance GP7200 EA GP7200 43 P&W PW6000 77 RR Trent 500 P&W PW6000 R-R Trent 500 11 G500 Gulfstream Aerospace G500 44 Pratt & Whitney PW6000 78 Rolls-Royce Trent 500 12 G600 Gulfstream Aerospace G600 45 P&W T-34-P-7WA Pratt & Whitney T-34-P-7WA 79 Rolls Royce Trent 500 13 GE CF34 46 P&WC PW300 P&WC PW300 80 R-R Trent 550 Rolls-Royce Trent 550 14 GE CF34 General Electric CF34 47 Pratt & Whitney Canada PW300 81 R-R Trent 700 15 CF34 48 P&WC PW500 Pratt & Whitney Canada PW500 82 RR Trent 700 R-R Trent 700 16 GE CF6 49 P&WC PW600 Pratt & Whitney Canada PW600 Rolls-Royce Trent 700 GE CF6 83 17 General Electric CF6 50 P&WC PW800 84 Rolls Royce Trent 700 General Electric GE Aviation CF6- 51 P&WC PW800 Pratt & Whitney Canada PW800 18 GE CF-80C2 85 R-R Trent 800 80C2 52 Pratt & Whitney PW800 86 RR Trent 800 19 GE GE90 53 P&W JT9D R-R Trent 800 Pratt & Whitney JT9D 87 Rolls-Royce Trent 800 20 GE90-110B1 54 Pratt & Whitney JT9D 88 Rolls Royce Trent 800 21 GE GE90 GE90-115BL 55 Lotarev D-18T Progress D-18T 89 R-R Trent 900 22 General Electric GE90 56 Progress D-18T 90 RR Trent 900 23 General Electric GE90-115B 57 Rolls-Royce RB211 R-R Trent 900 Rolls-Royce RB211 91 Rolls-Royce Trent 900 GE GEnx 24 58 RB211-524B 92 Rolls Royce Trent 900 GEnx-1B54 25 59 R-R AE 3007 93 R-R Trent XWB 26 GEnx-1B58 60 Rolls-Royce AE 3007 R-R AE 3007 94 RR Trent XWB 27 GE GEnx GEnx-1B64 R-R Trent XWB 61 RR AE 3007 95 Rolls-Royce Trent XWB 28 GEnx-1B67 62 Rolls Royce AE 3007 96 Rolls Royce Trent XWB 29 GEnx-1B70 63 R-R BR700 97 Silvercrest 30 GEnx-2B67 64 Rolls-Royce BR700 Silvercrest R-R BR700 98 Snecma Silvercrest 31 GE Passport 65 RR BR700 GE Passport 99 Williams FJ33 Williams International/Rolls-Royce FJ33 32 General Electric Passport 66 Rolls Royce BR700 100 Williams FJ44 Williams International/Rolls-Royce FJ44 33 GE-H HF120 GE Honda HF120 67 R-R RB211-524H-T Rolls-Royce RB211-524H-T ......
Source: ISSEK HSE 16 Use case: evaluation of the quality of technologies, products, services – assessment of positivity of aircraft mentions
Representation in the professional media resources
Name of engine 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 Overall result CFM International LEAP & 3 9 28 85 61 44 60 54 60 404 LEAP-X Snecma Silvercrest 2 5 7 13 15 9 10 6 24 95 54 54 18 312 Rolls Royce Trent XWB 8 9 2 17 23 62 22 23 31 197 GE Passport 2 6 2 6 22 4 42 GE GEnx 2 1 2 2 2 4 2 1 1 2 19 First mentions Peak of popularity
Source: ISSEK HSE 17 Use case: evaluation of the quality of technologies, products, services – assessment of positivity of aircraft mentions
Indicator of Indicator of Index of positivity of Aircraft engine positivity of negativity of mentions mentions mentions Snecma Silvercrest 376 -226 1,7 CFM International 339 -87 3,9 LEAP General Electric 77 -26 3,0 Passport Rolls Royce Trent XWB 64 -16 4,0 …
18 Source: ISSEK HSE Use case: reputation analysis of aerospace and defense companies Positive Negative
Boeing
Aircraft crashes
Boston Dynamics
Company listing for sale
DuPont Running the cost optimization program
Lockheed Martin Contracts for equipment for US Navy helicopters
Rolls-Royce
Operational problems with engines
2005 2007 2009 2011 2013 2015 2017 Source: ISSEK HSE 19 Use case: intelligent automatic classification of organization documents
Expert choice of documents that are Collection of documents the thematic core of the branch / discipline Structuring documents by groups Organizer of documents
Choice of global markers for a thematic area
Group 1 Group 2 Group 3 Group 4
Flexible automated thematic classification of documents
Structuring sections of documents by topic
The text is automatically divided into sections
Collection of documents
Selection Entering new Selected of documents documents documents Intelligent reassembly of documents in the form of texts and tables Document Analyzer
Source: ISSEK HSE 20 Use case: automated selection of employees for new projects
Development of a new project
iFORA Project Comparison
with job with previous with documents on with resume of descriptions of projects structural divisions employees employees Analysis of the texts of publications, Information dissertations, presentations, data on about certificates, diplomas on additional. participants education and training, etc.
Extended list of candidates for participation in the project
Analysis of employee activity in the organization
Intensity and efficiency Semantic analysis of work of texts
Identification of employees who meet project requirements
Source: ISSEK HSE 21 Conclusion and discussion
Big data augmented horizon scanning allow solving many tasks in the sphere of strategic and marketing analytics:
- STI landscape structural dynamics, trends and weak signals - Networks and alliances (organizations and individuals), centers of excellence - Automated quantitative consensus-forecasts - Emerging markets and technologies, STI-based start-ups - Sentiment analysis-based assessment of technologies, products, organizations - and many more…
Source: ISSEK HSE 22 Thank you for your attention!
http://issek.hse.ru [email protected] http://foresight.hse.ru https://prognoz2030.hse.ru/ [email protected]