A PRACTICAL GUIDE TO BIG DaTA Opportunities, Challenges & Tools 3DS.COM/EXALEAD “Give me a lever long enough and a fulcrum on which to place it, and I shall move the world.” 1 Archimedes ABOUT THE AUTHOR Laura Wilber is the former founder and CEO of California-based AVENCOM, Inc., a software development company specializing in online databases and database-driven Internet applications (acquired by Red Door Interactive in 2004), and she served as VP of Marketing for Kintera, Inc., a provider of SaaS software to the nonprofit and government sectors. She also developed courtroom tutorials for technology-related intellectual property litigation for Legal Arts Multimedia, LLC. Ms. Wilber earned an M.A. from the University of Maryland, where she was also a candidate in the PhD program, before joining the federal systems engineering division of Bell Atlantic (now Verizon) in Washington, DC. Ms. Wilber currently works as solutions ana- lyst at EXALEAD. Prior to joining EXALEAD, Ms. Wilber taught Business Process Reengineering, Management of Informa- tion Systems and E-Commerce at ISG (l’Institut Supérieur de Gestion) in Paris. She and her EXALEAD colleague Gregory Grefenstette recently co-authored Search-Based Applications: At the Confluence of Search and Database Technologies, published in 2011 by Morgan & Claypool Publishers. A Practical Guide to Big Data: Opportunities, Challenges & Tools 2 © 2012 Dassault Systèmes ABOUT EXALEAD Founded in 2000 by search engine pioneers, EXALEAD® is the leading Search-Based Application platform provider to business and government. EXALEAD’s worldwide client base includes leading companies such as PricewaterhouseCooper, ViaMichelin, GEFCO, the World Bank and Sanofi Aventis R&D, and more than 100 million unique users a month use EXALEAD’s technology for search and information access. Today, EXALEAD is reshaping the digital content landscape with its platform, EXALEAD CloudView™, which uses advanced semantic technologies to bring structure, meaning and accessibility to previously unused or under-used data in the new hybrid enterprise and Web information cloud. CloudView collects data from virtually any source, in any format, and transforms it into structured, pervasive, contextualized building blocks of business information that can be directly searched and queried, or used as the foundation for a new breed of lean, innovative information access applications. EXALEAD was acquired by Dassault Systèmes in June 2010. EXALEAD has offices in Paris, San Francisco, Glasgow, London, Amsterdam, Milan and Frankfurt. © 2012 Dassault Systèmes 3 A Practical Guide to Big Data: Opportunities, Challenges & Tools EXECUTIVE SUMMARY The Internet: Home to Big Data Innovation Not surprisingly, most of these game-changing technologies What is Big Data? were born on the Internet, where Big Data volumes collided While a fog of hype often envelops the omnipresent discussions with a host of seemingly impossible constraints, including the of Big Data, a clear consensus has at least coalesced around the need to support: definition of the term. “Big Data” is typically considered to be • Massive and impossible to predict traffic a data collection that has grown so large it can’t be effectively • A 99.999% availability rate or affordably managed (or exploited) using conventional data • Sub-second responsiveness management tools: e.g., classic relational database manage- • Sub-penny per-session costs ment systems (RDBMS) or conventional search engines, • 2-month innovation roadmaps depending on the task at hand. This can as easily occur at 1 terabyte as at 1 petabyte, though most discussions concern col- To satisfy these imposing requirements constraints, Web entre- lections that weigh in at several terabytes at least. preneurs developed data management systems that achieved supercomputer power at bargain-basement cost by distributing Familiar Challenges, New Opportunities computing tasks in parallel across large clusters of commodity If one can make one’s way through the haze, it also becomes servers. They also gained crucial agility – and further ramped clear that Big Data is not new. Information specialists in fields up performance – by developing data models that were far like banking, telecommunications and the physical sciences more flexible than those of conventional RDBMS. The best have been grappling with Big Data for decades.2 These Big known of these Web-derived technologies are non-relational Data veterans have routinely confronted data collections that databases (called “NoSQL” for “Not-Only-SQL,” SQL being the outgrew the capacity of their existing systems, and in such standard language for querying and managing RDBMS), like situations their choices were always less than ideal: the Hadoop framework (inspired by Google; developed and • Need to access it? Segment (silo) it. open-sourced to Apache by Yahoo!) and Cassandra (Facebook), • Need to process it? Buy a supercomputer. and search engine platforms, like CloudView (EXALEAD) and • Need to analyze it? Will a sample set do? Nutch (Apache). • Want to store it? Forget it: use, purge, and move on. Another class of solutions, for which we appropriate (and What is new, however, is that now new technologies have expand) the “NewSQL” label coined by Matthew Aslett of the emerged that offer Big Data veterans far more palatable op- 451 Group, strives to meet Big Data needs without abandon- tions, and which are enabling many organizations of all sizes ing the core relational database model.4 To boost performance and types to access and exploit Big Data for the very first time. and agility, these systems employ strategies inspired by the Internet veterans (like massive distributed scaling, in-memory processing and more flexible, NoSQL-inspired data models), or they employ strategies grown closer to (RDBMS) home, like in- “In the era of Big Data, more isn’t just more. memory architectures and in-database analytics. In addition, a More is different.” 3 new subset of such systems has emerged over the latter half of 2011 that goes one step further in physically combining high performance RDMBS systems with NoSQL and/or search plat- forms to produce integrated hardware/software applicances for deep analytics on integrated structured and unstructured data. This includes data that was too voluminous, complex or fast- moving to be of much use before, such as meter or sensor The Right Tool for the Right Job readings, event logs, Web pages, social network content, email Together, these diverse technologies can fulfill almost any Big messages and multimedia files. As a result of this evolution, the Data access, analysis and storage requirement. You simply need Big Data universe is beginning to yield insights that are chang- to know which technology is best suited to which type of task, ing the way we work and the way we play, and challenging and to understand the relative advantages and disadvantages of just about everything we thought we knew about ourselves, particular solutions (usability, maturity, cost, security, etc.). the organizations in which we work, the markets in which we operate– even the universe in which we live. Complementary, Not Competing Tools In most situations, NoSQL, Search and NewSQL technologies play complementary rather than competing roles. One excep- tion is exploratory analytics, for which you may use a Search A Practical Guide to Big Data: Opportunities, Challenges & Tools 4 © 2012 Dassault Systèmes platform, a NoSQL database, or a NewSQL solution depending on your needs. A search platform alone may be all you need if 1) you want to offer self-service exploratory analytics to gen- eral business users on unstructured, structured or hybrid data, or 2) if you wish to explore previously untapped resources like log files or social media, but you prefer a low risk, cost-effective method of exploring their potential value. Likewise, for operational reporting and analytics, you could use a Search or NewSQL platform, but Search may once again be all you need if your analytics application targets human decision- makers, and if data latency of seconds or minutes is sufficient (NoSQL systems are subject to batch-induced latency, and few situations require the nearly instanteous, sub-millisecond latency of expensive NewSQL systems). While a Search platform alone may be all you need for analyt- ics in certain situations, and it is a highly compelling choice for rapidly constructing general business applications on top of Big Data, it nonetheless makes sense to deploy a search engine alongside a NoSQL or NewSQL system in every Big Data scenario, for no other technology is as effective and efficient as Search at making Big Data accessible and meaningful to human beings. This is, in fact, the reason we have produced this paper. We aim to shed light on the use of search technology in Big Data environments – a role that’s often overlooked or misunderstood even though search technologies are profoundly influencing the evolution of data management – while at the same time providing a pragmatic overview of all the tools available to meet Big Data challenges and capitalize on Big Data opportunities. Our own experience with customers and partners has shown us that for all that has been written about Big Data recently, a tremendous amount of confusion remains. We hope this paper will dispel enough of this confusion to help you get on the road to successfully exploiting your own Big Data. © 2012 Dassault Systèmes 5 A Practical Guide to Big Data: Opportunities, Challenges & Tools