VLDB Prerequisite for the success of Digital India 02 Content Foreword 05 Introduction to Very Large Database 06 Adoption of VLDB 07 Overview of Digital India Programme 10 How VLDB Can Enable Digital India Programme 12 Key VLDB Challenges and Solutions 13 Conclusion 20 References 21 Contacts 21 03 VLDB | Prerequisite for the success of Digital India 04 VLDB | Prerequisite for the success of Digital India Foreword A few decades ago, data was considered a byproduct To tap ongoing momentum of digitizing India, there is a of algorithms or processes, not quite an integral great need to develop an atmosphere of impregnable part. But as the algorithms started being used for association between government, industry and businesses, it was realized that data generated is common man. A new kind of professional has not just a byproduct, rather an essential part of the emerged, the data scientist, who possesses the skills of process. Personal desktops also began using client- software programmer, statistician and artist to extract server databases regularly. Two decades later, we see the data. With time, the data generated and processed databases being involved in activities we perform on a will further increase and new solutions will have to be daily basis. The presence of the “Industrial Revolution devised, but this first step is essential in ensuring that of Data” is being felt all over the world, from science the whole country moves towards digitization as one. to the arts, from business to government. Digital information increases tenfold every five years that The purpose of this report is to promote discussions results in a vast amount of data being shared. This and share our point of view on the recent can be attributed to the improvements in algorithms developments around Digital India. Deloitte believes driving computer applications. this report provides you with insights into the India- scale data problems and how technology can be In 2014, the Prime Minister of India, Narendra Modi, instrumental in solving those. introduced the Digital India Programme; a programme which aims to connect every citizen of India, using the digital infrastructure. An initiative of this magnitude has never before been seen/attempted in India. It comes with its fair share of challenges, one of which is the immensity of the data it will generate, store and handle. With the Internet completing 25 years of its existence, India should utilize this platform to reap the benefits by aggregating and analyzing data in the various fields under concern. Data associated with the services of Digital India Programme will range in terabytes and will have to be stored in Very Large Databases (VLDB). Processing and accessing this data will also be more challenging due to the enormous amounts of data involved due to the large population of around 1.3 billion and and an area of 1.3 million square miles. Moreover, ensuring data security and protecting privacy is becoming harder Hemant Joshi as the information is shared ever more widely around the world. For the Digital India programme to be successful, large databases is an important aspect that needs to be acknowledged. 05 VLDB | Prerequisite for the success of Digital India Introduction to Very Large Database “The high velocity, white-water flow of data from innumerable real-time data sources such as market data, Internet of Things, mobile, sensors, clickstream, and even transactions remain largely unnavigated by most firms. The opportunity to leverage streaming analytics has never been greater.”- Forrester According to NASSCOM & Hansa Cequity’s Very large database (VLDB) emerges as report on Indian Analytics, the explosion a solution to existing data management in data has increased the importance of problems. It is a type of database that analytics. Experts believe that 90% of the consists of very high number of database world’s data was created in the last two records, rows and entries, which are years. It is expected that, by 2020, the spanned across a wide file system. amount of digital information existence Although there is no standard size of the will have grown from 3.2 Zettabytes to database to be classified as VLDB but it 40 Zettabytes (1 Zettabytes= 1 trillion is typically in terabyte range and contains Gigabytes).1 Businesses are clear that they billions of table rows. It is primarily an need the right set of products, tools and enterprise class database. techniques to manage such high volume of data. A VLDB is generally a repository for big data, a transactional processing system As the technology becomes cheaper and or a combination of the two, and is more widely accessible to the end users, maintained through standard relational there is more data getting produced database management system (RDBMS) than ever before. The advent of new software with levels of normalization technologies such as Internet of Things to reduce redundancy. A VLDB not only (IoT), Big Data and rise in number of requires capable hardware computing analysis to make an informed decision and storage resources but also needs the for the business, also creates a massive underlying system to have the ability to demand for databases that can store scale up to address its ever-growing size. huge volume of data for various purposes without compromising the accessibility or processing speed. 06 VLDB | Prerequisite for the success of Digital India Adoption of VLDB The enterprises are facing several • Many enterprises or Government challenges for management of organisation face regulations for storing heterogeneous, unstructured and data for a minimum period of time. These enormous data that is being produced regulations generally result in more data every second. The data originating from being stored for longer periods of time. various known and unknown sources has many forms of complexity. They need VLDB addresses these challenges and the databases that are technologically problems faced by modern enterprises by equipped to reduce the magnitude of helping them adapt to the ever increasing these challenges. VLDB appears to be a complexity of data management and cost- solution to this problem. It is being seen as effectiveness of performing operations a remedy for complex data management against a system of that size. with its ever increasing flow of churn for the factors listed below: Fig.1: Survey from ISACA Journal Vol 3 Do you have the data you Need? • For a long time, systems have been 37 developed and managed in isolation. 29 21 Enterprises have now started to 13 recognise benefits of combining these systems to enable cross-departmental No Yes, poor Good set Good set analysis while reducing system quality of data of enriched maintenance costs. 2 Consolidation of data databases and applications is a key factor According to an article published in ISACA in the ongoing growth of database size. Journal Volume 3 in which a survey was • Enterprises grow by expanding sales conducted amongst 100 attendees, 58% and operations or through mergers of attendees say they either have no and acquisitions, causing the amount data available or data available is of poor of generated and processed data to quality (as represented in the Fig.1), which increase. At the same time, the user ascertains the need for very large database population that relies on the database for among enterprises. daily activities increases. One principal advantage of large volume • The high demand, popularity and usage of data that is collected and stored on of social media (Facebook, WhatsApp, every click or tap of the user is that it can YouTube, Twitter, etc.) where various provide new insights that enable enterprise types of data such as images, videos, to make more informed decisions. This is etc. are being uploaded, shared and not only true for organisations but also downloaded every moment makes it for customers. Customers are better imperative for enterprises to have a informed to make better choices resulting robust database management solution in in improved satisfaction. place to meet these challenges. 07 BREXIT – Trigger for larger things to come? | Discussion on consequential scenarios: Focus on India While it may appear that enterprises that Starting early 2000s, the introduction stories. Departments like Income Tax, are venturing into various innovative of Telco OTT (Over the Top) services Passport Office have completely automated solutions to manage these large sets of where a telecommunications service their workflows and made them digital. data, Government organisations and provider delivers one or more services This not only helped them in running the departments are making an equal use like voice & messages, content, data, etc. business efficiently but also proved to be of these technologies. However, the This telecom growth has been further more effective than the manual processes challenges and their solutions are slightly propelled by the introduction and rapid and resulted in better outcomes and different or more than the enterprises. penetration of first multimedia phones customer satisfaction. and more recent smart phones. Such Both corporate and government rapid growth in telecommunication sector The vision of Digital India is such that organisations are grappling with the issue has been majorly possible because of citizens of the country can use government of managing large data coming from the liberalisation policies and hyper- services at their disposal which can only multiple known and unknown sources. competitive market. be achieved by making it digital. The The Digital India programme launched With the infrastructure in place and adaptability of digital services can’t be by the Government will prepare India for far reachability nature of telecom, the questioned anymore as people have a 'Knowledge Future." The focus of this infrastructure has been complimented embraced many digital businesses like programme is to make technology central by growing Information Technology ecommerce, passport services, etc.