Lab Assignment 3 on JSON CIS 612 Big Data Processing Sunnie Chung

Semi-Structured Data Processing:

Creating NoSQL Database and Transforming JSON Data to CSV

Create a database for the given Yelp data in NoSQL Mongo DB and Convert the data in JSON format to a structured file format (CSV, TSV) so that the converted files can be created as database tables in any server in a table format (any SQL server or Hive/PIG). You can write this lab in your choice of any language.

For this lab, we assume that you already requested a file for business data and received a JSON file ( business100.json ) from the Yelp site.

Your task is

1. Create a database for the given Yelp data in NoSQL Mongo DB. 2. Retrieve the Yelp Data in MongoDB to Transform the data in a JSON format to the table structured format (one business info with all the column values per line) so that the converted files can be stored to Relational Database tables. Usually it is not a good idea to create one big dirty CSV file. You can create multiple CSV files in Relational database scheme to convert one JSON file in your program. You need to create a database in your SQL Server from the converted files so that you can build your Web Service to display any restaurant information on your webpage per your customer’s request in a real time . Download the zip file on the Lab3 JSON section on the class webpage for the Jason data file that has 100 business data (business100.json) and a file (OnebusinessDataFormat_yelp.json) that shows the JSON file format.

To see the Yelp Business Data Format, open the file “OnebusinessDataFormat_yelp.json” using NotePad++.

For the MongoDB database creation and JDBC/ODBC set up for MongoDB, See MongoDB download and installation guide and MongoDB CRUDE Tutorial in the Class Lecture Notes section or Lab3_3 on JSON section.

Automatic database table creation in a SQL Server in your program using JDBC/ODBC would be the best and recommended. However, you don’t need to make the database table creation part in your codes if you don’t know how. This is because there is some set up procedure if you are writing in Java that uses JDBC and try to connect to MS SQL Server which uses ODBC. You can use MySQL instead of MS SQL Server to avoid the extra JDBC-ODBC Bridge set up if you are using JDBC in JAVA. You can create a Stored Procedure separately to create database tables from your outputs.

This lab is to learn JSON data processing with NoSQL MongoDB and how to handle a multivalued columns, nested and irregular data in semi-structured data to transform them to a correct relational database scheme, which is a common task in real life applications.

There are many ways to transform JSON to Table. You are to find a way to do this task, any way would be good. Two common ways could be:

1) Create a big dirty table in CSV in your program and

2) Create multiple tables in correct scheme reading from the big table in a Stored Procedure in a SQL Server.

II.

1) Design a scheme (multiple CSV file formats) and create multiple CSV files in your program and

2) Create database tables directly from each CSV in a Stored Procedure in a SQL Server.

Design your CSV file formats. There is no one strict CSV file format as a solution. Think about what would be a good format to transform those irregular multi valued data or nested data to a table (or connected multiple tables) so that you can retrieve them from a database easily and efficiently without losing information. As long as it is transformed correctly without losing data, any form would be good.

Invalid JASON format handling:

If there is any invalid data format is found in the input file, you can change it to the correct JSON format. For example, $$ in the “Price Range” key value pair in your input json file, the value $$ is not in quotes and this will cause to fail. If , is missing between objects, add it.