A RESTFUL API WITH MONGODB
A Project
Presented to the faculty of the Department of Computer Science
California State University, Sacramento
Submitted in partial satisfaction of the requirements for the degree of
MASTER OF SCIENCE
in
Computer Science
by
Satia Ahmed
SPRING 2018
© 2018
Satia Ahmed
ALL RIGHTS RESERVED
ii
A RESTFUL API WITH MONGODB
A Project
by
Satia Ahmed
Approved by:
______, Committee Chair Dr. Ying Jin
______, Second Reader Dr. Jun Dai
______Date
iii
Student: Satia Ahmed
I certify that this student has met the requirements for format contained in the University format manual, and that this project is suitable for shelving in the Library and credit is to be awarded for the project.
______, Graduate Coordinator ______Dr. Jinsong Ouyang Date
Department of Computer Science
iv
Abstract
of
A RESTFUL API WITH MONGODB
by
Satia Ahmed
MongoDB is a document based NoSQL database system which stores data in the form of JSON documents. The de-normalized structure of MongoDB document makes query performance efficient and cost effective. In this project, a daycare application is implemented using MongoDB as the database. To emulate real life data, Big Data documents are generated programmatically using data generator tool from Redgate Software Ltd, called SQL Data Generator, Visual Studio
SSIS tool, and python scripting.
To connect with the database and perform CRUD (Create, Read, Update, Delete) operations on the data using RESTful API, multiple methods were surveyed in this project. These studies include dependencies and methods to wrap MongoDB with Java, C# and nodeJS. Further, this project implemented RESTful API using nodeJS, wrapping MongoDB as the backend database.
The frontend was rendered with AngularJS which communicates with the backend and allows users to query and operate on the data stored in MongoDB. A MVC (Model View Controller)
v
model was used to implement the REST API, where Model is our MongoDB database and
AngularJS is our View.
______, Committee Chair Dr. Ying Jin
______Date
vi
TABLE OF CONTENT Page
List of Tables ...... x
List of Figures ...... xi
Chapter
1. INTRODUCTION ...... 1
2. CORE CONCEPT OF MONGODB ...... 3
2.1 Database Data Model...... 3
2.2 Always-On High Availability Feature ...... 4
2.3 Horizontal scaling ...... 4
2.4 Security and storage...... 5
2.5 Download and installation process ...... 5
3. CREATING A BIG DATA DATABASE ...... 8
3.1 Designing the Document Structure ...... 8
3.2 Creating the Database ...... 18
3.3 Generating JSON Documents ...... 19
3.4 Populating the database ...... 19
3.5 Exporting and Importing Collection ...... 20
3.6 Changing Schema ...... 22 vii
4. COMMANDS AND QUERIES IN MONGODB ...... 23
4.1 CRUD Operations...... 24
4.1.1 Create ...... 24
4.1.2 Read...... 29
4.1.3 Update ...... 36
4.1.4 Delete ...... 39
4.2 Query Documents ...... 41
4.2.1 Query Embedded Documents ...... 41
4.2.2 Query Operator ...... 42
4.2.3 Projection of Array Elements ...... 46
4.3 Aggregate Operation...... 48
4.3.1 Pipeline ...... 48
4.3.2 $lookup for Left Outer Join ...... 49
4.4 Queries used in the REST API ...... 51
5. IMPLEMENT REST API USING MONGODB ...... 55
5.1 Introduction to REST...... 55
5.2 RESTful API With MongoDB as Backend ...... 57
5.2.1 MongoDB drivers ...... 57
5.2.2 Connecting with .Net: C# framework ...... 59
5.2.3 Connecting with Java framework ...... 61
5.2.4 Connecting with Node.js: JavaScript framework ...... 65
viii
5.3 MEAN stack ...... 67
5.3.1 Functional Requirements for the REST API ...... 67
5.3.2 Setting up a MEAN project ...... 69
6. CONCLUSION ...... 82
References ...... 83
ix
LIST OF TABLES
Tables Page
2.5 MongoDB Version Supported by Different OS...... 6
5.2.1 MongoDB Drivers ...... 57
x
LIST OF FIGURES
Figure Page
3.1.a Collection daycareInfo for daycare branch information ...... 9
3.1.b Collection for Employee information ...... 10
3.1.c 1:N Relationships between Program documents and Student documents ...... 11
3.1.d N:N Relationship between Program documents and Employee documents ...... 12
3.1.e Document Structure in Collection programInfo containing information about
programs ...... 13
3.1.f Array of Documents in Collection programInfo containing information about all
Students in that program...... 14
3.1.g Embedded related documents in Array of Documents Collection programInfo ...... 15
3.1.h Embedded Documents in Array of Document Structure in Collection programInfo ...... 16
3.1.h Related Embedded Documents in Collection programInformation ...... 17
3.5.a Importing documents to populate collection ...... 20
3.5.b Export Operation to download documents from the collection...... 21
4.1.1.a Create operation by single document insertion ...... 25
4.1.1.b Create Operation by multiple documents insertion ...... 26
4.1.1.c Document Inserted with JavaScript and mongoDB Insert Operation...... 27
4.1.1.d Populate Collection in database with import Operation ...... 28
4.1.2.a Read Operation Having One Filter And No Projection ...... 30
4.1.2.b Multiple Field-Value Pairs in Filter That Acts as AND Condition ...... 31
4.1.2.c Projection and Filer in Find Operation ...... 32
xi
4.1.2.d Excluding “_id” from the retrieved documents ...... 33
4.1.2.e Retrieve All Except the Boolean Zero Assigned Fields in Projection ...... 34
4.1.2.f Read Operation to Retrieve All Data ...... 35
4.1.2.g Read Operation to Retrieve the First Element...... 36
4.1.3.a display the document needed to be updated ...... 37
4.1.3.b Update operation to change value of field ...... 37
4.1.3.c Display updated document ...... 38
4.1.4.a Delete operation ...... 39
4.1.4.b Drop operation to Delete Collection ...... 40
4.2.1.a Retrieve an Embedded Document with Dot Notation ...... 41
4.2.1.b Display a Field in Embedded Document Using Projection ...... 42
4.2.2.a $in Query operator ...... 43
4.2.2.b $or Query operator ...... 44
4.2.2.c $lt operator inside $or Query operator ...... 45
4.3.1 Pipeline Aggregation Operation...... 49
4.3.2.a $Lookup for Left Outer Join ...... 50
4.3.2.b Results for $Lookup Operation ...... 51
5.2.1 MongoDB Drivers ...... 57
5.2.1.a Driver Installation via Nuget ...... 60
5.3.2.a Directory Structure of The Project ...... 71
5.3.2.b RESTful Page for CRUD Operation ...... 75
5.3.2.c Page displaying the Dashboard ...... 81
xii
1
1. INTRODUCTION
For a long time, relational database system has served applications reliably, and offered features that remain critical to date. However modern web applications raise requirements that are not addressed by the traditional relational databases [1]. This has led to the evolution of NoSQL databases. NoSQL databases meet these new imposing requirements by offering innovative new features but have failed to deliver the crucial aspects that have been anticipated and counted upon from relational databases. MongoDB binds the innovations of NoSQL while sustaining the groundwork of relational databases, according to [2].
MongoDB is an immerging NoSQL document based Big data database. Among all the immerging new NOSQL databases MongoDB is currently one of the most popular ones [3]. The flexible schema of the document structure offered by MongoDB can represent a superset of other database models. MongoDB’s multi-model design significantly reduces developer and operational complexity, as several distinct database systems can be addresses to meet application requirements [1]. The Storage engine supported by MongoDB provides encryption which ensures data security. MongoDB permits users to configure combination of multiple storage engines which helps to design a simple and steady approach to meet various application requirements for data. Conventionally, if multiple database systems are involved in one application, these are managed with complex, custom integration code so that data can be moved consistently and securely between the database systems. “With MongoDB’s flexible storage architecture, the database automatically manages the movement of data between storage engine technologies using
2 native replication” according to [1]. MongoDB is still evolving and new features and tools are being added to it with every release [1] [2].
MongoDB supports high write load that can write real time data, horizontal scalability that deploys large data set with higher throughput, high availability that makes easy recovery from a node failure, flexible schema that supports changing data, geo-spatial features that supports location based Query [3] [4], which makes MongoDB a database for modern web applications and internet infrastructure [5]. Many web applications are built on REST architecture. A RESTful
API follows a consistent design to follow REST architectural best practices to create web service.
Based on REST technology which uses HTTP request to process data. Creating a RESTful API with NoSQL database addresses all the modern web application demand and requirements. In this project, we implemented a Big Data database using MongoDB, and how RESTful APIs can be built wrapping MongoDB with three popular frameworks such as Spring MVC, ASP.NET and nodeJS. There are dependencies that needs to be addressed, to connect to these frameworks.
Studying all these dependencies, we chose to implement a RESTful API in nodeJS with
MongoDB as the backend database.
3
2. CORE CONCEPT OF MONGODB
MongoDB database management system is available as free open source and has an enterprise version, both of which are supported in Linux and Windows OS. MongoDB stores data in a binary representation called BSON (Binary JSON).
2.1 Database Data Model
Database in MongoDB consists of collections of documents that hold similar kind of data. These documents are JSON objects comprising of key value pairs. Supported data types for these values are strings, numbers and dates. Fields can support all the data types supported by relational database, more over a field can have arrays of values, or a JSON document as its value or arrays of JSON documents as its values. MongoDB requires a primary key called ObjectID for every document. ObjectID identifies each document in the collection of MongoDB database. This primary key has field name _id and value as a 24-byte hexadecimal string, which are unique for every single document. Additional secondary indexes can also be created on fields which supports ascending, descending, unique, compound-key and geospatial indexes [6].
Value for primary key, _id can be created by the user while creating the document. But as it is the primary key and index, it must be unique. If a document is inserted in the collection without a _id field then MongoDB generates unique value for this field. If a document is inserted with duplicate value in _id field then a duplicate key error is generated.
4
2.2 Always-On High Availability Feature
Replication process provides real time availability and disaster recovery solution for always on global deployments [1]. Replication process is done by a set of replica sets, which is a group of process that maintains same data set by distributing the datasets across multiple machines called nodes. This results in redundancy which makes recovery from primary failover due to server or network outage fast. Replica set consists of a topology where there is a primary node and one or more secondary nodes. Read operation is handled by both primary and secondary nodes, which is why read intensive web applications prefer MongoDB. The write operations are only handled by primary node. Whenever the primary node fails, a secondary node is selected by the cluster and promotes it to be primary, thus supporting the automatic failover. Whenever the failed primary node comes back online, it will continue running as a secondary node [7].
2.3 Horizontal scaling
“Scalability is the capability of a system to handle a growing amount of work, or its potential to be enlarged to accommodate that growth”, according to [8]. Scalability is accomplished by horizontal scaling and vertical scaling. Vertical scaling is accomplished by increasing the capacity of existing hardware or software by adding CPU and storage resources. Vertical scaling has the limitation to grow only as big as the server. On the other hand, horizontal scaling means distribute the data across multiple machines by adding more nodes/computers to the system.
Horizontal scaling can connect to the hardware or software of these machines such that they can work like a single unit [4] [9].
5
MongoDB supports horizontal scaling by a process called sharding. Sharding distributes a large collection over multiple servers called clusters or shards where each of these shards are independent database but collectively these shards make up the complete database system. The sharding process in MongoDB is automated. Adding a new cluster to the database system gets auto-sharded by distributing the data evenly among all the clusters [10].
2.4 Security and storage
MongoDB provides authentication, access control, encryption, limits network exposure, Audits system activities etc. to secure the data in deployment.
MongoDB provides a variety of storage engines to support applications with different requirements. The default storage engine is WireTiger. MongoDB enterprise version offers In- memory Storage Engine for more productivity by storing the data in memory instead of storing them in disk [11] [12].
2.5 Download and installation process
MongoDB supports multiple operating systems. There are multiple versions of MongoDB available. Table 2.5.a lists all the versions of MongoDB that are supported by Mac OS, Ubuntu and Windows operating system [1].
6
Table 2.5 MongoDB Versions Supported by Different OS
MongoDB Version Operating System 3.6 3.4 3.2 3.0
Mac 10.7+ 64-bit
Ubuntu 14.04+ 64-bit
Windows 7+ 32-bit / 64-bit
Windows Vista
For this project we installed latest release of MongoDB which is version 3.4 on Windows 10 OS by downloading the 64-bit windows version from https//www.MongoDB.org/downloads [13].
Extracted and ran the downloaded file in our project location. By default, it is installed in C drive.
After installing MongoDB, there will be a folder created with the name MongoDB. Create a folder named data in that directory. In the data folder create a folder named db. This is the default location for MongoDB data directory. Lastly, the path of these folders that were created is needed to be set up. For this we need to go to a folder named bin inside the folder MongoDB and set up the path within mongod.exe. Following are the commands ran to execute these steps.
7
C:\Users\sahmed\MongoDB>md data
C:\Users\sahmed\MongoDB>cd data
C:\Users\sahmed\MongoDB\data>md db
C:\Users\sahmed\MongoDB\data>cd..
C:\Users\sahmed\MongoDB>cd bin
C:\Users\sahmed\MongoDB\bin>mongod.exe --dbpath "C:\Users\sahmed\MongoDB\data"
To check if MongoDB is installed successfully the following command can be run.
C:\Users\sahmed\MongoDB\bin>mongo.exe
MongoDB shell version v3.4.1
connecting to: MongoDB://127.0.0.1:27017
MongoDB server version: 3.4.1
The result of this command displays the sell version, connection port and the server version of the installed MongoDB.
8
3. CREATING A BIG DATA DATABASE
The MongoDB database we decided to create is a daycare system named DayCareDB. It has a big data element and relational property among its data. The day care has 20 branches. Each branch has about 5 to 10 programs or classes offered for kids. Each of these programs has a class capacity of 20 students and two instructors. Including the instructors involved with the programs, there are management, office Clerks and kitchen workers, making about 20 employees involved in each of the branch. The big data document is generated on daily basis reporting activities performed by each kid, with additional information requested by the parents.
3.1 Designing the Document Structure
Based on the different data to be stored, four collections were created. In MongoDB, to establish relationships between logically related data, there are two approaches available: embedded and referenced approach. In this database for daycare system, mostly embedded approach was taken to store related information in the same document. To test how referenced approach works, two collections were created with referenced approach.
In this project report to demonstrate the document structures, they are represented as a block enclosed in a box for visual clarity. For fields containing a document as its value, these documents are shown as a box overlapping the original box of the document for which the field value pair belongs.
9
Figure 3.1.a Collection daycareInfo for daycare branch information
Depending on the kind of data that would be used together, four collections were created for the database. These are daycareInfo, programInfo, employeeInfo and kidReport. Figure 3.1.a illustrates the document structure for collection daycareInfo. This is a collection of documents that holds information about each branch of the daycare.
Figure 3.1.b illustrates the document structure of the collection employeeInfo. The field value pairs for these documents stores all data related to employees. These documents store the
ObjectID of related daycareInfo document, referring which employee works for which branch.
This represents a N:N relationship between the documents. This representation of relationship in
MongoDB is called referenced approach. With this approach 1:N, N:1 and N:N relationships also can be achieved.
10
Figure 3.1.b Collection for Employee information
Figure 3.1.c and figure 3.1.d presents a visual representation of how 1:N, N:1 or N:N relationships can be established between documents. In this figure 1:N relationship is displayed between the documents about program Information and documents about Student Information.
Each student has their own document containing individual student data. Each of these student document contains an ObjectID as a unique identifier of that document. In The program document, all the students who are enrolled in that program are captured by listing all the
ObjectIDs of student documents as that array elements. When a document contains ObjectID of multiple other documents, it represents a one to many (1:N) relationship between that document.
11
The many to many relationship (N:N), as shown in Figure 3.1.d, captures how multiple Program documents reference to all the employee documents and how multiple employee documents reference to multiple program documents.
Figure 3.1.c 1:N Relationships between Program documents and Student documents
12
Figure 3.1.d N:N Relationship between Program documents and Employee documents
13
Figure 3.1.e Document Structure in Collection programInfo containing information about
programs
The collection programInfo stores documents that holds information about all the programs offered by a branch of the database. In these documents all the employees involved, students enrolled and waitlist information are stored. Figure 3.1.e illustrates the document structure where the field value pairs holds data about the program information and instructor information.
14
Figure 3.1.f Array of Documents in Collection programInfo containing information about all
Students in that program
In programInfo, addition to program information and instructor information, information about all the students enrolled for that program are listed as arrays of documents as shown in Figure 3.1.f.
15
This figure, demonstrates document structure to embed documents within documents to store related information.
Figure 3.1.g Embedded related documents in Array of Documents Collection programInfo
Figure 3.1.g displays the document structure for parent information containing documents for father’s name, mother’s name and contact.
16
Figure 3.1.g Embedded Documents in Array of Document Structure in Collection programInfo
Figure 3.1.g displays the document structure containing fields for photograph permission, roll number and sex of the student. It contains a document that holds information about pickup authorization. Figure 3.1.e, Figure 3.1.f and Figure 3.1.g are part of Student document containing all student information who are enrolled in that program. Documents of all the students are embedded in programInfo as array documents.
17
Figure 3.1.h Related Embedded Documents in Collection programInformation
In Figure 3.1.h WaitList field contains arrays of documents that contains information about all the wait listed students for a program. Each of the documents in this array contains further embedded documents that capture all the details about parent information, contact information etc.
18
The collection called KidReport holds documents about kid activities which does not have a fixed schema. All the documents have few common fields such as Name, Program, Branch but additional fields can be added according to requirement. The concept of this document is to enable the day care staff to report or inform the parent about the learning activity, play activity, food consumed, nap time or any other information that the parent might have requested about their child.
3.2 Creating the Database
To create the database, we need to locate the folder named MongoDB, in the directory where
MongoDB was installed. We need redirect to data\db directory that we created in the folder
MongoDB and create the database there. Following is the syntax for creating a database. This command creates the database and switches the user to it. To switch to any other database, same command is used.
C:\Users\sahmed\MongoDB\bin> Mongo
>use dayCareDB
To confirm the creation of database, the following command displays all the databases created,
> show dbs
dayCareDB 0.003GB
local 0.000GB
19
3.3 Generating JSON Documents
Programmatically 1115600 JSON documents were generated for the collection dailyReport. For the collection daycareInfo, 20 documents were generated. For employeeInfo, 205 documents were generated. For programInfo, 122 documents were generated. Generated documents were imported in the respective collections. Import command used for this purpose is discussed in section 3.6.
3.4 Populating the database
To populate the database, we imported the programmatically generated JSON text files. The import command creates the database, creates the collection, and imports all JSON documents in the newly created collection.
Following import command was used to import the JSON documents for the collection employeeInfo in to the database DayCareDB.
:\Users\sahmed\MongoDB\bin>Mongoimport --db DayCareDB --collection
employeeInfo --type JSON --file
C:\Users\sahmed\Desktop\MongoDBmaPro\Employees9.JSON --JSON Array
017-10-15T17:07:44.467-0700 connected to: localhost
017-10-15T17:07:44.470-0700 imported 18 documents
20
3.5 Exporting and Importing Collection
It is easy to export out the whole MongoDB collection in as a JSON text file. It is useful for creating quick backups, transfer documents to another database collection etc. It is easy to import collection if JSON documents as well. Command to import JSON text file in to collection is as follows,
Mongoimport --db database Name --collection collectionName –type json --file file.json
Command to export documents from a collection and save in a file is as follows, mongoexport --db database Name --collection collectionName --out file.json
In Figure 3.5.a and Figure 3.5.b, we transferred a collection from one database to another. The export command in Figure 3.5.b exports all the documents from a collection called dailyReport, from a test database called cccDB to a json text file called kidReport2.json. We transferred this json text file to the database we are using in this project. As demonstrated in figure 3.5.a, the import command takes the text file kidReport2.json that contains all the JSON documents for the collection kidReport and populates it. It imports the file from the specified location and populates the collection kidReport in our daycare database DayCareDB.
Figure 3.5.a Importing documents to populate collection
21
Figure 3.5.b Export Operation to download documents from the collection
22
3.6 Changing Schema
One of the feature of MongoDB is dynamic schema. Schema or data model in database is designed based on application requirements. According to this feature the schema or data model structure in MongoDB is not fixed or in other words flexible or dynamic. As requirements change, MongoDB database schema can be re-modeled accordingly. During building the
RESTful API for this project, a need for schema redesigning appeared. While querying the data from the database, a need for additional fields in the documents in collection daycareInfo arose.
The change in schema was executed by a bulk update operation was performed to update values for fields and insert additional new fields with new values.
In Chapter 4, we explored all the query operations offered by MongoDB. These queries that were used to update the schema of the collection are demonstrated in that chapter.
23
4. COMMANDS AND QUERIES IN MONGODB
MongoDB shell is a standalone MongoDB client that can interpreter commands written in
JavaScript programs.
To start the MongoDB Shell from command prompt, the user has to start from the directory where MongoDB was installed. The default data directory \data\db is needed to be created beforehand for the server to run. The command ./bin/mongo starts the mongo shell. To demonstrate the commands and queries we are going to use database named dayCareDB and collections called daycareInfo, employeeInfo, programInfo and dailyReport.
Display all existing databases: >show dbs
Switch to Database: >use dayCareDB
Creating database: New database is created when the first entry of a collection to a database is made. The database in which the collection belongs to, user is needed to switch to that database before performing the query.
24
4.1 CRUD Operations
CRUD acronym stands for create, read, update and delete operations on the data. Create operation involves two types of inserts, single JSON document insert and multiple JSON documents inserts.
4.1.1 Create
Create operation that inserts one document in the collection daycareInfo, contains the document needed to be inserted. If this database is nonexistent, this create operation creates the database, create the collection according to the collection name specified in the query and the collection would contain the document.
For ease of use in this project we took the advantage of MongoDB generated Object id. While creating the documents, the object id field _id was excluded from the document and during insertion operation in the database, MongoDB included a unique Object id for each document created for the database.
Figure 4.1.1.a demonstrates command to insert one document in to the database DayCareDB.
First command is to switch to DayCareDB. Second command db.daycareInfo.insertOne() inserts a single document to the collection daycareInfo. Here insertOne function performs a create operation, daycareInfo is the collection name and db is a global variable that points to the database DayCareDB that the user is currently using. After the document is inserted MongoDB shell returns a confirmation document that consists of two field value pairs.
"acknowledged" : true, states that the document was inserted.
25
"insertedId" : ObjectId("5a0df1376faa7acf806c1563") states that MongoDB generated an object id which is the primary key for that document.
Figure 4.1.1.a Create operation by single document insertion
Multiple documents can be inserted with a single query. Documents to be inserted are put in an array and the array of documents are inserted in the collection using a single create operation. As
MongoDB supports dynamic schema, the documents may contain different schema or data model.
Best practice for working with dynamic schema is that, the developers needs to review the schema or data model thoroughly so that working with different structured documents, stored in a common collection is manageable.
In Figure 4.1.1.b, this is a Create operation that inserts multiple documents in to the collection daycareInfo with dynamic schema. After the documents are inserted MongoDB shell returns confirmation document with the “acknowledge” field and “insertedIds” field. Figure 4.3 demonstrates that the insertMany() function performs create operation by taking arrays of two
26 documents and inserts them. MongoDB shell confirms the operation by returning a document with “acknowledge”; true field value pair and ObjectIds for the inserted documents in the field inseretIds.
Figure 4.1.1.b Create Operation by multiple documents insertion
27
The documents that are inserted in the figure 4.1.1.a and 4.1.1.b does not contain a field _id. As
_id field is the primary key for the documents if user does not provide this value in the documents being created, then MongoDB auto generates this field value pair for each of these documents.
Figure 4.1.1.c Document Inserted with JavaScript and mongoDB Insert Operation
Since MongoDB shell is JavaScript shell, we can use JavaScript along with mongoDB commands to perform operations on the database. In Figure 4.1.1.c a create operation is performed by creating a local variable which is a JavaScript object representing the document we want to insert in collection called daycareInfo. A local variable named doc is created where it has a JSON document as its value. The JSON Object has the document structure which we desire to inset in collection daycareInfo, The insert command, db.daycareInfo.insertOne(doc) operates on the daycareInfo to inserts the JSON document stored in the object called doc.
28
We can create a database and populate collection by importing a JSON text file. Figure 4.1.1.d demonstrates the command. The command mongoimport imports a JSON text file from the specified location and populates the collection in the specified database. if the specified database does not exist, then the query will create a database named DayCareDB, with a collection named programInfo and will populate it with the JSON documents from the text file.
Figure 4.1.1.d Populate Collection in database with import Operation
29
4.1.2 Read
Basic syntax to Query a read operation in MongoDB is as following,
db.collection.find(
{filter },
{projection}
)
Filter consists of field value pairs and projection consists of fields. The query finds the documents that contains specified field value pairs in filter and according to the fields in projection, the query displays those specified field value pairs from the document. Filter and projection part of the query are optional.
The query uses a function find() that performs the read operation. The pretty() function displays the document in an indented format. Without this function the document is wrapped together and becomes difficult to read.
A simple read operation having one filter and no projection is demonstrated in figure 4.1.2.a, which finds all the documents in the collection employeeInfo having where the field Daycare has the value “Daycare 1”. This command displays the whole documents.
30
Figure 4.1.2.a Read Operation Having One Filter And No Projection
Filter may contain one or more field value pairs. More than one field value pair in filter is treated as AND condition between the fields. Query containing multiple field value pairs in the filter part, will display only those documents which contains all the specified fields with those specified values. this is demonstrated in figure 4.1.2.b. the command finds and displays all the documents in the collection employeeInfo where the fields Daycare have the value “Daycare 1” and field Program have the value “Program 6”.
31
Figure 4.1.2.b Multiple Field-Value Pairs in Filter That Acts as AND Condition
To add projection, we mention the fields we want to display from the filtered documents in the projection part of the query and assign those fields a Boolean value of 1. But in case if we want to display the filtered documents by excluding few fields, then in the projection part of the query we
32 assign a Boolean value of zero to those fields. As shown in figure 4.1.2.c the query displays only the specified fields. For this the fields “Name”, “Daycare” and “Program” are assigned a Boolean value of 1 in the projection part of the query. Which displays only these fields and the object id
“_id” from the filtered documents.
Figure 4.1.2.c Projection and Filer in Find Operation
We can exclude the object id by assigning it a value of zero in the same query as demonstrated in
Figure 4.1.2.d. This query in Figure 4.1.2.d, finds all the documents in employeeInfo who work in
Daycare 1 and in Program 6, and display the document with three field value pairs, Name,
Daycare and Program. In read operation ObjectID is displayed in the documents by default. But in this query since the field for ObjectID which is, _id is assigned a Boolean value of 0 in projection, the document will not display ObjectID value.
33
Figure 4.1.2.d Excluding “_id” from the retrieved documents
In figure 4.1.2.e displays the whole document excluding the specified fields “Name” and
“Contact” which are assigned a Boolean value of zero. There is restriction for assigning Boolean value of both zeros and ones in a single query. Only object id can be assigned 0 in the projection part of the query where fields are assigned value 1, as shown in Figure 4.1.2.d. In the same query display some fields and exclude some fields is not possible. Which means a read query either may contain fields to be projected or excluded.
34
Figure 4.1.2.e Retrieve All Except the Boolean Zero Assigned Fields in Projection
To retrieve and display all documents from a collection, no filter or projection is required. Figure
4.1.2.f demonstrates the read operation that retrieves all documents stored in collection daycareInfo.
35
Figure 4.1.2.f Read Operation to Retrieve All Data
A read operation performed by fineOne() function that retrieves only the first documents it finds in the collection daycareInfo. Figure 4.1.2.g demonstrate the command db.daycaeInfo.findOne() which retrieves the first document it finds in the collection daycareInfo.
36
Figure 4.1.2.g Read Operation to Retrieve the First Element.
4.1.3 Update
Update operation modifies existing documents in a collection using three functions. During building the database, for one of the collection programInfo, value for a field was entered wrong.
To correct this value an update operation is performed. In the figure 4.1.3.a the find() function containing filter and projection finds all the documents in collection programInfo where the field
Daycare has the value 4 and displays only the field programName from the retrieved documents.
The query retrieves all the documents where value of Daycare is 4 and displays the field programName. In Figure 4.1.3.b, the query updates the value of field programName for the
Document where value of Daycare is 4. Previously the value for programName was 6 and after the update, value for programName is “Program 6”. Figure 4.1.3.c retrieves the updated document to confirm the change.
37
Figure 4.1.3.a display the document needed to be updated
Figure 4.1.3.b Update operation to change value of field
38
Figure 4.1.3.c Display updated document
According to the MongoDB manual there are three functions using which update operation is performed. These are, db.collection.updateOne(
39
The update operator $set is used to modify the value of a field. We can put a field value pair in filter to filter out the document that we want to modify. In the option part of the query if we put
“upsert: true”, during an update operation, then, if no document was found that matched the filter, then, a new document will be inserted in the collection. If there is matching documents found according to the filter, then, that document will be modified or replaced [14].
4.1.4. Delete
Delete operation is performed with the functions deleteOne() and deleteMany().
Figure 4.1.4.a Delete operation
Figure 4.1.4.b demonstrate a query that deletes a collection from the database. After switching to database, we see there are four collections presents. Db.collection.drop() deletes the collection specified in the query. In Figure 4.1.4.b the query deletes the collection programInfo. In completion of the delete operation MongoDB shell returns a value true to indicate the collection is deleted.
40
Figure 4.1.4.b Drop operation to Delete Collection
41
4.2 Query Documents
Apart from the read operations discussed in section 4.1.2, there exists more operations with which documents can be retrieved or read.
4.2.1 Query Embedded Documents
To use fields in query filter for embedded documents, dot notation is used to access the nested fields.
Figure 4.2.1.a Retrieve an Embedded Document with Dot Notation
In Figure 4.2.1.a the read operation retrieves a document which contains the value “Daycare 1” for the field “Daycare” and contains embedded document with field value pair “FirstName”:
“Carol” as the value for field “Name”.
42
Figure 4.2.1.b Display a Field in Embedded Document Using Projection
In Figure 4.2.1.b, we used the same query filter but used projection to display specific field from the embedded document. Here the Contact field consists of a document named Address, which contains several field value pairs. We are interested to retrieve only the city field value pair.
Hence in the query, in the projection part, we nest the fields using dot notation to assign it a
Boolean value of one.
4.2.2 Query Operator
In a query filter, query operator can be used. Some of the query operators are $in, $or, $lt, $gt.
In Figure 4.2.2.a displays documents from the collection employeeInfo where “Daycare” is
“Daycare 1” and “FirstName” for field “Name” is “Carol” or “Cecilia”. For “Daycare” and
“Name.FirstName” implicitly logical AND is performed. The $in operator is used when the equality check is performed on the same field. In figure 4.2.2.b a logical OR operation is performed to find documents from the collection employeeInfo where Name.FirstName is
“Carrol” or “Cecilia”.
43
Figure 4.2.2.b retrieves documents from the employeeInfo collection where “Name.FirstName” is
“Carrol” OR “Name.LastName” is “Walters”. Here the logical operator OR is used to check the equality of two different fields.
Figure 4.2.2.a $in Query operator
44
Figure 4.2.2.b $or Query operator
45
Figure 4.2.2.c $lt operator inside $or Query operator
$lt, $gt are less than and greater than operator respectively. Figure 4.2.2.c displays a query where
$lt operator is used inside a OR operation to retrieve documents from the collection employeeInfo where “Program” is “Program 1” AND, “Age” is less than 25 OR “EmployeeID” is 20.
46
4.2.3 Projection of Array Elements
To project a Documents in which arrays of documents are value for a field, following projection operators are used [14].
$elemMatch projects the array element that match the value. It is useful to match field value pairs in arrays of documents for embedded structures. For example, the following query returns only the documents from the array of “Students” where "PhotographPermission": "Yes”. All the other embedded documents will not be displayed.
>db.programInfo.find({},{"Students": {$elemMatch: {"PhotographPermission": "Yes"}}})
$slice operator limits the number of elements returns from an array of documents [14]. The following query would return the first two elements from the array Students, db.programInfo.find({
"ProgramName": "Program 1",
"Daycare": 1
},
{
"Students": {$slice: 2}
})
47
The following query would return the last two elements from the array Students, db.programInfo.find({
"ProgramName": "Program 1",
"Daycare": 1
},
{
"Students": {$slice: -2}
})
$ operator is used with a field that has the documents of array as its value, to project only the first matching array document [14]. Following query displays the first element in the array Students where ProgramName is Program 1, Daycare is 2 and FirstName of the student is “Macy”. db.daycareInfo.find({
"ProgramName" :"Program 1",
"Daycare": 2,
"Students.Name.FirstName" : "Macy"
},{
"Students.$": 1
48
})
4.3 Aggregate Operation
Aggregate operations perform on multiple documents where fields are grouped together to form a single value of more significant meaning or measurement.
4.3.1 Pipeline
With aggregation pipeline documents can be displayed in aggregated result. In figure 4.3.1 the documents are grouped by the field Daycare. There are 20 Daycare documents and each have number of documents with Program information. $sum operator counts the aggregated documents for each of the Daycare and displays it in the user defined field Total_Programs_Offered.
>db.programInfo.aggregate([
{$group: {
_id: “$Daycare”,
“Total_Programs_Offered”: {$sum: 1}
}
},{$sort: {_id:1}}
])
49
Figure 4.3.1 Pipeline Aggregation Operation
4.3.2 $lookup for Left Outer Join
One of the recent feature of MongoDB which immerged in the version MongoDB 3.4 is the feature of left outer join. Using $lookup operator two collections can be joined over a common field. The aggregation query in Figure 4.3.2.a performs a left outer join between the collection employeeInfo and daycareInfo. Both are contained in the same database. The object id for the documents in daycareInfo are stored in the field _id, The object id of the daycareInfo is stored in a field called DaycareID in the documents of the collection employeeInfo. We are executing the query from employeeInfo collection, which is our local collection. From contains the name of the other collection. localField and foreignField are the common field over which the join operation will take place. Daycare_Info is a user defines field where the joined information will be documented.
Figure 4.3.2.b displays the result of the join operation. The new joined document is formed over the common value of DaycareID and _id fields from employeeInfo collection and daycareInfo
50 collection respectively. This document contains the fields from employeeInfo collection and the user defined field Daycare_Info, which contains the fields from daycareInfo collection.
Figure 4.3.2.a $Lookup for Left Outer Join
51
Figure 4.3.2.b Results for $Lookup Operation
4.4 Queries used in the REST API
Based on the database created for the daycare system a Rest API is built. The Rest API performs
CRUD operations as well as few aggregation operations on the database collections.
The following query is performed on the collection employeeInfo. This collection contains information about each employee. In each of these documents, along with employee information,
52 the daycare for which the employee works is stored. The following query groups the document based on the daycares. As a result, each group will contain all the documents of employees who work for that daycare. Counting these grouped documents gives us the total employees working in each daycare. var collection = db.collection('employeeInfo');
// Group documents by Daycare, count the number of aggregated documents for each daycare
//which proves total employees in that daycare. collection.aggregate([ { $group: { _id: "$Daycare", "count": { $sum: 1 } } },{ $sort: { _id: 1 } } ])
The following query performs a similar aggregation operation on the collection programInfo, which stores the information about all the programs offered by the daycare. These documents contain, along with the program related information, the information about which daycare they are offered. The following query groups the documents by Daycare value and count the aggregated documents which gives us the total number of programs offered by each daycare
53 var collection = db.collection('programInfo');
// Group by documents, count total number of programs offered by each document collection.aggregate([ { $group: { _id: "$Daycare", "count": { $sum: 1 } } }, { $sort: { _id: 1 } } ])
The following query is used to display total number of students in each program that is offered in a daycare.
54 var collection = db.collection('daycareInfo'); // group by document, project program count, students count collection.aggregate([ { $group: { _id: { "Daycare": "$Daycare", "ProgramName": "$ProgramName", "Students": "$Students.StudentID" }, "count": { $sum: 1 } } }, { $sort: { _id: 1 } } ])
55
5. IMPLEMENT REST API USING MONGODB
MongoDB is a database for modern web application [15]. MongoDB can handle exponentially increasing web user and growth in data. To build a REST api with MongoDB we are going to explore three different options for the programming language and related dependencies.
5.1 Introduction to REST
Representational state transfer (REST) is a standard architecture for web services that allows requesting systems to access and manipulate web resources using HTTP protocols.
REST uses a client-server model, where client sends HTTP requests (GET, POST, PUT,
DELETE), along with a URL. The URL contains encoded variable parameters that describes the object to act upon and the server responds with a valid JavaScript Object Notation (JSON) [16].
The REST architecture is a collection of six constraints which serve as a collection of standards
[17]. These are as follows,
• Client-Server: there must be a separation between server and client
• Stateless: session information is not retained by the server for multiple requests.
• Cacheable: Clients and intermediates can cache responses.
• Layered system: a transparent middleware is required for a consistent interaction between
service and consumer. The system is designed such that the client is not able to detect if
the connection between the client and the end server is direct, or there is an intermediary
in between.
• Code on demand: by transferring the executable code, improve flexibility of the client
56
• Uniform interface: this is a fundamental constraint to the design of REST service which
simplifies and decouples the architecture and allows each part to evolve independently
[18].
The CRUD acronym is used to describe database operations. CRUD stands for CREATE, READ,
UPDATE, and DELETE which map to the HTTP verbs [16], as follows:
POST: insert or create data.
GET: read data.
PUT: update data.
DELETE: delete data.
Some of the common HTTP result codes that are often used inside REST APIs according to [16] are as follows:
200 - “OK”.
201 - “Created” (for POST).
400 - “Bad Request” (missing required parameters).
401 - “Unauthorized” (Missing authentication parameters).
403 - “Forbidden” (lacking required privileges).
404 - “Not Found”.
57
5.2 RESTful API With MongoDB as Backend
Application Programming interface (API) allows two applications to communicate with each other [19]. Representative State Transfer (REST) Web API provides an interface for Web applications that communicate through internet using HTTP calls with action verbs (CRUD).
5.2.1 MongoDB drivers
MongoDB drivers are client libraries using which API communicates with MongoDB database.
These drivers handle all interaction with the database in a language appropriate to the application
[19]. MongoDB supports 11 drivers across 10 programming languages [19]. This driver as listed in the Table 5.2.1, facilitates flexibility on choosing programming language to build the API using MongoDB as the backend database [20]. The subsequent sections discuss how to use these drivers, three popular programming languages, C#, Java, and JavaScript can be paired with
MongoDB and the dependencies it carries to build REST API.
Table 5.2.1 MongoDB Drivers
Languages Release Source
C mongo-c-driver 1.8.1 github.com/MongoDB/ mongo-c-driver github.com/MongoDB/mongo-c- driver/releases/latest
C++ mongocxx 3.2.0, the MongoDB C++11 github.com/MongoDB/ MongoDB.github.io/mongo-cxx- mongo-cxx-driver driver/mongocxx-v3/installation/
58
github.com/MongoDB/ C# .NET Driver Version 2.4.4 mongo-csharp-driver github.com/MongoDB/mongo-csharp- driver/releases/latest
Java Java Driver 3.6.0-rc0 github.com/MongoDB/ github.com/MongoDB/mongo-java- mongo-java-driver driver/releases/latest
NodeJS Node Driver V2.2.33 github.com/MongoDB/n github.com/MongoDB/node-MongoDB- ode-MongoDB-native native/releases/tag/v2.2.33
Perl MongoDB-v1.8.0 github.com/MongoDB/ metacpan.org/release/MongoDB mongo-perl-driver
PHP Version 1.3.3 github.com/MongoDB/ pecl.php.net/package/MongoDB mongo-php-driver
Python pymongo 3.5.1 github.com/MongoDB/ pypi.python.org/pypi/pymongo/ mongo-python-driver
Motor motor 1.1 github.com/MongoDB/ pypi.python.org/pypi/motor/ motor
Ruby Mongo 2.4.3 github.com/MongoDB/ rubygems.org/gems/mongo mongo-ruby-driver
Scala mongo-scala-driver 2.1.3 github.com/MongoDB/ MongoDB.github.io/mongo-scala- mongo-scala-driver driver/?_ga=2.2056628.440227938.1511883513 -913280439.1484764396
59
5.2.2 Connecting with .Net: C# framework
MongoDB .NET Driver provides asynchronous interaction with MongoDB. MongoDB Driver is built on top of a Core library using which alternative or experimental high-level APIs can be built. A standalone BOSON library with a serialization infrastructure can be built using high performance serialization [21].
System Requirements for the driver requires .NET 4.5 or later to utilize the libraries. Using
Visual studio 2015 the current version of the C# Driver has been built and tested.
Dependency management based on the driver and version for MongoDB is 3.2, Supported operating systems are Mac OS X 10.7 or up, Ubuntu 15.04 or up, Windows 7 or up. For all these operating systems only 64-bit is supported. For web browser, Internet explorer is not supported.
Firefox 39.0 or up and Chrome 43 or up is supported [22].
Installation Process:
There are two options to install MongoDB driver for C# and .NET framework. These are as following,
Nuget Installation: Simplest way to use driver is through Nuget. Figure 5.2.1.a displays the UI for nuget installation from MongoDB website. [MongoDB.github.io/mongo-csharp-driver/]
Following are the 4 packages available on nuget, MongoDB.Driver, MongoDB.Driver.Core,
MongoDB.Bson, mongocsharpdriver [23].
60
Figure 5.2.1.a Driver Installation via Nuget
Binary Installation: From the release section in the GitHub repository of MongoDB, zip files for each release can be downloaded.
Connecting with MongoDB:
The following code in the client program establishes connection directly to a single MongoDB server [24],
var client = new MongoClient();
61 alternatively, using a connection string, connection with MongoDB server can be established
[24],
var client = new MongoClient("MongoDB://localhost:27017");
MongoClient is an instance of connection to database from a pool of connections. Only one instance of MongoClient is required for a single or multithreaded connection to the database.
Documentation on MongoDB .NET Driver API is available in the official website of MongoDB in the documentation section for C# drivers [25]
5.2.3 Connecting with Java framework
The recommended procedure to install driver for Java is through the dependency management system. The MongoDB-driver is the updated Java driver that includes both the legacy API as well as a new generic MongoCollection interface [26].
62
[26]
The legacy mongo-java-driver uber jar is still available. The uber jar contains: the BSON
library, the core library, and the MongoDB-driver.
63
[27]
System requirements for MongoDB Java Driver are: 1) MongoDB version 3.2 or up, 2) operating system version for Mac OS X is 10.7 or later, Ubuntu 14.04 or later, Windows 7 or later.
MongoDB is only supported in 64-bit OS. Software requirements are Java7, MongoDB Java
Driver 3.0.0, spark 1.1.1, Maven 2, mongoProc 3.0 or up. MongoDB Java driver is available as eclipse plug-in.
To connect to MongoDB from Java program a running deployment of MongoDB, MongoDB driver and some import statements are required. The import statements required in the Java program are listed as follows, [28]
import com.MongoDB.MongoClient;
import com.MongoDB.MongoClientURI;
import com.MongoDB.ServerAddress;
import com.MongoDB.MongoCredential;
64
import com.MongoDB.MongoClientOptions;
import java.util.Arrays;
According to [29], to connect to a standalone MongoDB instance the following code in the Java program is used.
MongoClient mongoClient = new MongoClient();
This creates a MongoClient() instance which establish connections to the database. One instance of class MongoClient is sufficient for multiple threads that connects to the database. If the object of the MongoClient is instantiated without any parameters then it connects to a MongoDB instance that is running on localhost on port 27017. Alternatively, explicit specification of hostname during the instantiation will connect to a MongoDB instance running on the specified host on port 27017. this can be coded as
MongoClient mongoClient = new MongoClient( "host1" );
Or connection can be established by explicit specification of the hostname and the port number as follows, MongoClient mongoClient = new MongoClient( "host1" , 27017 );
Or MongoClientURI connection string can be specified in the program to connect to a running instance of MongoDB as follows,
MongoClient mongoClient = new MongoClient (
new MongoClientURI("MongoDB://host1:27017")
65
);
5.2.4 Connecting with Node.js: JavaScript framework
NPM is a package manager for JavaScript. It is software registry with packages of code that developers can share and reuse. Node.js is installed using npm and all the dependencies are managed by it.
MongoDB node Drive installation:
MongoDB driver and its dependencies are also installed in the following command,
>npm install mongod –save
This downloads the MongoDB driver and adds dependency in the package.json file of the project.
Package.json file is where all the dependencies are recorded for a node.js project.
System Requirements for using driver for node.js requires web browser to have Firefox 39.0 or higher, or Chrome 43 or higher version. Internet explorer is not supported for drivers.
Requirement for operating system includes, Mac OS X 10.7 or higher, Ubuntu 14.04 or higher,
Windows 7 or higher. For all operating system 64 bit is supported. Required MongoDB version is
3.2 or higher. MongoDB node.js drivers are supported in Visual Studio 15 and Linux OS [29].
To connect to a single MongoDB instance MongoClient.connect method is used. A running
MongoDB deployment and driver is required. The code to be used in the nodeJS program, according to [29], to connect to a MongoDB instance running on localhost using port 27017 using
URI connection string is as follows,
66 const MongoClient = require('MongoDB').MongoClient; const assert = require('assert');
// Connection URL const url = 'MongoDB://localhost:27017';
// Database Name const dbName = 'DayCareDB';
// Use connect method to connect to the server
MongoClient.connect(url, function(err, client) {
assert.equal(null, err);
console.log("Connected successfully to server");
const db = client.db(dbName);
client.close();
});
67
5.3 MEAN stack
Among the three programming frameworks we discussed in the previous section with which the process of binding MongoDB as backend database can be achieved, we picked nodeJS to implement RESTful API on top of the MongoDB database that we created. In this RESTful Web
API MongoDB and nodeJS is used for backend and AngularJS for front end. The programming stack using these programming languages is called MEAN.
The MEAN stack can be summarized as follows
M = MongoDB: NoSQL Object Data Model for node.js.
E = ExpressJS: a lightweight web application framework.
A = AngularJS: a framework for creating HTML5 and JavaScript-rich web applications.
N = NodeJS: a server-side JavaScript interpreter.
5.3.1 Functional Requirements for the REST API
In this API two major functional features are created in two pages of the application using the front-end programming language AngularJS. These features are, 1) a dash board that displays data from the database in graphical format and 2) a single page REST interface that lets the client perform CRUD operation on the database.
These two features are explained in detail as following,
68
1) Dashboard for the day care database displays data from four different collections in graph format. i) Total number of all the employees in each of the branch of the day care is displayed in horizontal bar graph format, ii) total number of programs offered in each of the branch are displayed in line graph, iii) for each of the branches, for every program, total number of students enrolled are displayed in vertical graphs and iv) for a branch, total students enrolled, total employees working and total programs offered are displayed in pie chart format.
2) CRUD operation page where loaded for the first time all the information about all the branches are displayed by performing a read operation. Any update on the database refreshes the page automatically and the new data is displayed on the page. On this page the client can perform, i) insert new record in daycare, ii) edit an existing daycare record in the database and the page will be updated with the updated information iii) An existing daycare record can be deleted from the database.
69
5.3.2 Setting up a MEAN project
To start a Node application the package.JSON file is created first. The dependency file package.JSON , includes all the dependencies and their versions for the MEAN API.
Package.json file for this project is as follows,
{
"name": "day-care-dash-board",
"version": "1.0.0",
"description": "Daycare Dashboard",
"main": "app.js",
"author": {
"name": "sahmed"
},
"dependencies": {
"angular-chart.js": "^1.1.1",
"body-parser": "^1.18.1",
"chart.js": "^2.7.0",
"express": "^4.15.4",
"MongoDB": "^2.2.21",
"mongoose": "^4.11.11",
"node-restful": "^0.2.6",
}
}
70
In the list of dependencies MongoDB is the native driver for interacting with MongoDB instance in the project. Express is a nodeJS web application framework that will help us create our application. App.js is the main node application.
In app.js, all the modules (libraries) used in the project are declared, configuration of MongoDB database is done and connection to the database is established. ExpressJS code also resides with in app.js. ExpressJS creates an “application” which listens on port 3000 for incoming HTTP requests from the front-end AngularJS, and creates routes that map the incoming HTTP requests with MongoDB query strings
When requests are received, they are queued in the middleware. Each of these requests from the middleware is given two objects to store the results, a req (request) object and a res (results) object. The app.use() function creates new middlewares. The main middleware is called “router”, which routes each different URL/verb combination to a specific callback function. We followed
MVC structure where MongoDB is our data model. Connecting to it, first a simple get request retrieves data from a collection and displays it to the console. Next to build controller a folder called controller was created which resides in a folder called public in the application directory.
The view of the MVC model are HTML files which also resides in the public folder. Figure 5.4.a displays the directory structure of this project.
71
Figure 5.3.2.a Directory Structure of The Project
The two controllers for the two REST Web pages are controller_dashboard.js and controller.js.
These two controller files are located in the public/controller folder. The views for these two pages are dashboard.html and index.html, which are located in the public folder. The controllers take the http requests from the frontend, at the backend the data is retrieved by the corresponding query. The retrieved data is manipulated by the code in the controller and passed on to the view.
72
The view which are the HTML files, contains the AngularJS and bootstrap code from CDN links.
AngularJS is a structural framework for dynamic web applications. It allows us to use HTML as template language with which we can extend the syntax of HTML to the component of the application program. The data binding and dependency injection in AngulerJS eliminate much of the codes required to be written by the programmers. The CDN links used in both the html files are as follows, AngularJS scripts are included from Google CDN, [30]
" ajax.googleapis.com/ajax/libs/angularjs/1.6.6/angular.min.js">
The following is part of nodeJS code in the file app.js. that creates routes by mapping URL in browser with the string or path that performs a query on the database.
// Routes app.use(express.static(__dirname + '/public'));
73 app.get('/daycareList', function (req, res) {
// Use connect method to connect to the Server
MongoClient.connect(url, function (err, db) {
console.log("Connected correctly to server");
var collection = db.collection('daycareInfo');
// Get documents that match the query
collection.find({}, { "Daycare": 1, "Address.Street": 1, "Address.City": 1
}).toArray(function (err, docs) {
assert.equal(null, err);
console.log(docs);
res.setHeader('Content-Type', 'application/JSON ');
res.JSON (docs);
db.close();
});
});
});
74
In the controller.js file, the controller receives the arrays of documents returned by the query, and assigned to the variable daycare.
var refresh = function () {
$http.get("/daycareList" ).then(function (success) {
$scope.daycareList = success.data;
$scope.daycare = {};
}, function (error) {
});
});
In the View index.HTML, for each one of these arrays of documents, the variable document accesses the fields in the document and display them in tabular form as shown below,
The retrieved data is displayed as shown in Figure 5.3.2.b. where when the page is loaded it displays name and address of all the daycare branches from the database. The client can enter new daycare branch information and the ‘Add Daycare’ button will insert that data in the database. There is a ‘Delete’ and ‘Edit’ button next to each daycare information. ‘Delete’ button,
75 deletes the daycare branch information from the database. ‘Edit’ button will put the day care information in the form. Client can edit the data and ‘Update’ button will update the information in the database.
Figure 5.3.2.b RESTful Page for CRUD Operation
Using chart.js in View dashboard.html, we display the data in four different graphical representation, horizontal bar, bar, line, and pie. For the graphical representation, data is manipulated in controller, all of which are HTTP get requests performing advanced queries to group and project the data as required.
The following nodeJS code is part of the app.js code that maps URL in browser with the string/path that performs query on the database to retrieved documents groups by the specified field.
76
app.get('/dashBoardAllStudents', function (req, res) {
// Use connect method to connect to the Server
MongoClient.connect(url, function (err, db) {
console.log("Connected correctly to server");
var collection = db.collection('programInfo');
// Get first documents that match the query
collection.aggregate([{ $group: { _id: { "Daycare": "$Daycare", "ProgramName":
"$ProgramName", "Students": "$Studnets.StudentID" }, "count": { $sum: 1 } } }, { $sort: { _id: 1
} }]).toArray(function (err, docs) {
assert.equal(null, err);
console.log(docs);
res.setHeader('Content-Type', 'application/JSON ');
res.JSON (docs);
db.close();
});
});
});
77
The following controller code in controller_dashboard.js received the arrays of documents from app.js, iterates through them and set up the modules for chart, and passes them to the View dashboard.html
$http.get("/dashBoardStudents").then(function (success) {
$scope.daycareList = success.data;
$scope.daycare = {};
var dataTypeArr = [];
var dataValueArr = [];
//for (var entry in $scope.daycareList) {
for (var i = 0, len = $scope.daycareList.length; i < len; i++) {
dataTypeArr.push("Branch " + $scope.daycareList[i]._id.Daycare + " " +
$scope.daycareList[i]._id.ProgramName );
if ($scope.daycareList[i]._id.Students) dataValueArr.push($scope.daycareList[i]._id.Students.length);
else dataValueArr.push(0);
//break;
}
78
//Setting up bar chart modules var ctx = document.getElementById("myChart3");
var myChart = new Chart(ctx, {
type: 'bar',
data: {
labels: dataTypeArr,
datasets: [{
label: 'Students Enrolled',
data: dataValueArr,
backgroundColor: [ 'rgba(255, 99, 132, 0.2)', .....],
borderColor: [ 'rgba(255,99,132,1)',...],
borderWidth: 1
}]
},
options: {
scales: {
79
yAxes: [{
ticks: {
beginAtZero: true
}
}]
}
}
});
Following is the part of code in View dashboard.html that displays the third chart in Figure
5.3.2.c.
|
|
80
Figure 5.3.2.b displays four charts representing 1) total number of all the employees in each of the branch by horizontal bar chart, 2) displays total number of programs offered in each of the branch using line chart, 3) displays for each of the branches, for every program, total number of students enrolled using bar chart, and 4) displays for a branch, total students enrolled, total employees working and total programs offered using pie chart.
81
Figure 5.3.2.c Page displaying the Dashboard
82
6. CONCLUSION
MongoDB is a NoSQL BIG data management system for modern web applications. In this project, we designed and implemented a document based NoSQL database system to support a daycare system. Creating the schema design, document structure, and populating the collections were main parts of this project. This project explored the MongoDB query language and the functionality of MongoDB shell to run JavaScript based scripts. The performance of the queries and execution of them were satisfactory. This project explored ways the database can be interfaced with a RESTful API, with the drivers offered by MongoDB. These drivers facilitate the backend programming language to connect to the database. The project surveyed the dependencies for C#, JAVA and nodeJS framework to build RESTful API with the database.
Among these three frameworks that we researched, we chose nodeJS framework. This framework is known as MEAN stack. This RESTful API that we created using MEAN stack technology is a modern replacement for LAMP (Linux, Apache, MySQL, PHP/Python). The drivers provided by mongoDB manual were sufficient to perform CRUD operations on the database programmatically. Retrieved JSON documents from the database were easy to manipulate in the controller program. This project also implemented a RESTful dashboard to display the data in a graphical format.
For future work, this RESTful API can be further developed to add additional CRUD operations to enhance more user functionalities. The dashboard can be further developed to process dynamic queries from the users to display the data in various user selected format. The database can be enhanced and optimized by implementing indexes on various fields.
83
References
[1] MongoDB official website, "MongoDB Architecture," MongoDB Manual 3.4. [Online].
Available: www.MongoDB.com/MongoDB-architecture. [Accessed: Apr. 20, 2017]
[2] MongoDB official website, "Relational Vs Non-Relational Database," MongoDB
Manual 3.4. [Online]. Available: www.MongoDB.com/scale/relational-vs-non-relational-
database. [Accessed: Apr. 20, 2017]
[3] Knowledge Base of Relational and NoSQL Database Management Systems, "Complete
Ranking: DB-Engines," Jan. 2016. [Online]. Available: https//db-ngines.com/en/ranking.
[Accessed: Apr. 20, 2017]
[4] C. Isaacson, Understanding Big Data Scalability, 1st ed., Prentice Hall, 2014, pp 40-45.
[5] K. Banker, P. Bakkum, S. Verch, D. Garret and T. Hawkins, MongoDB In Action, 2nd ed.,
Manning Publication Co, 2016, pp. 3-4.
[6] K. Chodorow, MongoDB a Definitive Guide, 2nd ed., O'Rielly Media Inc, 2013, p. 13.
[7] MongoDB official website, "MongoDB Replication Process," MongoDB manual 3.4.
[Online]. Available: docs.MongoDB.com/manual/replication. [Accessed: Nov. 17, 2017]
[8] A.B. Bondi, “Characteristics of scalability and their impact on performance”, Proc. 2nd
international workshop on Software and performance (WOSP '00) New York, NY, 2000;
doi: 10.1145/350391.350432
[9] K. Chodorow, Scaling MongoDB, 1st ed., O’Rielly Media Inc, 2011, pp. 2-4.
84
[10] MongoDB official website, "Sharding,” MongoDB Manual 3.4. [Online]. Available:
https//docs.MongoDB.com/manual/sharding. [Accessed: Nov. 17, 2017]
[11] MongoDB official website, "Storage,” MongoDB manual 3.4. [Online]. Available:
https//docs.MongoDB.com/manual/storage. [Accessed: May 24, 2017]
[12] MongoDB official website, "Security,” MongoDB manual 3.4. [Online]. Available:
https//docs.MongoDB.com/manual/security. [Accessed: May 24, 2017]
[13] MongoDB official website, "Windows 64-bit MongoDB installation,". [Online].
Available: www.MongoDB.org/downloads. [Accessed: Nov. 17, 2017]
[14] MongoDB official website, "Update Documents,” MongoDB manual 3.4. [Online].
Available: https//docs.MongoDB.com/manual/tutorial/update-documents. [Accessed:
May 24, 2017]
[15] D. Golden, "Server Selection in Next Generation MongoDB Driver," blog, 27 Mar. 2015;
https//docs.MongoDB.com/manual/reference/operator/projection/positional/#proj.S.
[Accessed May 24, 2017]
[16] N. Leite, "Creating REST API using MongoDB," blog, Apr. 2015;
www.MongoDB.com/blog/post/building-your-first-application-MongoDB-creating-rest-
api-using-mean-stack-part-1. [Accessed: Nov. 17, 2017]
[17] R. T. Fielding. "REST APIs must be hypertext driven", blog, Oct. 2008;
https//roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven. [Accessed: May
24, 2017]
85
[18] M. Stowe, "Undistributed REST, A guide to designing the perfect API," blog, May 2015;
www.mulesoft.com/lp/ebook/api/restbook. [Accessed: Jan. 20, 2018]
[19]. MongoDB manual, "Application Drivers," MongoDB Manual 3.4. [Online]. Available:
https//docs.MongoDB.com/manual/applications/drivers. [Accessed: May 24, 2017]
[20] MongoDB official website, "MongoDB Drivers," MongoDB Manual 3.4. [Online].
Available: www.MongoDB.com/blog/post/server-selection-next-generation-MongoDB-
drivers?jmp=docs&_ga=2.93201688.440227938.1511883513-913280439.1484764396.
[Accessed: Nov. 17, 2017]
[21] MongoDB official website, "MongoDB Drivers and client Libraries," MongoDB
Manual 3.4. [Online]. Available: https//docs.MongoDB.com/manual/applications/drivers.
[Accessed: May 24, 2017]
[22] MongoDB official website, "MongoDB .NET Driver," [Online]. Available:
http://MongoDB.github.io/mongosharpdriver/?jmp=docs&_ga=2.258369935.440227938.
1511883513-913280439.1484764396. [Accessed: May 24, 2017]
[23] MongoDB official website, "MongoDB C# Drivers," MongoDB Manual 3.4. [Online].
Available: http://MongoDB.github.io/mongo- csharpdriver/2.2/getting_started.
[Accessed: May 24, 2017]
[24] MongoDB official website, "Installing MongoDB driver for C#," MongoDB Manual 3.4.
[Online]. Available: http://MongoDB.github.io/mongo-csharp-driver/2.2/getting_started.
[Accessed: May 24, 2017]
86
[25] MongoDB official website, "Projects with MongoDB and C# using mongo-csharp-
drivers," MongoDB Manual 3.4. [Online]. Available:
http://api.MongoDB.com/csharp/2.2/html/R_Project_CSharpDriverDocs.htm.
[Accessed May 24, 2017]
[26] MongoDB official website, "Java Driver for connecting with MongoDB," MongoDB
Manual 3.4. [Online]. Available: http://MongoDB.github.io/mongo-java-driver.
[Accessed: Nov. 17, 2017]
[27] MongoDB official website, "Java API connector for MongoDB," MongoDB Manual 3.4.
[Online]. Available: https//docs.MongoDB.com/spark-connector/current/java-api.
[Accessed: Apr. 20, 2017]
[28] MongoDB official website, "Driver Tutorial for java to connect to MongoDB,"
MongoDB Manual 3.4. [Online]. Available: https//github.com/mongodb/mongo-java-
driver. [Accessed: Apr. 20, 2017]
[29] MongoDB official website, "MongoDB NodeJS Driver," MongoDB Manual 3.4.
[Online]. Available: https//university.MongoDB.com/courses/M101N/about.
[Accessed: Apr. 20, 2017]
[30] AngularJS CDN link, [Online]. Available: https//docs.angularjs.org/misc/downloading.
[Accessed: Apr. 20, 2017]
[31] Bootstrap CDN link, V3.3.7, [Online]. Available: www.bootstrapcdn.com. [Accessed:
Apr. 20, 2017]