“Real Time Stock Market Prediction Using Back Propagation Algorithm”

DEPARTMENT OF INFORMATION SCIENCE AND ENGINEERING

A PROJECT REPORT ON

ON “REAL TIME STOCK MARKET PREDICTION USING BACK PROPAGATION ALGORITHM”

Submitted in the partial fulfillment of the requirements in the 8th semester of

BACHELOR OF ENGINEERING IN INFORMATION SCIENCE AND ENGINEERING

By Girish B K 1NH15IS037

Under the guidance of

Mrs. VANDANA Sr.Assistant Professor, Dept. of ISE, NHCE

DEPARTMENT OF INFORMATION SCIENCE AND ENGINEERING

CERTIFICATE

Certified that the project work entitled “Real Time Stock Market Prediction Using Back propagation Algorithm ”, carried out by Mr. GIRISH B K,1NH15IS037, a bonafide student of NEW HORIZON COLLEGE OF ENGINEERING, Bengaluru, in partial fulfillment for the award of Bachelor of Engineering in Information Science and Engineering of the Visveshwaraiah Technological University, Belgaum during the year 2018-19.It is certified that all corrections/suggestions indicated for Internal Assessment have been incorporated in the Report deposited in the departmental library. The project report has been approved as it satisfies the academic requirements in respect of Project work prescribed for the said Degree.

……………………. ……………………. ……………………. Prof. Vandana Dr. R J Anandhi Dr. Manjunatha Internal Guide HOD, Dept. of ISE Principal,NHCE

External Viva Name of the Examiners Signature with Date

1. ______

2. ______

DEPARTMENT OF INFORMATION SCIENCE AND ENGINEERING

DECLARATION

I hereby declare that I have followed the guidelines provided by the Institution in preparing the project report and presented report of project titled “Real Time Stock Market Prediction Using Backpropagation Algorithm”, and is uniquely prepared by me after the completion of the project work. I also confirm that the report is only prepared for my academic requirement and the results embodied in this report have not been submitted to any other University or Institution for the award of any degree.

Signature of the Student

Name: GIRISH BK USN: 1NH15IS037

ABSTRACT

Stock market index forecasting is one of the most challenging tasks for one who wants to invest in stock market. This challenge is because of the uncertainty and volatility of the stock prices in the market. Due to advancement in technology and globalization of business and share markets it is important to forecast the stock prices more quickly and accurately. Machine Learning techniques are most accepted due to the capability of identifying stock trend from massive amounts of data that capture the underlying stock price movement. Multi Linear Regression architecture with back propagation algorithm has the capability to predict with greater accuracy. A feed-forward neural network using multiple back propagation algorithm has been used to forecast next day’s OHLC data. This model has used the pre- processed dataset of Open price (O), High price (H), Low price (L), Close price (C), Volume Traded (V) and Turnover (T) for the period of 1 year. The root mean square error (RMSE) is chosen as indicators of performance of the network.

ACKNOWLEDGEMENT

Any achievement, be it scholastic or otherwise does not depend solely on the individual efforts but on the guidance, encouragement and cooperation of intellectuals, elders and friends. A number of personalities, in their own capacities have helped me in carrying out this project. I would like to take an opportunity to thank them all.

First and foremost I thank the management, Dr. Mohan Manghnani, Chairman, New Horizon Educational Institutions for providing us the necessary state of art infrastructure to do Project.

I would like to thank Dr.Manjunatha, Principal, New Horizon College of Engineering, Bengaluru, for his valuable suggestions and expert advice.

I would like to thank Dr. R J Anandhi, Professor and Head of the Department, Information Science and Engineering, New Horizon College of Engineering, Bengaluru, for constant encouragement and support extended towards completing my Project.

I deeply express my sincere gratitude to my guide Mrs.Vandana, Sr. Assistant Professor, Department of ISE, New Horizon College of Engineering, Bengaluru, for her able guidance, regular source of encouragement and assistance throughout my project period.

Last, but not the least, I would like to thank my peers and friends who provided me with valuable suggestions to improve my project.

Girish B K (1NH15IS037)

TABLE OF CONTENTS

Abstract i

Acknowledgement ii

Table of Contents iii

List of Figures iv

1. Preamble 1

1.1 Introduction 1

1.2 Relevance of the Project 2

1.3 Purpose 3

1.4 Scope of the Project 3

1.5 Problem Statement and Definition 4

1.6 Objective of study 4

1.7 Existing System 4

1.8 Proposed system 5

2. Literature Survey 6

3. System Requirements and specifications 8

3.1 General Description of the system 8

3.1.1 Overview of Functional requirements 8

3.1.2 Overview of Data requirements 9 3.2 Technical Requirements of the System 9 3.2.1 Hardware Requirements 10 3.2.2 Software Requirements 10 3.3 Language specification 11 3.3.1 Python 12 3.3.2 Machine Learning Features 13

3. 3. 3 Logistic Regression 14

4. System Design and Analysis 15

4.1 Preliminary Design 15

4.2 System Architecture 15

4.3 Data Flow Diagram 16

4.3.1 DFD for Data Extraction 16

4.3.2 DFD for Classification of Data 16 4.3 Use Case Diagram 17 5. Implementation 18 5.1 Different modules of the project 26 5.2 Flow Chart Of The Proposed System 28 6. Experimental Results 29 6.1 Outcomes of the Proposed System 33 7. Testing 34 7.1 Testing and Validations 34 7.2 Testing Levels 34 7.2.1 Functional Testing 35 7.2.2 Non-Functional Testing 36 7.3 White Box Testing 37 7.4 Different Stages of Testing 38 7.4.1 Unit Testing 38 7.4.2 Integration Testing 39 7.4.3 System Testing 42 7.4.4 Acceptance Testing 43 8. Conclusion and Future Enhancement 44 8.1 Conclusion 45 Reference 45

iii

LIST OF FIGURES

Figure no Figure Name Page no 1 Preliminary Design 15 2 System Architecture 15 3 Data Flow Diagram 15 4 DFD for Data Extraction 16 5 DFD for Classification of 16 Data 6 Use Case Diagram 17 7 Training Data 27 8 Flow Chart Of The Proposed 28 System 9 Data distribution 31 10 Non-Functional Testing 36 11 Unit Testing 39

Stock Market Prediction

CHAPTER 1 PREAMBLE

1.1 INTRODUCTION:

A share market is a market in which shares of publicly held companies or people are traded to raise money. The fluctuation of prices of shares depends upon the demand and supplies of shares. Only the registered Companies are allowed to carry out trading. Stock market forecasting is the process of trying to predict the future stock value of a company. The successful prediction of a stock's prices would return significant profit. Forecasting stock market index is an important financial problem that is receiving increasing diligence. A number of neural network models have been proposed for attaining accurate prediction results. Economies of a country are strongly linked and heavily influenced by the performance of the stock market. In India stock prices are basically influenced by sentimental factors and event driven activities. The investors at a large are still at a nascent stage and lack required knowledge and experience before investing in stock market.

Most of them just do it for the sake of trying their luck. A few who own good resources and are serious about it often tend to invest based on good fundamentals, technical indicators and news channel based advises. We often see a sudden spurt or spike in stock prices as a result of these. There are many techniques in the literature and applications to predict short term movements based on different stochastic models. We have proposed here a model which will not require any previous experience of trading for investing into the stock market. The model suggests trend of stock index based on historical prices.

Dept of ISE, NHCE Page 1

Stock Market Prediction

1.2 Relevance of the Project

Predicting how the stock market will perform is one of the most difficult things to do. There are so many factors involved in the prediction – physical factors vs. physhological, rational and irrational behaviour, etc. All these aspects combine to make share prices volatile and very difficult to predict with a high degree of accuracy. Using features like the latest announcements about an organization, their quarterly revenue results, etc., machine learning techniques have the potential to unearth patterns and insights we didn’t see before, and these can be used to make unerringly accurate predictions. In this article, we will work with historical data about the stock prices of a publicly listed company. We will implement a mix of machine learning algorithms to predict the future stock price of this company, starting with simple algorithms like averaging and linear regression, and then move on to advanced techniques like RSME.

The core idea behind this article is to showcase how these algorithms are implemented. I will briefly describe the technique and provide relevant links to brush up on the concepts as and when necessary. In case you’re a newcomer to the world of time series, I suggest going through the following articles first:

1.3 PROBLEM STATEMENT AND EXPLANATION

From the given data create a Machine Learning model that can predict Stock market movement (Closing price) most accurately. We’ll dive into the implementation part of this article soon, but first it’s important to establish what we’re aiming to solve. Broadly, stock market analysis is divided into two parts – Fundamental Analysis and Technical Analysis.

• Fundamental Analysis involves analyzing the company’s future profitability on the basis of its current business environment and financial performance. • Technical Analysis, on the other hand, includes reading the charts and using statistical figures to identify the trends in the stock market.

Dept of ISE, NHCE Page 2

Stock Market Prediction

As you might have guessed, our focus will be on the technical analysis part. We’ll be using a dataset from Nifty 50(you can find historical data for various stocks here) and for this particular project, I have used the data for ‘NSE India site’.

Purpose

A Machine Learning Model for Stock Market Prediction. Stock market prediction is the act of trying to determine the future value of a company stock or other financial instrument traded on a financial exchange. Proposed model is based on the study of stocks historical data and technical indicators.

1.4 Objective of the study

The main objective of this research is to predict the market performance of Karachi Stock Exchange (KSE) on day closing using different machine learning techniques. The prediction model uses different attributes as an input and predicts market as Positive & Negative. The attributes used in the model includes Oil rates, Gold & Silver rates, Interest rate, Foreign Exchange (FEX) rate, NEWS and social media feed. The old statistical techniques including Simple Moving Average (SMA) and Autoregressive Integrated Moving Average (ARIMA) are also used as input. The machine learning techniques including Single Layer Perceptron (SLP), Multi-Layer Perceptron (MLP), Radial Basis Function (RBF) and Support Vector Machine (SVM) are compared. All these attributes are studied separately also. The algorithm MLP performed best as compared to other techniques. The oil rate attribute was found to be most relevant to market performance. The results suggest that performance of KSE-100 index can be predicted with machine learning techniques.

Stock market is a very volatile in-deterministic system with vast number of factors influencing the direction of trend on varying scales and multiple layers. Efficient Market Hypothesis (EMH) states that the market is unbeatable. This makes predicting

Dept of ISE, NHCE Page 3

Stock Market Prediction the uptrend or downtrend a very challenging task. This research aims to combine multiple existing techniques into a much more robust prediction model which can handle various scenarios in which investment can be beneficial. Existing techniques like sentiment analysis or neural network techniques can be too narrow in their approach and can lead to erroneous outcomes for varying scenarios. By combing both techniques, this prediction model can provide more accurate and flexible recommendations. Embedding Technical indicators will guide the investor to minimize the risk and reap better returns.

1.6 Existing System

There have been numerous attempt to predict stock price with Machine Learning. The focus of each research project varies a lot in three ways. (1) The targeting price change can be near-term (less than a minute), short-term (tomorrow to a few days later), and long-term (months later). (2) The set of stocks can be limited to less than 10 particular stock, to stocks in a particular industry, to generally all stocks. (3) The predictors used can range from global news and economy trend, to particular characteristics of the company, to purely time series data of stock price.

LIMITATIONS

• ANN, MLP, Feed Forward, Back Propagation: Data train very slowly and requires lots of training data.

• Single Layer Perceptron, LP, RBF and SVM: The most related attribute to the KSE performance was found to be the petrol prices.

• Linear Regression and Neural Network Standard FFB: Applicable to the Single trading company.

Dept of ISE, NHCE Page 4

Stock Market Prediction

1.7 Proposed system

Our system is based on real time data thus it can be used anytime. The system takes historical data from yahoo finance as the input and then uses the back-propagation algorithm to predict the future values of the stock using concept of artificial neural networks Our system provides features like comparing two or more stocks and helping the user to select best available stock from the selected once. System provides detail analysis regarding including the graph of a particular stock. The system also has a feature of sending alert messages to the user regarding the information about the stock.

Advantages of Proposed System

• More accurate result. • Able to detect different fraudulent behavior. • Cost and Time efficient. • Biases, gone forever. • Cross-checking results become important. • Consistency assured.

Dept of ISE, NHCE Page 5

Stock Market Prediction

CHAPTER 2

LITERATURE SURVEY

1. Statistical Computational Approach:

Debadrita Banerjee’s Stock Prediction Model is based on the statistical computational approach that utilizes Autoregressive Integrated Moving Average (ARIMA) [1]. The author has analyzed month-wise data that has been obtained from the S&P Bombay Stock Exchange Sensitive Index. The dataset spans across a duration of six consecutive years from 2007 to 2012. Based on the analysis, the author has tried to predict the unobserved sensex indices. Durbin-Watson Test is applied on the accumulated dataset to determine whether Regression Analysis is required or Time-Series Analysis is required to be performed. Once the time-series characteristic of the dataset is ensured, Auto-Correlation and Partial Auto-Correlation are calculated to obtain a basic idea of the optimal stock prediction model. The most suitable ARIMA model is shortlisted by trying out different combinations of the number of auto-regressive terms, non-seasonal differences and lagged forecast errors in the prediction equation. Quantitative evaluation of this model is carried out for measuring the prediction accuracy

2. Artificial Neural Networks Approach: Abdalla, A. Elfaki, and Y. Murtadha have introduced a new methodology for the optimization of ANN parameters [2]. Their method introduces a process of training the ANN which is effective and also less humandependent. Based on their survey of the related work in this field, they have addressed ANN designing problems with the help of a Genetic Algorithm. They have gathered datasets from two sources. The first is the PETRONAS Penapisan (Melaka) Sdn Bhd from January, 2007 to February, 2007 while the other dataset is the XOR standard problem dataset. Data collection is followed by the pretreatment and analysis of the dataset during which outlier errors and invalid observations are removed using MATLAB functions. Then the preparation and scaling of the dataset is carried out. Their ANN Development Model utilizes the combination of ANN with Genetic Algorithm from which the latter is used to obtain the optimal ANN design and training parameters. The parameters represented using Genetic Algorithm Encoding are namely, network architecture, activation function, training algorithm, learning rate, momentum rate and number of epochs.

Dept of ISE, NHCE Page 6

Stock Market Prediction

In the last two decades forecasting of stock returns has become an important field of research. In most of the cases the researchers had attempted to establish a linear relationship between the input macroeconomic variables and the stock returns. But with the discovery of nonlinear trends in the stock market index returns[4], there has been a great shift in the focus of the researchers towards the nonlinear prediction of the stock returns. Although, there after many literature have come up in nonlinear statistical modeling of the stock returns, most of them required that the nonlinear model be specified before the estimation is done. But for the reason that the stock market return being noisy, uncertain, chaotic and nonlinear in nature, ANN has evolved out to be better technique in capturing the structural relationship between a stock’s performance and its determinant factors more accurately than many other statistical techniques[5] In literature, different sets of input variables are used to predict stock returns. In fact, different input variables are used to predict the same set of stock return data.

Dept of ISE, NHCE Page 7

Stock Market Prediction

CHAPTER 3 SYSTEM REQUIREMENTS AND SPECIFICATIONS

3.1 GENERAL DESCRIPTION OF THE SYSTEM

A proper and thorough literature survey concludes that there are various methods that can be used to detect credit card fraud detection. Some of these approaches are:

• Artificial Neural Network • Bayesian Network • Back Propagation Algorithm • Multiple Linear Regression Algorithm

3.1.1 Overview of Functional requirements

Preprocess Data

Data Preprocessing is a technique that is used to convert the raw data into a clean data set. In other words, whenever the data is gathered from different sources it is collected in raw format which is not feasible for the analysis.

Therefore, certain steps are executed to convert the data into a small clean data set. This technique is performed before the execution of Iterative Analysis. The set of steps is known as Data Preprocessing. It includes

 Data Cleaning  Data Integration  Data Transformation  Data Reduction

Data Preprocessing is necessary because of the presence of unformatted real-world data. Mostly real-world data is composed of –

Dept of ISE, NHCE Page 8

Stock Market Prediction

Inaccurate data (missing data) - There are many reasons for missing data such as data is not continuously collected, a mistake in data entry, technical problems with biometrics and much more.

The presence of noisy data (erroneous data and outliers) - The reasons for the existence of noisy data could be a technological problem of gadget that gathers data, a human mistake during data entry and much more.

Inconsistent data - The presence of inconsistencies are due to the reasons such that existence of duplication within data, human data entry, containing mistakes in codes or names, i.e., violation of data constraints and much more.

Train and Test Data Creation

The data we use is usually split into training data and test data. The training set contains a known output and the model learns on this data in order to be generalized to other data later on. We have the test dataset (or subset) in order to test our model’s prediction on this subset.

Model Creation

 The process of training an ML model involves providing an ML algorithm (that is, the learning algorithm) with training data to learn from. The term ML model refers to the model artifact that is created by the training process.  The training data must contain the correct answer, which is known as a target or target attribute. The learning algorithm finds patterns in the training data that map the input data attributes to the target (the answer that you want to predict), and it outputs an ML model that captures these patterns.  You can use the ML model to get predictions on new data for which you do not know the target. For example, let's say that you want to train an ML model to predict if an email is spam or not spam. You would provide training data that contains emails for which you know the target (that is, a label that tells whether an email is spam or not spam). Machine would train an ML model by using this data, resulting in a model that attempts to predict whether new email will be spam or not spam.

Dept of ISE, NHCE Page 9

Stock Market Prediction

 In our project we are using Multiple Linear Regression Algorithm to build our Model on Stock Price Prediction Dataset.

Result Analysis

In this final phase, we will test our model on our prepared dataset and also measure the Fraud detection performance on our dataset. To evaluate the performance of our created classification and make it comparable to current approaches, we use Accuracy to measure the effectiveness of classifiers. We consider the Fraud class as a negative class and Non-Fraud class as a positive class.

3.1.2 OVERVIEW OF NON FUNCTIONAL REQIREMENTS

Nonfunctional requirements describe how a system must behave and establish constraints of its functionality. This type of requirements is also known as the system’s quality attributes. Attributes such as performance, security, usability, compatibility are not the feature of the system, they are a required characteristic. They are "developing" properties that emerge from the whole arrangement and hence we can't compose a particular line of code to execute them. Any attributes required by the customer are described by the specification. We must include only those requirements that are appropriate for our project.

Some Non-Functional Requirements are as follows:

• Reliability The structure must be reliable and strong in giving the functionalities. The movements must be made unmistakable by the structure when a customer has revealed a couple of enhancements. The progressions made by the Programmer must be Project pioneer and in addition the Test designer. • Maintainability The system watching and upkeep should be fundamental and focus in its approach. There should not be an excess of occupations running on diverse

Dept of ISE, NHCE Page 10

Stock Market Prediction

machines such that it gets hard to screen whether the employments are running without lapses.

• Performance The framework will be utilized by numerous representatives all the while. Since the system will be encouraged on a single web server with a lone database server outside of anyone's ability to see, execution transforms into a significant concern. The structure should not capitulate when various customers would use everything the while. It should allow brisk accessibility to each and every piece of its customers. For instance, if two test specialists are all the while attempting to report the vicinity of a bug, then there ought not to be any irregularity at the same time. • Portability The framework should to be effectively versatile to another framework. This is obliged when the web server, which s facilitating the framework gets adhered because of a few issues, which requires the framework to be taken to another framework. • Scalability The framework should be sufficiently adaptable to include new functionalities at a later stage. There should be a run of the mill channel, which can oblige the new functionalities. • Flexibility Flexibility is the capacity of a framework to adjust to changing situations and circumstances, and to adapt to changes to business approaches and rules. An adaptable framework is one that is anything but difficult to reconfigure or adjust because of diverse client and framework prerequisites. The deliberate division of concerns between the trough and motor parts helps adaptability as just a little bit of the framework is influenced when strategies or principles change.

Dept of ISE, NHCE Page 11

Stock Market Prediction

3.2 Technical Requirements of the System

3.2.1 Hardware Requirements

Hardware Requirements • System Processor : Core i3 / i5 • Hard Disk : 500 GB. • Ram : 4 GB.  Any desktop / Laptop system with above configuration or higher level.

Software Requirements

• Operating system : Windows 8 / 10. • Programming Language : Python • DL Libraries : Numpy, Pandas

3.3 LANGUAGE SPECIFICATION

3.3.1 PYTHON INTRODUCTION:

Python is an easy to learn, powerful programming language. It has efficient high-level data structures and a simple but effective approach to object-oriented programming. Python’s elegant syntax and dynamic typing, together with its interpreted nature, make it an ideal language for scripting and rapid application development in many areas on most platforms.

The Python interpreter and the extensive standard library are freely available in source or binary form for all major platforms from the Python Web site, https://www.python.org/, and may be freely distributed. The same site also contains distributions of and pointers to many free third party Python modules, programs and tools, and additional documentation.

The Python interpreter is easily extended with new functions and data types implemented in C or C++ (or other languages callable from C). Python is also suitable as an extension language for customizable applications.

Dept of ISE, NHCE Page 12

Stock Market Prediction

Python is a high-level, interpreted, interactive and object-oriented scripting language. Python is designed to be highly readable. It uses English keywords frequently where as other languages use punctuation, and it has fewer syntactical constructions than other languages.

• Python is Interpreted − Python is processed at runtime by the interpreter. You do not need to compile your program before executing it. This is similar to PERL and PHP.

• Python is Interactive − you can actually sit at a Python prompt and interact with the interpreter directly to write your programs.

• Python is Object-Oriented − Python supports Object-Oriented style or technique of programming that encapsulates code within objects.

• Python is a Beginner's Language − Python is a great language for the beginner-level programmers and supports the development of a wide range of applications from simple text processing to WWW browsers to games.

3.3.2 MACHINE LEARNING FEATURES

Machine Learning is a method of statistical learning where each instance in a dataset is described by a set of features or attributes. In contrast, the term “Deep Learning” is a method of statistical learning that extracts features or attributes from raw data. Deep Learning does this by utilizing neural networks with many hidden layers, big data, and powerful computational resources. The terms seem somewhat interchangeable, however, with Deep Learning method, The algorithm constructs representations of the data automatically. In contrast, data representations are hard- coded as a set of features in machine learning algorithms, requiring further processes such as feature selection and extraction, (such as PCA).

Both of these terms are in dramatic contrast with another class of classical artificial intelligence algorithms known as Rule-Based Systems where each decision is manually programmed in such a way that it resembles a statistical model.

Dept of ISE, NHCE Page 13

Stock Market Prediction

In Machine Learning and Deep Learning, there are many different models that fall into two different categories, supervised and unsupervised. In unsupervised learning, algorithms such as k-Means, hierarchical clustering, and Gaussian mixture models attempt to learn meaningful structures in the data. Supervised learning involves an output label associated with each instance in the dataset. This output can be discrete/categorical or real-valued. Regression models estimate real-valued outputs, whereas classification models estimate discrete-valued outputs. Simple binary classification models have just two output labels, 1 (positive) and 0 (negative). Some popular supervised learning algorithms that are considered Machine Learning: are linear regression, logistic regression, decision trees, support vector machines, and neural networks, as well as non-parametric models such as k-Nearest Neighbors.

Dept of ISE, NHCE Page 14

Stock Market Prediction

CHAPTER 4

SYSTEM DESIGN AND ANALYSIS

4.1 System Architecture

4.2 Data Flow Diagram

Dept of ISE, NHCE Page 15

Stock Market Prediction

4.2.1 DFD for Data Extraction

4.2.2 DFD for Classification of Data

Dept of ISE, NHCE Page 16

Stock Market Prediction

4.3 Use Case Diagram

Dept of ISE, NHCE Page 17

Stock Market Prediction

CHAPTER 5 IMPLEMENTATION

In this work, a business intelligent model has been developed, to classify different animals, based on a specific business structure deal with Animal classification using a suitable machine learning technique. The model was evaluated by a scientific approach to measure accuracy. We are using Convolutional Neural Network (CNN) to build our model.

Analysis:

Prediction in the stock market is challenging and complicated for investors. Many researches have performed to sense the future market movements. In the stock market, social media data have high impact today than ever. In this work, various prediction algorithms are analyzed to build a prediction model. The prediction model will be based on monthly prediction and daily prediction to forecast the next day market price. This model estimates the open value of the next day in the market. Sentiment Analysis needs to identify and extract sentiments from each individual in the social media. The correlation between the sentiments and the stock value is to be determined. A comparative study of these three algorithms: Multiple Linear Regression, Support Vector Machine and Artificial Neural Network are done. The stock price is predicted by sentiment analysis with the best forecasting algorithm.

After model building, knowing the power of model prediction on a new instance, is very important issue. Once a predictive model is developed using the historical data, one would be curious as to how the model will perform on the data that it has not seen during the model building process. One might even try multiple model types for the same prediction problem, and then, would like to know which model is the one to use for the real-world decision making situation, simply by comparing them on their prediction performance (e.g., accuracy). To measure the performance of a predictor, there are commonly used performance metrics, such as accuracy, recall etc. First, the most commonly used performance metrics will be described, and then some famous estimation methodologies are explained and compared to each other. "Performance Metrics for Predictive Modeling In classification problems, the primary source of performance measurements is a coincidence matrix (classification matrix or a

Dept of ISE, NHCE Page 18

Stock Market Prediction

contingency table)”. Above figure shows a coincidence matrix for a two-class classification problem. The equations of the most commonly used metrics that can be calculated from the coincidence matrix are also given in Fig 2.7.

As being seen in above figure, the numbers along the diagonal from upper-left to lower- right represent the correct decisions made, and the numbers outside this diagonal represent the errors. "The true positive rate (also called hit rate or recall) of a classifier is estimated by dividing the correctly classified positives (the true positive count) by the total positive count. The false positive rate (also called a false alarm rate) of the classifier is estimated by dividing the incorrectly classified negatives (the false negative count) by the total negatives. The overall accuracy of a classifier is estimated by dividing the total correctly classified positives and negatives by the total number of samples.

The architecture of a Stock Price Prediction is analogous to that of the connectivity pattern of Neurons in the Human Brain and was inspired by the organization of the Visual Cortex. Individual neurons respond to stimuli only in a restricted region of the visual field known as the Receptive Field. A collection of such fields overlap to cover the entire visual area.

Flexibility

Sometimes you just don’t want to use what is already there but you want to define something of your own (for example a cost function, a metric, a layer, etc.). Although Keras 2 has been designed in such a way that you can implement almost everything you want but we all know that low-level libraries provides more flexibility. Same is the case with TF. You can tweak TF much more as compared to Keras.

Functionality

Although Keras provides all the general purpose functionalities for building Deep learning models, it doesn’t provide as much as TF. TensorFlow offers more advanced operations as compared to Keras. This comes very handy if you are doing a research or developing some special kind of deep learning models. Some examples regarding high level operations are:

Dept of ISE, NHCE Page 19

Stock Market Prediction

Threading and Queues

Queues are a powerful mechanism for computing tensors asynchronously in a graph. Similarly, you can execute multiple threads for the same Session for parallel computations and hence speed up your operations.

Debugger

Another extra power of TF. With TensorFlow, you get a specialized debugger. It provides visibility into the internal structure and states of running TensorFlow graphs. Insights from debugger can be used to facilitate debugging of various types of bugs during both training and inference.

Control

The more control you have over your network, more better understanding you have of what’s going on with your network. With TF, you get such a control over your network. You can control whatever you want in your network. Operations on weights or gradients can be done like a charm in TF.

Numpy

Numpy, which stands for Numerical Python, is a library consisting of multidimensional array objects and a collection of routines for processing those arrays. Using NumPy, mathematical and logical operations on arrays can be performed. This tutorial explains the basics of NumPy such as its architecture and environment. It also discusses the various array functions, types of indexing, etc. An introduction to Matplotlib is also provided. All this is explained with the help of examples for better understanding.

Numpy is a Python package. It stands for 'Numerical Python'. It is a library consisting of multidimensional array objects and a collection of routines for processing of array.

Numeric, the ancestor of NumPy, was developed by Jim Hugunin. Another package Numarray was also developed, having some additional functionality. In 2005, Travis Oliphant created NumPy package by incorporating the features of Numarray into Numeric package. There are many contributors to this open source project.

Dept of ISE, NHCE Page 20

Stock Market Prediction

Operations using NumPy

Using NumPy, a developer can perform the following operations −

• Mathematical and logical operations on arrays.

• Fourier transforms and routines for shape manipulation.

• Operations related to linear algebra. NumPy has in-built functions for linear algebra and random number generation.

NumPy – A Replacement for Mat Lab

NumPy is often used along with packages like SciPy (Scientific Python) and Mat−plotlib (plotting library). This combination is widely used as a replacement for MatLab, a popular platform for technical computing. However, Python alternative to MatLab is now seen as a more modern and complete programming language.

It is open source, which is an added advantage of NumPy.

The most important object defined in NumPy is an N-dimensional array type called ndarray. It describes the collection of items of the same type. Items in the collection can be accessed using a zero-based index.

Every item in an ndarray takes the same size of block in the memory. Each element in ndarray is an object of data-type object (called dtype).

Any item extracted from ndarray object (by slicing) is represented by a Python object of one of array scalar types. The following diagram shows a relationship between ndarray, data type object (dtype) and array scalar type −

An instance of ndarray class can be constructed by different array creation routines described later in the tutorial. The basic ndarray is created using an array function in NumPy as follows −

numpy.array

It creates an ndarray from any object exposing array interface, or from any method that returns an array.

The ndarray objects can be saved to and loaded from the disk files. The IO functions available are −

Dept of ISE, NHCE Page 21

Stock Market Prediction

• load() and save() functions handle /numPy binary files (with npyextension)

• loadtxt() and savetxt() functions handle normal text files

NumPy introduces a simple file format for ndarray objects. This .npy file stores data, shape, dtype and other information required to reconstruct the ndarray in a disk file such that the array is correctly retrieved even if the file is on another machine with different architecture.

numpy.save()

The numpy.save() file stores the input array in a disk file with npyextension.

import numpy as np

a = np.array([1,2,3,4,5])

np.save('outfile',a)

To reconstruct array from outfile.npy, use load() function.

import numpy as np

b = np.load('outfile.npy')

print b

It will produce the following output −

array([1, 2, 3, 4, 5])

The save() and load() functions accept an additional Boolean parameter allow_pickles. A pickle in Python is used to serialize and de-serialize objects before saving to or reading from a disk file. savetxt()

The storage and retrieval of array data in simple text file format is done with savetxt() and loadtxt() functions.

Example

import numpy as np

Dept of ISE, NHCE Page 22

Stock Market Prediction

a = np.array([1,2,3,4,5])

np.savetxt('out.txt',a)

b = np.loadtxt('out.txt')

print b

It will produce the following output −

[ 1. 2. 3. 4. 5.]

The savetxt() and loadtxt() functions accept additional optional parameters such as header, footer, and delimiter.

CODE:

# import standard libraries

import numpy as np

import scipy.stats as st

import pandas as pd

import matplotlib.pyplot as plt

from sklearn import metrics

# import libraries for preparing model from sklearn.model_selection import train_test_split

from sklearn.preprocessing import StandardScaler

from sklearn.linear_model import LinearRegression

import warnings

warnings.filterwarnings("ignore")

Dept of ISE, NHCE Page 23

Stock Market Prediction

# Load data stockData=pd.read_csv("Nifty 50 data.csv")

print(stockData.describe())

# Prepare measures of volatility stockData['HL_percent'] = (stockData['High'] - stockData['Low']) / stockData['Low'] * 100.0

stockData['Change_percent'] = (stockData['Close'] - stockData['Open']) / stockData['Open'] * 100.0

X = stockData.drop(['Date','Turnover (Rs. Cr)','High','Low','Open','Close','Next day Close'],axis=1)

y = stockData['Next day Close']

# initiate lists to contain all the 1000 intercepts and coeffficients

RSME=[]

Coef1=[]

Coef2=[]

Coef3=[]

Intercept=[]

for k in range(1,1000):

# Split data for training and testing

X_train, X_test, y_train, y_test = train_test_split(X, y)

Dept of ISE, NHCE Page 24

Stock Market Prediction

scaler = StandardScaler()

# Fit only to the training data

scaler.fit(X_train)

StandardScaler(copy=True, with_mean=True, with_std=True)

# Now apply the transformations to the data:

X_train = scaler.transform(X_train)

X_test = scaler.transform(X_test)

lm=LinearRegression()

# fit the model

lm.fit(X_train,y_train)

pred_test=lm.predict(X_test)

Coef1.append(lm.coef_[0])

Coef2.append(lm.coef_[1])

Coef3.append(lm.coef_[2])

Intercept.append(lm.intercept_)

RSME.append(metrics.mean_squared_error(y_test, pred_test))

minRSME=min(RSME)

#find the index where minimium RSME is stored indexRSME=RSME.index(minRSME)

Dept of ISE, NHCE Page 25

Stock Market Prediction

# print the best weights possible

print("The below are the best possible coefficient and intercept in 100 trials for the Linear Regression model:")

print("The below values have been arrived at using the backpropagation using RSME values and using the best combination")

print("Best RSME is:"+str(RSME[indexRSME]))

print("Best Intercept is:"+str(Intercept[indexRSME]))

print("Best Coefficients are as below:")

print("Best Coeffcieint1 is:"+str(Coef1[indexRSME]))

print("Best Coeffcieint2 is:"+str(Coef2[indexRSME]))

print("Best Coeffcieint3 is:"+str(Coef3[indexRSME]))

plt.scatter(lm.predict(X_train), lm.predict(X_train)-y_train, c='b', s=40, alpha=0.5)

plt.scatter(lm.predict(X_test), lm.predict(X_test)-y_test, c='g', s=40)

plt.hlines(y=0, xmin=0, xmax=50)

plt.title("Residual plot using training (blue) and test (green) data")

plt.ylabel("Residuals")

plt.show()

5.1 Different modules of the project

 Training Testing

 Index Data

 Nifty50 Dataset

Dept of ISE, NHCE Page 26

Stock Market Prediction

Training Testing:

X-test

Index Data:

Nifty 50 Dataset:

Dept of ISE, NHCE Page 27

Stock Market Prediction

5.2 Flow Chart Of The Proposed System

Dept of ISE, NHCE Page 28

Stock Market Prediction

6. Experimental Results

6.1 Outcomes of the Proposed System

The proposed system is depicted. Proposed research work introduces a framework to predict the models using Nifty 50 data set. In this frameworks dataset is given to pre processing stage which further predicted by selected classifier. Machine learning tools WEKA[2] is used to analyze the performance of models.

The below are the best possible coefficient and intercept in 100 trials for the Linear Regression model:The below values have been arrived at using the backpropagation using RSME values and using the best combination.

Best RSME is:72758.8056644591

Best Intercept is:10871.058108108107

Best Coefficients are as below:

Best Coeffcieint1 is:15.07998645174593

Best Coeffcieint2 is:-105.78275096348388

Best Coeffcieint3 is:-0.7277352582113563

Dept of ISE, NHCE Page 29

Stock Market Prediction

Snapshot 1

Dept of ISE, NHCE Page 30

Stock Market Prediction

Snapshot 2

The below are the best possible coefficient and intercept in 100 trials for the Linear Regression model:

The below values have been arrived at using the backpropagation using RSME values and using the best combination

Best RSME is:72758.8056644591

Best Intercept is:10871.058108108107

Best Coefficients are as below:

Best Coeffcieint1 is:15.07998645174593

Best Coeffcieint2 is:-105.78275096348388

Best Coeffcieint3 is:-0.7277352582113563

Dept of ISE, NHCE Page 31

Stock Market Prediction

Snapshot 3

Snapshot 4

Dept of ISE, NHCE Page 32

Stock Market Prediction

Snapshot 5

Dept of ISE, NHCE Page 33

Stock Market Prediction

CHAPTER 7 TESTING

7.1 Testing and Validations

Validation is a complex process with many possible variations and options, so specifics vary from database to database, but the general outline is:

• Requirement Gathering

o The Sponsor decides what the database is required to do based on regulations, company needs, and any other important factors.

o The requirements are documented and approved. • System Testing

o Procedures to test the requirements are created and documented.

o The version of the database that will be used for validation is set up.

o The Sponsor approves the test procedures.

o The tests are performed and documented.

o Any needed changes are made. This may require another, shorter round of testing and documentation.

• System Release

o The validation documentation is finalized.

o The database is put into production.

7.2 Testing Levels

7.2.1 Functional Testing:

This type of testing is done against the functional requirements of the project.

Types:

Unit testing: Each unit /module of the project is individually tested to check for bugs. If any bugs found by the testing team, it is reported to the developer for fixing.

Integration testing: All the units are now integrated as one single unit and checked for bugs. This also checks if all the modules are working properly with each other.

System testing: This testing checks for operating system compatibility. It includes both functional and non functional requirements.

Dept of ISE, NHCE Page 34

Stock Market Prediction

Sanity testing: It ensures change in the code doesn’t affect the working of the project.

Smoke testing: this type of testing is a set of small tests designed for each build.

Interface testing: Testing of the interface and its proper functioning.

Regression testing: Testing the software repetitively when a new requirement is added, when bug fixed etc.

Beta/Acceptance testing: User level testing to obtain user feedback on the product.

7.2.2 Non-Functional Testing:

This type of testing is mainly concerned with the non-functional requirements such as performance of the system under various scenarios.

Performance testing: Checks for speed, stability and reliability of the software, hardware or even the network of the system under test.

Compatibility testing: This type of testing checks for compatibility of the system with different operating systems, different networks etc.

Localization testing: This checks for the localized version of the product mainly concerned with UI.

Security testing: Checks if the software has vulnerabilities and if any, fix them.

Reliability testing: Checks for the reliability of the software

Stress testing: This testing checks the performance of the system when it is exposed to different stress levels.

Usability testing: Type of testing checks the easily the software is being used by the customers.

Compliance testing: Type of testing to determine the compliance of a system with internal or external standards.

Dept of ISE, NHCE Page 35

Stock Market Prediction

Dept of ISE, NHCE Page 36

Stock Market Prediction

The framework should to be effectively versatile to another framework. This is obliged when the web server, which s facilitating the framework gets adhered because of a few issues, which requires the framework to be taken to another framework. • Scalability The framework should be sufficiently adaptable to include new functionalities at a later stage. There should be a run of the mill channel, which can oblige the new functionalities. • Flexibility Flexibility is the capacity of a framework to adjust to changing situations and circumstances, and to adapt to changes to business approaches and rules. An adaptable framework is one that is anything but difficult to reconfigure.

7.3 White Box Testing

White Box Testing is defined as the testing of a software solution's internal structure, design, and coding. In this type of testing, the code is visible to the tester. It focuses primarily on verifying the flow of inputs and outputs through the application, improving design and usability, strengthening security. White box testing is also known as Clear Box testing, Open Box testing, Structural testing, Transparent Box testing, Code-Based testing, and Glass Box testing. It is usually performed by developers.

It is one of two parts of the "Box Testing" approach to software testing. Its counterpart, Blackbox testing, involves testing from an external or end-user type perspective. On the other hand, Whitebox testing is based on the inner workings of an application and revolves around internal testing.

The term "WhiteBox" was used because of the see-through box concept. The clear box or WhiteBox name symbolizes the ability to see through the software's outer shell (or "box") into its inner workings. Likewise, the "black box" in "Black Box Testing" symbolizes not being able to see the inner workings of the software so that only the end-user experience can be tested.

• Internal security holes

Dept of ISE, NHCE Page 37

Stock Market Prediction

• Broken or poorly structured paths in the coding processes • The flow of specific inputs through the code • Expected output • The functionality of conditional loops • Testing of each statement, object, and function on an individual basis

The testing can be done at system, integration and unit levels of software development. One of the basic goals of whitebox testing is to verify a working flow for an application. It involves testing a series of predefined inputs against expected or desired outputs so that when a specific input does not result in the expected output, you have encountered a bug.

7.4 Different Stages of Testing

7.4.1 Unit Testing

UNIT TESTING is a level of software testing where individual units/ components of software are tested. The purpose is to validate that each unit of the software performs as designed. A unit is the smallest testable part of any software. It usually has one or a few inputs and usually a single output. In procedural programming, a unit may be an individual program, function, procedure, etc. In object-oriented programming, the smallest unit is a method, which may belong to a base/ super class, abstract class or derived/ child class. (Some treat a module of an application as a unit. This is to be discouraged as there will probably be many individual units within that module.) Unit testing frameworks, drivers, stubs, and mock/ fake objects are used to assist in unit testing.

Dept of ISE, NHCE Page 38

Stock Market Prediction

Unit Test Plan:

• Unit Test Plan

o Prepare

o Review

o Rework

o Baseline • Unit Test Cases/Scripts

o Prepare

o Review

o Rework

o Baseline • Unit Test

o Perform

Benefits

• Unit testing increases confidence in changing/ maintaining code. If good unit tests are written and if they are run every time any code is changed, we will be able to promptly catch any defects introduced due to the change. Also, if codes are already made less interdependent to make unit testing possible, the unintended impact of changes to any code is less. • Codes are more reusable. In order to make unit testing possible, codes need to be modular. This means that codes are easier to reuse.

Dept of ISE, NHCE Page 39

Stock Market Prediction

• Development is faster. How? If you do not have unit testing in place, you write your code and perform that fuzzy ‘developer test’ (You set some breakpoints, fire up the GUI, provide a few inputs that hopefully hit your code and hope that you are all set.) But, if you have unit testing in place, you write the test, write the code and run the test. Writing tests takes time but the time is compensated by the less amount of time it takes to run the tests; You need not fire up the GUI and provide all those inputs. And, of course, unit tests are more reliable than ‘developer tests’. Development is faster in the long run too. How? The effort required to find and fix defects found during unit testing is very less in comparison to the effort required to fix defects found during system testing or acceptance testing. • The cost of fixing a defect detected during unit testing is lesser in comparison to that of defects detected at higher levels. Compare the cost (time, effort, destruction, humiliation) of a defect detected during acceptance testing or when the software is live. • Debugging is easy. When a test fails, only the latest changes need to be debugged. With testing at higher levels, changes made over the span of several days/weeks/months need to be scanned.

7.4.2 Integration Testing

INTEGRATION TESTING is a level of software testing where individual units are combined and tested as a group. The purpose of this level of testing is to expose faults in the interaction between integrated units. Test drivers and test stubs are used to assist in Integration Testing.

Dept of ISE, NHCE Page 40

Stock Market Prediction

• Integration testing: Testing performed to expose defects in the interfaces and in the interactions between integrated components or systems. See also component integration testing, system integration testing. • Component integration testing: Testing performed to expose defects in the interfaces and interaction between integrated components. • System integration testing: Testing the integration of systems and packages; testing interfaces to external organizations (e.g. Electronic Data Interchange, Internet).

Tasks

• Integration Test Plan

o Prepare

o Review

o Rework

o Baseline • Integration Test Cases/Scripts

o Prepare

o Review

o Rework

o Baseline • Integration Test

Dept of ISE, NHCE Page 41

Stock Market Prediction

7.4.3 System Testing

SYSTEM TESTING is a level of software testing where a complete and integrated software is tested. The purpose of this test is to evaluate the system’s compliance with the specified requirements.

• system testing: The process of testing an integrated system to verify that it meets specified requirements.

7.4.4 Acceptance Testing

ACCEPTANCE TESTING is a level of software testing where a system is tested for acceptability. The purpose of this test is to evaluate the system’s compliance with the business requirements and assess whether it is acceptable for delivery.

Dept of ISE, NHCE Page 42

Stock Market Prediction

• acceptance testing: Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system.

Dept of ISE, NHCE Page 43

Stock Market Prediction

CHAPTER 8 CONCLUSION AND FUTURE ENHANCEMENT

8.1 Conclusion

Data mining based predictive models are very beneficial to analysis of model in different domain. In financial domain, predictive models play very important role for stock price prediction and important key factor for investors. We have applied the BSE data set to the different predictive model to check the robustness and compare the performance of models in terms of CC, MAE, RMSE, RAE and RRAE. Among all these models, the SVM produces better result as compare to others. The SVM model produces 0.9986, 0.0055, 0.0075 ,4.30% and 5.38 of CC, MAE and RMSE RAE and RRSE respectively.

The below are the best possible coefficient and intercept in 100 trials for the Linear Regression model:

The below values have been arrived at using the backpropagation using RSME values and using the best combination

Best RSME is:72758.8056644591

Best Intercept is:10871.058108108107

Best Coefficients are as below:

Best Coeffcieint1 is:15.07998645174593

Best Coeffcieint2 is:-105.78275096348388

Best Coeffcieint3 is:-0.7277352582113563

Dept of ISE, NHCE Page 44

Stock Market Prediction

8.2 Future Enhancement

The project has covered almost all the requirements. Further requirements and improvements can easily be done since the coding is mainly structured or modular in nature. Improvements can be appended by changing the existing modules or adding new modules. One important development that can be potential improvement can be made to our data collection and analysis method. Future research can be done with possible improvements such as more refined data and more accurate algorithm. Implementation of discussion forums and economic news portal including other sector apart from hydropower and going in national level.

References

• https://www.nseindia.com/products/content/equities/indices/historical_index_ data.htm • https://www.cs.waikato.ac.nz/ml/weka/.htm • https://a16z.com/2016/06/10/ai-deep-learning-machines/

Dept of ISE, NHCE Page 45