Lecture 4: Measure of Dispersion
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
A Review on Outlier/Anomaly Detection in Time Series Data
A review on outlier/anomaly detection in time series data ANE BLÁZQUEZ-GARCÍA and ANGEL CONDE, Ikerlan Technology Research Centre, Basque Research and Technology Alliance (BRTA), Spain USUE MORI, Intelligent Systems Group (ISG), Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), Spain JOSE A. LOZANO, Intelligent Systems Group (ISG), Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), Spain and Basque Center for Applied Mathematics (BCAM), Spain Recent advances in technology have brought major breakthroughs in data collection, enabling a large amount of data to be gathered over time and thus generating time series. Mining this data has become an important task for researchers and practitioners in the past few years, including the detection of outliers or anomalies that may represent errors or events of interest. This review aims to provide a structured and comprehensive state-of-the-art on outlier detection techniques in the context of time series. To this end, a taxonomy is presented based on the main aspects that characterize an outlier detection technique. Additional Key Words and Phrases: Outlier detection, anomaly detection, time series, data mining, taxonomy, software 1 INTRODUCTION Recent advances in technology allow us to collect a large amount of data over time in diverse research areas. Observations that have been recorded in an orderly fashion and which are correlated in time constitute a time series. Time series data mining aims to extract all meaningful knowledge from this data, and several mining tasks (e.g., classification, clustering, forecasting, and outlier detection) have been considered in the literature [Esling and Agon 2012; Fu 2011; Ratanamahatana et al. -
Mean, Median and Mode We Use Statistics Such As the Mean, Median and Mode to Obtain Information About a Population from Our Sample Set of Observed Values
INFORMATION AND DATA CLASS-6 SUB-MATHEMATICS The words Data and Information may look similar and many people use these words very frequently, But both have lots of differences between What is Data? Data is a raw and unorganized fact that required to be processed to make it meaningful. Data can be simple at the same time unorganized unless it is organized. Generally, data comprises facts, observations, perceptions numbers, characters, symbols, image, etc. Data is always interpreted, by a human or machine, to derive meaning. So, data is meaningless. Data contains numbers, statements, and characters in a raw form What is Information? Information is a set of data which is processed in a meaningful way according to the given requirement. Information is processed, structured, or presented in a given context to make it meaningful and useful. It is processed data which includes data that possess context, relevance, and purpose. It also involves manipulation of raw data Mean, Median and Mode We use statistics such as the mean, median and mode to obtain information about a population from our sample set of observed values. Mean The mean/Arithmetic mean or average of a set of data values is the sum of all of the data values divided by the number of data values. That is: Example 1 The marks of seven students in a mathematics test with a maximum possible mark of 20 are given below: 15 13 18 16 14 17 12 Find the mean of this set of data values. Solution: So, the mean mark is 15. Symbolically, we can set out the solution as follows: So, the mean mark is 15. -
Applied Biostatistics Mean and Standard Deviation the Mean the Median Is Not the Only Measure of Central Value for a Distribution
Health Sciences M.Sc. Programme Applied Biostatistics Mean and Standard Deviation The mean The median is not the only measure of central value for a distribution. Another is the arithmetic mean or average, usually referred to simply as the mean. This is found by taking the sum of the observations and dividing by their number. The mean is often denoted by a little bar over the symbol for the variable, e.g. x . The sample mean has much nicer mathematical properties than the median and is thus more useful for the comparison methods described later. The median is a very useful descriptive statistic, but not much used for other purposes. Median, mean and skewness The sum of the 57 FEV1s is 231.51 and hence the mean is 231.51/57 = 4.06. This is very close to the median, 4.1, so the median is within 1% of the mean. This is not so for the triglyceride data. The median triglyceride is 0.46 but the mean is 0.51, which is higher. The median is 10% away from the mean. If the distribution is symmetrical the sample mean and median will be about the same, but in a skew distribution they will not. If the distribution is skew to the right, as for serum triglyceride, the mean will be greater, if it is skew to the left the median will be greater. This is because the values in the tails affect the mean but not the median. Figure 1 shows the positions of the mean and median on the histogram of triglyceride. -
Higher-Order Asymptotics
Higher-Order Asymptotics Todd Kuffner Washington University in St. Louis WHOA-PSI 2016 1 / 113 First- and Higher-Order Asymptotics Classical Asymptotics in Statistics: available sample size n ! 1 First-Order Asymptotic Theory: asymptotic statements that are correct to order O(n−1=2) Higher-Order Asymptotics: refinements to first-order results 1st order 2nd order 3rd order kth order error O(n−1=2) O(n−1) O(n−3=2) O(n−k=2) or or or or o(1) o(n−1=2) o(n−1) o(n−(k−1)=2) Why would anyone care? deeper understanding more accurate inference compare different approaches (which agree to first order) 2 / 113 Points of Emphasis Convergence pointwise or uniform? Error absolute or relative? Deviation region moderate or large? 3 / 113 Common Goals Refinements for better small-sample performance Example Edgeworth expansion (absolute error) Example Barndorff-Nielsen’s R∗ Accurate Approximation Example saddlepoint methods (relative error) Example Laplace approximation Comparative Asymptotics Example probability matching priors Example conditional vs. unconditional frequentist inference Example comparing analytic and bootstrap procedures Deeper Understanding Example sources of inaccuracy in first-order theory Example nuisance parameter effects 4 / 113 Is this relevant for high-dimensional statistical models? The Classical asymptotic regime is when the parameter dimension p is fixed and the available sample size n ! 1. What if p < n or p is close to n? 1. Find a meaningful non-asymptotic analysis of the statistical procedure which works for any n or p (concentration inequalities) 2. Allow both n ! 1 and p ! 1. 5 / 113 Some First-Order Theory Univariate (classical) CLT: Assume X1;X2;::: are i.i.d. -
An Empirical Analysis of Higher Moment Capital Asset Pricing Model for Karachi Stock Exchange (KSE)
Munich Personal RePEc Archive An Empirical Analysis of Higher Moment Capital Asset Pricing Model for Karachi Stock Exchange (KSE) Lal, Irfan and Mubeen, Muhammad and Hussain, Adnan and Zubair, Mohammad Institute of Business Management, Karachi , Pakistan., Iqra University, Shaheed Benazir Bhutto University Lyari, Institute of Business Management, Karachi , Pakistan. 9 June 2016 Online at https://mpra.ub.uni-muenchen.de/106869/ MPRA Paper No. 106869, posted 03 Apr 2021 23:51 UTC An Empirical Analysis of Higher Moment Capital Asset Pricing Model for Karachi Stock Exchange (KSE) Irfan Lal1 Research Fellow, Department of Economics, Institute of Business Management, Karachi, Pakistan [email protected] Muhammad Mubin HEC Research Scholar, Bilkent Üniversitesi [email protected] Adnan Hussain Lecturer, Department of Economics, Benazir Bhutto Shaheed University, Karachi, Pakistan [email protected] Muhammad Zubair Lecturer, Department of Economics, Institute of Business Management, Karachi, Pakistan [email protected] Abstract The purpose behind this study to explore the relationship between expected return and risk of portfolios. It is observed that standard CAPM is inappropriate, so we introduce higher moment in model. For this purpose, the study taken data of 60 listed companies of Karachi Stock Exchange 100 index. The data is inspected for the period of 1st January 2007 to 31st December 2013. From the empirical analysis, it is observed that the intercept term and higher moments coefficients (skewness and kurtosis) is highly significant and different from zero. When higher moment is introduced in the model, the adjusted R square is increased. The higher moment CAPM performs cooperatively perform well. Keywords: Capital Assets Price Model, Higher Moment JEL Classification: G12, C53 1 Corresponding Author 1. -
EART6 Lab Standard Deviation in Probability and Statistics, The
EART6 Lab Standard deviation In probability and statistics, the standard deviation is a measure of the dispersion of a collection of values. It can apply to a probability distribution, a random variable, a population or a data set. The standard deviation is usually denoted with the letter σ. It is defined as the root-mean-square (RMS) deviation of the values from their mean, or as the square root of the variance. Formulated by Galton in the late 1860s, the standard deviation remains the most common measure of statistical dispersion, measuring how widely spread the values in a data set are. If many data points are close to the mean, then the standard deviation is small; if many data points are far from the mean, then the standard deviation is large. If all data values are equal, then the standard deviation is zero. A useful property of standard deviation is that, unlike variance, it is expressed in the same units as the data. 2 (x " x ) ! = # N N Fig. 1 Dark blue is less than one standard deviation from the mean. For Fig. 2. A data set with a mean the normal distribution, this accounts for 68.27 % of the set; while two of 50 (shown in blue) and a standard deviations from the mean (medium and dark blue) account for standard deviation (σ) of 20. 95.45%; three standard deviations (light, medium, and dark blue) account for 99.73%; and four standard deviations account for 99.994%. The two points of the curve which are one standard deviation from the mean are also the inflection points. -
Robust Logistic Regression to Static Geometric Representation of Ratios
Journal of Mathematics and Statistics 5 (3):226-233, 2009 ISSN 1549-3644 © 2009 Science Publications Robust Logistic Regression to Static Geometric Representation of Ratios 1Alireza Bahiraie, 2Noor Akma Ibrahim, 3A.K.M. Azhar and 4Ismail Bin Mohd 1,2 Institute for Mathematical Research, University Putra Malaysia, 43400 Serdang, Selangor, Malaysia 3Graduate School of Management, University Putra Malaysia, 43400 Serdang, Selangor, Malaysia 4Department of Mathematics, Faculty of Science, University Malaysia Terengganu, 21030, Malaysia Abstract: Problem statement: Some methodological problems concerning financial ratios such as non- proportionality, non-asymetricity, non-salacity were solved in this study and we presented a complementary technique for empirical analysis of financial ratios and bankruptcy risk. This new method would be a general methodological guideline associated with financial data and bankruptcy risk. Approach: We proposed the use of a new measure of risk, the Share Risk (SR) measure. We provided evidence of the extent to which changes in values of this index are associated with changes in each axis values and how this may alter our economic interpretation of changes in the patterns and directions. Our simple methodology provided a geometric illustration of the new proposed risk measure and transformation behavior. This study also employed Robust logit method, which extends the logit model by considering outlier. Results: Results showed new SR method obtained better numerical results in compare to common ratios approach. With respect to accuracy results, Logistic and Robust Logistic Regression Analysis illustrated that this new transformation (SR) produced more accurate prediction statistically and can be used as an alternative for common ratios. Additionally, robust logit model outperforms logit model in both approaches and was substantially superior to the logit method in predictions to assess sample forecast performances and regressions. -
Accelerating Raw Data Analysis with the ACCORDA Software and Hardware Architecture
Accelerating Raw Data Analysis with the ACCORDA Software and Hardware Architecture Yuanwei Fang Chen Zou Andrew A. Chien CS Department CS Department University of Chicago University of Chicago University of Chicago Argonne National Laboratory [email protected] [email protected] [email protected] ABSTRACT 1. INTRODUCTION The data science revolution and growing popularity of data Driven by a rapid increase in quantity, type, and number lakes make efficient processing of raw data increasingly impor- of sources of data, many data analytics approaches use the tant. To address this, we propose the ACCelerated Operators data lake model. Incoming data is pooled in large quantity { for Raw Data Analysis (ACCORDA) architecture. By ex- hence the name \lake" { and because of varied origin (e.g., tending the operator interface (subtype with encoding) and purchased from other providers, internal employee data, web employing a uniform runtime worker model, ACCORDA scraped, database dumps, etc.), it varies in size, encoding, integrates data transformation acceleration seamlessly, en- age, quality, etc. Additionally, it often lacks consistency or abling a new class of encoding optimizations and robust any common schema. Analytics applications pull data from high-performance raw data processing. Together, these key the lake as they see fit, digesting and processing it on-demand features preserve the software system architecture, empower- for a current analytics task. The applications of data lakes ing state-of-art heuristic optimizations to drive flexible data are vast, including machine learning, data mining, artificial encoding for performance. ACCORDA derives performance intelligence, ad-hoc query, and data visualization. Fast direct from its software architecture, but depends critically on the processing on raw data is critical for these applications. -
The Method of Maximum Likelihood for Simple Linear Regression
08:48 Saturday 19th September, 2015 See updates and corrections at http://www.stat.cmu.edu/~cshalizi/mreg/ Lecture 6: The Method of Maximum Likelihood for Simple Linear Regression 36-401, Fall 2015, Section B 17 September 2015 1 Recapitulation We introduced the method of maximum likelihood for simple linear regression in the notes for two lectures ago. Let's review. We start with the statistical model, which is the Gaussian-noise simple linear regression model, defined as follows: 1. The distribution of X is arbitrary (and perhaps X is even non-random). 2. If X = x, then Y = β0 + β1x + , for some constants (\coefficients", \parameters") β0 and β1, and some random noise variable . 3. ∼ N(0; σ2), and is independent of X. 4. is independent across observations. A consequence of these assumptions is that the response variable Y is indepen- dent across observations, conditional on the predictor X, i.e., Y1 and Y2 are independent given X1 and X2 (Exercise 1). As you'll recall, this is a special case of the simple linear regression model: the first two assumptions are the same, but we are now assuming much more about the noise variable : it's not just mean zero with constant variance, but it has a particular distribution (Gaussian), and everything we said was uncorrelated before we now strengthen to independence1. Because of these stronger assumptions, the model tells us the conditional pdf 2 of Y for each x, p(yjX = x; β0; β1; σ ). (This notation separates the random variables from the parameters.) Given any data set (x1; y1); (x2; y2);::: (xn; yn), we can now write down the probability density, under the model, of seeing that data: n n (y −(β +β x ))2 Y 2 Y 1 − i 0 1 i p(yijxi; β0; β1; σ ) = p e 2σ2 2 i=1 i=1 2πσ 1See the notes for lecture 1 for a reminder, with an explicit example, of how uncorrelated random variables can nonetheless be strongly statistically dependent. -
Weighted Means and Grouped Data on the Graphing Calculator
Weighted Means and Grouped Data on the Graphing Calculator Weighted Means. Suppose that, in a recent month, a bank customer had $2500 in his account for 5 days, $1900 for 2 days, $1650 for 3 days, $1375 for 9 days, $1200 for 1 day, $900 for 6 days, and $675 for 4 days. To calculate the average balance for that month, you would use a weighted mean: ∑ ̅ ∑ To do this automatically on a calculator, enter the account balances in L1 and the number of days (weight) in L2: As before, go to STAT CALC and 1:1-Var Stats. This time, type L1, L2 after 1-Var Stats and ENTER: The weighted (sample) mean is ̅ (so that the average balance for the month is $1430.83.) We can also see that the weighted sample standard deviation is . Estimating the Mean and Standard Deviation from a Frequency Distribution. If your data is organized into a frequency distribution, you can still estimate the mean and standard deviation. For example, suppose that we are given only a frequency distribution of the heights of the 30 males instead of the list of individual heights: Height (in.) Frequency (f) 62 - 64 3 65- 67 7 68 - 70 9 71 - 73 8 74 - 76 3 ∑ n = 30 We can just calculate the midpoint of each height class and use that midpoint to represent the class. We then find the (weighted) mean and standard deviation for the distribution of midpoints with the given frequencies as in the example above: Height (in.) Height Class Frequency (f) Midpoint 62 - 64 (62 + 64)/2 = 63 3 65- 67 (65 + 67)/2 = 66 7 68 - 70 69 9 71 - 73 72 8 74 - 76 75 3 ∑ n = 30 The approximate sample mean of the distribution is ̅ , and the approximate sample standard deviation of the distribution is . -
Chapter 6: Random Errors in Chemical Analysis
Chapter 6: Random Errors in Chemical Analysis Source slideplayer.com/Fundamentals of Analytical Chemistry, F.J. Holler, S.R.Crouch Random errors are present in every measurement no matter how careful the experimenter. Random, or indeterminate, errors can never be totally eliminated and are often the major source of uncertainty in a determination. Random errors are caused by the many uncontrollable variables that accompany every measurement. The accumulated effect of the individual uncertainties causes replicate results to fluctuate randomly around the mean of the set. In this chapter, we consider the sources of random errors, the determination of their magnitude, and their effects on computed results of chemical analyses. We also introduce the significant figure convention and illustrate its use in reporting analytical results. 6A The nature of random errors - random error in the results of analysts 2 and 4 is much larger than that seen in the results of analysts 1 and 3. - The results of analyst 3 show outstanding precision but poor accuracy. The results of analyst 1 show excellent precision and good accuracy. Figure 6-1 A three-dimensional plot showing absolute error in Kjeldahl nitrogen determinations for four different analysts. Random Error Sources - Small undetectable uncertainties produce a detectable random error in the following way. - Imagine a situation in which just four small random errors combine to give an overall error. We will assume that each error has an equal probability of occurring and that each can cause the final result to be high or low by a fixed amount ±U. - Table 6.1 gives all the possible ways in which four errors can combine to give the indicated deviations from the mean value. -
Lecture 3: Measure of Central Tendency
Lecture 3: Measure of Central Tendency Donglei Du ([email protected]) Faculty of Business Administration, University of New Brunswick, NB Canada Fredericton E3B 9Y2 Donglei Du (UNB) ADM 2623: Business Statistics 1 / 53 Table of contents 1 Measure of central tendency: location parameter Introduction Arithmetic Mean Weighted Mean (WM) Median Mode Geometric Mean Mean for grouped data The Median for Grouped Data The Mode for Grouped Data 2 Dicussion: How to lie with averges? Or how to defend yourselves from those lying with averages? Donglei Du (UNB) ADM 2623: Business Statistics 2 / 53 Section 1 Measure of central tendency: location parameter Donglei Du (UNB) ADM 2623: Business Statistics 3 / 53 Subsection 1 Introduction Donglei Du (UNB) ADM 2623: Business Statistics 4 / 53 Introduction Characterize the average or typical behavior of the data. There are many types of central tendency measures: Arithmetic mean Weighted arithmetic mean Geometric mean Median Mode Donglei Du (UNB) ADM 2623: Business Statistics 5 / 53 Subsection 2 Arithmetic Mean Donglei Du (UNB) ADM 2623: Business Statistics 6 / 53 Arithmetic Mean The Arithmetic Mean of a set of n numbers x + ::: + x AM = 1 n n Arithmetic Mean for population and sample N P xi µ = i=1 N n P xi x¯ = i=1 n Donglei Du (UNB) ADM 2623: Business Statistics 7 / 53 Example Example: A sample of five executives received the following bonuses last year ($000): 14.0 15.0 17.0 16.0 15.0 Problem: Determine the average bonus given last year. Solution: 14 + 15 + 17 + 16 + 15 77 x¯ = = = 15:4: 5 5 Donglei Du (UNB) ADM 2623: Business Statistics 8 / 53 Example Example: the weight example (weight.csv) The R code: weight <- read.csv("weight.csv") sec_01A<-weight$Weight.01A.2013Fall # Mean mean(sec_01A) ## [1] 155.8548 Donglei Du (UNB) ADM 2623: Business Statistics 9 / 53 Will Rogers phenomenon Consider two sets of IQ scores of famous people.