DOCSLIB.ORG

Efficient Compression of PMU Data in WAMS

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Efficient Compression of PMU Data in WAMS IEEE TRANSACTIONS ON SMART GRID, SPECIAL ISSUE ON BIG DATA ANALYTICS FOR GRID MODERNIZATION Efficient Compression of PMU Data in WAMS Phani Harsha Gadde, Student Member, IEEE, Milan Biswal, Member, IEEE, Sukumar Brahma, Senior Member, IEEE, and Huiping Cao Abstract—Widespread placement and high data sampling rate Archived data can be used for purposes like model validation, of current generation of Phasor Measurement Units (PMUs) in testing new wide area protection and control applications Wide Area Monitoring Systems (WAMS) result in huge amount that use PMU data, or training/testing disturbance classifiers of data to be analyzed and stored, making efficient storage of such data a priority. This paper presents a generalized compression typically used in system visualization applications that are technique that utilizes the inherent correlation within PMU being commercialized. data by exploiting both spatial and temporal redundancies. A Clearly, compression methods are necessary to archive these two stage compression algorithm is proposed using Principal data. Such methods essentially seek to maximize the Com- Component Analysis in the first stage, and Discrete Cosine pression Ratio (CR), and minimize the loss of data, which is Transform in the second. Since compression parameters need to be adjusted to compress critical disturbance information with measured by certain error metrics. The parameters chosen for high fidelity, an automated but simple statistical change detection any compression technique heavily depend on the nature of the technique is proposed to identify disturbance data. Extensive data. In case of PMU data, most of the data will be relatively verifications are performed using field data as well as simulated constant or slow-varying with overriding noise, but in case of data to establish generality and superior performance of the a disturbance the data will vary. The speed and the nature of method. variation again depend on the type of a disturbance. Therefore, Index Terms—Data compression, phasor data concentrator, any compression technique chosen to compress PMU data with phasor measurement units, wide area monitoring systems. high fidelity should be able to detect a disturbance and change the compression parameters for efficiently compressing any I. INTRODUCTION disturbance data, and be robust enough not to be affected by noise in the data. Validation of any approach should S Wide area monitoring systems (WAMS) grow to mon- therefore encompass real world data, including disturbance itor large interconnected power systems, the number of A data corresponding to major power system disturbances. Phasor Measurement Units (PMUs) has also been growing A lossless compression approach with slack referenced rapidly. For example, in North America, after the initiation of encoding (SRE) is suggested for PMU data in [4], where the the Western Interconnection Synchrophasor Project (WISP), voltage data are compressed with a maximum CR of 10:12. the number of PMUs installed in the Western Electricity This approach is heavily biased towards fidelity at the expense Coordination Council (WECC) system has increased from 56 of CR, and is not comprehensively tested. In [6] authors have in 2005 to 584 till April 2015 [1], [2]. The WISP shares used wavelet packet transform (WPT) to compress the PMU operational data with 97 participants over a secured wide data resulting in a low CR of 2 with root mean square error area network spanning the entire Western Interconnection (RMSE) of 3:68×10−6. A comparative assessment of standard [2] and has over 77 Phasor Data Concentrators (PDCs). In compression algorithms has been presented in [7], where szip WISP, almost every PMU feeds data to a PDC. The detailed algorithm is used to achieve a CR of 2:77 for voltage data PMU datas-flow and interconnections in North America are and 3:77 for frequency data, which are low values. In [8] an mentioned in [3]. The rapid increase in the number of PMUs, embedded zerotree wavelet based denoising and compression higher sampling rate of the modern PMUs, and more stringent method is introduced, but it lacks proper analysis, and doesn’t criteria for system integrity violations are expected to result perform well for real PMU data. Recently, a dimensionality re- in several fold expansion in the already large volumes of duction algorithm has been proposed by Xie et. al. [9] for PMU PMU data. As pointed out in [4], 100 PMUs having 20 data utilizing principal component analysis (PCA) subspace measurements at 30 Hz sampling rate generate over 50 GB for detection of disturbances, but the issue of compression is of data in one day. Higher sampling rate of 60 Hz or 120 not adequately addressed. Another real time event detection Hz in modern PMUs will increase this data volume to 100 and data archival scheme has been proposed in [10], where GB or 200 GB respectively. This means even larger volume PCA is used for event detection, and least square curve fitting of PMU data will need to be stored in PDCs, or transmitted is used for compression. It reports a relatively better CR of to Super PDCs. The NERC PRC 002-2 [5] requires that the 63:3 for voltage data, but with a higher normalized RMSE actual recorded disturbance data be preserved for 10 calendar of 0:176 for steady state data. Moreover, the CR reduces to days. However, all PMU data are finally archived by utilities. 4 for the voltage data during a disturbance event. This paper P H. Gadde, M. Biswal, and S. M. Brahma are with the Klipsch School of does not consider current and frequency data which are also Electrical and Computer Engineering, and H. Cao is with Computer Science recorded by PMUs, and will have different variations during Department at New Mexico State Univesity, Las Cruces, NM, 88003, USA. disturbances. In addition, it uses data from PMUs in a campus (e-mail:fphgadde;milanb;sbrahma;[email protected]). This work was supported by the National Science Foundation, CREST microgrid, and hence their method is not tested extensively for Grant HRD-1345232. large scale data from transmission systems, which have quite IEEE TRANSACTIONS ON SMART GRID, SPECIAL ISSUE ON BIG DATA ANALYTICS FOR GRID MODERNIZATION different disturbance patterns than a microgrid. The aim of s this paper is to propose a generalized method that works with PL (P (n) − P 0 (n))2 RMSE = n=1 i i ; (2) any number of PMUs and significantly improves the CR while L maintaining excellent fidelity, and show the robustness of the method on extensive data from field as well as from accurate 0 simulations. MADE = maxfj(Pi(n) − Pi (n))jg ; n = 1; 2; : : : L; (3) With the occurrence of a power system disturbance, mul- 0 where Pi(n) and Pi (n) are the original and reconstructed tiple PMUs at different geographical locations are triggered data samples, respectively, from the ith PMU, and L is the simultaneously, capturing various snapshots of the same event. length of the data sequence. Note that Higher CR and lower Therefore, it stands to reason that the parametric signatures errors mean higher quality compression. of all snapshots would be similar. Thus, correlation among multiple local PMU data streams at the PDC can be exploited PMU 1 PMU 2 PMU N for the compression of data. Exploiting this crucial property of PMU data, in this work we use PCA to minimize the spatial Detect disturbance by comparing incoming sample redundancy among data captured by multiple PMUs. We with moving average using thresholds for V, f, df/dt further reduce temporal redundancies in the data by applying Discrete Cosine Transform (DCT) and Discrete Wavelet Trans- NO Sample ≥ YES form (DWT) on the principal coefficients (PCs). A method to Threshold? choose adaptive window length is developed to attain efficient Fix Data window Monitor SV and compression for both, normal as well as disturbance data. We length W = 10s determine W finally apply a classical lossless data compression technique on the data that are already compressed using the above approach PCA to make the storage even more efficient. DCT/DWT II. METHODOLOGY PDC ARCHIVE DATA Let Pi(n) denote the measured voltage, frequency or current data sample expressed in per unit at time index n recorded th Fig. 1. Block Diagram of proposed PMU Data Compression Technique by the i PMU with a reporting rate of fs samples per sec (sps). Let there be N number of PMUs in a geographical region, connected to one PDC. The block diagram describing the proposed approach for compression is shown in Fig.1. A. Statistical Change Detection The proposed method implements Statistical Change Detection In order to preserve critical changes in PMU measurements (SCD) to detect and isolate disturbance data. This technique due to disturbances, a higher fidelity is needed. However, will be described in section II-A. If no disturbance is detected, in an automated environment, it is required to detect the a fixed data window W of 10 s is used. This value is based inception and duration of disturbances, so the fidelity can on studies presented in section II-B, which shows that the be enhanced for that period. A statistical change detection 10 s window provides a good combination of data-fidelity (SCD) algorithm is used for this purpose. Instead of using and CR for normal PMU data. If, however, a disturbance is the standard 10 s window length, a variable window length detected, continuously computed Statistical Variance (SV) is that envelops the full disturbance is used for compression of used to select an appropriate W, which is used to capture disturbance data. The NERC PRC 002 disturbance triggering the disturbance details with higher fidelity, although with criteria [5] recommends either of the following to detect a some reduction in CR.
Load more

Recommended publications
  • Data Validation As a Main Function of Business Intelligence
    E-Leader Berlin 2012 Data Validation as a Main Function of Business Intelligence PhDr. M.A. Dipl.-Betriebswirt (FH) Thomas H. Lenhard, PhD. University of Applied Sciences Kaiserslautern Campus Zweibruecken Zweibruecken, Germany Abstract Thinking about Business Intelligence, Data Mining or Data Warehousing, it seems to be clear, that we generally use three levels to compute our data into information. These levels are validation of input data, transformation and making output data available. This study/paper will give a short overview about the importance of validation and the validation level of a Data Warehouse or any BI-System. Several examples shall prove that validation is not only important to get proper and useful output data. Some combinations of circumstances will show that validation might be the result that is requested. After an introduction concerning the levels of generating information and after explaining necessary basics about BI, different methods of validation will be discussed as well as it will be thought about different variations to use the validation level or the validated data. Examples from the practice of Business Intelligence in general and in the area of Health Care IT specifically will support the theory that the level of validation must be considered as the most important. Especially in the area of operational management, it will be proven that a profound use of data validation can cause benefits or even prevent losses. Therefore it seems to be a method to increase efficiency in business and production. The Importance of Quality in Business Intelligence Business Intelligence (BI) is a non-exclusive collection of methods that are used as a transformation process to generate information out of data.
    [Show full text]
  • Complete Report
    A Report on The Fifteenth International TOVS Study Conference Maratea, Italy 4-10 October 2006 Conference sponsored by: IMAA / CNR Met Office (U.K.) University of Wisconsin-Madison / SSEC NOAA NESDIS EUMETSAT World Meteorological Organization ITT Industries Kongsberg Spacetec CNES ABB Bomem Dimension Data LGR Impianti Cisco Systems and VCS Report prepared by: Thomas Achtor and Roger Saunders ITWG Co-Chairs Leanne Avila Maria Vasys Editors Published and Distributed by: University of Wisconsin-Madison 1225 West Dayton Street Madison, WI 53706 USA ITWG Web Site: http://cimss.ssec.wisc.edu/itwg/ February 2007 International TOVS Study Conference-XV Working Group Report FOREWORD The International TOVS Working Group (ITWG) is convened as a sub-group of the International Radiation Commission (IRC) of the International Association of Meteorology and Atmospheric Physics (IAMAP). The ITWG continues to organise International TOVS Study Conferences (ITSCs) which have met approximately every 18 months since 1983. Through this forum, operational and research users of TIROS Operational Vertical Sounder (TOVS), Advanced TOVS (ATOVS) and other atmospheric sounding data have exchanged information on data processing methods, derived products, and the impacts of radiances and inferred atmospheric temperature and moisture fields on numerical weather prediction (NWP) and climate studies. The Fifteenth International TOVS Study Conference (ITSC-XV) was held at the Villa del Mare near Maratea, Italy from 4 to 10 October 2006. This conference report summarises the scientific exchanges and outcomes of the meeting. A companion document, The Technical Proceedings of The Fourteenth International TOVS Study Conference, contains the complete text of ITSC-XV scientific presentations. The ITWG Web site ( http://cimss.ssec.wisc.edu/itwg/ ) contains electronic versions of the conference presentations and publications.
    [Show full text]
  • Ssfm BPG 1: Validation of Software in Measurement Systems
    NPL Report DEM-ES 014 Software Support for Metrology Best Practice Guide No. 1 Validation of Software in Measurement Systems Brian Wichmann, Graeme Parkin and Robin Barker NOT RESTRICTED January 2007 National Physical Laboratory Hampton Road Teddington Middlesex United Kingdom TW11 0LW Switchboard 020 8977 3222 NPL Helpline 020 8943 6880 Fax 020 8943 6458 www.npl.co.uk Software Support for Metrology Best Practice Guide No. 1 Validation of Software in Measurement Systems Brian Wichmann with Graeme Parkin and Robin Barker Mathematics and Scientific Computing Group January 2007 ABSTRACT The increasing use of software within measurement systems implies that the validation of such software must be considered. This Guide addresses the validation both from the perspective of the user and the supplier. The complete Guide consists of a Management Overview and a Technical Application together with consideration of its use within safety systems. Version 2.2 c Crown copyright 2007 Reproduced with the permission of the Controller of HMSO and Queen’s Printer for Scotland ISSN 1744–0475 National Physical Laboratory, Hampton Road, Teddington, Middlesex, United Kingdom TW11 0LW Extracts from this guide may be reproduced provided the source is acknowledged and the extract is not taken out of context We gratefully acknowledge the financial support of the UK Department of Trade and Industry (National Measurement System Directorate) Approved on behalf of the Managing Director, NPL by Jonathan Williams, Knowledge Leader for the Electrical and Software team Preface The use of software in measurement systems has dramatically increased in the last few years, making many devices easier to use, more reliable and more accurate.
    [Show full text]
  • Review Article Applied Machine Learning and Artificial Intelligence in Rheumatology
    Rheumatology Advances in Practice 20;0:1–10 Rheumatology Advances in Practice doi:10.1093/rap/rkaa005 Review article Applied machine learning and artificial intelligence in rheumatology Maria Hu¨ gle1, Patrick Omoumi2, Jacob M. van Laar3, Joschka Boedecker1,* and Thomas Hu¨ gle4,* Abstract Machine learning as a field of artificial intelligence is increasingly applied in medicine to assist patients and physicians. Growing datasets provide a sound basis with which to apply machine learning meth- ods that learn from previous experiences. This review explains the basics of machine learning and its subfields of supervised learning, unsupervised learning, reinforcement learning and deep learning. We provide an overview of current machine learning applications in rheumatology, mainly supervised learn- ing methods for e-diagnosis, disease detection and medical image analysis. In the future, machine learning will be likely to assist rheumatologists in predicting the course of the disease and identifying important disease factors. Even more interestingly, machine learning will probably be able to make treatment propositions and estimate their expected benefit (e.g. by reinforcement learning). Thus, in fu- ture, shared decision-making will not only include the patient’s opinion and the rheumatologist’s empir- ical and evidence-based experience, but it will also be influenced by machine-learned evidence. Key words: machine learning, neural networks, deep learning, rheumatology, artificial intelligence Key messages . Deep learning can assist rheumatologists by disease classification and prediction of individual disease activity. Artificial intelligence has the potential to empower autonomy of patients, e.g. by providing individual treatment propositions. Automated image recognition and natural language processing will be likely to pioneer implementation of artificial intelligence in rheumatology.
    [Show full text]
  • Data Migration
    Science and Technology 2018, 8(1): 1-10 DOI: 10.5923/j.scit.20180801.01 Data Migration Simanta Shekhar Sarmah Business Intelligence Architect, Alpha Clinical Systems, USA Abstract This document gives the overview of all the process involved in Data Migration. Data Migration is a multi-step process that begins with an analysis of the legacy data and culminates in the loading and reconciliation of data into new applications. With the rapid growth of data, organizations are in constant need of data migration. The document focuses on the importance of data migration and various phases of it. Data migration can be a complex process where testing must be conducted to ensure the quality of the data. Testing scenarios on data migration, risk involved with it are also being discussed in this article. Migration can be very expensive if the best practices are not followed and the hidden costs are not identified at the early stage. The paper outlines the hidden costs and also provides strategies for roll back in case of any adversity. Keywords Data Migration, Phases, ETL, Testing, Data Migration Risks and Best Practices Data needs to be transportable from physical and virtual 1. Introduction environments for concepts such as virtualization To avail clean and accurate data for consumption Migration is a process of moving data from one Data migration strategy should be designed in an effective platform/format to another platform/format. It involves way such that it will enable us to ensure that tomorrow’s migrating data from a legacy system to the new system purchasing decisions fully meet both present and future without impacting active applications and finally redirecting business and the business returns maximum return on all input/output activity to the new device.
    [Show full text]
  • DS18B20 Datasheet
    AVAILABLE DS18B20 Programmable Resolution 1-Wire Digital Thermometer DESCRIPTION . User-Definable Nonvolatile (NV) Alarm The DS18B20 digital thermometer provides 9-bit Settings to 12-bit Celsius temperature measurements and . Alarm Search Command Identifies and has an alarm function with nonvolatile user- Addresses Devices Whose Temperature is programmable upper and lower trigger points. Outside Programmed Limits (Temperature The DS18B20 communicates over a 1-Wire bus Alarm Condition) that by definition requires only one data line (and . Available in 8-Pin SO (150 mils), 8-Pin µSOP, ground) for communication with a central and 3-Pin TO-92 Packages microprocessor. It has an operating temperature . Software Compatible with the DS1822 range of -55°C to +125°C and is accurate to . Applications Include Thermostatic Controls, ±0.5°C over the range of -10°C to +85°C. In Industrial Systems, Consumer Products, addition, the DS18B20 can derive power directly Thermometers, or Any Thermally Sensitive from the data line (“parasite power”), eliminating System the need for an external power supply. PIN CONFIGURATIONS Each DS18B20 has a unique 64-bit serial code, which allows multiple DS18B20s to function on the same 1-Wire bus. Thus, it is simple to use one MAXIM microprocessor to control many DS18B20s N.C. 1 8 N.C. 18B20 MAXIM 18B20 distributed over a large area. Applications that N.C. 2 7 N.C. can benefit from this feature include HVAC 1 2 3 environmental controls, temperature monitoring VDD 3 6 N.C. systems inside buildings, equipment, or DQ 4 5 GND machinery, and process monitoring and control systems.
    [Show full text]
  • 17.0 Data Review, Verification and Validation
    QA Handbook Vol II, Section 17.0 Revision No: 1 Date: 12/08 Page 1 of 7 17.0 Data Review, Verification and Validation Data review, verification and validation are techniques used to accept, reject or qualify data in an objective and consistent manner. Verification can be defined as confirmation, through provision of objective evidence that specified requirements have been fulfilled1. Validation can be defined as confirmation through provision of objective evidence that the particular requirements for a specific intended use are fulfilled. It is important to describe the criteria for deciding the degree to which each data item has met its quality specifications as described in an organization’s QAPP. This section will describe the techniques used to make these assessments. In general, these assessment activities are performed by persons implementing the environmental data operations as well as by personnel “independent” of the operation, such as the organization’s QA personnel and at some specified frequency. The procedures, personnel and frequency of the assessments should be included in an organization’s QAPP. These activities should occur prior to submitting data to AQS and prior to final data quality assessments that will be discussed in Section 18. Each of the following areas of discussion should be considered during the data review/verification/validation processes. Some of the discussion applies to situations in which a sample is separated from its native environment and transported to a laboratory for analysis and data generation; others are applicable to automated instruments. The following information is an excerpt from EPA G-52: Sampling Design - How closely a measurement represents the actual environment at a given time and location is a complex issue that is considered during development of the sampling design.
    [Show full text]
  • Introduction to Data Validation (PDF)
    Introduction to Data Validation Hilary Hafner Sonoma Technology, Inc. Petaluma, CA for National Ambient Air Monitoring Conference St. Louis, MO August 8, 2016 STI-6550 2 VOC and PM Speciation Data • Differences from one measurement (such as ozone or PM mass) – More complex instruments (more to go wrong?) – Many species per sample – Data overload • Opportunity for intercomparison 3 Why You Should Validate Your Data (1) • It is the monitoring agency’s responsibility to prevent, identify, correct, and define the consequences of monitoring difficulties that might affect the precision and accuracy, and/or the validity, of the measurements. • Serious errors in data analysis and modeling (and subsequent policy development) can be caused by erroneous data values. • Accurate information helps you respond to community concerns. Validate data as soon after collection as practical – it reduces effort and minimizes data loss 4 Why You Should Validate Your Data (2) • Criteria pollutant data quality issues are important to national air quality management actions, including – Attainment/nonattainment designations – Clean data determinations – Petitions to EPA for reconsideration • Air quality data are very closely reviewed by stakeholders – Do data collection efforts meet all CFR requirements? – Have procedures outlined in the QA handbook or project-specific QA plans been followed? – Are agency logbooks complete and up-to-date? • Deviations are subject to potential litigation From Weinstock L. (2014) Ambient Monitoring Update. Presented at the National Ambient Air Monitoring Conference, Atlanta, GA, August 11-14, by the U.S. Environmental Protection Agency Office of Air Quality Planning and Standards. Available at https://www3.epa.gov/ttnamti1/files/2014conference/tueweinstock.pdf.
    [Show full text]
  • Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes
    Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes Jie Song∗ Yeye He University of Michigan Microsoft Research [email protected] [email protected] ABSTRACT Complex data pipelines are increasingly common in diverse ap- plications such as BI reporting and ML modeling. These pipelines often recur regularly (e.g., daily or weekly), as BI reports need to be refreshed, and ML models need to be retrained. However, it is widely reported that in complex production pipelines, upstream data feeds can change in unexpected ways, causing downstream applications to break silently that are expensive to resolve. Figure 1: Example code snippet to describe expected data- Data validation has thus become an important topic, as evidenced values, using declarative constraints in Amazon Deequ. by notable recent efforts from Google and Amazon, where the ob- jective is to catch data quality issues early as they arise in the pipelines. Our experience on production data suggests, however, Such schema-drift and data-drift can lead to quality issues in that on string-valued data, these existing approaches yield high downstream applications, which are both hard to detect (e.g., mod- false-positive rates and frequently require human intervention. In est model degradation due to unseen data [53]), and hard to debug this work, we develop a corpus-driven approach to auto-validate (finding root-causes in complex pipelines can be difficult). These machine-generated data by inferring suitable data-validation “pat- silent failures require substantial human efforts to resolve, increas- terns” that accurately describe the underlying data-domain, which ing the cost of running production data pipelines and decreasing minimizes false-positives while maximizing data quality issues the effectiveness of data-driven system [21, 53, 58, 60].
    [Show full text]
  • "Data Validation". In: Clinical Data Management
    SEQ 0014 JOB PDF8280-006-007 PAGE-0109 CHAP 6 109-122 REVISED 30NOV01 AT 16:46 BY TF DEPTH: 58.01 PICAS WIDTH 40 PICAS Clinical Data Management, 2nd Edition. Richard K. Rondel, Sheila A. Varley, Colin F. Webb Copyright 1993, 2000 John Wiley & Sons Ltd ISBNs: 0471-98329-2 (Hardback); 0470-84636-4 (Electronic) 6 Data Validation PANKAJ ‘PANNI’ PATEL SmithKline Beecham Pharmaceuticals, Harlow, Essex, UK INTRODUCTION The primary objective of Clinical Data Management (CDM) is to ensure timely delivery of high-quality data which are necessary to satisfy both good clinical practice (GCP) requirements and the statistical analysis and reporting requirements. CDM data validation activities play a critical role within the drug development programme involving many people, multiple systems and several data transfers. The quality of the data validation process has a direct impact on the quality of data presented as part of an NDA submission. There is a general misconception that data validation activities com- mence when clinical trial data are presented to the sponsor’s data man- agement department. The author will attempt to dispel this somewhat narrow view and discuss various stages of data validation activities which actually start when the investigator records the data on the case report form (CRF) and when the final medical report is issued as part of the overall clinical trial data handing and reporting process. CDM REQUIREMENT IN GCP CDM requirements within the ICH and EU CPMP GCP guidelines are not defined in any great detail, resulting in lack of clarity, or indeed misrepre- sentation. The FDA Code of Federal Regulations contains no mention of CDM! This should be considered as a major concern, given that CDM plays a vital role in protecting data integrity, and is charged with produc- ing high-quality databases that meet clinical and regulatory require- ments.
    [Show full text]
  • EDRM Glossary Version 1.002, April 22, 2016
    EDRM Glossary http://www.edrm.net/resources/glossaries/glossary Version 1.002, April 22, 2016 © 2016 EDRM LLC EDRM Glossary http://www.edrm.net/resources/glossaries/glossary The EDRM Glossary is EDRM's most comprehensive listing of electronic discovery terms. It includes terms from the specialized glossaries listed below as well as terms not in those glossaries. The terms are listed in alphabetical order with definitions and attributions where available. EDRM Collection Standards Glossary The EDRM Collection Standards Glossary is a glossary of terms defined as part of the EDRM Collection Standards. EDRM Metrics Glossary The EDRM Metrics Glossary contains definitions for terms used in connection with the updated EDRM Metrics Model published in June 2013. EDRM Search Glossary The EDRM Search Glossary is a list of terms related to searching ESI. EDRM Search Guide Glossary The EDRM Search Guide Glossary is part of the EDRM Search Guide. The EDRM Search Guide focuses on the search, retrieval and production of ESI within the larger e- discovery process described in the EDRM Model. IGRM Glossary The IGRM Glossary consists of commonly used Information Governance terms. The Grossman-Cormack Glossary of Technology-Assisted Review Developed by Maura Grossman of Wachtell, Lipton, Rosen & Katz and Gordon Cormack of the University of Waterloo, the Grossman-Cormack Glossary of Technology-Assisted Review contains definitions for terms used in connect with the discovery processes referred to by various terms including Computer Assisted Review, Technology Assisted Review, and Predictive Coding. If you would like to submit a new term with definition or a new definition for an existing term, please go to our Submit a Definition page, http://www.edrm.net/23482, and complete and submit the form.
    [Show full text]
  • Automation in ETL Testing Omkar Sanjay Kulkarni Computer Science Graduate (2013-2017) CSE Walchand College of Engineering, Sangli, India
    ISSN:0975-9646 Omkar Sanjay Kulkarni et al, / (IJCSIT) International Journal of Computer Science and Information Technologies, Vol. 8 (6) , 2017,586-589 Automation in ETL Testing Omkar Sanjay Kulkarni Computer Science Graduate (2013-2017) CSE Walchand College Of engineering, Sangli, India Abstract — ETL which stands for (Extract Transform Load) is on OLTP (transactional systems) where the data comes a process used to extract data from different data sources, from different applications into the transactional database. transform the data and load it into a Data Warehouse System. ETL testing emphasizes on data extraction, transformation ETL testing is done to identify data defects and errors that and loading for BI reporting, while Database testing occur prior to processing of data for analytical reporting. ETL stresses more on data accuracy, validation and integration testing is mainly done using SQL scripts and manual comparison of gathered data. This approach is slow, resource [1]. ETL Testing involves validation of data movement intensive and error-prone. Automating ETL testing allows from the source to the target system, verification of data testing without any user intervention and supports automatic count in the source and the target system, verifying data regression on old scripts after every new release. In this paper extraction, transformation as per requirement and we will examine the challenges involved in ETL testing and expectation, verifying if table relations are preserved during subsequently need for automation in ETL testing to deliver the transformation. On the other hand, Database testing better and quicker results. involves verification of primary and foreign keys, validating data values in columns, verifying missing data Keywords— ETL, Data Warehouse, Automation, SQL, and to check if there are null columns which actually should regression, scripts.
    [Show full text]