Peter McEldowney Mark Waddington | Capstone CSNT 255 Outputting Event Log Events to a Remote SQL Database Using PowerShell Objective: After completing this lab, the Administrator will have the Event Log from the computer of their choice uploading information to a SQL database. The step-by-step outlines how to create the tables that will be used to organize the data and will cover how to create a basic SQL script to create these tables. Within the script to create the tables, indexes will be covered to ensure quick access to the data by a particular column that is indexed and will also cover items to be aware of. After creating these tables, a script to upload the event log items to the SQL server will be created so that the script can be run from any machine to ensure scalability. After the script is created and tested, this step-by-step will go through how to create a trigger that will run the script whenever a new event log entry is created. Requirements: PowerShell 3.0, Microsoft SQL Server [w/ Management Tools] Step 1 : Create a diagram of the tables that will be used to store the data Considerations when designing your tables: a) What data is available? b) How will you identify the machine once the data is in the database? c) What log did the data come from and how will this be stored in the database? d) Is there an efficient way to organize the data to make retrieving the data quicker? e) What columns will people be searching from? f) What should the Primary Keys for the table(s) be? What is a Primary Key? A primary key is basically an index number. This is the unique number that identifies a particular row. For example, if 2 rows have the same information, how would we identify which row is which? g) What should the Foreign Keys be? What is a Foreign Key? Foreign Keys refer to data from another table. These are primarily used to link tables. For example, say you have data in one table and the computer that uploaded the data in another. The foreign key in the data table would reference an identifying key in another so that the data can be linked to the associated machine. 1 | P a g e The tables and relationships that are being made are referred to as an Entity-Relation Diagram. Not only will creating this make building the SQL script to create the databases easier, but it will also allow for future database users to access the data efficiently and know what data is available. a) What data is available? The best way to identify the available data is by looking at the data. For example, using PowerShell, we can pull event log data and look at what is available. Try this command in PowerShell to find the available data: Ex: Get-EventLog –LogName System –Newest 1 | Format-List Now we must decide what information we want to log in our database. [Remember: Databases can grow to be massive so far large environment, it may be more efficient to only take critical information. In this tutorial, we will be uploading all information because we can.] 2 | P a g e We can actually call specific values that are defined by incorporating the text before the pipe | within parenthesis () and then appending a period. to identify we want to call a value. Ex: (Get-EventLog –LogName System –Newest 1).Index Now since we have the information available, we can start grouping particular items together and deciding how we want to arrange everything. b) Identifying the original machine This can be done within the PowerShell script that will be created in Step 3. Try this command in PowerShell for a preview on how this will be accomplished. Ex: [Environment]::MachineName c) Identifying the originating system log Different logs are kept for different aspects of the system. For example, DNS has its own log. PowerShell has its own log, and so do System tasks. The available logs on a system can be obtain by running the following: Ex: Get-EventLog -LogName * 3 | P a g e d) Organizing the Data for efficiency. This step all depends on the volume of data that will be stored within the database. For centralize Event Log management, it may be beneficial to store only EntryType, Message, Source, and Username within 1 table so that less data is retrieved when searching for errors. An example is portrayed below. Notice how only crucial information is stored in the primary table. All other data must be referenced by joining tables together by their similar keys (foreign keys). e) Frequent columns. Generally people will be searching for information based off specific columns. These columns (like name of the computer, the type of entry, source, etc) are columns that should contain indexes. An index is simply a tag put onto a column so that it is monitored for efficient data retrieval. One caution of using indexes is that every time new data is entered, the index must be re-evaluated. As you can imagine, this can cause a great deal of decreased performance and may even crash systems that are inadequate for supporting the amount of data being uploaded. Indexes are not required (and are not covered here) but are an important aspect to consider if deploying on a large scale as it is much easier to incorporate when designing than trying to implement during production. f) Primary Keys Primary key columns are (as described above) a unique identifying key. No 2 primary keys in a table can be the same or the data would be inconsistent. This 4 | P a g e is why it is best to create a special column for the primary key and set it to auto- increment. In the entity-relationship diagram below, you can see how columns with PK are identified with some sort of text that is followed by_id g) Foreign Keys FK’s should be easy to identify and associate with other tables. The most difficult part is coding Foreign Keys. One nice part about foreign keys is that you can set constraints on the data that is referenced. For example, if we use the code exemplified below, then when the referenced data in the other table is deleted, the delete is cascaded that the table with the foreign key and all associated data is also removed. This helps with normalization to ensure that all data that is stored is related to other data and is not just fluff data. evl_id_fk INT FOREIGN KEY (evl_id_fk) REFERENCES [evl_data] ON DELETE CASCADE Step 2: Create the Database Script [This script should be based off the information outlined in your Entity-Relationship Diagram] Considerations when creating the database script: a) Identify what type of data will be stored in each column. b) Are there any columns where data entry should be mandatory? c) What are the constraints on the table and how are these foreign keys identified in the script? Will a delete of data from one table cascade to the others? How to create a SQL database from a script: [Reference your diagram and use the tables below to help with data types and optional parameters] 5 | P a g e The first item in order is to create the database. This can be done with CREATE DATABASE [dbname]; You must then USE [dbname]; to work with the database. This is demonstrated on Page 7. Generic CREATE TABLE structure for Microsoft SQL Server (or T-SQL): CREATE TABLE [table_name] ( [column_name] [data type] [optional parameters] [table delimitated by a comma , to indicate next column] index_number INT PRIMARY KEY IDENTITY, other_table_index INT FOREIGN KEY (other_column_name) REFERENCES [other_table_name], timestamp DATETIME NOT NULL, logname VARCHAR(50) NOT NULL, username VARCHAR(50) ) What are Data Types? A data type is what is held by that particular variable, column, etc. For example, if a column will always hold a number, it will have INT (or integer) as its data type to limit what the cell can hold. A field can also have DECIMAL(9,2) (where 9,2 represents up to 9 numbers and 2 decimal points). Things like DATETIME are data types (DATE only stores the date but DATETIME stores both depending on your needs). These can be found very easily through a quick search for almost any programming language. Some of the most common data types for SQL are below: Syntax Data Type Meaning (remember, variables are a number of your choice) INT(n) Integer of n length An Integer is just a whole number (not a fraction) CHAR(n) n Characters This is a fixed length field of length n VARCHAR(n) Up to n Characters This is a variable length character field with a max length of n DECIMAL(n,d) Number with a decimal Use this data type to specify a number of length n with the number of decimal points to store as d DATE Only stores date This field stores month, day, and year. Nothing more DATETIME Stores date and time This field can store date and hour, minute, second, & split second There are many more but these should handle your needs for this task. What kind of optional parameters are available? Parameter Meaning PRIMARY KEY A reference number for a row (there cannot be 2 identical values of a primary key) Indicates that the value should be auto -incremented for each new entry [default (1,1)] IDENTITY(s,i) s = seed [or starting number], i = increment [the number the value goes up by] Identifies a relationship to another table FOREIGN KEY (with constraints set, value must exist in another table) (col_name) (col_name) references a column in another table REFERENCES Comes after a Foreign Key to indicate the table that should be referenced when [table_name] associating the row with another table.