Peter McEldowney Mark Waddington | Capstone CSNT 255

Outputting Event Log Events to a Remote SQL Database Using PowerShell

Objective: After completing this lab, the Administrator will have the Event Log from the computer of their choice uploading information to a SQL database. The step-by-step outlines how to create the tables that will be used to organize the data and will cover how to create a basic SQL script to create these tables. Within the script to create the tables, indexes will be covered to ensure quick access to the data by a particular column that is indexed and will also cover items to be aware of. After creating these tables, a script to upload the event log items to the SQL server will be created so that the script can be run from any machine to ensure scalability. After the script is created and tested, this step-by-step will go through how to create a trigger that will run the script whenever a new event log entry is created. Requirements: PowerShell 3.0, Microsoft SQL Server [w/ Management Tools]

Step 1 : Create a diagram of the tables that will be used to store the data

Considerations when designing your tables: a) What data is available? b) How will you identify the machine once the data is in the database? c) What log did the data come from and how will this be stored in the database? d) Is there an efficient way to organize the data to make retrieving the data quicker? e) What columns will people be searching from? f) What should the Primary Keys for the table(s) be?

What is a Primary Key? A primary key is basically an index number. This is the unique number that identifies a particular row. For example, if 2 rows have the same information, how would we identify which row is which?

g) What should the Foreign Keys be?

What is a Foreign Key? Foreign Keys refer to data from another table. These are primarily used to link tables. For example, say you have data in one table and the computer that uploaded the data in another. The foreign key in the data table would reference an identifying key in another so that the data can be linked to the associated machine.

1 | P a g e

The tables and relationships that are being made are referred to as an Entity-Relation Diagram. Not only

will creating this make building the SQL script to create the databases easier, but it will also allow for future database users to access the data efficiently and know what data is available.

a) What data is available? The best way to identify the available data is by looking at the data. For example, using PowerShell, we can pull event log data and look at what is available.

Try this command in PowerShell to find the available data:

Ex: Get-EventLog –LogName System –Newest 1 | Format-List

Now we must decide what information we want to log in our database. [Remember: Databases can grow to be massive so far large environment, it may be more efficient to only take critical information. In this tutorial, we will be uploading all information because we can.]

2 | P a g e

We can actually call specific values that are defined by incorporating the text before the pipe | within parenthesis () and then appending a period. to identify we want to call a value.

Ex: (Get-EventLog –LogName System –Newest 1).Index

Now since we have the information available, we can start grouping particular items together and deciding how we want to arrange everything.

b) Identifying the original machine This can be done within the PowerShell script that will be created in Step 3. Try this command in PowerShell for a preview on how this will be accomplished.

Ex: [Environment]::MachineName

c) Identifying the originating system log Different logs are kept for different aspects of the system. For example, DNS has its own log. PowerShell has its own log, and so do System tasks. The available logs on a system can be obtain by running the following:

Ex: Get-EventLog -LogName *

3 | P a g e d) Organizing the Data for efficiency.

This step all depends on the volume of data that will be stored within the database. For centralize Event Log management, it may be beneficial to store only EntryType, Message, Source, and Username within 1 table so that less data is retrieved when searching for errors.

An example is portrayed below. Notice how only crucial information is stored in the primary table. All other data must be referenced by joining tables together by their similar keys (foreign keys).

e) Frequent columns. Generally people will be searching for information based off specific columns.

These columns (like name of the computer, the type of entry, source, etc) are columns that should contain indexes. An index is simply a tag put onto a column so that it is monitored for efficient data retrieval.

One caution of using indexes is that every time new data is entered, the index must be re-evaluated. As you can imagine, this can cause a great deal of decreased performance and may even crash systems that are inadequate for supporting the amount of data being uploaded.

Indexes are not required (and are not covered here) but are an important aspect to consider if deploying on a large scale as it is much easier to incorporate when designing than trying to implement during production. f) Primary Keys

Primary key columns are (as described above) a unique identifying key. No 2 primary keys in a table can be the same or the data would be inconsistent. This

4 | P a g e

is why it is best to create a special column for the primary key and set it to auto- increment. In the entity-relationship diagram below, you can see how columns with PK are identified with some sort of text that is followed by_id

g) Foreign Keys FK’s should be easy to identify and associate with other tables. The most difficult part is coding Foreign Keys.

One nice part about foreign keys is that you can set constraints on the data that is referenced. For example, if we use the code exemplified below, then when the referenced data in the other table is deleted, the delete is cascaded that the table with the foreign key and all associated data is also removed. This helps with normalization to ensure that all data that is stored is related to other data and is not just fluff data. evl_id_fk INT FOREIGN KEY (evl_id_fk) REFERENCES [evl_data] ON DELETE CASCADE

Step 2: Create the Database Script [This script should be based off the information outlined in your Entity-Relationship Diagram]

Considerations when creating the database script: a) Identify what type of data will be stored in each column. b) Are there any columns where data entry should be mandatory? c) What are the constraints on the table and how are these foreign keys identified in the script? Will a delete of data from one table cascade to the others?

How to create a SQL database from a script: [Reference your diagram and use the tables below to help with data types and optional parameters]

5 | P a g e

The first item in order is to create the database. This can be done with CREATE DATABASE [dbname]; You must then USE [dbname]; to work with the database. This is demonstrated on Page 7.

Generic CREATE TABLE structure for Microsoft SQL Server (or T-SQL): CREATE TABLE [table_name] ( [column_name] [data type] [optional parameters] [table delimitated by a comma , to indicate next column] index_number INT PRIMARY KEY IDENTITY, other_table_index INT FOREIGN KEY (other_column_name) REFERENCES [other_table_name], timestamp DATETIME NOT NULL, logname VARCHAR(50) NOT NULL, username VARCHAR(50) )

What are Data Types? A data type is what is held by that particular variable, column, etc. For example, if a column will always hold a number, it will have INT (or integer) as its data type to limit what the cell can hold. A field can also have DECIMAL(9,2) (where 9,2 represents up to 9 numbers and 2 decimal points). Things like DATETIME are data types (DATE only stores the date but DATETIME stores both depending on your needs). These can be found very easily through a quick search for almost any programming language. Some of the most common data types for SQL are below:

Syntax Data Type Meaning (remember, variables are a number of your choice) INT(n) Integer of n length An Integer is just a whole number (not a fraction) CHAR(n) n Characters This is a fixed length field of length n VARCHAR(n) Up to n Characters This is a variable length character field with a max length of n DECIMAL(n,d) Number with a decimal Use this data type to specify a number of length n with the number of decimal points to store as d DATE Only stores date This field stores month, day, and year. Nothing more DATETIME Stores date and time This field can store date and hour, minute, second, & split second There are many more but these should handle your needs for this task.

What kind of optional parameters are available? Parameter Meaning PRIMARY KEY A reference number for a row (there cannot be 2 identical values of a primary key) Indicates that the value should be auto -incremented for each new entry [default (1,1)] IDENTITY(s,i) s = seed [or starting number], i = increment [the number the value goes up by] Identifies a relationship to another table FOREIGN KEY (with constraints set, value must exist in another table) (col_name) (col_name) references a column in another table REFERENCES Comes after a Foreign Key to indicate the table that should be referenced when [table_name] associating the row with another table. NOT NULL Specifies that a value must be entered into t his column or an error will return. CONSTRAIN T Indicates that this column is referenced by another table [see below for an example]

I. Look at your Entity Relationship diagram and decide the data type for each field. The Entity Relationship diagram that has been converted into a SQL script to create tables:

6 | P a g e

The code for the servers, evl_primary, evl_data, and evl_timestamp table, would resemble: CREATE DATABASE server_event_logs; USE server_event_logs;

CREATE TABLE [servers] ( pc_id INT CONSTRAINT pc_id_fk PRIMARY KEY IDENTITY, pc_name VARCHAR(100) NOT NULL );

CREATE TABLE [evl_primary] ( evl_id INT CONSTRAINT evl_id_fk PRIMARY KEY IDENTITY, pc_id_fk INT FOREIGN KEY (pc_id_fk) REFERENCES [servers], entry_type VARCHAR(50) NOT NULL, message VARCHAR(1000) NOT NULL, source VARCHAR(100), username VARCHAR(100) );

CREATE TABLE [evl_data] ( data_id INT PRIMARY KEY IDENTITY, evl_id_fk INT FOREIGN KEY (pc_id_fk) REFERENCES [servers] ON DELETE CASCADE, index_id INT, category VARCHAR(100), category_num INT, replacement_str VARCHAR(100) );

CREATE TABLE [evl_timest] ( time_id INT PRIMARY KEY IDENTITY, evl_id_fk INT FOREIGN KEY (evl_id_fk) REFERENCES [evl_data] ON DELETE CASCADE, instance_id INT, time_gen DATETIME, time_written DATETIME ); Note: These must be executed query by query in SQL Server Management Studio (as an Administrator).

7 | P a g e

II. Pay attention to columns that are defined with NOT NULL. These columns must ALWAYS have data inputted or the data entry will fail. This is to make monitoring data entry a minimal task.

[Note: You can also code a database to upload error that happen in SQL but that is beyond the scope of this step-by-step.]

III. Notice the placement of the ON DELETE CASCADE parameters. This means that if data is deleted from one of the associated tables, so will the entry in the evl_primary table. This makes removing an event log entry much easier as the database will do it automatically.

IV. Also, notice the IDENTITY option for the primary keys. This option will auto-increment that column. How to perform this is explained in the table in the beginning of Step 2.

8 | P a g e

Step 3: Importing the proper SQL PowerShell module and testing connectivity

The first component of this step is to understand how to manage SQL through PowerShell. Once SQL has been installed on a machine, the SQLPS module is made available. Let’s go ahead and import it:

We can check all the available Cmdlet’s and functions that are available with the following cmdlet:

Ex: Get-Command –Module SQLPS

Now since we know what we can do, we must understand what we want to do. This will become more apparent the more you work with SQL as you will experience more applications for the various available cmdlets (notice how data backups, replication, and availability groups can all be automated).

For basic queries, let us take a closer look at the Invoke-Sqlcmd cmdlet. Try the following:

9 | P a g e

Ex: Get-Help Invoke-Sqlcmd

We can find more specific information about each parameter with the –detailed switch appended.

Ex: Get-Help Invoke-Sqlcmd –detailed

Let’s try a basic query. We will insert data into the servers table about the computer that we are sending the information from.

First, we need to define a variable that is a secure string so that it will pass the password to the database properly. Otherwise, we will receive authentication errors.

We can do this with the ConvertTo-SecureString cmdlet:

Ex: $userpass = ConvertTo-SecureString –String password –AsPlainText –Force

We now have password stored as a secure string in the $userpass variable. Notice how if we call the variable, we cannot see the password, we only get System.Security.SecureString

10 | P a g e

Next, we will define a variable that holds the Machine Name

Ex: $pc_name = [Environment]::MachineName

Notice how this is the same command we used in step 1, only we have put the result into the variable $pcname.

Now if we use the Invoke-Sqlcmd cmdlet, we can pass a SQL query to it with our variable so that we can query the database.

1 Invoke -SqlCMD -Database server_event_logs ` 2 -ServerInstance SCVMM \ADK ` 3 -Password $userpass ` 4 -Query "INSERT INTO servers VALUES ('$pc_name');"

When calling a cmdlet such as this, it is important to use the proper parameters with the cmdlet.

1. First we call the –Database parameter. This is used to signify what database on our SQL server we want to USE when we are sending queries. 2. The –ServerInstance parameter signifies the machine and Instance Name for our SQL server. This is the name that is logged into when SQL Server Management Studio is loaded. Check in the start menu under the SQL Server start menu group. Start –> All Programs –> Microsoft SQL Server –> SQL Server Management Studio 3. Next, we are passing the System.Security.SecureString password that we defined in the variable above. [Note: We cannot just pass text here or SQL authentication fails] 4. To finish the command, we must then pass a query to the server. For example, we can “INSERT INTO servers VALUES (‘$pc_name’);” Note the syntax that we use. We are INSERTing data INTO the table servers and the VALUES we want to insert should be enclosed within quotation marks “, not apostrophes ‘. Since the whole statement is in what are some called “double quotes,” PowerShell interprets the statement as a string so PS will pass single quotes. PowerShell will still pass variables with double quotes. This means that we should be passing “INSERT INTO servers VALUES (‘ ‘);”

11 | P a g e

[For more information, check out the following (you must have updated your help to view this documentation via Update-Help ] Ex: Get-Help about_Quoting_Rules | more

Step 4: Create the script that will INSERT the data into the appropriate tables

One of the most important parts in creating this particular PowerShell script is allowing the passing of parameters to the script. This eliminates the need to customize multiple scripts for each log and allows you to pass values to the script that can be used during data entry. For this particular script, we will be using a [CmdletBinding()] object to pass the name of the log that the event is coming from.

1 [CmdletBinding ()] 2 param( 3 [Parameter(Mandatory=$true,Position=0)] 4 [string]$ logname = "System" 5 )

1. CmdletBinding indicates that advanced functionality is available within the script or cmdlet. For example, here we are using this to mandate passing a parameter. 2. ‘param(‘ is the opening statement of the beginning part of your script or function being created. 3. [Parameter(Mandatory=$true,Position=0)] Parameter indicates that this value is passed as a parameter or switch. Since Mandatory has the Boolean (1 or 0) value or $true (which is 1), this value must be passed for the script or function to execute. Position means that the person does not have to specify –logname to pass a value to the variable $logname. The string that is passed for the variable must be in the absolute first position, which is position 0 (because binary starts counting at 0). 4. [string]$logname = “System” Here we begin defining variables that are passed to the script or function. [string] is the data type that the variable is stored as. $logname is the name of the variable where the value is stored. For example, if –logname “PowerShell” is passed to the script or function, then if $logname is called, it will contain the string “PowerShell”. [Note: We use double quotation marks to include everything included within so that more than one word can be stored.] = “System” means that the variable before the = will have “System” stored as the default string. 5. ) is simply indicating that it is the end of the parameters. More parameters can be specified but must be included before this end parenthesis specified here. Parameters must be separated by a ,

In the PowerShell script outlined below, notice how most variables have a specific data type. This helps when inputting data to the database because then PowerShell will pass the proper data type to the SQL database. One reason why all of the variables have data types is because I was receiving data type errors upon entering data into the database.

As well, notice how # comments are added with a # pound sign # to indicate a comment/ignore this line.

Each section of code has explanations as to what is being performed in each section.

12 | P a g e

# set arguments that can be passed to the script # 1 [CmdletBinding()] 2 param( 3 [Parameter(Mandatory=$true,Position=0)] 4 [string]$logname = "System" 5 )

# ensure that SQL powershell module is available # 6 import -module SQLPS

<# This first variable definition stores a System.Security.SecureString object for the $userpass variable.#> # define variables # 7 $userpass = ConvertTo -SecureString -String 555erv333@ -AsPlainText –Force

<# Here we take an environment variable and store it as a string in the $pc_name variable Next, we query the database and attempt to store the value in pc_id as the variable $pc_id_select For ServerInstance, Make sure it is your local machine. #> # get pc_id for specific machine # 8 [string]$ pc_name = [Environment]::MachineName 9 [int]$pc_id_select = (Invoke -SqlCMD -Database server_ event_logs -ServerInstance STOUT \ADK - Password $userpass -Query "SELECT pc_id FROM servers WHERE pc_name = '$pc_name';").pc_id

<# Here we are validating that the machine has an entry in the servers table. If the value is 0 (aka, does not exist), then the $pc_name is inserted into the servers table. Then we query the pc_id value for the specific pc_name and store it into a variable #> # check for pc_id, put it in to variable if it is 0 (or non -existent) # 10 if ($pc_id_select -eq 0) { 11 Invoke -SqlCMD -Database server_ event_logs -ServerInstance STOUT \ADK -Password $userpass - Query "INSERT INTO servers VALUES ('$pc_name');" 12 [int]$pc_id_select = (Invoke -SqlCMD -Database server_ event_logs -ServerInstance STOUT \ADK - Password $userpass -Query "SELECT pc_id FROM servers WHERE pc_name = '$pc_name';").pc_id 13 }

<# Now it is time to store the values of the event log entry into variables for the evl_primary table. This was demonstrated in Step 1, we are not just storing the returned value into variables. #> # get event log into into variables # #### table = evl_primary ### 14 [string]$ev_ent_type = (Get -EventLog -LogName $logname -Newest 1).EntryType 15 [string]$ev_mess = (Get -EventLog -LogName $logname -Newest 1).Message 16 [string]$ev_src = (Get -EventLog -LogName $logname -Newest 1).Source 17 [string]$ev_usr = (Get -EventLog -LogName $logname -Newest 1).Username

<# Here we are doing the same thing as above but for the evl_data table. Note the similarities. The only difference is the data stored in the variables. #> #### table = evl_data ### 18 [int]$ev_ind = (Get -EventLog -LogName $logname -Newest 1).Index

13 | P a g e

19 [string]$ev_cat = (Get -EventLog -LogName $logname -Newest 1).Category 20 [int]$ev_cat_num = (Get -EventLog -LogName $logname -Newest 1).CategoryNumber 21 [string]$ev_repl_str = (Get -EventLog -LogName $logname -Newest 1).ReplacementStrings

<# Now we must create a query to the database. To do this, we must follow SQL syntax. INSERT INTO table_name (column_id, next_column_id, …) VALUES ($variable_for_column_id, $variable_for_next_column_id, …) #> # input variables to evl_ primary table # 22 Invoke -SqlCMD -Database server_ event_logs -ServerInstance STOUT \ADK -Password $userpass ` 23 -Query "INSERT INTO evl_primary (pc_id _fk, entry_type, message , source, username) VALUES 24 ($pc_id_select, '$ev_ent_type', '$ev_mess', '$ev_src', '$ev_usr');"

<# Next, we set a value for $evl_id_sel by querying the database for the primary key of evl_primary Next we INSERT data from the newest entry into the evl_data table. #> # input variables to evl_data table # 25 [int]$evl_id_sel = (Invoke -Sqlcmd –Database server_event_logs –ServerInstance STOUT \ADK – Password $userpass –Query “SELECT evl_id FROM evl_primary WHERE pc_id_fk = $pc_id_select;”).evl_id 26 Invoke -Sqlcmd –Database server_event_logs –ServerInstance STOUT \ADK –Password $userpass - Query "INSERT INTO evl_data (evl_id_fk, index_id, category, category_num, replacement_str) VALUES ($evl_id_sel, $ev_ind, '$ev_cat', $ev_cat_num, '$ev_repl_str');"

<# Here we define the final 3 variables for the evl_timest table so that the information can be INSERTed into the table #> ### table = evl_timest ### 27 [int]$ev_inst_id = (Get -EventLog -LogName $logname -Newest 1).InstanceId 28 [string]$ev_tim_gen = (Get -EventLog -LogName $logname -Newest 1).TimeGenerated 29 [string]$ev_tim_writ = (Get -EventLog -LogName $ logname -Newest 1).TimeWritten

<# Finally, we use Invoke-Sqlcmd a final time to upload the final set of data to the evl_timest database #> # input variables to dbo # 30 Invoke -SqlCMD -Database server_ event_logs -ServerInstance STOUT \ADK -Password $ userpass ` 31 -Query "INSERT INTO evl_timest (evl_id _fk , pc_id, instance_id, time_gen, time_written) VALUES 32 ($evl_id, $pc_id_select, $ev_inst_id, '$ev_tim_gen', '$ev_tim_writ');"

Step 5: Set a trigger on the Event Log to upload the information upon an event

a) Go into Event Viewer This can be accomplish via run eventvwr.msc or This can also be accomplish through

Control Panel
Administrative Tools
Event Viewer

14 | P a g e

b) Next, expand Windows Logs (or the log of your choice) so that you can view the log

c) Right click the Event Log of your choice and Select

 Attach a Task To this Log…

d) The Name and Description field are only locally significant. This means that they are your choice and have no bearing on the event.

e) Click Next because no information about ‘When an Event is Logged’ can be changed as this was select previously in the right click menu.

f) Next, we want to start a program because we have a PowerShell script to execute.

15 | P a g e g) The next part is the most important part. This is where we choose how to run the script that we have created.

For the Program/script part, we want to call powershell Next we need to know which arguments (or switches, or parameters) that we want to pass to powershell.exe so that it will execute properly.

We can find these optional parameters by running powershell.exe /? from the command prompt. [for the full output, run the command in any command prompt window]

16 | P a g e

As we notice from the previous output, we can bypass the execution of scripts. This is handy because now we do not have to change the execution policy

-ExecutionPolicy Bypass

Next we notice how we can call a file. We will use this to point the script. In the syntax, it specifies that we can use [–File ]

The square brackets indicate that it is an optional field. Here is the syntax for the argument that we are passing. We are passing “System” as the “Name of the Log” because that is the log we are uploading information from.

-File C:\scripts\evl2sql.ps1 "Name of the Log"

Another switch to consider is the –WindowStyle argument. This will allow us to hide this task from displaying on the screen whenever an event occurs.

-WindowStyle Hidden

To summarize, the final result that will go into Add Arguments box is:

-ExecutionPolicy Bypass –WindowStyle Hidden -File C:\scripts\evl2sql.ps1 "Name of the Log"

Pay attention to the order as PowerShell demands a certain order for the arguments passed. This is found in the powershell /? output. h) The final part is the click Finish. If there are other configurations you would like to perform, check the check box. If not, the Event Log trigger should be functional.

After Finish is clicked, a window opens that telling that Task Scheduler is where the event can be viewed and edited in the future. This is covered in Step 6.

17 | P a g e

Step 6: Testing Your Setup to Ensure it is Functioning

Go into Task Scheduler. Here you can view the Event that was previously created. [Run
taskschd.msc]

In the Task Scheduler, go down the hierarchy to
Task Scheduler Library
Event Viewer Tasks

The next part involves testing to make sure that Events are properly being uploaded to the Database.

This is done with SELECT queries to the database.

This is where the foreign keys come into effect.

If you return no results from your queries, it is time to troubleshoot the script.

Go into PowerShell and run your script.

For the script that I have placed in the location C:\scripts\evl2sql.ps1, I run:

powershell -ExecutionPolicy Bypass -File C:\scripts\evl2sql.ps1 "System"

Notice how you do not include the –WindowStyle Hidden argument. This is so we can see the output of the script so that we can troubleshoot. The output is shown below.

18 | P a g e

The issue that is happening is on line 58, 59, and 70. We can see that on the 2 nd red line of each item returning an error. If we edit our script in PowerShell ISE, we can look for errors.

As for the general information uploaded to the SQL database, we can access it by using SELECT statement below.

SELECT * FROM servers, evl_primary WHERE servers.pc_id = evl_primary.pc_id_fk;

19 | P a g e