Yale University Incident Process Guide

Yale University Incident Management Process 1 of 17 Introduction

Purpose This document will serve as the official process of Incident Management for Yale University. This document will introduce a Process Framework and will document the workflow, roles, procedures, and policies needed to implement a high quality process and ensure that the processes are effective in supporting the . This document is a living document and should be analyzed and assessed on a regular basis.

Scope The scope of this document is to define the Incident Management Process, and process inputs from, and outputs to, other process areas. Other areas are detailed in separate documentation. The following is a specific list of items that are in scope for this document. Other items not listed here are considered out of scope for this document.

In scope:

 Incident Management Overview o Incident Definition o Incident Management Objectives o Incident Management Policies  Incident Management Process Flow  Incident Management Roles  Incident Management RACI  Incident Management Procedure Flows and Descriptions  Incident Management Prioritization scheme  Incident Management Service Categorization Model  Incident Management Process Metrics

Yale University Incident Management Process 2 of 17 Incident Management Overview

Incident Definition An Incident is an unplanned interruption to a technology service or reduction in quality of a technology service. Failure of a Configuration Item or product that has not yet impacted service is also an incident (i.e. failure of one disk from a mirror set).

Incident Management Objectives The goal of Incident Management is to restore service operations as quickly as possible. Timely and efficient resolution will minimize business impact and increase productivity.

Incident Management Policies Incident reporting must go through the Service Desk, providing Users with a single point of contact All incidents must be logged, prioritized and solutions recorded in the Incident Management System One standard Incident Management Process is defined and used to support all IT Service users The Service Desk manages, tracks, escalates, closes and communicates status of all incident records and is responsible for all incident assignments The Incident Management Process is the conduit of communication of any degradation of service, to the affected users and IT personnel Closure of incidents is dependent on validating with the user that the incident has been resolved and service is restored The Service Desk will own all incidents that they themselves log or that are assigned to them from a Tier 2 provider. Ownership will transfer to the Incident / Situation Manager for major incidents Once a major incident has been validated by the Service Desk, escalation and communication protocols for high-priority incidents are initiated and managed by the Service Desk

Yale University Incident Management Process 3 of 17 Incident Management Process Flow

V3 Incident Management Process

r

/ e

Incident Identification

r m

e Sources:

l

o l

t Event Mgt, Email, Self-Service, Phone

a s

u Call, Tech Staff (Tier 2 / Walkup)

C Management Hierarchic C Escalation Escalation Yes Process

Needed?

/

r

n

t

e

o

n

i

g

t

e

a

a

d

n

i

u

t

a

c

i

n

S

M

I

e

9.0 Major u

l No

e r

a Incident

u e

n Process

g

o

Q

i

a

t

-

n

c

p

a

n

u

u

M

o F r Yes G 6.0

Investigation &

r

l

t Diagnosis

e

i

s

a

T

y

n

l

o

a

–

i

t

n

p

c

A

u

n

o u

+ 2.0

r 1.0 F 2 Incident 4.0 G Incident Logging Categorization Initial Diagnosis 7.0 Resolution and

t Recovery

s Functional y

l Escalation

a Required? n 3.0 A Service No Major No Incident k Request? Incident? s Prioritization Yes 8.0

e Incident Closure D Yes

e 5.0 c i Functional

v To Request

r Fulfillment Escalation

e

S

t

s

r

n

s

e

e

e

n d

c Accountable for the Incident Process and its Evolution/Maturation

i

w

o

c

r

O

n I P

Roles The following roles have been identified within the Incident Management Process.

Role Description Incident Manager • Oversee day to day process execution • Often the Service Desk Manager • Manages major incidents until the appropriate situation manager is identified Situation Manager • Manages and owns major incidents Service Desk Manager • Manages the service desk function, including staffing management activities • Provides guidance to Service Desk Analysts Incident Process Owner • Owns the process end-to-end, including the RACI, process & procedural steps, role & definitions • Accountable for maturing and evolving the process, based on monthly/quarterly/yearly review of process KPIs • Adjusts the process to address performance or changing business needs Service Desk Site Lead • Responsible for the operations of Service Desk Analysts that are geographically disperse, reporting to the Service Desk Manager Service Desk Analyst • Logs incidents • Provides initial diagnosis • Resolve incidents at first point of contact if possible • Escalates incidents

Yale University Incident Management Process 4 of 17 Role Description • Owns non-major incidents

Caller / Customer • The end user having or reporting the service interruption Functional Group – Queue • Assigns incidents to individual Tier 2+ Analysts in the functional group Manager • Monitors and manages support resolution performance • May directly manage (reporting manager) the day to day activities of Tier 2+ analysts outside of process activities Functional Group – Tier 2+ • Group of technical support experts that will handle issues escalated by the Service Analyst Desk • For example, a Network Engineer • Receive process direction for a functional group queue manager, staff management from a reporting manager

RACI Caller / Service Service Incident Situation Functional Functional Incident Customer Desk Desk Manager Manager Group – Group – Tier Process Analyst Site Lead Queue 2+ Analyst Owner Manager 1.0 Incident C R A R Logging 2.0 Incident C R A R Categorization 3.0 Incident C R A, C, I Prioritization 4.0 Initial R A, C, I R Diagnosis 5.0 Functional R A, C Escalation 6.0 Investigation & A R

Diagnosis 7.0 Resolution I R A R & Recovery 8.0 Incident C R R A R Closure 9.0 Major Incident A R R

Process Process Maturity and C, I C R R C R C A Evolution

Yale University Incident Management Process 5 of 17 Process Procedures

1.0 Incident Logging

–

–

p

p 1.1 Verify Issue

u u

o Exists

r

o

r

G

G

l

l

a a

n 1.2 Update Incident

n

o i

o Existing Activity Log &

t t

i Yes Status Provided

t

t c

s Incident? Communicate

c

s

y

n

l y

n Status

u

l

a

u

a

F

n

F No

n

/

A

/

t

A

t

s

+ 1.3 Validate

s

+

y

2

l y

Caller / Customer

2

l

r

a r

a Contact details

e

n

i

e

n i

A and update if

T

A

T

k required

k

s

s

e

e

D

D

e e

c 1.4 Capture and i

c 4.0 2.0 i

v Document Incident

r

v r

e Details

e

S S

Step Activities 1.1 Verify Issue Exists Take steps to validate or replicate the interruption. Gather any data about the issue (screenshots, descriptions). Associate to any concurrent incident (e.g. major outage). 1.2 Update Incident Activity Log & If the Caller is inquiring about status of an existing incident, provide the caller with Communicate Status status as available in the incident record and update the record indicating that the caller was inquiring and update with additional details if available. 1.3 Validate Caller / Customer Complete caller data and ensure contact details are accurate and update if necessary. Contact details and Update if Required 1.4 Capture and Document Incident Complete the short and long description, ensuring they are clear and can be Details understood by others. Collect incident symptoms.

2.0 Incident Categorization

–

–

t

p

t

s

p

k

u

s k

y 2.2

u

s

l o

y 2.1 2.3

/

s

l

r

o

e

a

/ 2.0 Associate 3.0

t

r

e

a Identify Incident Complete Incident

n

t

D G

s Configuration

n

D

G

s

l y

A Type Categorization

l

e

l

y

A Item(s)

a

l

e

c

a

a

+

i

n

c

a

+

i

n

n

2

v

o

n

2

r

v

i

o

A

r

t

r

i

A

e

r

t

e

c

e

i

e

c

S

n

i

S

T

n

u

T

u F F

Yale University Incident Management Process 6 of 17 Step Activities

2.1 Identify Incident Type Capture the incident type based on the customer-reported symptoms.

2.2 Associate Configuration Items(s) If a System (CMS) is If there is no CMS present, present, associate the incident to the capture the device name or ID, Configuration Item(s) (CI) diagnosed to have and based on the primary failed failed and are causing the incident. Note, IT device, capture the component Business and Provider Services may be captured categorization. as CI’s, if implemented. 2.3 Complete Incident Capture IT Business Service categorization, as defined by the customer. Based on the Categorization symptoms and incident diagnosis, capture the IT Provider Service categorization.

3.0 Incident Prioritization

t t

s 2.0 3.1 Major 4.0 s

y No l

y Prioritize Incident Incident?

l

a

a

n

n

A

A

k

k Yes

s

s

e

e

D

D

e e

c 3.2

i

c i

v Escalate Incident

r

v r

e to Incident e

S Manager

S

r

t

r

t e

n 9.0

e

n

g

e

g

e

a

d

a

i

d

n

i

c

n

a

c

n

a

I

n

M

I M

Step Activities

3.1 Prioritize Incident Select the impact and urgency of the Incident according to guidelines if it is not present. This will determine the priority. If priority-based service level monitoring is enabled, the selected priority to define the response and resolution time service level targets for the incident. If service-based monitoring is enabled, the selected priority will only define the response time service level targets for the incident. If the reported service does not have any restoration service level targets defined, a generic priority-based restoration service level target may be used. 3.2 Escalate Incident to Incident Determine if this is a major incident. If so, the service desk agent will escalate to the Manager / Situation Manager incident manager accordingly.

Yale University Incident Management Process 7 of 17

4.0 Initial Diagnosis

r

/

e

r 4.3 Acquire

m

e

l o

l Additional

t a

s Information

u

C

C

l

l a

a 3.0 4.1 Perform Initial n 1.0

n Yes

o Diagnosis

i

o

t

t

i

t

t

c

s

c

s

y

n

l

y

n

u

l

a

u

a

F

n

F

n

/

A

/

t

A

t

s +

s 4.2 Search

+

y 2

l 4.4 4.6 Update Incident

y

2

l Knowledge Base, More

r

a Incident 4.5 Details linking to Known

r 5.0

a e n Known Error Information No

i No Recurring No

e n

i Resolution Error, Knowledge Article,

A T

Database and Required? A

Incident? T

Possible? Change as required

k

– Change Schedule

k

s

–

s

p

e

p

e u

D Yes Yes

u

o

D

r

o

e

r

e

c

G

i

c

G i

v 7.0 Problem

r

v r

e Management e

S Process S

Step Activities

4.1 Perform Initial Diagnosis Document all trouble-shooting steps within the incident record.

4.2 Search Knowledge Base, Known Use initial diagnosis details to search the knowledge base for relevant knowledge. Error, Database and Change Also check the known error database to see if a workaround exists and the change Schedule schedule to see if this is issue could be related to a recently implemented change. Ensure the incident record is coded appropriately. 4.3 Acquire Additional Information If additional information is required, contact the customer. If the customer cannot be reached, place the incident on hold.

4.4 Incident Resolution Possible? If a resolution is possible, proceed to step 7.0 Resolution and Recovery. If resolution is not possible, the incident may need to be assigned to a functional group for resolution. 4.5 Recurring Incident? Determine if other incidents of the same nature have been experienced. If others exist and no root cause has been determined, this may be a good candidate for . 4.6 Update Incident Details linking to Confirm that the incident record is updated and coded according to the diagnosis Known Error, Knowledge Article, steps. Selection of a knowledge record may update (e.g. provider service, component Change as required category, urgency) incident categorization and details.

Yale University Incident Management Process 8 of 17 5.0 Functional Escalation

4.0

t

t s

s 5.1 5.3 Assign y

l Monitor Incident y

l Internal Yes Incident to

a (Incident Ownership Activity) a

n Provider? Functional Group n

A 6.0

A

k k

s No

s

e

e

D

D

e 8.0 5.2 Dispatch to e

c 6.0

i c

i Vendor and

v

r v

r Monitor Incident

e

e S

S 9.0

r

r

e

d

r

e

d

d

r

i 3

d Vendor

i

3

v

/

v

/ Diagnosis

o

r

r

o 7.0

r r

o and Incident

P

o

d

P Resolution If

d

y

n

t

y n

e Possible

r

t

e

r

a

V

a

V P P

Step Activities

5.1 Internal Provider? If assignment is necessary, determine if the functional group that is equipped to resolve the incident is an internal support group or an established external partner that has a support agreement and process established for incident resolution. 5.2 Dispatch to Vendor and Monitor If the functional group is an external group, ensure that the established incident Incident process is followed.

5.3 Assign Incident to Functional If the functional group is an internal group, determine the proper group for Group assignment and assign it to the Group.

5.4 Monitor Incident Optimally, the Service Desk owns the monitoring of incident to resolution and closure. Guidelines for ownership/monitoring include: - Providing customers with desk contact info for updates - Progress notifications originate from a desk monitored email account - Incidents that have not been accepted within response time targets should be initially escalated to the assignment group manager, and ultimately to the Incident Manager if required - Ownership of major incidents should be transferred to the incident manager/ situation manager

Yale University Incident Management Process 9 of 17

6.0 Investigation and Diagnosis

r

r

/

e

/

e

r

m 6.3 Contact

r 5.0

e

m

l

o

e

l t

l Requestor for

o

l

t

a

s a

s Additional

u

C u

C Information if C

C required and

t update Activity

t s

s Log y

l 6.2

y l

a 5.0 Update Incident

a n

n and Assign to

A

A Yes

Service Desk for

+ +

2 Re-Diagnosis and

2 r

Re-assignment

r

e i e Functional

i 7.0

T No

T Escalation?

No

–

–

p

p

u u

o 6.1 Perform Initial Yes r o Accept r Incident Review

G Incident?

G 6.4 Tier 3 l

and Diagnosis l

a Functional Diagnosis

a n

n Escalation to Tier and Incident

o

i

o t

i 3 Resolution

t

c

c

n

n

u

u F F

Step Activities

6.1 Perform Initial Review & Perform initial review to determine if the incident has been properly assigned. Diagnosis

6.2 Update Incident and Assign to If the incident was improperly assigned, the Functional Group assigns it back to the Service Desk for Re-diagnosis and Service Desk for further diagnosis and assignment. Re-assignment 6.3 Contact Requestor for If assignment is proper, accept the incident (work in progress) and determine if Additional Information if Required further information is required contact the customer to obtain then proceed to step 7.0 Resolution and Recovery. 6.4 Functional Escalation to Tier 3 if If the incident requires further assignment to Tier 3 or an external Vendor, the required Functional Group is responsible to work with the external partners and maintain oversight of the incident record.

Yale University Incident Management Process 10 of 17

7.0 Resolution and Recovery

k

k 7.4 Service Desk s

s 4.0 e

t Analyst Contact

e

t

s D

s Functional Group

y

D l

8.0

y e

l or Vendor for

a

e

c

a

i n

c Additional

i

n

v

A r

v 7.2 Work to Resolution Details

A r

e Problem Management

e 7.1 Communicate Resolve Incident if Required S Process - Document Known S Workaround if Updating Incident No Error or Workaround (if Appropriate with Necessary 5.0 applicable)

Details

+

+

2

2

r

r

e

i

e

i

T

T

–

6.0

–

t

p

t Resolution

s

p u

s Incident y

u Yes

l requires Change No

o

y l

r Resolved?

o a

r Request?

a

n

G

n

G

l

A

l A

a Yes 7.3 Update

a n

n Incident

o

i

o t i Resolution Details

t Change c

c and Assign to

n Management n

u Service Desk

u Process

F F

Step Activities

7.1 Communicate Workaround if Investigate sources of information to see if a workaround exists. Check relevant Appropriate knowledge, known error database, problem records, etc. and provide the work around to the customer. 7.2 Work to Resolve Incident If no workaround exists, begin resolution activities making sure to update the incident Updating Incident with Necessary record with all details related to resolution activities. Details If resolution requires that a change be introduced, a Request for Change must be submitted and flow through the Process. 7.3 Update Incident Resolution Once the incident has been resolved it is good practice to review the solution and Details and Assign to Service Desk determine if knowledge could be authored for future occurrences, or if there is a systemic issue that needs to be addressed through the Problem Management Process. Upon resolution, the incident is updated with the proper resolution information and coding, and is assigned to the Service Desk for final closure activities. 7.4 Service Desk Analyst Contact In preparation for closure activities, review the incident details to ensure it is Functional Group or Vendor for completed properly and has the appropriate resolution details. Additional Resolution Details if Required

Yale University Incident Management Process 11 of 17

8.0 Incident Closure

r

r

/

e

/

e

r

m

r

e m

l 8.3 Trigger

o

e

l

t

l

o

l t

a Customer

s Incident a s Yes Incident Closed

u 9.0

C Satisfaction

u Resolved? C

C Survey C

8.1

t No t

s Validate Incident

s

y l

y Resolution with

l a

a Caller / Customer

n n

A 8.2 Update

A 7.0 5.0

k Incident for Re-

k s

s Diagnosis

e

e

D

D

e

e

c

i

c

i

v

r

v

r

e

e

S S

Step Activities

8.1 Validate Resolution with Caller / Follow proper procedures to validate with the Customer that the incident has in fact Customer been resolved. If it has been resolved, the incident will be closed according to procedures. 8.2 Update Incident for Re-Diagnosis If the Customer indicates that the incident has not yet been resolved, it must be sent back for further diagnosis before the incident is closed. NOTE: If the incident is in a closed state when the customer indicates it was not resolved, a new incident should be opened and associated to the original incident. 8.3 Trigger Customer Satisfaction Once the customer has confirmed resolution and the incident is in process of being Survey officially closed, a customer satisfaction survey is to be provided to inform future improvement opportunities.

Yale University Incident Management Process 12 of 17

9.0 Major Incident Process

r

t

r

t

e

n

e

n

g

e g

e 9.1 Validates

a 3.0

d

a

i

d n

i Major Incident 9.4 Prepare Major Incident

c

n

a

c n

a documentation as defined 8.0

I

n

M I

M & Perform Post Resolution Review

9.3 Trigger and execute communications as required until

r Incident Resolved

e

g

a n

a 9.2 Ensure

M Assignment of 5.0

n Incident to o

i Functional Team

t

a

u t i Consult with S Business & IT to invoke Business/IT Service Continuity Plans as defined

Step Activities

9.1 Validate Major Incident If an incident is escalated to a “Major Incident” status, the Incident Manager must first ensure that it should be treated as a Major Incident and be given the enhanced communication and management attention that a Major Incident requires. 9.2 Ensure Assignment of Incident to Ensure that the incident has been assigned to the appropriate team for Functional Team resolution and works with the management structure to coordinate a cross- functional team to address the situation if needed and where the underlying issue is unclear. 9.3 Trigger and execute communications as Ensure that the communication is planned and executed according to internal required until Incident Resolved procedures and triggers. At a minimum communication is to be shared at the beginning and end of a Major Incident and perhaps at specific intervals throughout the resolution process. This communication can be to either internal IT stakeholders or Customers or a combination of both. 9.4 Prepare Major Incident documentation Upon resolution of a Major Incident, documentation must be prepared that as defined & Perform Post Resolution summarizes the issue, actions taken and resolution details. It should also Review trigger root case analysis if required and allow for improvements that can be made to avoid the situation in the future.

Yale University Incident Management Process 13 of 17 Prioritization scheme The prioritization scheme is based on a combination of Impact and Urgency and used to:

 Generate a simplified value in order to drive escalation and notification  Establish appropriate ‘order and extent of work effort to achieve resolution

Impact:

 Measure of scope and criticality to business  Often equal to the extent to which an incident leads to distortion of agreed or expected service levels  Impact is often measured by the number of people affected, critical systems, or financial loss as a result of the service interruption

Urgency:

 Measure of how quickly an incident needs to be responded to based on the business needs of the customer  Prioritizes workload; incidents with higher urgency will be addressed before others, unless SLA breaches are imminent

The follow table shows the resulting priority based on impact and urgency.

Service Categorization Model A Service Categorization model is an important aspect that:

 Recognizes need to capture service vs. technology details  Future-proofed for introduction of Service Asset and Configuration Management  Enhances value of reporting by defining IT service view in terms the business should understand

Yale University Incident Management Process 14 of 17 The following shows the 2 Service Categorization levels that enable views from both Customer/Business and Provider perspectives:

Process Metrics The following table describes the Incident Management KPIs identified.

Yale KPI KPI Name OM DESCRIPTION # 2.1 % Incidents logged How many tickets are incidents? Incidents are tickets excluding service requests and complaints DERIVED Incidents by Source (e.g. Call, Email etc.) How many incidents are logged via calls 11 First incident resolution How many incidents were resolved correctly in first attempt? 12 Incidents resolved at Help Desk How many incidents were resolved at the help desk? 13 Incidents that bypassed help desk support How many incidents were not reported to help desk 17 Reassignment Count How many incidents are reassigned to another work- group? 14 % Incidents assigned incorrectly How many incidents are assigned to the wrong work- group? % Incidents escalated Inaccurately How many incidents are assigned from Tier 2 to Tier 2, vs. being reassigned to Tier 1 15 % Incidents escalated Incidents that are escalated to another group 17 Average time for second level support to Time elapsed between a call being assigned to second respond level support and the call being accepted

Yale University Incident Management Process 15 of 17 Yale KPI KPI Name OM DESCRIPTION # 18 Incidents reopened How many incidents were reopened? 19 Ageing incidents How many incidents are unresolved past threshold 20 Incidents responded on time (Tier 2) How many incidents were responded on time by Tier 2

NEW Incidents responded on time (customer- How many incidents were responded to on time from perspective) logging to last in-progress 21 Incidents resolved on time How many incidents were resolved on time DERIVED Incidents by Priority 24.1 Cost per Incident Ratio of operational budget to users of IT services

Yale University Incident Management Process 16 of 17 Document History Version Date Changes Author 01 12/31/2011 Initial Document Angie Massicotte / Michael Oas 02 01/02/2012 Minor Updates Michael Oas

Yale University Incident Management Process 17 of 17