Got Backups? Log Shipping: Disaster Recovery, Backup and More

Ted Davidson, Director of IT Things to think about…

• Are you confident your database backup is valid? • When is the last time you had to restore a database? • When is the last time you tested a backup? • How much data would you lose if your LAST backup was corrupt? • “CPS just crashed and I lost my document. Can you get it back?” Agenda

• Company Overview • Benefits of Log Shipping • Setup and Configure Log Shipping for CPS • Beyond Backups Preferred Primary Care Physicians • Ted Davidson: 25+ years in Information Technology • Private for-profit medical group founded in 1995 • 24 offices, southwest Pennsylvania • 56 family/internal medicine providers • GE customer since 2002 (PM, EMR, CPS) • CPS Database 700GB – 300 concurrent users • Data center, DR site, Offsite archive • Hyper-V all servers except DB and Application Database Transactions

• A transaction is a sequence of operations performed as a single logical unit of work – Microsoft Technet • The transaction log is a record of all database modifications made by each transaction • Used to maintain database consistency in the event of a error or failure. • Every transaction is assigned a unique Log Sequence Number (LSN) Side note

• ALWAYS use transactions when executing data manipulation statements What is Log Shipping?

• Process to maintain database availability (similar to replication) – Backup transaction log from primary instance

– Copy log file to secondary instance

– Restore transactions to secondary server • Transactions must be restored ”in LSN order” Benefits of Log Shipping

• Low Cost – included with Microsoft SQL Standard • Simple to configure and maintain • Not subject to WAN latency (managed bandwidth) • Scheduled copy & restore • Restore verification • Backup server available during CPS upgrades Disadvantages

• Not real time

• Manual failover / recovery

• Potential for data loss Recovery Model Database Note: ServerSetup must use changes Full or model to bulk- Simple during logged CPS upgrade recovery model What Breaks Log Shipping

• Missing or corrupt transaction log backup file • Changing recovery model to simple on production database • Bring secondary database online • Revert from a database snapshot

– Msg 4214, Level 16, State 1, Line 1 BACKUP LOG cannot be performed because there is no current database backup. Msg 3013, Level 16, State 1, Line 1 BACKUP LOG is terminating abnormally. Preparation

• Prepare Secondary SQL Server or instance • Copy SQL users • Install new CPS database on secondary server • Configure Log Shipping • Restore production database to secondary server • Enable Log Shipping Configure Log Shipping for CPS

• 2 instances of Microsoft SQL Server – Monitor server is NOT required • Setup SQL server based on GE recommendations • Transfer SQL Server users to secondary server – exec sp_help_revlogin – https://support.microsoft.com/en-us/kb/91899 – CPS Application User, Report, Hermes Sp_help_revlogin Output Secondary CPS Servers

• Run ServerSetup to create new CPS database – Use same name as production database – Training license is sufficient for now • Create website on backup App server – Use same application user as production • Verify Centricity access in the backup environment Configure Log Shipping

Adjust backup schedule based on database activity and Recovery Point Objective (RPO) Frequent Log Backups

• The busier your server the more frequent you want to backup the transaction log • Our backups average 4 seconds every 5 minutes • (48 seconds every hour) • Brent Ozar – https://www.brentozar.com/archive/2014/02/back-transaction- logs-every-minute-yes-really/

Stagger backup, copy and restore schedules to eliminate overlap.

Primary Server Secondary Server If you stop nightly FULL backups: Change default backup job on primary server Configure Log Shipping

• Do a full backup of production CPS database

• Restore it to secondary server – RESTORE database CentricityPS FROM DISK = ‘D:\CentricityPS_full.BAK’ WITH NORECOVERY, REPLACE

Configure Log Shipping

• Enable Backup log job on primary server

• Enable Copy and Restore jobs on secondary server Checking Status of Log Shipping Beyond Backup

• Bring backup CPS system online – testing, available during upgrade, disaster • Restart log shipping after recovery • Backup verification • Recover corrupt document • Find misrouted document Bring backup CPS system online

• Disable Restore job on secondary server – Run each job to get latest transactions • Bring secondary database online – restore database CentricityPS with recovery • You may need to run these – ALTER DATABASE CentricityPS SET TRUSTWORTHY ON; – USE CentricityPS; – Exec sp_changedbowner "sa”; Bring backup CPS system online

• Run ServerSetup – License (use production license code, should be visible) – Security, needed to reset RSA encryption key • Restart JBOSS on backup App server

Restart Log Shipping

• Stop JBOSS on backup App server • Put secondary database into recovery mode – BACKUP LOG CentricityPS TO DISK = ‘E:\CentricityPS_tail.trn’ WITH NORECOVERY • Restore latest production full backup – RESTORE database CentricityPS FROM DISK = ‘D:\CentricityPS_full.BAK’ WITH NORECOVERY, REPLACE Restart Log Shipping

• Restore any differential or transaction log backups as necessary – Transaction logs take time to restore

• Enable Restore Job on secondary server – As long as all transaction logs are available restore will be successful

• Use Transaction Log Shipping Status report Backup Verification Recover Corrupt/Lost Document

Scenario:

“I was working on a office visit and CPS crashed and now I cannot open the document.” Recover Corrupt/Lost Document Recover Corrupt/Lost Document

• Stop restore job on secondary server • Restore latest full backup (within last 2 weeks) • Restore latest differential (last night) • Point in time transaction log restore – WITH STOPAT = ‘2016-09-08 10:51:40’ • Bring database online • Use CPS Client to print document or SSMS to extract data Recover Corrupt/Lost Document

Find a Misrouted Document Audit Scenario (Advanced):

A physician signs and routes a document to a triage desktop. The staff claims they never received it. The physician asks you to confirm the document was really routed and if so, who removed it from the desktop.

For demo: tdavidson = physician, sdargay = triage desktop Find a Misrouted Document

• CPS does not create audit entries for document Routes or Removals from the desktop

• CPS stores document routing information in DOCROUTE table

• Utilize third-party tool to search the transaction log files Find a Misrouted Document

• Information needed: – Patient ID (CPS), PID (database) – Doc ID (CPS), SDID (database) – Final sign date and time (CPS) – Doctor ID number (CPS), PVID (database) • Event_Type_ID for “View a normal document”

Find a Misrouted Document • Information needed: – Patient ID (CPS) 379988, PID (database) – Doc ID (CPS) 173, SDID (database) – Final sign date and time (CPS) 9/16/2016 3:56PM – Doctor ID (CPS) 2593, PVID (database) • Event_Type_ID for “View a normal document” = 132 Find a Misrouted Document Find a Misrouted Document • Information needed: – Patient ID (CPS) 379988, PID (database) 1769682495098340 – Doc ID (CPS) 173, SDID (database) 1789660534586090 – Final sign date and time (CPS) 9/16/2016 3:56PM – Doctor ID (CPS) 2593, PVID (database) 1582203703027530 • Event_Type_ID for “View a normal document” = 132 Find a Misrouted Document

• Use third-party tool to inspect transaction log backup • Document was signed on 9/16/2016 @ 3:56PM • Transaction log backups every 5 minutes • Use log backup for 9/16/2016 4:00PM – CentricityPS_20160916200000.trn

Find a Misrouted Document Find a Misrouted Document

• Who accessed this document?

Summary

• Log shipping is a process to maintain database availability (backup, copy restore) • Created new database for CPS backup environment • Transferred SQL users • Configured log shipping • Restored production database to secondary server • Uses for backup environment and backup files Things to think about…

E-mail: [email protected]