Musings of a Data professional

Stuart Moore

Tag: powershell Page 4 of 5

Day 6 of 31 days of SQL Server Backup and Restore using PowerShell: Redirecting backups

A very useful feature of performing your backups with PowerShell is that you can redirect your backups easily on pretty much any condition you want, or even run multiple backups asynchronously to different network drives to save saturating a single NIC.

At the simplest level you can push each database backup into it’s own folder. However, just like with T-SQL backups, if the folder doesn’t exist the backup will fail, so we’ll introduce a quick check:

Import-Module "SQLPS" -DisableNameChecking

$ServerName = "Server1"
$SQLSvr = New-Object -TypeName  Microsoft.SQLServer.Management.Smo.Server($ServerName)

$Db = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Database

foreach ($db in $SQLSvr.Databases | Where-Object {$_.Name -ne "tempdb"}){
    $Backup = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Backup
    $Backup.Action = [Microsoft.SQLServer.Management.SMO.BackupActionType]::Database
    $backup.BackupSetDescription = "Full Back of "+$Db.Name
    $Backup.Database = $db.Name
    $BackupFolder = "c:\psbackups\"+$Db.name+"\"
    if ((Test-Path $BackupFolder) -eq $False){
        New-Item $BackupFolder -type Directory
    }
    $BackupName = $BackupFolder+$Db.Name+"_"+[DateTime]::Now.ToString("yyyyMMdd_HHmmss")+".bak"
    $DeviceType = [Microsoft.SqlServer.Management.Smo.DeviceType]::File
    $BackupDevice = New-Object -TypeName Microsoft.SQLServer.Management.Smo.BackupDeviceItem($BackupName,$DeviceType)

    $Backup.Devices.Add($BackupDevice)
    $Backup.SqlBackup($SQLSvr)
    $Backup.Devices.Remove($BackupDevice)
}

Or we can monitor the space left on the backup drive, and if it falls below what we need (plus a margin for safety) the backups move to another area?:

Import-Module "SQLPS" -DisableNameChecking

$folders = @()
$folders += "c:\psbackups"
$folders += "\\server2\backups$"
$folders += "\\server3\panicspace$"

$FolderCount = 0
$CurrentDrive = New-Object -Com Scripting.FileSystemObject
$MarginForError = 200

$ServerName = "Server1"
$SQLSvr = New-Object -TypeName Microsoft.SQLServer.Management.Smo.Server($ServerName)

$Db = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Database

foreach ($db in $SQLSvr.Databases | Where-Object {$_.Name -ne "tempdb"}){
    $Backup = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Backup
    $Backup.Action = [Microsoft.SQLServer.Management.SMO.BackupActionType]::Database
    $backup.BackupSetDescription = "Full Back of "+$Db.Name
    $Backup.Database = $db.Name

    $CurrentDrive = $folders[$FolderCount]
    if (($Db.Size-$Db.SpaceUnused)+$MarginForError -gt ($CurrentDrive.AvailableSpace/1024/1024)){
        $FolderCount++
    }
    $BackupName = $Folders[$FolderCount]+"\"+$Db.Name+"_"+[DateTime]::Now.ToString("yyyyMMdd_HHmmss")+".bak"
    $DeviceType = [Microsoft.SqlServer.Management.Smo.DeviceType]::File
    $BackupDevice = New-Object -TypeName Microsoft.SQLServer.Management.Smo.BackupDeviceItem($BackupName,$DeviceType)

    $Backup.Devices.Add($BackupDevice)
    $Backup.SqlBackup($SQLSvr)
    $Backup.Devices.Remove($BackupDevice)
}

Here we build up an array of possible backup folders ($Folders), set a $MarginForError in Mb (in this case I’m setting 200MB) and creating a new Scripting object to hold our drive object. As we loop through the database we check that the AvailableSpace on the drive is greater that our requirements (AvailableSpace returns space in bytes, so we need to change that to MB), I check each time just in case something else is using the disk and don’t get caught out. If we are short of space we increase the $FolderCount variable by 1.

When building up the backup target we us $FolderCount to pick the currently used path from the $Folders array, and then continue with the backup as normal.

Using $Db.Size gives us a worst case scenario for the potential size of the backup. Though if you have a lot of free space in your DB it’ll be skewed, this could be corrected by using $Db.spaceavailable as well (correcting for the differences in return values (Size is in MB, SpaceAvilable in KB).

Tomorrow we’ll be looking at Asynchronous backups, allowing you to fire off multiple concurrently running backups from a single script

This post is part of a series posted between 1st September 2013 and 3rd October 2013, an index for the series is available here.

Day 5 of 31 days of SQL Server Backup and Restore using PowerShell: File and Filegroup backups

So far we’ve only looked at backup types supported by the Maintenance Plans (Full, Differential and Log), now we’re going to start going past those and look at File and FileGroup backups. Normally to perform these you’d need a 3rd party tool or you own home rolled SQL scripts.

The script examples are based on using the following database, with 3 filegroups, 1 of which (tertiary) is readonly:

USE [master]
GO

CREATE DATABASE [fg_test]
 ON  PRIMARY
( NAME = N'fg_test', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\DATA\fg_test.mdf' ),
 FILEGROUP [secondary]
( NAME = N'fg_test_2', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\DATA\fg_test_2.ndf' ),
( NAME = N'fg_test_2a', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\DATA\fg_test_2a.ndf' ),
 FILEGROUP [tertiary]
( NAME = N'fg_test_3', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\DATA\fg_test_3.ndf' )
 LOG ON
( NAME = N'fg_test_log', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\DATA\fg_test_log.ldf');
GO

alter database fg_test modify filegroup tertiary readonly;
go

Using SMO, to backup up a single filegroup you use the Add method of the Backup object’s DatabaseFileGroup property, and everything else stays the same:

Import-Module "SQLPS" -DisableNameChecking

$ServerName = "WIN-C0BP65U3D4G"
$SQLSvr = New-Object -TypeName  Microsoft.SQLServer.Management.Smo.Server($ServerName)

$Db = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Database
$Db = $SQLSvr.Databases.Item("fg_test")
$FileGroupName="Secondary"

$Backup = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Backup
$Backup.Action = [Microsoft.SQLServer.Management.SMO.BackupActionType]::Files
$Backup.BackupSetDescription = "Filegroup Backup of Filegroup "+$FileGroupName+" of "+$Db.Name
$Backup.Database = $db.Name
$backup.DatabaseFileGroups.add($FileGroupName)

$BackupName = "c:\psbackups\"+$Db.Name+"_"+[DateTime]::Now.ToString("yyyyMMdd_HHmmss")+".bak"
$DeviceType = [Microsoft.SqlServer.Management.Smo.DeviceType]::File
$BackupDevice = New-Object -TypeName Microsoft.SQLServer.Management.Smo.BackupDeviceItem($BackupName,$DeviceType)

$Backup.Devices.Add($BackupDevice)
$Backup.SqlBackup($SQLSvr)
$Backup.Devices.Remove($BackupDevice)

The line that does the work is:

$Backup.DatabaseFileGroups.Add($FileGroupName)

If you want to backup multiple FileGroups then you can call the method multiple times:

$Backup.DatabaseFileGroups.Add("primary")
$Backup.DatabaseFileGroups.Add("secondary")

Or you can pass them in via the AddRange operator:

$Backup.DatabaseFileGroups.AddRange(("primary","secondary"))

It can also be done with a loop if you can identify which filegroups you want to backup. For example, if you wanted to backup all non readonly filegroups you could replace the line with this script snippet:

foreach ($fg in $db.FileGroups | where-object {$_.ReadOnly -eq $FALSE}){
    $Backup.DatabaseFileGroups.Add($fg.Name)
}

This is one of the cases where the Backup-SQLDatabase is slightly harder to use. To replicate the first SMO example you’d use:

Import-Module SQLPS -DisableNameChecking

Backup-SqlDatabase -ServerInstance WIN-C0BP65U3D4G -Database fg_test  -DatabaseFileGroup "secondary" -BackupFile "c:\psbackups\"+$Db.Name+"_"+[DateTime]::Now.ToString("yyyyMMdd_HHmmss")+".bak"

For the second:

Import-Module SQLPS -DisableNameChecking

Backup-SqlDatabase -ServerInstance WIN-C0BP65U3D4G -Database fg_test  -DatabaseFileGroup "primary","secondary" -BackupFile "c:\psbackups\"+$Db.Name+"_"+[DateTime]::Now.ToString("yyyyMMdd_HHmmss")+".bak"

Note passing in the filegroup names as a comma separated list of strings.

And for the looped version:

Import-Module "SQLPS" -DisableNameChecking

$ServerName = "WIN-C0BP65U3D4G"
$SQLSvr = New-Object -TypeName  Microsoft.SQLServer.Management.Smo.Server($ServerName)

$Db = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Database
$Db = $SQLSvr.Databases.Item("fg_test")
$Filegroup = ""
$i=0
foreach ($fg in $db.FileGroups | where-object {$_.ReadOnly -eq $FALSE}){
    if ($i -eq 0){
        $Filegroup = $FileGroup+"`"$fg.Name`""
    }else{
        $Filegroup = $FileGroup+",`"$fg.Name`""
    }
}
Backup-SqlDatabase -ServerInstance WIN-C0BP65U3D4G -Database fg_test  -DatabaseFileGroup $Filegroup -BackupFile "c:\psbackups\"+$Db.Name+"_"+[DateTime]::Now.ToString("yyyyMMdd_HHmmss")+".bak"

Here we still have to build the connection to the SQL Server instance so we can get the information about the Filegroups. And then loop through it building up the correct string format to pass in as a parameter. I use a simple counter $i to keep track of how many filegroups we’ve found and to ensure the “,”s only go where they’re needed.

Database file backups work in exactly the same way, but using the $Backup.Files.Add() method or the -DatabaseFile parameter.

Tomorrow we’ll be looking at redirecting the backup files.

This post is part of a series posted between 1st September 2013 and 3rd October 2013, an index for the series is available here.

Day 4 of 31 days of SQL Server backup and Restore using Powershell: Catching new databases for a full backup

As the PowerShell backups are just calling the standard T-SQL backup commands, all the usual backup rules a SQL Server DBA is used to still apply.

The main one of these, is that before a Transaction or Differential Backup can be taken the database must have had at least one full backup. This ensures that there is a start to the backup chain.

A common setup at many ISVs is for a new database to be created for each new customer that sign’s up. If that’s an automated process kicked off by an online payment, then the new database could appear at any time, and if you’re only taking full backups once a day the database may not get a full backup for nearly 24 hours! If you’re lucky your developers or suppliers wrote in a check to do an initial backups, do ya feel lucky?

And it’s usually in that first 24 hours that the new customer will be very keen, loading up lots of data and making lots of configuration choicses; but not used to the product, potentially leading to an unexpected deletion of data. If that happens, you’l have no way to recover the data.

By using the following PowerShell script to run your transaction log backups, it will catch any database without a previous backup.

Import-Module "SQLPS" -DisableNameChecking

$Server = "Server1"
$SQLSvr = New-Object -TypeName  Microsoft.SQLServer.Management.Smo.Server($Server)
$Db = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Database

foreach ($Db in $SQLSvr.Databases | Where-Object {$_.Name -ne "tempdb" -and $_.RecoveryModel -eq "Full"}){
    $Backup = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Backup
    if ($Db.LastBackupDate -lt $Db.CreateDate){
        $Backup.Action = [Microsoft.SQLServer.Management.SMO.BackupActionType]::Database
        $BackupName = "c:\psbackups\"+$db.Name+"_"+[DateTime]::Now.ToString("yyyyMMdd_HHmmss")+".bak"
        $Backup.BackupSetDescription = "Full Back of "+$db.Name
    }else{
        $Backup.Action = [Microsoft.SQLServer.Management.SMO.BackupActionType]::Log
        $BackupName = "c:\psbackups\"+$db.Name+"_"+[DateTime]::Now.ToString("yyyyMMdd_HHmmss")+".trn"
        $Backup.BackupSetDescription = "Log Back of "+$Db.Name
        $Backup.LogTruncation = [Microsoft.SqlServer.Management.Smo.BackupTruncateLogType]::Truncate
    }

    $DeviceType = [Microsoft.SqlServer.Management.Smo.DeviceType]::File
    $Backup.Database = $Db.Name

    $BackupDevice = New-Object -TypeName Microsoft.SQLServer.Management.Smo.BackupDeviceItem($BackupName,$DeviceType)
    $Backup.Devices.Add($BackupDevice)
    $Backup.SqlBackup($sqlsvr)
    $Backup.Devices.Remove($BackupDevice
}

We make use of the fact that if a new database doesn’t have a backup then SQL Server returns LastBackupDate as “Monday, January 01, 0001 12:00:00 AM”. We don’t match for that value as if a database has previously existed with the same name and someone hasn’t run sp_delete_database_backuphistory to clear it’s backup history then SQL Server will return the old databases last backup date. But that will still be less than the CreateDate of the new database, so we use that instead.

Tomorrow we’ll be looking at FileGroup and File backups

This post is part of a series posted between 1st September 2013 and 3rd October 2013, an index for the series is available here.

Day 3 of 31 days of SQL Server backup and Restore using Powershell: Transaction and Differential backups

So far we’ve only looked at performing Full Databse backups. PowerShell and SMO are perfectly happy to handle all the other types available in SQL Server:

  • Transaction Log backup
  • Differential
  • File or FileGroup
  • File or FileGroup Differential

In this post we’ll just look at the first 2 types, and then Files and Filegroups in tomorrow’s post But before we do, be aware that SQL Server and PowerShell do not care nor correct your backup file names or extensions, so it will quite happily write out your transaction log backups to filename.bak. As we’ll cover in the restore section, this isn’t a problem if you’re looking into the files, but may cause some confusion if someone just looks in the backup folder! For an SMO Transaction Log Backup we just need to change the backup action to Log:

$Backup.Action = [Microsoft.SQLServer.Management.SMO.BackupActionType]::Log

By default under this action the backup will truncate the logs. If that’s not what you want to happen, then you can override it by using the LogTruncation property of the Backup object:

#To Not truncate the log and leave all the transactions in the log use
$Backup.LogTruncation = "NoTruncate"

#For completeness if you just want to truncate the log, and not back anything up use
$Backup.LogTruncation = "TruncateOnly"

For the Backup-SQLDatabase you use the following parameters:

Backup-SqlDatabase -BackupAction Log -LogTruncationType Truncate ...

Backup-SqlDatabase -BackupAction Log -LogTruncationType NoTruncate ...

Backup-SqlDatabase -BackupAction Log -LogTruncationType TruncateOnly ...

For a SMO Differential backup Action is left as file, but you set the backup objects’s Incremental property to $TRUE:

$Backup.Action = [Microsoft.SQLServer.Management.SMO.BackupActionType]::Database
$Backup.Incremental = $TRUE

And for the Backup-SQLDatabase cmdlet you would use:

Backup-SqlDatabase -BackupAction Database -Incremental ...

So putting this together, a simple script to backup a database with the following schedule:

  • Sunday Evening at 18:00 – Full Backup
  • Every Evening (expect Sunday) at 18:00 – Differential Backup
  • Every hour – Transaction Log

would look like:

$ScriptStartTime = Get-Date
$BackupFileSuffix = ""
Import-Module "SQLPS" -DisableNameChecking

$ServerName = "Server1"
$SQLSvr = New-Object -TypeName  Microsoft.SQLServer.Management.Smo.Server($ServerName)
$DataBase = "DB1"

$Db = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Database
$Db = $SQLSvr.Databases.Item($DataBase)
$Backup = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Backup
$Backup.Database = $db.Name

if ($ScriptStartTime.Hour -eq 18){
    if ($ScriptStartTime.DayOfWeek -eq "Sunday"){
        $Backup.Action = [Microsoft.SQLServer.Management.SMO.BackupActionType]::Database
        $Backup.Incremental = $false
        $backup.BackupSetDescription = "Full Backup of "+$Db.Name
        $BackupFileSuffix = ".bak"
    }else{
        $Backup.Action = [Microsoft.SQLServer.Management.SMO.BackupActionType]::Database
        $Backup.Incremental = $true
        $backup.BackupSetDescription = "Differential Backup of "+$Db.Name
        $BackupFileSuffix = ".bck"
    }
}else{
    $Backup.Action = [Microsoft.SQLServer.Management.SMO.BackupActionType]::Log
    $Backup.Incremental = $true
    $backup.BackupSetDescription = "Log Backup of "+$Db.Name
    $BackupFileSuffix = ".trn"

}

$BackupName = "c:\psbackups\"+$Db.Name+"_"+[DateTime]::Now.ToString("yyyyMMdd_HHmmss")+$BackupFileSuffix
$DeviceType = [Microsoft.SqlServer.Management.Smo.DeviceType]::File
$BackupDevice = New-Object -TypeName Microsoft.SQLServer.Management.Smo.BackupDeviceItem($BackupName,$DeviceType)

$Backup.Devices.Add($BackupDevice)
$Backup.SqlBackup($SQLSvr)
$Backup.Devices.Remove($BackupDevice)

The sections doing the work are:

$ScriptStartTime = Get-Date
...
if ($ScriptStartTime.Hour -eq 18){
    if ($ScriptStartTime.DayOfWeek -eq "Sunday"){
        ....
    }else{
        ....
    }
}else{
    ....
}

As the script starts we get the time it started and use this as a reference. This prevents an issue if we were looping through a number of databases on a Sunday evening and it took more than an hour to get to the lasts one, they would end up getting just transaction backups. By referencing the start time we ensure that all the databases touched by that running of the script get the appropriate backup. We then check to see if the script started at 18:* (we match it a little fuzzily to allow for a delayed start due to other system jobs), if it has we check to see if it’s a Sunday. If it is we set for a Database backup, ensure that we aren’t doing a differential, populate the Description appropriately and set the suffix to .bak as normal. If it’s 18:00 on any other day we set for a Database backup but this time an Incremental backup, and set the suffix to .bck and populate the description correctly. Any other running of the script, and we set the parameters up for a Transaction log backup.

Tomorrow we’ll look at an example of how to use these ideas to catch newly created databases and ensure we start a new backup chain before it’s too late!

Day 2 of 31 days of SQL Server backup and Restore using Powershell: Looping through Databases

Yesterday we looked at backing up a single SQL Server database with PowerShell. There is certainly more code involved than a good old T-SQL style backup. But thanks to the wonders of PowerShell we now have a very reusable piece of code.

As a good example, if you wanted to loop through every database in a SQL Server instance we can now take the central part of the script and loop through it as many times as we want. And if we want to ignore certain DBs then that’s simple as well:

Import-Module "SQLPS" -DisableNameChecking

$ServerName = "WIN-4B40IEFH4CR\SQL2012"
$SQLSvr = New-Object -TypeName  Microsoft.SQLServer.Management.Smo.Server($ServerName)

$Db = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Database

foreach ($db in $SQLSvr.Databases | Where-Object {$_.Name -ne "tempdb"}){
    $Backup = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Backup
    $Backup.Action = [Microsoft.SQLServer.Management.SMO.BackupActionType]::Database
    $backup.BackupSetDescription = "Full Back of "+$Db.Name
    $Backup.Database = $db.Name

    $BackupName = "c:\psbackups\"+$Db.Name+"_"+[DateTime]::Now.ToString("yyyyMMdd_HHmmss")+".bak"
    $DeviceType = [Microsoft.SqlServer.Management.Smo.DeviceType]::File
    $BackupDevice = New-Object -TypeName Microsoft.SQLServer.Management.Smo.BackupDeviceItem($BackupName,$DeviceType)

    $Backup.Devices.Add($BackupDevice)
    $Backup.SqlBackup($SQLSvr)
    $Backup.Devices.Remove($BackupDevice)
}

By changing one line we suddenly make the script a lot more useful:

foreach ($db in $SQLSvr.Databases | Where-Object {$_.Name -neq "tempdb"}){
    code
}

We get our SQL Server to return an object containing all the database objects present on the server($SQLSvr.Databases. We use the Where-Object cmdlet to filter down to only the databases we want. In this case, we’ve asked for all Databases where their name is not equal (-ne to TempDB. You can modify this fitering to exclude anything you’d like based on any Database Property, for example:

  • {_.Name -notlike "*_test"} – returns all databases that don’t end in _test
  • {$_.IsSystemObject -eq $FALSE} – returns all non system databases

Then using Foreach we loop through all the DB objects in the Databases object executing our backup code.

This can be extended further to loop through a list of servers as well:

Import-Module "SQLPS" -DisableNameChecking

$ServerList = @("server1","server2","server3")

foreach ($ServerName in $ServerList){
    $SQLSvr = New-Object -TypeName  Microsoft.SQLServer.Management.Smo.Server($ServerName)

    $Db = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Database

    foreach ($db in $SQLSvr.Databases | Where-Object {$_.Name -ne "tempdb"}){
        $Backup = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Backup
        $Backup.Action = [Microsoft.SQLServer.Management.SMO.BackupActionType]::Database
        $backup.BackupSetDescription = "Full Back of "+$Db.Name
        $Backup.Database = $db.Name

        $BackupName = "c:\psbackups\"+$Db.Name+"_"+[DateTime]::Now.ToString("yyyyMMdd_HHmmss")+".bak"
        $DeviceType = [Microsoft.SqlServer.Management.Smo.DeviceType]::File
        $BackupDevice = New-Object -TypeName Microsoft.SQLServer.Management.Smo.BackupDeviceItem($BackupName,$DeviceType)

        $Backup.Devices.Add($BackupDevice)
        $Backup.SqlBackup($SQLSvr)
        $Backup.Devices.Remove($BackupDevice)
    }
}

Now to look at getting the same outcome using the Backup-SQLDatabase cmdlet:

Import-Module "SQLPS" -DisableNameChecking

$ServerName = "WIN-4B40IEFH4CR\SQL2012"
$SQLSvr = New-Object -TypeName  Microsoft.SQLServer.Management.Smo.Server($ServerName)

$Db = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Database

foreach ($db in $SQLSvr.Databases | Where-Object {$_.Name -neq "tempdb"}){
    $BackupFile = "c:\psbackups\"+$Db.Name+"_"+[DateTime]::Now.ToString("yyyyMMdd_HHmmss")+".bak"
    Backup-SQLDatabase -InputObject $SQLSvr -Database $Db.name -BackupFile $BackupFile  -BackupAction Full -BackupSetDescription "Full Back of "+$Db.Name
}

Again the backup script is shorter, though in this case we still need to use the SMO methods to create a connection to the SQL Server instance so we can loop through it’s collection of databases. And as we have it open, we can also pass it to the Backup-SQLDatabase cmdlet, though note that we have to use the -InputObject parameter rather than the -ServerInstance we used previously

This can be extended to multiple servers in exactly the same way as the SMO version:

Import-Module "SQLPS" -DisableNameChecking

$ServerList = @("server1","server2","server3")

foreach ($ServerName in $ServerList){
    $SQLSvr = New-Object -TypeName  Microsoft.SQLServer.Management.Smo.Server($ServerName)
    $Db = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Database

    foreach ($db in $SQLSvr.Databases | Where-Object {$_.Name -neq "tempdb"}){
        $BackupFile = "c:\psbackups\"+$Db.Name+"_"+[DateTime]::Now.ToString("yyyyMMdd_HHmmss")+".bak"
        Backup-SQLDatabase -InputObject $SQLSvr -Database $Db.name -BackupFile $BackupFile  -BackupAction Full -BackupSetDescription "Full Back of "+$Db.Name
    }
}

Now we’ve looked at performing full database backups, tomorrow we’ll move onto looking at transaction log and differential backups.

This post is part of a series posted between 1st September 2013 and 3rd October 2013, an index for the series is available here.

PowerShell

Day 1 of 31 days of SQL Server backup and Restore using Powershell: Simple Database Backup

We’ll start with the simplest form of backup script, taking a full backup of a database. This demonstrate’s the basic principles that will then be expanded on to perform more complex operations later on in the series.

The script is:

Import-Module "SQLPS" -DisableNameChecking

$ServerName = "Server1\SQL2012"
$SQLSvr = New-Object -TypeName  Microsoft.SQLServer.Management.Smo.Server($ServerName)

$Db = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Database
$Db = $SQLSvr.Databases.Item("psdb1")

$Backup = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Backup
$Backup.Action = [Microsoft.SQLServer.Management.SMO.BackupActionType]::Database
$backup.BackupSetDescription = "Full Backup of "+$Db.Name
$Backup.Database = $db.Name

$BackupName = "c:\psbackups\"+$Db.Name+"_"+[DateTime]::Now.ToString("yyyyMMdd_HHmmss")+".bak"
$DeviceType = [Microsoft.SqlServer.Management.Smo.DeviceType]::File
$BackupDevice = New-Object -TypeName Microsoft.SQLServer.Management.Smo.BackupDeviceItem($BackupName,$DeviceType)

$Backup.Devices.Add($BackupDevice)
$Backup.SqlBackup($SQLSvr)
$Backup.Devices.Remove($BackupDevice)

Now to break it down:

Import-Module "SQLPS" -DisableNameChecking

$ServerName = "Server1\SQL2012"
$SQLSvr = New-Object -TypeName  Microsoft.SQLServer.Management.Smo.Server($ServerName)

$DbName = "psdb1"
$Db = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Database
$Db = $SQLSvr.Databases.Item($DbName)

First we import the SQL PowerShell module, we use the -DisableNameChecking to hide this error message:

WARNING: The names of some imported commands from the module 'SQLPS' include unapproved verbs that might make them less discoverable. To find the commands with unapproved
verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.

which is just letting us know that Microsoft don’t always follow Microsoft’s recommended best practice

Next we build our connection to our SQL Server by putting the Server and Instance name into a string variable and then passing it into the constructor for a new SQL Server object. This examples assumes that you are using Windows Authentication to connect to your server and you are running the script under an account that has permissions to the server. You can pass the value straight to the constructor, but I find having well name variables towards the top of the script much easier to find and you can reuse them throughout, so you don’t have to search and replace each time you want to work on a different server,

Then we create a new Database object by passing the name of the database we want to backup to the constructor

The next section:

$Backup = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Backup
$Backup.Action = [Microsoft.SQLServer.Management.SMO.BackupActionType]::Database
$backup.BackupSetDescription = "Full Backup of "+$Db.Name
$Backup.Database = $Db.Name

This creates a new Backup object, and then we say what type of backup it is. In this case, we used the Database type. For later ease we name the backup. Then attach the database to be backup up.

$BackupName = "c:\psbackups\"+$Db.Name+"_"+[DateTime]::Now.ToString("yyyyMMdd_HHmmss")+".bak"
$DeviceType = [Microsoft.SqlServer.Management.Smo.DeviceType]::File
$BackupDevice = New-Object -TypeName Microsoft.SQLServer.Management.Smo.BackupDeviceItem($BackupName,$DeviceType)

Now we create our backup device. $BackupName for the File backup device is the path to the .bak file. Remeber that the path is relative to the SQL Server Instance you’re asking to perform the backup. ie; that c:\psbackups folder is on Server1, not the machine you’re running the PowerShell on. The type of device and path are passed into the constructor for a new backup device.

And now finally, we come to do the actual backup:

$Backup.Devices.Add($BackupDevice)
$Backup.SqlBackup($SQLSvr)
$Backup.Devices.Remove($BackupDevice)

Add the newly created backup device to the backup object. Perform the actual backup, by calling the SqlBackup passing it the SQL server object we created before. The referenced SQL Server instance will now perform the backup before handing back to the script, so if you’ve pointed this at a 50GB database and are backing it up to slow disks this might be a good time to grab a coffee.

Once the backup has completed the script continues, and we remove the backup device from the backup object

In this case the same operation using Backup-SQL requires much less script:

Import-Module SQLPS -DisableNameChecking
$ServerName = "Server1\SQL2012"
$DbName = "psdb1"
$BackupFile ="c:\psbackups\"+$Db.Name+"_"+[DateTime]::Now.ToString("yyyyMMdd_HHmmss")+".bak"
Backup-SQLDatabase -ServerInstance $ServerName -Database $DbName -BackupFile $BackupFile -BackupAction Database

As you can see, this new cmdlet removes the need for the scriptwriter to manually build the SMO objects manually as here it’s just taking in strings, and doing the work on your behalf.

Tomorrow, we’ll be looking at looping this script across all the databases in a SQL Server Instance excluding those we don’t want.

This post is part of a series posted between 1st September 2013 and 3rd October 2013, an index for the series is available here.

Day 0 of 31 days of SQL Server backup and Restore using Powershell: Why not T-SQL?

Last day of introductory material, tomorrow will feature an actual example and some code, promise.

So why do I think that it’s better to step outside of SQL Server into the world of PowerShell for doing backups?

SQL Server Maintenance plans are pretty good for nice simple environments. A few clicks and 2 maintenance plans later you’ve got some nice scheduled jobs that back your database up once a day, and your transaction logs each hour. Excellent.

But what about if you want to treat your system databases (master, model, tempdb, msdb) differently? Ok, you create another set of maintenance plans. Those databases are pretty fixed (you hope!) so it’s not too much work.

But what if you have hundreds of databases in a single instance and you want to vary when their backups run to spare your IO system? You could build up a number of maintenance plans, but they’ll still need manual maintenance. How much easier would it be to have a set of simple scripts that so you can define rules such as ‘Backup first 10% of databases at 01:00’, ‘Backup second 10% of databases at 02:00’? Much less of your time will be spent on keeping on top maintaining maintenance plans, so you can get on with yet more of the fun parts of your role.

SQL Sever doesn’t even offer a ‘Maintenance Plan’ to help automate restoring your databases to check your backups, so you’re on your own from the start. If you’re only wanting to perform a very simple automated restore, then you could write some simple T-SQL and schedule it via SQL Server Agent. But this would be very hard to write so it could work with any database, or be able to catch unusual setups. You end up in the world of cursors and Dynamic SQL, and are limited by SQL Server’s limited access to the filesystem (assuming you’ve set your security up for security rather than ease of administration (and we all do that don’t we?)).

By using PowerShell we can create scripts that can loop through folders and build up complex restores, can access OS and filesystems to work out how best to layout files, or to abort cleanly if there’s a problem. And if we combine that with a database behind it containing schedules and information, then we can have an automated system that will cope with restoring any of our databases. So when management (or the auditors) ask me how I can be sure that our backups work and that my RTO estimates are accurate I can just pull the numbers out and show them.

Hopefully the next 31 days of posts and scripts will help you towards that goal as well.

Tomorrow, the first script!

This post is part of a series posted between 1st September 2013 and 3rd October 2013, an index for the series is available here.

Day -1 of 31 days of SQL Server backup and Restore using Powershell: SQL Server Powershell Components

Continuing my pre 31 Days  of SQL Server backup and Restore using Powershell into, today we’ll look at getting the PowerShell SQL Server components, and a quick mention of GUIs

All Microsoft operating systems since Windows XP have shipped with PowerShell either installed or as an option. At the time of writing the current version is 3.0 (Download PowerShell v3.0). (PowerShell 4.0 is currently in public beta, but won’t be covered in this series)

Version 3 introduced enough new features to make it worthwhile upgrading to, especially for automating jobs. These include:

  • Better Scheduled Job Integration
  • Improved Integrate Scripting Environment
  • Workflows

A basic install of PowerShell doesn’t include the SQL PowerShell module which allows you to access the SQL Management Objects. These are installed when you install the client tools from your SQL Server media. They are also now freely available as part of SQL Server 2012 Express Service Pack 1 (download here).

The SMO capabilities are the same across the PowerShell module in each SQL Server edition, the limitations in usage are controlled by the edition of SQL Server you connect to. This means you can install the free SQL Express version on your desktop to control your Enterprise edition Servers, and that scripts developed on Express will also work with Standard and Enterprise editions.

SMO also relies on  standard T-SQL calls, so as long as the version of SQL Server supports the functionality you can use it. For instance, all the scripts in this series will work back to SQL Server 2005 (and the basic backups will also work backwards to SQL Server 7 and 2000). But just because PowerShell support AlwaysOn availability groups, it doesn’t mean you can enable them on SQL Server 2005

PowerShell 3.0 ships with a fairly usable  Integrated Scripting Environment GUI. This gives you Intellisense, some basic debugging functionality and an easier to read interface to Get-Help

If you start developing more complex PowerShell scripts then a more fully featured IDE may be preferable. There are a number available, both free and paid for. Commonly used ones are:

Which one you prefer will mostly come down to personal preference, so I’d suggest trying them all and then choosing one.

Tomorrow I’ll lay out some of the reasons why I’ve moved away from T-SQL for a lot of my backup and restore automation.

This post is part of a series posted between 1st September 2013 and 3rd October 2013, an index for the series is available here.

Day -2 of 31 days of SQL Server backup and Restore using Powershell: Introduction

Welcome to day -2 of 31 days of posts talking about using PowerShell to automate SQL Backup and Recovery.

Yes, it is day minus 2. I thought before I got onto the how tos and scripts, and to save wasting some of the 31 days I’d post some background information about why I’d use PowerShell for these tasks, how to set yourself up if you’re not already using PowerShell and also explain some of the assumptions I’ll be making in my scripts throughout this series.

So before we talk about the how, let’s think the why?

SQL Server database backups are pretty simple to setup and schedule. If you’re running a simple setup, then you can use the Maintenance Plan wizard to quickly get your databases backing up on a schedule. But how about if you want to do something not so simple? How about being able to choose where a particular database is being backed up to? Or catch newly created databases for a full backup during the day? And what can you do about scheduling SQL Express backups when it doesn’t come with SQL Server Agent? The scripts and ideas presented should help you to come up with simple solutions to achieve all this

We all know we should be regularly restoring our databases to make sure that our backups are valid, and that we have a proper idea of our Recovery Time Objective. But it’s not the most riveting part of our job, and SQL Server doesn’t offer any way of easily automating database restores without having to write scripts for each individual case. Using PowerShell and it’s SQLPS module you can easily write scripts that will cope all restore eventualities. This means you can automate the testing, and only have to intervene when a problem is found.

And once you can automate backup and restores it becomes easy to create scripts to migrate environments between servers, or restore production data into a dev environment or speed up creating Mirrored or AlwaysOn High Availability setups.

Tomorrow’s post will discuss the components needed to perform these backups – Day -1 of 31 days of SQL Server backup and Restore using Powershell: SQL Server Powershell Components

This post is part of a series posted between 1st September 2013 and 3rd October 2013, an index for the series is available here.

Chunking files into sets of a certain size or number of files with PowerShell

Using PowerShell to scratch another itch. This time though it wasn’t a work or SQL Server related itch for once. I’m a keen cyclist and like to keep track of all my rides, so I’ve been experimenting with various online ride trackers. And I’ve settled on Strava as I can have cool little online badges like this:

And race myself on my own segments.

The only problem was transferring a couple of years worth of data across from other sites or off of my hard drive. Strava has a handy multiple file uploader, but it only allows you to upload up to 25MB or 25 files at a time. To streamline this I wanted to run through all my old files and put them into folders so that each folder contained less than 25MB of data or less than 25 files. Enter PowerShell:

#folder containing files to be "chunked"
$file_to_process = Get-ChildItem F:\strava-tmp

#Hold the current total size of files
$current_total_size = 0

#Set the maximum total size of files for one folder (in this case 24MB)
$max_size = 24*1024*1024

#Set maximum number of files in a folder
$max_count = 25

#Folder numbering count
$current_folder_index = 1

#Count of files in folder
$current_file_count = 0

#Base folder for output folders
$output_base = "f:\strava2"

new-item "$output_base\folder$current_folder_index" -ItemType directory
$fullout = "$output_base\folder$current_folder_index"
foreach ($file in $file_to_process){
    $tmp_size = $file.Length + $current_total_size
    if (($tmp_size -lt $max_size) -and ($current_file_count -lt $max_count)){
        copy-item $file.FullName -destination $fullout
        $current_total_size = $current_total_size + $file.Length
        $current_file_count++
    }else{
        $current_folder_index++
        new-item "$output_base\folder$current_folder_index" -ItemType directory
        $fullout = "$output_base\folder$current_folder_index"
        $current_total_size= $file.length
        copy-item $file.FullName -destination $fullout
        $current_file_count = 1
    }
}

I’d have then liked to have used <code>Invoke-WebRequest</code> to do the actual uploading for me. But it appears that Strava’s v3 API is invite only, which isn’t that useful really.

This script also comes in handy for splitting files up for emailing if you have a maximum attachment limit on your account.

Page 4 of 5

Powered by WordPress & Theme by Anders Norén