If you’re looking to record the error and warning outputs from your PowerShell commands whilst keeping your console free of nasty red text, then the ErrorVariable and WarningVariable parameters are just what you’re looking for.
Which assuming you don’t have C:\DoesNotExist\ on your workstation, then you should see output like this:
From this quick example a couple of things stand out:
* you pass the name for the Error variable in without the $
* If you resuse a variable name it’ll get clobbered, so be careful (or read on)
* The error is still output to the screen
The last one is not what we want. So one more little tweak will make that disapper
That’s it, just write an advanced function and PowerShell puts all this in behind the scenes for you
Now, what happens if you’re calling multiple commands and want to store the error output from all of them. Do you need to specify a new ErrorVariable for each one? That could get horribly messy with auto generating variable names in a loop. Luckily there’s a built in method for handling that, prepend the variable name with + and the new data will be added to the variable (which is an array):
So you can reuse the same name throughout your script. Don’t worry about when to add the + to the string, behind the scenes PS checks if the variable exists and if it doesn’t creates it the first time it’s used.
These Error and Warning paramters are very useful for collecting information about failed runs, and from hiding nasty screens of red text from end users if you don’t want to worry them.
Whilst going through my blog roll on Feedly this weekend, I spotted Kevin Marquette’s blog post announcing his new Chronometer module.
At a high level it generates a script profiler timeline as you run through a script. Showing you how long each line takes to run, which lines weren’t run and how many times that line has been called.
I’ve just finished writing a number of functions for the dbatools module to implement some new restore functionality, this seemed like a great way to peer into my functions and see what was going on inside, and spot where I was spending most of the time. This should make it easier to work out where I needed to focus attention to improve performance.
Whilst having a play with someone else’s code I wanted to quickly find all the function definitions within a module, and then all the function calls within a function definition.
Having had a quick bingle around for a prewritten regex example I didn’t come up with much that fitted the bill. So in the hope that this will help the next person trying to do this here they are:
Assumptions:
A PowerShell function name is of the form Word-Word
A PowerShell function definition is of the form “Function Word-Word”
A Powershell function call can be preceeded by a ‘|’,'(‘, or ‘ ‘
The script is written using a reasonable style, so there is a ‘ ‘ post call
So to find the function definition I ended up using:
-match 'function\s(\w+-\w+)'
The function name ends up in $Matches[1]
And for a PowerShell function call:
-match '[^\s|(]\w+-\w+'}
The function name ends up in $Matches[0]
This works accurately enough for what I needed it to do. Feel free to let me know if you can spot any improvements on it.
Probably a bit niche this one, but as I scratched my head trying to work it out I’ll share it to save others in future.
The Problem
Working on some new functionality for DbaTools, adding the ability to cope with striped SQL Server backupsets in fact. Everything’s going pretty swimmingly, I’ve got a process for finding the backup files and grouping them together.
Adding them to the Restore device collection is nice and easy as I have a Object to loop through:
The problem comes when I want to reuse my restore object as I loop through the Differentials and the transaction logs. If these Devices aren’t removed, then SQL Server tries to reuse them on every run. As this code is designed to be run anywhere by anyone I’ve know idea of the number of devices needed. Not only will people stripe their backups across differing numbers of files, but it’s possible to strip your backups differently for each type! So this is perfectly fine:
Full Backup – Across 4 files
Differential Backup – Across 2 files
Transaction Backup – Across 1 file (you CAN strip transaction backups if you want)
and you can even do it differently for different runs.
I’ve found examples for c# if you know the devices names that don’t work in PowerShell. I’ve found methods using Get-Member that don’t appear to be documented, and generally worn a dent in my work desk with my forehead (annoying my co-workers with the dull thud).
The classic PowerShell way of using ForEach fails:
ForEach ($Device in $Restore.Devices)
{
$Restore.Devices.Remove($Device)
}
Fails with the nice red error message:
An error occurred while enumerating through a collection: Collection was modified;
enumeration operation may not execute..
The Solution
The oft forgotten while loop!
while ($Restore.Devices.count -gt 0)
{
$device = $restore.devices[0]
$restore.devices.remove($Device)
}
No need to know beforehand how many objects you need to remove, though here I’m expecting at least 1. If there was a chance of 0 runs I’d use a do…while loop instead. But as if there wasn’t even 1 backup device to remove I’d have had an error during the restore I’m fairly safe here.
As part of my talk at the Nottingham SQL Server User group last night, I offered some tips on how to track SQL Server index usage over time.
sys.dm_db_index_usage_stats is a very handy table for seeing how many seeks/scans and updates your indexes get. But there are a few issues with. Firstly it’s a DMV, so it will be flushed on a number of events:
SQL Server Instance Restart
Database Detach
Database Close
SQL Server 2012/2014 – Alter Index rebuild
For a good thorough check normally a full months worth of stats are needed, this normally covers most of the business processes. I may event be useful to compare stats across months so you also capture events such as Financial Year end.
A better break down of when the scans and seeks happened can be very useful. It’s all well and good knowing your index was scanned 5000 times and Seeked across 15000 times, but what processes were doing that? Was it OLTP workload during 9-5 or was it the overnight processing? This could make a big difference to any planned Index changes.
So here’s a technique I use. It’s pretty simple and allows you to tune it to your own needs,
Lining up a few SQL Server Usergroups speaking sessions for the year already:
All sessions will be:
Indexing Nightmare – Cut Through the Clutter
Inherited a database with 30 indexes on every table? Has the vendor for your 3rd party app recommended more indexes over the years than you can remember? Got indexes that were added to fix a data load problem 4 years ago, but not sure if they’re still being used? Indexes are key to SQL Server performance, but like everything too much of a good thing is a bad thing. In this sessions we’ll look at how you can analyse your current indexes with the aim of consolidating them into useful ones, even removing some completely and how to improve the ones you’ve got left
Except for Southampton, where it’ll be:
Get on the Bus
As time goes by, more systems are crossing hosting boundaries (On Premises, Azure, multi-cloud provider, ISVs). We need a simple reliable mechanism to transfer data between these systems easily, quickly and reliably. Microsoft have taken their Message Bus technology and moved it to the cloud as Service Bus. This session will introduce you to this service and give examples of how internal databases can safely process data from cloud hosted applications without having to be exposed to the InterTubes. Examples are predominantly .Net C#, but aren’t complex!
Dates
Nottingham SQL Server Usergroup – 12th January 2017
(Also presenting will be Steph Middleton, talking about Building a Robust SSIS Solution)
(More details and registration here
Hope to see some of you there. And if there’s any other usergroups out there that are looking for speakers then let me know, have presentations on SQL, Powershell and general IT process to offer.
As part of my work on the Backup/Restore portions of DBATools I was interested to see if I could avoid the overhead of having to parse files using SQL Server to find out if they were backup files, and if they were what they contained. (I’m a DBA, ergo I’m paranoid, ergo I’m not trusting a file extension!)
Working on the assumption that SQL Server has a RESTORE HEADERONLY option, and that the speed results are returned with even from large backup files, I was hoping to find all the information I needed fairly early on in the file headers.
In this post I’m going to show how I worked through this using PowerShell (DBATools is trying to avoid using anything other than easily available PS cmdlets, so pulling dlls and other .Net objects would feel a little wrong!).
tl;dr
It sort of works. Not very useful with compressed backups, but reliable for filtering normal backups in a hurry.
So for this month’s T-SQL Tuesday Kenneth asked us to talk about backups.
If you’ve been reading my blog for a while, you’ll be aware that I have a ‘thing’ about backups and restores (31 days of posts in a row on a topic shows a slight interest!), and I’ve been speaking at SQL user groups on the subject for years as well.
Why?
During my whole DBA career I’ve been aware that the absolute worst thing that can happen to me is having to admit to my employer that I can’t recover their precious data to the point they want (RPO), or do it in a time that they’re happy with (RTO).
To me, if either of those happens it’s a Resume Generating Event! And that’s an event I’ve spent years avoiding. And having those bases covered also stops other mistakes (what, I ran that DROP TABLE in prod?) from becoming RGEs as you’ve got a safety net.
As I’ve moved into management, I’ve also come to realise it’s even worse when you hear from someone who reports to you that they aren’t 100% sure about their backups, this could be 2 RGEs for the price of one!
So I’ve always been hugely keen on backups, and more importantly on ensuring that they’ve been tested and you’ve practiced your restores as much as possible.
SQL Server makes it very easy to perform backups, plenty of options to perform and unless you’ve a fairly large or specialised database setup then there’s fairly little setup required.
Restoring a database, now thats a whole other story. If you’re like most people, you’re taking a full backup each night and then transaction log backups hourly through the day. So at worst case you’re got 25 files to restore.
On a larger system you could have a weekend full backup a twice daily diff backup and 10 minute transaction log backups. Now, these things always fail just before the next full backup so you’re looking as 1 full backup, 13 differential backups and 71 transaction backups to process and restore! Now that’s going to take a while to script out and get going!
SSMS will help you out if you have access to the server that made the backups and it can see msdb, if that whole instance is now a smoking crater, all you’re going to have is a folder full of backup files.
So you’re thinking about stepping up to speak at a SQL Server (or any other technical event), or are having your arm gently twisted by an organiser to do so. How bad is it going to be?
tl;dr version:
– Just do it, it’s easy, and it’s great!
Long Version:
Not very. Let’s break down the most comment arguments:
1) – I’m not using the later version
Doesn’t matter. Most people out there won’t be. As of writing this, there are very few people running SQL Server 2016, but there are a lot of people still on SQL Server 2012 (and older!). So don’t think you have to be talking about the latest greatest feature
2) – I’m not using the coolest technology
Yes, each SQL Server release has a must-use technology which people preach about. But that’s not always what people want to hear about. Replication is as old as dust, but it’s still something people want to learn about or know how to fix, a good replication talk aways gets listeners. I talk a lot about backups, and not the new features either, and those talks go down well. What about indexing and performance, well those are perennial favourites, and everyone does them differently so maybe you’ve got something to add there
3) – I’m not doing anything exciting
Neither are most people out there! The lie in the marketing papers is that everyone should be doing a billion transactions a second and have a multi terabyte Web Scale database!
Truth is, 90% of your audience aren’t doing that either. Most of us have the same issues, too many databases and not enough time to look after them all. Those are topics that will grab people
4) – I’m not going in depth enough.
I admit it, I love a good Bob Ward (w|t) or Bradley Balls (w|t) 500 level session on deep SQL Server internals, but then that’s me!
For most people a good level 200 session on a topic is a great introduction to that topic, pushing into 300 for someone who wants to move on to the next level. So don’t worry if you’re not breaking out the debugger or tracing into dll calls
4) – I’m not an MVP or other high end consultancy title
Neither are most of us doing the speaking. Don’t let that hold you back. You think they got those titles before they started speaking? It’s putting yourself out there that get’s you noticed.
5) – I don’t have enough content
You’ll be surprised how easy it is to fill up 50 minutes with content. And that’s without questions, once they come into the picture you’ll find yourself accelerating to get everything in. Demos always take longer than you plan as well, seriously, never underestimate how long demo can take in front of an audience!
6) – I’ve not done it before
We all start somewhere (Birmingham SQL User group for me many years ago), local user groups are good as you’ll have friendly faces around. If you want to dip a toe in the water then keep an eye out for events offering a shorter quicker intro, for example lightning talks of 10-12 minutes for you to have a go with, or there’s webinars, so you can present from the security of home.
7) – Don’t be afraid of questions, or you answers (An addition suggested by Rob Sewell (w|t))
Yes, people will ask questions. But don’t be scared of them. I’ve yet to see someone throw in a question explicitly to be nasty to a presenter. Most of the questions will be because someone’s not quite followed what you’re saying so repeat yourself and see how that goes. If you get a question you really can’t answer, you can’t answer in a reasonable amount of time, or is going to lose the rest of your audience you can always arrange to take it afterwards or give them your contact details and discuss it offline.
So there’s nothing insurmountable there. All group leaders and organisers want to see new speakers, so don’t be afraid to ask for help. We’ll happily let you know of any topic requests we’ve had from our members, or give you feedback on your topic. They’re also happy to go through your presentation with you before the big day to make sure it’s going.
Post up below if there’s anything else you’re worrying about. And if there isn’t, go and start writing that presentation
The SQLAutoRestores module as currently stands (04/08/2016 v 0.9.0.0) is very much based on my personal usage patterns, but the plan is to make it as flexible as possible. So if you see something in this workflow that you’d like to change, or need something to cope with your SQL Sever backup procedures, then please leave a comment below, or drop in a feature request at github – https://github.com/Stuart-Moore/SQLAutoRestores/issues
Current practice at work is that all SQL Server databases are backed up, either to a central share, or locally and then transferred to the central share. So we end up with a backup folder structure like:
I want to randomly pick a set of backup files to restore. So the first thing I want to do is to get all the ‘Bottom folders’ of this directory tree:
$folders = Get-BottomFolders \\backupserver\share
And if I’ve split the backups across multiple shares I can add more:
$folders += Get-BottomFolders \\server2\backups$\
So now I want to pick a folder at random from those. But, not everyone cleans up after themselves, so there could be folders that don’t contain anything we’re interested in. So there’s a Test-DBBackupsExist function that makes sure we have a file. So we loop until we get something useful:
Great, we’ve got a folder with some SQL Server backups in it. Now we need to see what’s in the backup files. So we do a scan of the file headers, which needs a SQL Server, so we build a SQL Server connection as well:
This returns a simple PowerShell Object containing the header hightlights from each file in the folder.
Note; at this point we’ve not checked we’ve gotten a complete restorable set of files. For all we know, we got 30 Transaction log files and no Full Backup to start from!
I prefer to restore databases to random points in time rather than just the latest available. This gives a wider range of options to compare, and might just mean that you’ll discover than your SAN is corrupting the 22:15 t-log backup.
The next function checks we’ve got at least one ‘anchoring’ full backup, picks the earliest point in time that backup covers, and then gets the latest point in time covered by the backup files, and returns a random point between those 2 extremes. This will be our Recovery Point Objective
Then we check we have enough space to restore. This includes checking for any file growth during the restore (if your transaction log grows to a stupid size during the day, then it’ll be grown to that size during the restore and sized down later on, so you need to accomdate the largest amount of space your database occupies, not just it’s final size):
And then we test the difference beteen the SQL Server version of the instance that did the backup and the SQL Server instance we’re asking to perform the restore. Microsoft state that restoring more that 2 major versions isn’t allowed, so we fail it in this case (non SQL Server backups aren’t supported (yet!))
Rinse, repeat ad infinitum. I’ve this process running 24×7 on a dedicated restore instance. On average I restore 80 databases a day and cover every production database in a 2 week windows (it’s random so not guaranteed, but I have a priority list that skews it!)
Currently I collect my statistics with some simple Send-MailMessage usage, but I want something more robust in this module, so thats on the list of things to get fixed before we go to 1.0.0.0 properly.
Hopefully that’s given some ideas on how to use the module. I’d love to hear any ideas on improvements or how you’d want to use it in your environment. Comment here, drop me an email, or ping me on twitter (accounts all linked top right).