Automating Perfmon with Perfcollect

[ Post by Paul Galjan ]

When I started here at EMC, I was pleased to see that most of us would use actual host data (perfmon) to perform size out our storage.  We have a variety of cool tools that will analyze perfmon output and help visualize trends, size out replication bandwidth required and a whole lot more.

But performance data gives you only half of what you need in order to size storage.  You need the capacity of the disks in order to do a complete sizing.  This resulted in a fair number of conversations that went like this:

Sales Rep: Hey Paul, did you get that perfmon data from the customer?
Paul: Sure did.  Based on the performance data, they’ll need about 12 15k disks for the database, and 4 for the logs.
Sales Rep: So sixteen 146G 15k disks?
Paul: Well, I’m not sure; I don’t have the capacity information.
Sales Rep: But you looked at the performance data.
Paul: Yes.
Sales Rep: And based on that, it looks like they’ll need sixteen disks to address the performance requirement.
Paul: Yes

Sales Rep:  So I can go ahead and quote 16 146G 15k disks, right?

You only need to have that conversation about three dozen times before you realize that something must be done.

So I wrote a tool called perfcollect.  It runs on Windows 2003 and later, tested on x86, x64, and even ia64.  Once I started writing it, I figured out that I could solve a lot more problems than I actually set out to solve.

First, I decided not to limit the counter to just storage.  The tool collects a wide variety of counters related to CPU, memory and even the application context.  It will collect up to 350 counters, based on the XML profiles from the very cool PAL tool. The counters include all sorts of stuff relevant to Exchange, SQL Server, SharePoint, AD, Hyper-V and more.

Second, it collects configuration information that is available only by doing WMI queries on the server, but is nonetheless still relevant to performance troubleshooting.

Operation of the tool is very simple.  You download the tool from my own blog site, and run it as administrator on your server (it automatically escalates privileges if you’re running it on 2008, Vista, or 7).  Select the duration of the collection, the sample frequency, hit enter and let it go.  Come back and look in c:\perflogs\EMC, and you’ll see a directory tree of text files and csv’s.

Here’s the progress of what perfcollect actually does:

  • Detects the version of Windows running.  If it’s running 2000 or earlier, it exits
  • Presents the UI portion, where you select the duration and frequency of samples
  • Detects all available counters on the system
  • Builds a list of relevant “interesting” counters based on what is available
  • Builds a list of services running on the machine
  • Gets boot options of the machine
  • Builds a list of applications installed on the machine
  • Builds a list of disks on the system and their capacity information – outputs in CSV and human-friendly text formats
  • Dumps event logs of error and above to CSV
  • Builds a list of disks on the system and their offsets
  • Gets network configuration information
  • Enumerates hardware on the system – processors and types, disks, SCSI and iSCSI adapters, tape drives, and media changers
  • Executes “systeminfo”
  • Executes “driverquery”
  • Consolidates information relevant to PAL (Number of processors, boot options, system type, and memory)
  • Starts the perfmon collection – output in CSV format.

The whole process usually takes less than a minute – excluding the time it takes to actually sample the data, of course.

Once you’re done, the perfmon CSVs are ready for use with any tool you use to manipulate and visualize perfmon data; PAL, perfmon itself, excel, etc.  If you’re worried about the size, I’ve never seen an uncompressed perfmon file generated by perfcollect of over 40MB.

The tool “belongs” to EMC (in that I used EMC’s money to feed my family while I was developing it, and I tested it in EMC’s incredible labs).  But it’s free of charge to use, and the output is yours.  If you use it to collect data just to get a baseline of your servers’ performance, or troubleshoot a problem, we’re cool with that.

You can get more information in the Perfcollect README file.

License
The software is licensed “as-is.” The contributors give no express warranties, guarantees or conditions. You may have additional consumer rights under your local laws which this license cannot change. To the extent permitted under your local laws, the contributors exclude the implied warranties of merchantability, fitness for a particular purpose and non-infringement.
The Microsoft Corporation tools are packaged herein through permission granted by Microsoft Corporation through the premier contract with EMC.  Grep, gawk, and printf are distributed unmodified under the GNU General Public License.

Notice of Copyright
This program is the confidential unpublished intellectual property of EMC Corporation.  It includes without limitation exclusive copyright and trade secret rights of EMC throughout the world.

6 thoughts on “Automating Perfmon with Perfcollect

  1. vTrooper

    This is a great tool! I can’t tell you how much of a benefit it is to have a consistant collection of stats from a customer or a family member that helps you troubleshoot an issue of performance.

    Thanks for putting this one out there Paul. I’m glad you kept your family fed.

    vTrooper

    Reply
  2. Pingback: Proven Professional Roundup Week Ending July 16, 2010 | Adventures in Corporate Education

    1. pgaljan Post author

      Hi Brian – Thanks for pointing that out. I’ve used logwiz, and I think it’s a great tool. Perfcollect is a bit of a different use case. The selection of the counters is completely automated (they collect what the PAL tool wants for analysis), so there’s really very little for the user to do with perfcollect. It’s ideal for our use because each collection on a particular type of server is identical, and we’re sure that what we need to do our sizing and high level troubleshooting (in order to determine issue attribution) is in there.

      Perfwiz is an excellent tool if you have specific counters to collect that perfcollect doesn’t gather, or if you want remote collection.

      Reply
  3. karabela33

    Hi Paul, its a very nice tool to use. i have a question about the following. youre saying that perfcollect can collect 350 counters based on the xml profiles of PAL. how can i run perfcollect based on such a profile. do i have to add those manually after running perfcollect? the possibility to add different counters to the tool would be nice. if its already possible, can you help me out?

    Karabela

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s