How to Measure the DBA’s Effectiveness?

Every now and then when I am talking to clients (and database professionals of all types) somebody raises the age old question of how to measure the effectiveness and quality of their DBA staff. Now I’ve tackled a related question on this blog a little bit ago, which is How Many DBAs?, but it can be a difficult question to effectively determine the appropriate staffing level for a DBA group. Basically, it boils down to the techies usually thinking that more DBAs are needed, and management saying that there are already enough (or, even worse, too many) DBAs on staff.

Actually, it always seems like the answer to “How many DBAs do we need?” is always “One more!” no matter how many are on staff… but I digress.

So let’s get back to today’s entry, in which we look at what type of metrics are useful for measuring the DBA’s quality of work. So, what is a good way to manage how effective your DBA group is?

This is not a very easy question to answer because a DBA has to be a Jack of all Trades… and to quote a retired friend of mine (Hello Roger!), “DBAs should be a Jack of all Trades and a master of some.” The important consideration here is that each of these “Trades” can have multiple metrics for measuring success. For example, a metric suggested by some DBAs is to measure the number of SQL statements that are processed successfully. But what does “successfully” mean? Does it mean simply that the statement returned the correct results, or does it mean it returned the correct results in a reasonable time? And what is a “reasonable” time? Two seconds? One minute? A half hour? Unless you have established service level agreements it is unfair to measure the DBA on response time. And the DBA must participate in establishing reasonable SLAs (in terms of cost and response time) lest he be handed a task that cannot be achieved. Of course, service levels are not just a good idea for SQL performance, you should develop SLAs for change management, recovery, and so on.

Measuring the number of incidence reports is another oft-cited potential metric. Well, this is fine if it is limited to only true problems that might have been caused by the DBA. But not all database problems are legitimately under the control of the DBA. Should the DBA be held accountable for bugs in the DBMS (caused by the DBMS vendor); or for design elements forced on him or her by an overzealous development team (happens all the time these days with agile development and e-rushing around).

I like the idea of using an availability metric, but it should be tempered against your specific environment and your organization’s up-time requirements. In other words, what is the availability required? Once again, back to SLAs. And the DBA should not be judged harshly for not achieving availability if the DBMS does not deliver the possibility of availability (e.g. online reorg and change management) or the organization does not purchase reasonable availability solutions from a third party vendor. Many times the DBA is hired well after the DBMS has been selected. Should the DBA be held accountable for deficiencies in the DBMS itself if he or she had no input at all into the DBMS purchase decision?

And what about those DBA tools that can turn downtime into up-time and ease administrative tasks? Well, most DBAs want all of these tools they can get their hands on. But if the organization has no (or little) budget, then the tools will not be bought. And should the DBA be held responsible for downtime when s/he is not given the proper tools to manage the problem?

OK then, what about a metric based on response to problems? This metric would not necessarily mean that the problem was resolved, but that the DBA has responded to the “complaining” entity and is working on a resolution. Such a metric would lean toward treating database administration as a service or help desk type of function. This sounds more reasonable, at least from the perspective of the DBA, but I actually think this is much too narrow a metric for measuring DBAs.

Any fair DBA evaluation metric must be developed with an understanding of the environment in which the DBA works. This requires in-depth analysis of things like:

• number of applications that must be supported,
• number of databases and size of those databases,
• number of database servers,
• use of the databases (OLTP, OLAP, web-enabled, data mining, ad hoc, etc.),
• number of different DBMSs (that is, Oracle, DB2, Sybase, MySQL, IMS, etc.),
• number of OS platforms to be supported (Windows 2000, UNIX, OS/390, AS/400, etc.),
• special consideration for ERP applications due to their non-standard DBMS usage,
• number of users and number of concurrent users,
• type of Service Level Agreements in effect or planned,
• availability required (24/7 or something less),
• the impact of database downtime on the business ($$$),
• performance requirements (subsecond or longer – gets back to the SLA issue),
• type of applications (mission critical vs. non-mission critical),
• frequency of change requests.

This is probably an incomplete list, but it accurately represents the complexity and challenges faced by DBAs on a daily basis. Of course, the best way to measure DBA effectiveness is to judge the quality of all the tasks that they perform. But many aspects of such measurement will be subjective. Keep in mind that a DBA performs many tasks to ensure that the organization’s data and databases are useful, useable, available, and correct. These tasks include data modeling, logical and physical database design, database change management, performance monitoring and tuning, assuring availability, authorizing security, backup and recovery, ensuring data integrity, and, really, anything that interfaces with the company’s databases. Developing a consistent metric for measuring these tasks in a non-subjective way is challenging.

You’ll probably need to come up with a complex formula of all of the above — and more — to do the job correctly. Which is probably why I’ve never seen a fair, non-subjective, metric-based measurement program put together for DBAs. If any of my readers have a measurement program that they think works well, I’d like to hear the details of the program — and how it has been accepted by the DBA group and management.