And now.

Welcome to Log Buffer #120. My name is Warner, and I’m a SQL Server DBA at The Pythian Group. This is my first time on Log Buffer duties ever, so here’s hoping I can give everyone a fair and unbiased look at this week in the database blogging world (and related).

I admit I had no idea of the community or state of the PostgreSQL RDBMS, and so I definitely learned some new stuff this week. First off, over on “The Scale-out Blog” Robert Hodges invites us all to get our shoephone and get smart about the new world of PostgreSQL replication.

Moving over to “ad’s corner”, Andreas Scherbaum gives us a glimpse of the glitz and glam of PGDay opening. Then he entices us with the title, “Party in the evening” just to horrify us by revealing that—you’d better sit down—by the end of the event half of the beer was still untouched. Next time Andreas, who you gonna call?

“Everything is a DNS problem,” is my new voicemail message, and also the title for Kris Buytaert’s blog, where we go deep into uncovering once again the 10 month-old enigma of why did Sun buy MySQL.

Next up, I will speak to you, Mr. (or Ms.) Cross-Platform DBA. You think you know all of Oracle’s exp command-line switches? All of SQL Server’s bcp formats? And all of the things you can do with . . .  whatever you use with DB2? Well, I bet you can still learn something from Eric Bergen and the different locking behaviors you can get out of mysqldump.

I take my hat off to Schlomo Priymak for contributing to our dream of cross-platform love by helping us uninstall before upgrading the MySQL Connector for .NET. Because both platforms can benefit from each other and people like him keep our dream strong.

On Sunday, I was data charmed by Giuseppe Maxia with this warning post about handling MySQL relay slaves when changing the replication format. After reading that I had to put on my 3-D glasses, having discovered Julian Cash’s Photo Gallery of MySQL DBAs including some very psychedelic shots of Pythian’s very own Sheeri K. Cabral.

Let’s jump right over to Oracle. Pete Finnigan’s Oracle security weblog pointed out a new paper going by the incendiary title “Assault on Oracle PL/SQL – Injection” by a hacker called AELPHAEIS MANGARAE. N.B.: If your kids are the kind that like to read DB security documentation, be aware that there are some four letter words on the document.

If you are a Tom Kyte Fan (look who I found on the front page, none other than Mr. Don Seiler) then you have to head over right now to the Oracle Scratchpad where Jonathan Lewis has posted the dates for Tom’s events in Europe.

Moving to current world trends, Gints Plivna shows us how to calculate our mortgages using the Oracle MODEL clause. Time is running out, in case you haven’t noticed, and there are Only Four Left…. Or so says John Piwowar, showing us how to run oidadmin on Mac.

Now, join me where the air is warm and the food is plentiful.

I call all my fellow SQL Server DBAs to gather for a road trip to Redmond, Boston, Seattle, and Montreal, as we follow Adam Machanic’s trail through the world of SQL Server conferencing. Adam, how about we pitch the reality show to Microsoft? (Suggestions for the show’s title on the Comments section, thank you).

I’m bringing myself up-to-date on SQL 2008 and Tibor Karaszi gave me a nice heads-up on what to expect with resource governor behavior on multiple CPUs. And just so we don’t forget to keep working on the fundamentals, Andrew Calvett gives us a clear look at the overhead of non-unique clustered indexes with the help of Danny Gould’s really cool Internals Viewer for SQL Server.

SQL injection is running rampant on all platforms, so over on SSQA.net, SQL Master (yes, that’s his real name) recommends us some tools to help secure your SQL Server environment to plug all those holes before it’s too late.

There are few things more embarrassing for a DBA than getting locked out of their own databases. If that ever happens to you though, don’t say you were not warned by Mohammed Mawla as he shows us how to properly use logon triggers or save some face with the SQL 2005 DAC connection.

And in case you are on the fence about upgrading to SQL Server 2008, Chris Miller will convince you with the reasons why he is loving the new SQL Server 2008 Management Studio: “ . . . it actually does suck 85% less than SQL Server Management Studio 2005″. Nuff said!

Okay, I’m winding down now, but I cannot go before letting you know that you might want to break the piggy bank for the $1.4 million Leon Katsnelson tells us you will need to recreate IBM’s DB2 TPC-C Benchmark on an x86-64 PC server.

And last but not least, I would like to close it out with two database-agnostic posts.

First Ken Downs, the Database Programmer, goes into our everlasting philosophical dilemma of normalization and denormalization. Second, Craig Mullins gives some food for thought in the form if his DBA rules of thumb. My personal favorite is #4: “Don’t Panic”.

I hope you enjoyed this 120th episode (that’s as long as The Muppet Show’s original run and the 1960’s Batman!). Until next time.

select * from sys.dm_db_mirroring_auto_page_repair

It looks like grouping by db or file id would be useful to zoom in on possible disk issues causing the page errors. Beware—only 100 rows are kept for any database, so if you want to keep a full history you should move those records on a scheduled basis.

select * from sys.dm_db_mirroring_past_actions

This one could be useful for monitoring when there is no witness on a mirroring setup; or for creating a mirroring history report.

select * from sys.dm_db_persisted_sku_features

This one should be used as a warning sign before moving databases between editions, specifically between Enterprise and Standard. Paul Randal has a great post on it: SQL Server 2008: Does my database contain Enterprise-only features?.

select * from sys.dm_db_script_level

The only thing I could find on this was this KB article: Error message when you try to run a query in SQL Server 2008: “Msg 916, Level 14, State 1, Line 1″, but no Books Online official page. It looks like some change-tracking internal view for hotfixing and service packing purposes. Oh well, it’s good to know what’s in your SQL anyway.

select * from sys.dm_exec_procedure_stats

This one is very similar to dm_exec_query_stats but on the Stored Procedure level instead of the statement level. It’s useful if you are interested only in stored procedures as a whole, or if you want to filter ad hoc SQL batches from your analysis.

select * from sys.dm_exec_trigger_stats

This is very nice, as trigger-related issues are a somewhat common cause of performance hiccups on client environments.

select * from sys.dm_os_memory_brokers

This is the DMV form of information that was returned in previous versions by DBCC MEMORYSTATUS. It is useful for identifying and diagnosing possible memory pressure issues with regular T-SQL querying. It was a real pain to get it with the DBCC MEMORYSTATUS output.

select * from sys.dm_os_memory_node_access_stats

This one is not documented in Books Online. It looks like read and write stats for different page types, aggregated by memory nodes. I wonder if internal memory pressure can be analyzed or diagnosed with some help from this view.

select * from sys.dm_os_memory_nodes

Books Online says, “Nodes are created per physical NUMA memory nodes.” Again, this is taken from the DBCC MEMORYSTATUS output.

select * from sys.dm_os_nodes

Tied to the SQL OS architecture*, this view has some interesting stats like active_worker_count and avg_load_balance. I will definitely be running some tests and checking the output of this.

select * from sys.dm_os_process_memory

General memory stats from the server, useful for SQL-based monitoring on columns such as process_physical_memory_low that could alert after N minutes have passed on low memory warning. Also for baselining with the page_fault_count. This is an accumulated value though, so the differences should be measured and stored on a separate monitoring table.

select * from sys.dm_os_sys_memory

These are system-wide memory statistics, including columns that show if the system is on high- or low-memory overall. Definitely useful for monitoring and correlating with SQL memory statistics on non-SQL Server exclusive machines (Which should not exist. But the ideal world and the real one are so different, aren’t they?)

select * from sys.dm_sql_referenced_entities
select * from sys.dm_sql_referencing_entities

These are both tools for tracking dependencies, covered in depth by Aaron Bertrand on Keeping sysdepends up to date in SQL Server 2008.

Now, I’m obviously just scratching the surface. There are many other new DMVs I haven’t had the chance to look at, many of them related to specific features like Full Text Search, Change Data Capture, FILESTREAM, Auditing, Extended Events (definitely a topic for upcoming posts) and Resource Governor.

If any of those interest you, take the time to browse around Books Online and explore them on your server (a trial version is out for free download!). Denis Gobo has a handy query to get all the names of the new DMVs. Start looking around and get ready for the upgrades to 2008 coming your way.

* Required reading for all SQL Server DBAs: A new platform layer in SQL Server 2005 to exploit new hardware capabilities and their trends. back

Blocking chain of 200 procedures, your server is crawling with high CPU and requests are coming out a funnel. Now, don’t worry, the waitresource column provides us with the information to start zooming in on our problem. In this case, we have value “TAB: 7:357576312 [[COMPILE]]”. Disregarding the fact that the resource description says TAB, run the following query with the resource information to get the database and procedure being locked:

select db_name(7)

Use [PerformanceReports] -- the database name we got from the select above
GO

select object_name(357576312)

Knowing the procedure, it’s then a matter of discarding possibilities. The common causes of COMPILE locking are well documented on the Microsoft KB, “Description of SQL Server blocking caused by compile locks”, so if you have a chance, go on and read that. If you don’t, then this is the gist of it:

1. Ensure all references to stored procedures in your code are owner-qualified. Do not call ‘exec mysp’, call ‘exec john.mysp’. If you don’t, then you can get COMPILE locking and a Cache Miss.
2. Do not begin your stored procedures’ names with ’sp_’. This makes the engine go looking into the master database for the stored procedure, and will get you a Cache Miss (or godforbid, someone created a procedure in master with the same name and you are actually executing that). There are many options: ‘usp_’, ‘appsp_’, etc. Whatever works for you.
3. If you do owner-qualify your call, but you execute the procedure with a different case than what it was created as, then you could get a Cache Miss or a COMPILE lock. I tested this and got Cache Hits, but why take the chance? Use the same case.

   The last one is not included in the KB; I found out about it when working this specific case:

4. Do not open and close an encryption key inside a stored procedure in order to use encryption routines that will be called concurrently in heavy volume. This will get you Cache Miss and COMPILE locks too.

I am talking about something like this:

CREATE PROCEDURE dbo.LookupByID @ValueID int
AS

SET NOCOUNT ON;

OPEN SYMMETRIC KEY PythianSymmetricKey DECRYPTION BY ASYMMETRIC KEY PythianAssymKey

SELECT CAST(DecryptByKey(encryptedField) AS VARCHAR(30)) FROM EncryptedValues
where ID = @ValueID

CLOSE SYMMETRIC KEY PythianSymmetricKey

If you don’t follow the points above, will you get the blocking chain I showed before? Not necessarily — if the server is fast enough, it will acquire and release those compile locks before you notice anything. As your workload increases, however, the issue will start coming to the surface until it materializes as a full-blown blocking chain.

Here is a detailed look through Profiler. I traced Lock, Batch, and Cache Hit events, and removed from the output the events that were not relevant to this problem. This is the output from the call with no owner specified and our encryption OPEN and CLOSE statements inside the procedure:

As you can see, we are getting an initial Cache Miss that corresponds to just the actual call ‘exec LookupByID 7′. After that, we get the real Cache Miss from the procedure, the Lock Acquired event with COMPILE type and locking for both encryption keys.

Now, let’s qualify the owner, take out the OPEN and CLOSE SYMMETRIC KEY instructions from the procedure, and just leave the call to DecryptByKey inside:

At first glance, it looks like the same thing. Sure, there is no more encryption key locking, but compile locks are being acquired, and our plan is not getting a Cache Hit. But now we got the Cache Insert fired. And that means that our next run of the procedure looks like this:

And that is ideally how it should look all the time: no compilation locking, cache hit and only the shared lock to the table we are querying.

So remember, follow the known Microsoft recommendations, OPEN and CLOSE the key once and you can execute as many times as you want the stored procedure calling DecryptByKey (one approach for this can be found in A Simple Approach to SQL Server 2005 Encryption by Mike Good). Not only will you avoid the COMPILE locks, but also CPU overhead. Back to work.