The highlight of the meeting was an interesting presentation on security by the founder of OVSAGE, Scott Murphy. The focus was on the fact that security is a mindset, not a product. Scott’s presentation looked at a large number of security issues and explained in detail while technology alone cannot fix security issues. The presentation was a response to the Amrit Williams Blog post Top 10 Reasons Your Security Program Sucks and Why You Can’t Do Anything About It.

Scott’s presentation can be viewed here: security-quagmire-pdf. I hope you find it useful.

It’s been another busy week here and shows no sign of slowing, but here’s a few of the things we found interesting this week.

Operating Systems

The H Online is reporting that Linus Torvalds named one of the 100 most influential inventors by “The Britannica Guide to the World’s Most Influential People”. More info can be found there, as well as a link to some free sample pages from the book. Of course, Bill Gates was also on the list.

Could it be that Microsoft ranks third in Linux sales? As part of their 2006 agreement with Novell, they were required to purchase $240 million worth of “subscription certificates” to sell or distribute. Microsoft says that so far 475 of the coupons have been used which is equivalent to about a half million dollars worth. See Sold out: Microsoft’s Linux business is booming at The H Online for the skinny, and Microsoft exhausts coupons for SuSE Linux at SD Times for a much more detailed analysis.

Ubuntu has released Ubuntu 8.04.4, the fourth patchset for “Hardy Heron”, which is a long term support release. See the release notes for the full list of updates. Alternately, schedule a change window and apt-get update and apt-get upgrade today!

Did anyone realize that SCO vs. Novell was still a going concern? Well, apparently it is. According to Groklaw SCO and Novell File Proposed Witness Lists. Again. If you enjoy a good licensing battle, I don’t think this qualifies, but you can decide for yourself. Why can’t they both just get along?

Internet

According to the political news site C-Span’s page for Solving Video Problems, it will be discontinuing support for Real Networks RealVideo streaming format citing as the cause lack of interest. They do, however, still support IE6 unlike . . . 

The official Google Enterprise Blog has announced that as of March 1^st, site functionality may no longer support IE6. New features for Google apps will be built without support for IE6 as well. Check out the full deal, called Modern browsers for modern applications.

Apache HTTP Server 1.3.42 Released by the Apache foundation spells EOL (end of life) for the popular web server, though some security patches may still be released. I know of a lot of Oracle app servers out there running apache 1.3, so perhaps one of our crack DBAs can chime in with some details on how to handle it, what versions support 2.0, etc.

The popular Squid proxy server released a patch this week for a vulnerability to specially crafted DNS packets which can trigger a queue overflow and eventually a DOS. If you make heavy use of Squid, check out the creatively named Squid Proxy Cache Security Update Advisory SQUID-2010:1.

We have all heard by now of the Google attack from China. Now Google is seeking the help of the NSA to prevent future attacks of this sort. Ars Technica has the full scoop, see In wake of hack, Google negotiating cooperation with the NSA.

Virtualization

VMWare has announced security flaws in a number of its products. Patches have yet to be released, so if you run VMWare, keep your eye out for them. The full info and affected products can be found in the announcement from the VMWare Security Announce mailing list, see [Security-announce] VMSA-2010-0002 VMware vCenter update release addresses multiple security issues in Java JRE.

Cloud

Are you running services in the “cloud”? If so, you’ll need to monitor them. Data Center Knowledge has a roundup of the current players in the cloud-monitoring game. Check out Cloud Monitoring Services: A Resource Guide.

That’s all the time we have for this week, folks. I know there have been other interesting things going on over the last two weeks, so please add them in the comments. Have a wonderful week!

PostgreSQL

David Fetter shares his recipe for adding only new rows: “Let’s say you have a table and a data set, and would like to add only those rows in your data set that aren’t already in the table. There are hard ways, but here’s an easy one.”

Simon Riggs, the Database Explorer, offers his thoughts on parallel query in Postgres: “I’m disappointed we’ve not made much progress with parallel operations and partitioning in core Postgres in last few releases. Recent Greenplum results show we have much work to do in improving things.”

David Christensen shares a PostgreSQL tip: using pg_dump to extract a single function.

Roppert Kalmar shares a screenshot of the new Sun Oracle PostgreSQL. That doesn’t even sound right, does it?

Oracle

Let’s begin with Chen Shapira, since she is so excited about the NoCOUG Winter Conference, and here gives a quick rundown of what attendees can expect to see.

Contrariwise (or not), Iggy Fernandez offers five reasons not to attend a NoCOUG conference in 2010. “Most Oracle professionals will benefit a lot from attending a NoCOUG conference in 2010. However, the following categories will not benefit much:  . . .  Those Oracle professionals who believe that Oracle’s goal in buying Sun is to replace Oracle Database with MySQL. This is probably a very small group . . . ”

Uwe Hesse, the Oracle Instructor and his readers share a discussion on sharing READ ONLY Tablespaces between databases. Uwe says, “ . . . the question was raised, whether it is possible to use the same READ ONLY Tablespace in multiple Databases. At first glance, I thought that this should of course be possible, though the answers where somewhat discouraging. So I have done a quick test to prove it . . . ”

Krishna Manoharan of Performance Engineering and Capacity Planning steps up with an item on understanding CPU time as an Oracle Wait event. Krishna pursues the question, “[What] if the stats from the system show that CPU Utilization (% Util and Run queue) are well within thresholds and show plenty of available capacity, but Oracle continues to report CPU time as a Top 5 wait event?”

Charles Hooper grapples the question, Which Plan is Better? Charles writes, “A recent post appeared in the OTN forums that indirectly asked the question: which execution plan is better?  . . .  If you are attempting to conclude which plan is faster/better based on the estimates in [a] first plan and an altered plan with a hinted cardinality estimate, you might be setting yourself up for failure.”

“Seems like all I ever write about these days is SQL Profiles,” writes Kerry Osborne. “I do other stuff, honest! It just seems like getting Oracle to do what you want when you can’t touch the code is the closest thing to ‘Magic’ that DBAs get to do.” Here’s Kerry’s post on single-hint SQL profiles, inspired by a discussion with Jonathan Lewis.

And now here is Jonathan with a post on SQL Server.  . . .  Wait a minute–what the?!

SQL Server

Ahem. And now here is Jonathan Lewis with a post on SQL Server. Yes, Jonathan Lewis, famous Oracle guy. “A few days ago,” Jonathan writes, “I did a presentation on SQL Server.  . . .  The title was ‘What the Enterprise needs in an RDBMS’ . . . and the presentation was about whether or not you could find everything you needed in SQL Server 2008, where you’d have to look in the manuals, and supplementary questions you’d have to ask.”

Buck Woody also has been, as it were, treading the boards: “I give series of classes and presentations on Data Design. I say ‘data’ design instead of ‘database’ design because we should consider more than just the database.  . . .  Here are the links I use in that presentation. Although this isn’t a comprehensive list of Data Design topics, I’ll visit this topic from time to time so you may want to bookmark this page in your favorites[.]”

Merrill Aldrich admonishes, don’t get burned by replication of SQL Server files: “ . . . if you try to use file system replication (robocopy, xcopy, repli-whatever) to maintain a DR server from your production SQL Server, you might be in for a nasty surprise.”

Buck Woody relays more nastiness in his post, Transparent Data Encryption and the Latest Data Breach: “Well, It’s happened again. Hundreds of thousands of private records were stolen from a database. This one, however, was different. No one stole any passwords, no one did any social engineering, nothing was captured in-line. No, this one was accomplished by stealing the actual hard drives themselves!”

One way to circumvent this–no hard drives! But wait–before you send them all to the kilns, Aaron Bertrand says, your laptop may be ready for SSDs, but are your SQL Servers? “I am not trying to be Debbie Downer here,” he explains,” … SSDs sound great . . .  But right now, if you are looking at expanding or upgrading your I/O under SQL Server, I’d give the vendors some time to shake off these early jitters.”

Here’s Thomas LaRock, DBA Survivor, with a Name That Caption Contest. The deal is: “The person who provides the best caption will win a copy of my book,” DBA Survivor, “and I will figure out a way to incorporate your caption into the main page.”

MySQL

On Serge’s blog appears a useful item on using UNIX_TIMESTAMP as a partitioning function for MySQL 5.1.43

Falko Timme shares How To Back Up MySQL Databases With mylvmbackup On Debian Lenny on HowtoForge.

Simon Mudd has some further thoughts on mysql upgrades.

From atop his MySQL Soapbox, Lachlan Mulcahy address the matter of a replicate-do-db gotcha, discovered lurking in the config file.

Gotcha again. Dathan Vance Pattishall says, “INNODB has some irritating gotchas that makes disk space management hard. In 2002ish INNODB, added innodb_file_per_table to get around allot of these issues, but it does not fix everything.” The post is innodb_file_per_table, shrinking table spaces and the data dictionary.

On EXPLAIN EXTENDED, Quassnoi examines some details of join on overlapping date ranges in answering the question, “Is there any way to optimize the query for overlapping ranges in MySQL if both ranges are dynamic?”

Last, The InfiniDB Team Blog announces that InfiniDB 1.0 is Now Available!

That’s all there’s room and time for. Let’s hear your favourite DB blogs from this week, and we’ll see each other again in the next one. Till then!

1. direct load/SQL load
2. direct-path inserts result from insert or merge statement
3. ALTER TABLE commands
4. CREATE and ALTER INDEX commands
5. INSERT /*+APPEND*/
6. partition manipulation
7. database object that has explicitly set with nologging option
8. Oracle eBusiness Suite concurrent job execution identified in Oracle metalink note: 216211.1
9. Oracle eBusiness Suite patches activities that involve database object manipulation

The database recovery operations will look completed, but those data blocks used by the nologging objects in the data file will be marked corrupted when they are recovered. Accessing those nologging data objects in the recovered database instance will return a data block reading error such as ORA-1578 and ORA-26040, and the logical corruption in the data file will prevent the database object from being useful in the recovered database instance.

How do we detect unrecoverable operations?

Unrecoverable data files are those that involve nologging operations since the last successful backup took place. There are several ways to identify them. You can locate those data files either with RMAN or by querying V$ tables.

A. Retrieve unrecoverable data file information from RMAN

RMAN> report unrecoverable database;

Report of files that need backup due to unrecoverable operations
File Type of Backup Required Name
---- ----------------------- -----------------------------------

14   full or incremental     +DGDATA/A1/data file/apps_ts_tx_data.286.662131793

32   full or incremental     +DGDATA/A1/data file/apps_ts_tx_data.326.667991737

33   full or incremental     +DGDATA/A1/data file/apps_ts_tx_data.327.667991741

…

80   full or incremental     +DGDATA/A1/data file/A1_obiee_data.386.682596961

81   full or incremental     +DGDATA/A1/data file/A1_obiee_data.387.682596961

82   full or incremental     +DGDATA/A1/data file/A1_obiee_data.388.684801365

RMAN> exit

B. Retrieve unrecoverable data file information directly from v$ table

SQL> SELECT df.name data file_name, df.unrecoverable_time
FROM v$data file df, v$backup bk
WHERE df.file#=bk.file#
and df.unrecoverable_change#!=0
and df.unrecoverable_time >  (select max(end_time) from v$rman_backup_job_details
* where INPUT_TYPE in ('DB FULL' ,'DB INCR'))

DATAFILE_NAME                                                UNRECOVERABLE_TIME

------------------------------------------------------------ -------------------

+DGDATA/A1/data file/apps_ts_tx_data.286.662131793          2010-01-08:00:13:13

+DGDATA/A1/data file/apps_ts_tx_data.326.667991737          2010-01-08:00:13:13

+DGDATA/A1/data file/apps_ts_tx_data.327.667991741          2010-01-08:00:13:10

…

+DGDATA/A1/data file/A1_obiee_data.386.682596961           2010-01-08:06:18:34

+DGDATA/A1/data file/A1_obiee_data.387.682596961           2010-01-08:06:18:34

+DGDATA/A1/data file/A1_obiee_data.388.684801365           2010-01-08:06:18:34

21 rows selected.

What database objects are affected by the unrecoverable operation?

We can locate the database objects affected by these unrecoverable operations by retrieving these objects with the following script:

select distinct dbo.owner,dbo.object_name, dbo.object_type, dfs.tablespace_name,
dbt.logging table_level_logging, ts.logging tablespace_level_logging
from v$segstat ss, dba_tablespaces ts, dba_objects dbo, dba_tables dbt,
v$data file df, dba_data_files dfs, v$tablespace vts
where ss.statistic_name ='physical writes direct'
and dbo.object_id = ss.obj#
and vts.ts# = ss.ts#
and ts.tablespace_name = vts.name
and ss.value != 0
and df.unrecoverable_change# != 0
and dfs.file_name = df.name
and ts.tablespace_name = dfs.tablespace_name
and dbt.owner = dbo.owner
and dbt.table_name = dbo.object_name

OWNER           OBJECT_NAME                    OBJECT_TYPE         TABLESPACE_NAME  TAB TABLESPAC

--------------- ------------------------------ ------------------- ---------------- --- ---------

APPLSYS         WF_LOCAL_ROLES_STAGE           TABLE               APPS_TS_TX_DATA  YES LOGGING

APPLSYS         WF_LOCAL_USER_ROLES_STAGE      TABLE               APPS_TS_TX_DATA  NO  LOGGING

APPLSYS         WF_UR_ASSIGNMENTS_STAGE        TABLE               APPS_TS_TX_DATA  YES LOGGING

APPS            DR$IBE_CT_IMEDIA_SEARCH_IM$I   TABLE               APPS_TS_TX_DATA  YES LOGGING

MSC             MSC_TP_ID_LID                  TABLE               APPS_TS_TX_DATA  YES LOGGING

MSC             MSC_TP_SITE_ID_LID             TABLE               APPS_TS_TX_DATA  YES LOGGING

OBIEE_OWNER     GL_DETAIL                      TABLE               A1_OBIEE_DATA    NO  LOGGING

OBIEE_OWNER     GL_EXPENDITURES                TABLE               A1_OBIEE_DATA    NO  LOGGING

OBIEE_OWNER     A1_DEPARTMENT_DIM              TABLE               A1_OBIEE_DATA    NO  LOGGING

OBIEE_OWNER     A1_FUNCTION_CHANNEL            TABLE               A1_OBIEE_DATA    NO  LOGGING

OBIEE_OWNER     A1_GEOGRAPHY                   TABLE               A1_OBIEE_DATA    NO  LOGGING

OWNER           OBJECT_NAME                    OBJECT_TYPE         TABLESPACE_NAME  TAB TABLESPAC

--------------- ------------------------------ ------------------- ---------------- --- ---------

OBIEE_OWNER     A1_PRODUCT_CLASS               TABLE               A1_OBIEE_DATA    NO  LOGGING

12 rows selected.

Directly knowing the objects or tables affected in the recovery operation due to nologging, can help both us and the data owners to make an informed decision on whether nologging needs to be disabled, based on the transactional or analytical nature of the data in those tables.

How do we fix unrecoverable data files?

One way to force these nologging objects operation to be captured in the redo log, is to implement FORCE_LOGGING, a feature introduced in Oracle 9i to allow Oracle to force logging in redo even if nologging operations are performed. Force logging can be enforced at the database and tablespace level. In Oracle ERP e-Business suite, the default installation of EBS sets the database FORCE_LOGGING option to NO. The intent of this setup is to enhance overall system performance. If this setting is not changed to YES, all “nologging” operations mentioned in this section will suppress the generation of redo log, causing unrecoverable operations issues with all nologging objects.

Before altering the database or tablespace to force logging, it is important to evaluate the benefit of force logging versus those of nologging.  Once force logging at the database level is turned on, redos are generated for all operations, dramatically increasing redo size. In addition, since there is an extra operation of capturing to redo log, performance of data inserts and updates will also be affected.

If a decision has been made to keep the database and tablespace in NO force logging mode, it is imperative for the business owner and database administrator to collectively ensure a good backup is taken immediately prior to any patching activities, direct load, and other nologging operations take place. In addition, DBAs should periodically query database to identify data files that contain nologging activities, and whenever that happens, to perform a database backup to ensure data file recovery integrity. For example, in the case of Oracle ERP e-Business Suite, this would mean a database backup will need to be performed every time prior to any patching activities.

Conclusion

If there are unrecoverable data files identified in the database, it is important for DBAs to provide the  affected non-recoverable nologging objects to the business owner to determine if these objects are of important recovery value. Examples of nologging objects which can be safely ignored includes tables that can easily be generated from source table, or tables designed to be temporary in nature.

It is important for business owners to decide if the performance gain from setting database or tablespace to NO force logging, offsets the recoverability of nologging database objects identified. Should a decision be made to keep the database and tablespace in NO force logging mode–which makes data files unrecoverable–both DBA and business owner need to be vigilant to ensure that database recoverability is achievable by taking more frequent backups.

In this post, we’re going to take a closer look at various queries that the Extract process uses against the database. As before, we will start by examining the strace output:

nanosleep({1, 0}, NULL)                 = 0
...
read(20, "\1\"\0\0\255\1\0\0\217\0\0\0H\200\366\256\5\24\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024000) = 1024000
...
write(16, "\0$\0\0\6\0\0\0\0\0\3N'\7\0\0\0\2\0\0\0`\0\0\0\0\0\0\0\7\0011"..., 36) = 36
read(17, "\0\351\0\0\6\0\0\0\0\0\6\1\"\375\2\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 8208) = 233
write(16, "\0 \0\0\6\0\0\0\0\0\3N(\10\0\0\0\2\0\0\0`\0\0\0\0\0\0\0\7\0011", 32) = 32
read(17, "\0\343\0\0\6\0\0\0\0\0\6\1\"\7\1\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 8208) = 227
write(16, "\0K\0\0\6\0\0\0\0\0\3N)\t\0\0\0\2\0\0\0`\0\0\0\0\0\0\0\7,/"..., 75) = 75
read(17, "\0\341\0\0\6\0\0\0\0\0\6\1\"\375\1\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 8208) = 225
write(16, "\0Q\0\0\6\0\0\0\0\0\3N*\n\0\0\0\2\0\0\0`\0\0\0\0\0\0\0\7,/"..., 81) = 81
read(17, "\0\254\0\0\6\0\0\0\0\0\4\1\0\0\0)\0\1\0\0\0\0{\5\0\0\0\0\n\0\0\0"..., 8208) = 172
lseek(20, 227328, SEEK_SET)             = 227328
nanosleep({1, 0}, NULL)                 = 0

File descriptors 16 and 17 are the pipes for one of the bequeath connections we have with the database. There are four queries being submitted each cycle (following the same order as those being sent to a database):

SELECT DECODE(archived, 'YES', 1, 0), status  FROM v$log WHERE thread# = :ora_thread AND sequence# = :ora_seq_no

SELECT MAX(sequence#)  FROM v$log WHERE thread# = :ora_thread AND status in ('INVALIDATED', 'CURRENT', 'ACTIVE')

SELECT DECODE(status, 'STALE', 1, 0) FROM v$logfile WHERE member = :log_name

SELECT 1  FROM V$LOGFILE WHERE(STATUS NOT IN ('STALE', 'INVALID') OR STATUS IS NULL) AND MEMBER <> :log_name AND EXISTS ( SELECT 1 FROM V$LOG WHERE GROUP#  = V$LOGFILE.GROUP# AND THREAD# = :ora_thread AND SEQUENCE# = :ora_seq_no ) AND ROWNUM = 1

The purpose of these statements is to constantly keep an eye on what’s happening inside the database by regularly polling the contents of the above views. What’s worth mentioning about the above queries is that all of them will cause extra I/O to the controlfile. On my test database, that equaled 640KB each cycle (40 I/O requests, 16KB each). In most cases, this is nothing to worry about–just keep the additional I/O in mind in case your controfile is already a hot spot.

The redo log stores object identifiers (a number), which means that when the Extract process encounters a supported operation, it needs a way to find out more details. This is achieved by a couple of statements against the data dictionary. The following statement will be issued first:

SELECT u.name, o.name, o.dataobj#, o.type#, (SELECT bitand(t.property, 1) FROM sys.tab$ t WHERE t.obj# = :ora_object_id) FROM sys.obj$ o, sys.user$ u WHERE o.obj# = :ora_object_id  AND decode(bitand(o.flags, 128), 128, 'YES', 'NO') = 'NO'  AND o.owner# = u.user# AND decode(bitand(o.flags, 16), 0, 'N', 16, 'Y', 'N') = 'N' AND (o.type# in (1, 19, 20, 34) OR EXISTS (SELECT 'x' FROM sys.tab$ t WHERE t.obj# = :ora_object_id))

In case the object turns out to be a table, it will be checked whether it is an overflow segment for an IOT:

SELECT nvl(iot_name, 'NULL')   FROM all_tables WHERE owner = :owner AND table_name = :object_name

This allows the Extract process to figure out whether it needs to process changes, in case the overflow segment belongs to an IOT from which we’re capturing the data. In case the object in question turns out to be an index, a corresponding check will be made to see whether it’s an underlying index for an IOT:

SELECT table_owner, table_name FROM all_indexes WHERE index_name = :object_name AND         owner = :owner AND index_type = 'IOT - TOP'

This is required so that the changes made to an IOT can be captured, in case it belongs to an interested tables list.

The above queries will be executed regardless of whether or not you’re interested in capturing changes from the particular object, because the queries are required before you can make that decision. In case this is something we’re interested in, additional information will be requested:

select object_type, object_name, subobject_name from dba_objects where object_id = :ora_object_id

The above statement is necessarily in case we’re dealing with the partitioned object and, depending on the result, one of the following two statements will be executed:

select ts.bigfile from dba_tablespaces ts,  all_tables t  where t.table_name = :ora_object_name and  t.tablespace_name = ts.tablespace_name and rownum = 1

select t.bigfile from dba_tablespaces t,  all_tab_partitions p  where p.partition_name = :ora_subobject_name and   p.tablespace_name = t.tablespace_name and rownum=1

There is obviously an issue with the above two statements. Neither of them specify the object owner and, in case you have two (or more) objects with the same name but in different schemas, the above statements may return incorrect information, if these objects are located in different tablespace types.

It’s interesting how the issue is shoved away using rownum = 1 condition. What could the potential impact be? One thought that immediately comes to mind is that the way ROWIDs are organized is different between small- and big-file tablespaces (the part being used for a relative data file number in a small file tablespace is used for a block number in case of a big file tablespace), so some functionality that potentially relies on that could be affected. I’ve got a couple of ideas, but I’ll hold these until I do some testing.

What else is interesting? The big thing is that none of the information on columns is being resolved. All I’m going to say right now is that column information will be fetched by the Replicat process using the destination system’s data dictionary. Combine this with the fact that an online data dictionary does not store historical information about an object’s metadata, and you have a perfect recipe to get yourself into various nasty situations (which is exactly the reason why Oracle Streams relies on MVDD instead of the online data dictionary). But more on that when we get to the Replicat process internals series.

CPAN Wants You!

Want to contribute to your favorite CPAN module, or maybe create your own, but don’t have the foggiest idea how to do it? Here are a few notes, tips, tricks, and links that might help you get started.

PAUSE id

While bringing awesome street cred, having a PAUSE id is strictly necessary only if you want to maintain or co-maintain a module. If you just want to contribute code, you’ll perfectly be able to do without, as it will usually be done via patches submitted to a bug tracking system, a code repository or using good ol’ email.

Becoming a co-maintainer

Becoming the co-maintainer of a module gives you the power to upload authorized releases of the modules on CPAN. To become one, the maintainer of the module simply has to promote you as such on PAUSE.

Creating a new module

You want to create your own module? Excellent! But before you do it, make sure that you . . . 

• you verified on CPAN that there’s not already a module that already scratch that particular itch.
• You made sure that the module uses a descriptive and pertinent namespace.

Once this is done, go ahead: log on PAUSE and register the module.

Adopting an abandoned module

There’s a module apparently abandoned by its owner that you’d love to take over? The procedure for that is given in the CPAN FAQ, and it can be boiled down to:

• Try to get in contact with the current maintainer.
• If you reach him or her, make puppy eyes and ask if you can take over.
• If they are agreeable, they will flip over the maintenance of the module to you on PAUSE, and that’s all there is to it.
• If you tried their email addresses, poked around in mailing lists, forums and other places without any luck, wait a little bit (a couple of weeks) and contact the PAUSE admins.

Module created! How do I upload releases, now?

You upload new versions of a module via the PAUSE interface or via ftp.

Alternatively, and more conveniently, you can also use the cpan-upload script provided by CPAN::Uploader.

Bug Tracking

Contributing to a module means fixing bugs or implementing new enhancements. To find those, who are you gonna call? The Bug Tracker!

rt.cpan.org

By default, every module on CPAN has a bug tracking queue on rt.cpan.org. For example, the one for Git::CPAN::Patch is at http://rt.cpan.org/Public/Dist/Display.html?Name=Git-CPAN-Patch.

Bug reporting can be done via the web interface, or by sending an email to bug-module@rt.cpan.org (e.g., bug-git-cpan-patch@rt.cpan.org).

…or somewhere else

Sometimes, the module’s maintainer uses a different bug tracking system. You’ll typically find it mentioned in the POD, or in the META.yml:

$ curl -L http://search.cpan.org/dist/Git-CPAN-Patch/META.yml | \
  perl -MYAML -0 -E'say Load(<>)->{resources}{bugtracker}'

http://rt.cpan.org/NoAuth/Bugs.html?Dist=Git-CPAN-Patch

If you don’t feel inclined to dig into META.yml for information, there is a Greasemonkey script that will do it for you, and automatically add a link to the bugtracker (along some other goodies) on the module’s CPAN page.

…or here, there and everywhere

What if the bugs are kept on rt.cpan.org, and on the local bug tracking system of Github, and a few other places? You can either follow them manually, or you can get funky and experiment with SD, a peer-to-peer ticket tracking system that can sync with and merge the information of several online bug tracking systems. SD, it goes without saying, can be found on CPAN as App::SD.

Code Repository

Most, but not all, modules have a public code repository somewhere out there. Just like for the bug tracking system look for it in the POD, or in the META.yml. It will be displayed on the module’s cpan page as well.

$ curl -L http://search.cpan.org/dist/Git-CPAN-Patch/META.yml | \
    perl -MYAML -0 -E'say Load(<>)->{resources}{repository}'

git://github.com/yanick/git-cpan-patch.git

<singing name=”Adam West”>Na na na na, Na na na na, GitPAN!</singing>

Thanks to Schwern, all CPAN distributions now have a GitHub repository tracking its releases. They can all be found under the GitHub gitpan account, and follow the pattern http://github.com/gitpan/Git-CPAN-Patch.

Git::CPAN::Patch

Speaking of Git::CPAN::Patch, you can use its set of scripts to ease the creation of CPAN-related git repositories. It also includes scripts to send patches directly from your local repository to a rt bug queue.

As we know, prior to 11g, Oracle did not have a simple table pivot feature. 11g has changed that, and the pivot clause is certainly useful. It requires, however, an aggregation calculation in the intersection (at least that’s what I got from the documentation). But what if you already have the data to populate in the intersection area? Or, you may no longer have the raw data to aggregate it again. In that case, you are forced to trick it to get an aggregation in.

I’ve written a set of fairly simple PL/SQL code that handles this task in all versions of Oracle that supports associative nested VARRAYs (I have no clue when that started getting supported. It sure works great with 10g).

Here is how my code works:

1. A function is used to pass in your query that retrieves your raw data.

   The function needs a query that returns three columns. The first one for the row labels, the second one for the column labels, and the third one for the intersection data.

2. The function converts the result set of the supplied query into a pivot as CSV strings (don’t worry, I’ll explain later how to make those separate columns).
3. Each line (row) is summarized in an extra trailing column.
4. An extra trailing line is also generated with the column summaries and a Grand Total at the end.
5. The function pipes the result out, as it gets readied, to the calling query, which receives it as a single column set of rows.

This is the core of the work. The resulting CSV strings can be spooled in a file and loaded into any decent spreadsheet editor.

Some of you may say, yeah okay, but I’m really looking at getting back a cursor with this data in separate columns. That’s not so difficult to do, assuming you are willing to live with some minor details, which I will explain later.

Alright, here is the function:

-- we need this TYPE created to return a cursor of varchar2.
create type varchar2_type as table of varchar2(2000);
/
create or replace function pivot_three_cols_table_func
(p_query_string varchar2, sum_cols_ind INTEGER default 0, sum_lines_ind INTEGER default 0)
return varchar2_type pipelined
as
   type tb1  is table of number index by varchar2(1000);
   type ntb1 is table of tb1 index by varchar2(500);
   nvar ntb1;

   type coltb is table of varchar2(20) index by varchar2(20);
   cols coltb;

   col_1 varchar2(1000);
   col_2 varchar2(500);
   col_3 number;

   line varchar2(1000);
   col  varchar2(500);

   sum_lines number :=0;
   sum_cols  number :=0;
   sum_pivot number :=0;

   type item_cur_type is ref cursor;
   item item_cur_type;

   str varchar2(32767);
begin
   -- initialize the empty array
   nvar := nvar;

   -- load the array
   open item for p_query_string;
   loop
       fetch item into col_1, col_2, col_3;
       -- replace commas with space from the col_1 and col2 as
       -- these are used as column delimiters in the csv
       col_1 := replace(col_1,',',' ');
       col_2 := replace(col_2,',',' ');
       nvar(col_1)(col_2) := col_3;
       exit when item%notfound;
   end loop;
   close item;

   -- print the crosstab/pivot header
   --  -- first load the list of distinct column names in an associative array
   line := nvar.first;
   for t in 1..nvar.count loop
       col := nvar(line).first;
       for d in 1..nvar(line).count loop
           cols(col) := col;
           col := nvar(line).next(col);
       end loop;
       line := nvar.next(line);
   end loop;
   -- -- print the list of disctinct column names
   str := 'TITLE,';
   col := cols.first;
   for d in 1..cols.count loop
       str := str||cols(col)||',';
       col := cols.next(col);
   end loop;
   if sum_cols_ind = 1 then
      str := str||'TOTAL';
   else
      str:= substr(str,1,length(str)-1);
   end if;
   pipe row (str);

   -- sum and print the column data
   line := nvar.first;
   for t in 1..nvar.count loop
       str := line||',';
       sum_cols :=0;
       col := cols.first;
       for d in 1..cols.count loop
           begin
             str := str||to_char(nvar(line)(col))||',';
             if sum_cols_ind = 1 then
                sum_cols := sum_cols + nvar(line)(col);
             end if;
           exception when NO_DATA_FOUND then
             str := str||',';
           end;
           col := cols.next(col);
       end loop;
       if sum_cols_ind = 1 then
          str := str||to_char(sum_cols);
       else
          str:= substr(str,1,length(str)-1);
       end if;
       pipe row (str);
       line := nvar.next(line);
   end loop;

   if sum_lines_ind = 1 then
      -- sum and print the column totals
      str := 'TOTAL RESULT,';
      line := nvar.first;
      col  := cols.first;
      for d in 1..cols.count loop
        sum_lines :=0;
        line := nvar.first;
        for t in 1..nvar.count loop
          begin
            sum_lines := sum_lines + nvar(line)(col);
          exception when NO_DATA_FOUND then
            null;
          end;
          line := nvar.next(line);
        end loop;
        str := str||to_char(sum_lines)||',';
        sum_pivot := sum_pivot + sum_lines;
        col := cols.next(col);
      end loop;

      -- print the Grand Total
      str := str||to_char(sum_pivot);
      pipe row (str);
   end if;
   return;
end;
/

Lots to say about it:

1. It does the job. At least it did–and very well–during my testing.
2. It can summarize the intersection data. So that third column in your query must be a number. It would be pretty easy to change that for someone who wanted to show text data in the intersection.
3. The first two columns must be characters. If you want numbers in the labels, simply get them converted in your driving query as in to_char(num_col).
4. I have included parameters to turn on and off the SUM calculations. The default is to not sum nor print the column and line totals.
5. The function produces CSV strings so that it will support any number of columns. No need to know how many columns it will have, nor what the names are. This meant that I had to make it replace all commas in the input strings with a space. Otherwise, it would have caused the CSV parsing to fail which would in turn have created some pretty ugly spreadsheets. So, your labels will lose any single quotes they may have (not a big deal in my opinion).
6. Line labels can be as large as 1000 characters. That is huge and very inconvenient, I would argue, as it would make the pivot table difficult to read. It’s up to you to control the width of the data your query provides in the first column.
7. Column labels can be as large as 500 characters. Same issue as for line labels. If you have wide column labels, your pivot table may be difficult to read, so set them appropriately in the second column.
8. Each CSV string/line is limited to 32767 characters. This would certainly create a very ugly pivot table if you have too many columns or very wide data. But that’s the limit, which I haven’t actually tested.

Let’s see a sample query and output. It creates a pivot of Oracle-archived logs showing the sum of log sizes created by weekday across 24 hours for the last year. DBAs will be familiar with that.

First here’s the source query and a data subset:

SELECT To_Char(completion_time,'Day') WD,
       To_Char(completion_time,'HH24') HR,
       round((Sum(blocks)*block_size)/1048576) mb
  FROM v$archived_log
 WHERE completion_time >= SYSDATE - 365
 GROUP BY to_Char(completion_time,'Day'), To_Char(completion_time,'HH24'), block_size;

WD        HR   MB
Friday    02   80
Friday    04   131
Thursday  17   128
Thursday  19   75
Thursday  18   130
Friday    03   75
Thursday  20   80
Thursday  21   542
Thursday  15   18
Thursday  22   207
Friday    00   85
Friday    06   56
Thursday  16   128
Thursday  23   88
Friday    01   92
Friday    05   72
...

This data gets pivoted using this query:

select column_value
  from table(pivot_three_cols_table_func(
       'select d, h, round((Sum(blocks)*block_size)/1048576) MB
          from (SELECT To_Char(completion_time,''Day'') D,
                       To_Char(completion_time,''HH24'') H,
                       blocks,
                       block_size
                  FROM v$archived_log
                 WHERE completion_time >= SYSDATE - 365
               )
         GROUP BY d, h, block_size
       ',1,1))
;

Note that single quotes needed to be doubled in the subquery string. Also, I’ve generated the line and column sums.

The output is as follows:

COLUMN_VALUE
------------
TITLE,00,01,02,03,04,05,06,07,08,09,10,11,12,13,14,15,16,17,18,19,20,21,22,23,TOTAL
Friday   ,299,261,233,216,404,242,265,328,420,482,443,465,465,396,403,412,309,263,221,168,142,1340,386,144,8707
Monday   ,186,172,160,157,277,168,184,206,292,320,306,295,288,232,261,278,278,266,204,184,171,1730,436,192,7243
Saturday ,152,155,166,146,275,146,251,162,213,185,150,158,155,145,161,144,142,148,141,140,149,1854,278,153,5769
Sunday   ,146,141,156,144,259,142,156,163,190,174,162,165,158,141,141,141,170,202,190,172,149,1815,240,161,5678
Thursday ,209,167,198,151,274,158,186,203,291,281,338,264,296,286,259,259,272,285,257,186,172,1641,620,251,7504
Tuesday  ,160,161,164,162,280,146,174,206,269,314,313,328,276,302,284,279,287,266,261,182,199,989,362,179,6543
Wednesday,196,178,161,172,288,181,175,220,313,347,316,277,264,281,243,312,292,272,270,207,159,1026,360,165,6675
TOTAL RESULT,1348,1235,1238,1148,2057,1183,1391,1488,1988,2103,2028,1952,1902,1783,1752,1825,1750,1702,1544,1239,1141,10395,2682,1245,48119

Voila! The pivot is done and can be saved in a file and imported into a spreadsheet, or whatever other tools you use that reads CSV data.

Now, for those who want distinct columns for this data. I’ve created a simple CSV parser function that can be used to extract distinct columns. Here’s the code for this CSV parser function:

create or replace function csv_element(string varchar2, element_number number)
return varchar2
as
i number := element_number;
r varchar2(2000);
begin
  case
    when (i=1 and instr(string,',',1)=0) then r:= string;
    when (i=1) then r := substr(string,1,instr(string,',',1,1) -1 );
    when (i>1 and (instr(string,',',1,i-1)>0 and instr(string,',',1,i)=0)) then r:= substr(string,instr(string,',',-1)+1);
    when (i>1) then r := substr(string,instr(string,',',1,i-1)+1,instr(string,',',1,i)-instr(string,',',1,i-1)-1 );
    else r:= null;
  end case;
  return r;
end;
/

The csv_element function handles strings with no commas, and returns that value as the first and only element. It also takes care of the trailing string, and it will also return a null value for columns that are beyond the number of elements in the strings. That is as sophisticated as I needed it to be.

Here’s the previous sample query using the function to return the data as separate columns:

select csv_element(column_value,1) c1, csv_element(column_value,2) c2,
       csv_element(column_value,3) c3, csv_element(column_value,4) c4,
       csv_element(column_value,5) c5, csv_element(column_value,6) c6,
       csv_element(column_value,7) c7, csv_element(column_value,8) c8,
       csv_element(column_value,9) c9, csv_element(column_value,10) c10,
       csv_element(column_value,11) c11, csv_element(column_value,12) c12,
       csv_element(column_value,13) c13, csv_element(column_value,14) c14,
       csv_element(column_value,15) c15, csv_element(column_value,16) c16,
       csv_element(column_value,17) c17, csv_element(column_value,18) c18,
       csv_element(column_value,19) c19, csv_element(column_value,20) c20,
       csv_element(column_value,21) c21, csv_element(column_value,22) c22,
       csv_element(column_value,23) c23, csv_element(column_value,24) c24,
       csv_element(column_value,25) c25, csv_element(column_value,26) c26,
       csv_element(column_value,27) c27, csv_element(column_value,28) c28
  from table(pivot_three_cols_table_func(
       'select d, h, round((Sum(blocks)*block_size)/1048576) MB
          from (SELECT To_Char(completion_time,''Day'') D,
                       To_Char(completion_time,''HH24'') H,
                       blocks,
                       block_size
                  FROM v$archived_log
                 WHERE completion_time >= SYSDATE - 365
               )
         GROUP BY d, h, block_size
       ',1,1))
;

I’ve deliberately added two extra columns in the query (27 and 28) so that you can see it handling it fine. Here’s the output of this query (with some columns removed for clearer formatting):

C1           C2   C3   C4   ... C23   C24  C25  C26        C27  C28
TITLE        00   01   02   ... 21    22   23   TOTAL
Friday       299  261  233  ... 1340  386  144  8721
Monday       186  172  160  ... 1730  436  192  7243
Saturday     152  155  166  ... 1854  278  153  5769
Sunday       146  141  156  ... 1815  240  161  5678
Thursday     209  167  198  ... 1593  620  251  7456
Tuesday      160  161  164  ... 989   362  179  6543
Wednesday    196  178  161  ... 1026  360  165  6675
TOTAL RESULT 1348 1235 1238 ... 10347 2682 1245 48085

As you can see, columns 27 and 28 are there, but null. This way, you can have a generic wrapper query with a hundred columns if you’d like, without worrying about how many columns will be produced from your base query. Just make sure your app can handle that.

Enjoy!

Here’s the video:

There were heaps of mostly technical posts this week. I think bloggers are tired of kicking around the ins-and-outs of Sun and Oracle, and wanted to talk about what really matters. So let’s start with . . . 

Oracle

Harald van Breederode shows how to setup a private DNS for your virtual cluster.

Pythian’s Alex Fatkulin discusses Oracle GoldenGate Extract Internals.

From Charles Hooper comes this investigation: Simple Query Generates Complex Execution Plan, the Mysterious 4063.88 Second Single Block Read Wait.

Coskan Gundogar was also in a deductive frame of mind. Here is his Working with statspack-part-1a-Diagnosis, featuring both a challenge and purty pictures (pastels!).

Here is Jonah H. Harris with an introduction to the NEXTGRES Gateway, a MySQL Emulator for Oracle. Jonah writes: “So, a few people have asked me what NEXTGRES Gateway is. My short answer, the ultimate database compatibility server.  . . . I’ve been working on this personal project non-stop for the last 8 months and am really excited about it.”

John Hallas lays out the use of functions in a .profile file.

MySQL

In a different part of config file land, Ronald Bradford cautions us, be sure to know your my.cnf [sections]. “The MySQL configuration file,” he writes, “e.g. /etc/my.cnf has a number of different section headings including [mysql], [mysqld], [mysqld_safe]. It is important that you ensure you put the right variables into the right section.”

Sheeri K. Cabral responds with Know your my.cnf groups, part II.

Baron Schwartz inspires a lot of conversation with his post, My wishlist for SQL: the UNTIL clause. Baron says, “I’d like an UNTIL clause, please. I’d use it sort of like LIMIT in MySQL and PostgreSQL, except that it would define when to stop returning looking for rows, instead of defining how many to return.”

Roland Bouman has gone about adding stuff to MySQL on his own, as he demonstrates in Easter Eggs for MySQL and Kettle with a MySQL stored function to calculate easter day.

Speaking of things hidden, here is a note from Andrew Morgan pointing out the location of MySQL documentation and binaries.

Marc Alff is looking forward, and offers a performance schema overview, “ . . . an introduction to the new ‘performance schema’ feature, which will be part of the upcoming MySQL 5.5 release.”

We must have at least one post on the whole Sun/Oracle thing, and I found a good one–Oracle, MySQL, the EU and Wayne Gretzky, by Mike Hogan. The keynote: “‘A good hockey player plays where the puck is. A great hockey player plays where the puck is going to be’ — Wayne Gretzky”

SQL Server

To quote Oracle blogger Kevin Closson, “little things doth crabby make.” Or as Aaron Bertrand would put it, Sometimes it’s the small things: match column names in subqueries. Aaron writes, “The behavior of column matching in subqueries is a little peculiar, to say the least. If you’ve been bitten by this behavior once, you’re unlikely to have been bitten a second time, but for some of us it just takes a while to sink in.”

Jonathan Kehayias has another puzzler: is NT AUTHORITY\SYSTEM a sysadmin in your SQL Server and why? “I was prompted to ask this question while configuring a server early today and after installing SQL Server 2008 on Windows Server 2008 R2, I noticed that despite being installed from default using a Domain Account for the Services, the Local System account was still a sysadmin in SQL Server.”

Another little thing, courtesy Roman Rehak: “One of the things that I found dissapointing in Management Studio 2008 is that in the event of a crash (and in addition to crashing in the first place), is that on a restart it doesn’t show me a dialog with a list of auto-saved files, like my SSMS 2005 did.” But, he says, your work may not be lost.

Michelle Ufford has updates on her index defrag script – beta testers are needed.

Simon Sabin shares SQLBits videos for SQLBits V.

PostgreSQL

Bruce Momjian relays an appeal for a new project slogan for Postgres, and shares the leading candidates thus far. “PostgreSQL: The ‘Open’ Open Source Database” Who can they be comparing themselves to? “PostgreSQL: Unpronounceably awesome!” That’s the one!

Bruce also enumerates what he sees as threats to Postgres.

Hubert Lubacziewski notes that he is now waiting for … 9.0: “I’ve written 29 posts about new features in 8.5. And now core team decided to name it 9.0. Great.”

And so the Database Explorer appears with a brief update on 9.0 Replication Features.

Last, Kendal Van Dyke exhorts you to use your tech skills to help Haiti. He writes, “So what do you do if you want to help?  . . .  I came across an article on cnn.com which highlighted “Crisis Camp Haiti”, a group of volunteers working to build digital maps, mobile apps, and searching through data to help relief groups in Haiti coordinates their efforts. They’ve got a website up at http://crisiscommons.org/ and you can follow them on Twitter (@crisiscamp).”

That’s all for now. See you for LB #177!

Update 9-Feb-10: Want to schedule a meeting with Pythian folks? See Pythian Events page.

I’m so much looking forward to the next conference in my schedule — RMOUG Training Days 2010. It would be only my second time I’m presenting at the RMOUG but it was enough to go there once to understand that it’s one of the top rated Oracle User Group conferences in the world. Some of the great speakers are presenting and registration fees are very low compare to other events of comparable quality. If your conference budget is low this year — that’s the conference you don’t want to miss!

Two of us from Pythian are going to speak at RMOUG Training Days 2010 that starts in just 4 week. I present the following session:

Alex Gorbachev: Oracle ASM 11g — The Evolution.
Oracle Automatic Storage Management has proven to be one of the most widely adopted new features in Oracle Database 10g and it has been dramatically improved in the later 11g releases. This presentation will explain what changes are solved by ASM, how these challenges are solved, what barriers there are to ASM adoptions, and how 11g Release 2 addresses these barriers.

My colleague, Christo Kutrovsky is presenting the following:

Christo Kutrovsky: RAC+ASM — Three Years in Production. Stories to Share.
This presentation will cover Migration via LUN re-assignment; migrating online migration from EMC to 3par; removing ASMLIB and using EMC multipathing; gotchas when building ASMLIB devices with multipathing; corrupting your production database beyond repair with duplicate database on ASM; overwriting your data with ASM due to incorrect naming convention; corrupting your production database with NETAPP/NFS due to Oracle bug; two nodes RAC is not HA; performing network maintenance; and RAC + SAME = LAME.

Coincidentally, out topics are very close. However, they don’t overlap but rather complement each other when it comes to ASM. While my session covers more the history of ASM, why it arrived in Oracle stack and, of course, why ASM 11g versions is so much better in many respects compare to ASM 10g, Christo delivers a more practical speech with some interesting real world scenarios.

Update 9-Feb-10: Unfortunately, Jay Caviness session is canceled so there will be only 3 ASM presentations.

Actually, there are four 3 sessions around ASM at the conference and all of them are back to back starting at 11:45 on the second day of the conference. Christo’s session is the first followed by Marshall Presser from Oracle presenting on ASM Cluster Filesystem (ACFS). I’ve had a pleasure to meet Marshall literally couple weeks ago here in Ottawa during a joint workshop on Oracle RAC (well, Marchall was actually running the show and I contributed some examples of real-world experience with Pythian customers). Marshall has long history with Oracle and definitely knows what he is talking about. He is also a very entertaining speaker so I recommend you attend his session if you are interested in the topic.

My session is on next and I’ll make sure not to cover much of ACFS — I have way more to go through during my slot. In addition, there is a dedicated presentation on Oracle ASM 11gR2 new features from Jay Caviness so my job will be to cover enough to warm up the audience for the Jay’s session that is up next according to the schedule. I haven’t met Jay yet but I’ll try to get in touch and coordinate to avoid significant overlaps in our presentations. Jay’s session has been canceled.

See you in Denver soon and, if you are a skier, make sure to catch the ski lift somewhere around Loveland Pass — great slopes there! The photo below was taken couple years ago after the conference (courtesy of Tim Gorman) — there is interesting story behind so ask me or Tim at the conference.

That’s all for now. Time to feed my new dog:

She is a 10 weeks old puppy (mix of Bull Mastiff, Doberman and Rottweiler). She is usually a good girl but, sometimes, throws surprises like this when she is left on her own for too long:

It’s not the worst that happened so far and I’m sure more to come!