The last time I wrote this was just under 2 years ago!!! WoW. Things have changed. Sun bought MySQL, Oracle bought Sun. Those were bombshell deals. At least you can rest assured that some things can be constant. I still eat my daily serving of broccoli (among other healthful “things”). I urge you all to go the fridge and grab some veggies prior to sitting down for this week’s… ahem… digest.

Starting with Oracle, Pythian’s own Alex Fatkulin illustrates a bug (?…likely) that could lead to logically corrupted data. Doug Burns provides an update on his Hotsos 2010 trip with some serious thumbs-up action pointing in Tanel Poder’s direction and his affinity toward SQL*Plus.

Over at Mark Rittman’s Oracle Weblog, Venkatakrishnan J talks about Oracle Warehouse Builder 11gR2 – Importing Essbase Cubes using ODI Knowledge Modules – Part 1. This should drive the MySQL folks bonkers (in a good way) since he points to an example by David Allen to import MySQL Metadata over and vice versa.

Oracle Virtualization Blog’s Adam Hawley let everyone know about a Best Practices around Oracle VM with RAC RAC SIG webcast. It was on March 18th, but it was recorded and should be available online. I’m interested in this stuff so I included it here. I hope you can enjoy it too.

I had the honour to recently provide a training session at a customer site on tuning methods and tools. A key topic of the discussion was related to Oracle statistics and the CBO. It was timely that the Oracle Scratchpad’s Jonathan Lewis posted a series created by Doug Burns all about stats. Nice.

Has anyone ever told you (or maybe you deduced it on your own) that leaving out where clauses is a bad thing. Well, leave it to Charles Hooper on his Oracle Notes to prove to us that sometimes, they are actually more efficient for solving certain types of problems. In another excellent post titled Physical Reads are Very Fast, Why is this SQL Statement Slow Charles also dives into why a slow query is “actually” slow. There’s an interesting discussion taking place. More performance tuning goodness was posted by Joel Goodman discussing some interesting behaviour with Oracle Index Leaf Blocks contention. Tell RAC to Leave Your Leaves Alone! Kerry Osborne illustrates the use of an interesting hidden parameter (_high_priority_processes) to resolve “log file sync” issues.

Over on the other side of the fence in the MySQL world, Jay Pipes @ Join-Fu gives us some background on the MySQL Transaction Log. Vadim Tkachenko with the MySQL Performance Blog has a number of posts on Percona 9.1 as well as a list of related sessions at the 2010 MySQL conference. Check it out.

On a more somber note (and by somber I mean legal, we all hate legalese don’t we?), Giuseppe Maxia over at The Data Charmer discusses Protocol, the GPL, and how Bazaar can help. He also has some good takeaways from the Linux MySQL distros meeting in Brussels. Baron Schwartz at xaprb has a new tool he’d like the MySQL folks to take for a spin. Try mk-query-advisor, a new Maatkit tool. It uses heuristics to find problems in SQL. Please use it and give feedback!

Are you interested in MySQL Clustering? I am. Andew Morgan has a new post introducing a tutorial to Build MySQL Cluster 7.1 from source – including MySQL Cluster Connector for Java. With Alex Fatkulin maybe running into an as yet possibly, sort of , kinda non-discovered bug, it’s only fair we also get some vision into the dark side on the MySQL front. Shlomi Noach states But I DO want MySQL to say “ERROR”!.

Brian Aker invites readers to participate at this year’s O’Reilly MySQL User’s Conference where they will be doing their first ever Ignite talk series.

Lastly, Stewart Smith has a very good set of posts illustrating Stored Procedures/Functions for Drizzle. Check this out (from Stewart’s post).

drizzle> select libtcc("#include <string.h>\n#include <stdlib.h>\nint foo(char* s) { char *a= 0x199c610; strcpy(s, a); return strlen(s); }") as result;

+--------------+
| result       |
+--------------+
| Hello World! |
+--------------+
1 row in set (0.01 sec)

I dare you.

On to SQL Server….

Scary DBA SQL MVP Grant Fritchey discusses Undocumented Virtual Column: %%lockres% and also recaps the SNESSUG March Meeting.

Jamie Thomson gives keyboard junkies some tips to Kill your temp tables using keyboard shortcuts : SSMS. Anyone interested in distributed queries should read Buck Woody’s Using linked servers, OPENROWSET and OPENQUERY.

Finally, Aaron Bertrand says with conviction “Yes, you can benefit from both data and backup compression”.

Having now exhausted my supply of munchies, it is time for me to retire. I bid you all a fantastic week. Keep your data safe, folks.

Shakir

Operating Systems

In case anyone was wondering about SCO vs. Linux it is still going on. If anyone has a lot of free time on their hands and is interested in lost causes, check out SCO vs. Linux: The story so far at The H Online. Even more details can be found at the prolific GrokLaw in Summary of SCO v IBM.

If you are running Max OSX you may be vulnerable to at least 20 major security flaws in you system. Security researcher Charlie Miller will be presenting full details at the upcoming CanSecWest conference. Read more in Mac OS X: “safer, but less secure” – Update.

If you, like me, are part of the “never put a .0 release into production” school of thought, then you probably have not rolled out Windows 7 on your company desktops. Fear not, SP1 is coming. Check out Microsoft announces Windows 7 SP1 at Computer World.

Internet

Is Google set to leave China and close the www.google.cn site entirely? Read the story on Cnet news in Report: Google to leave China on April 10. The sources seem to be dubious, so I would take this with a grain of salt until the official announcement.

There has been stuff floating around the internet about DNS tunneling exploits for years. The H Online is reporting that hacker Ron Bowes has now released a command line tool and code that could easily be integrated into an exploit. Check out Exploit code with DNS tunnel for more.

Have you upgraded from FireFox 3.0 to 3.6 yet? You probably should soon. See Firefox 3.0 approaches end-of-life.

Security

The Register is reporting that almost 2 weeks after being notified the Energizer site still plagued by data-stealing trojan. The file in question is “UsbCharger_setup_V1_1_1.exe” so try to avoid downloading that one.

There’s reports of a security flaw in milter, part of the spamassassin utility for scanning email. The flaw allows a specially crafted RCPT command to trick postfix into running commands as root. Sendmail servers seem to not be affected. You can find more details in [Full-disclosure] Spamassassin Milter Plugin Remote Root.

Here’s another follow-up to a previous item about the Waledec botnet being taken down by Microsoft. See Waledac Botnet Now Completely Crippled, Experts Say. Microsoft has also waxed poetic on the subject in What we know (and learned) from the Waledac takedown on the technet blog.

Open Source

Ars Technica has a cute item on hidden gems in open source applications, affectionately knows in geek nomenclature as “Easter Eggs”. Check out Cracking open five of the best open source easter eggs for a larf, and read over the comments for a bunch more.

That’s all the time I will have this week. I know there’s a lot more stuff from the last 2 weeks that I could have added, so feel free to leave your favourites in the comments. ‘Til next week.

Who you trying to get crazy with SA, don’t you know I’m loco?

Let’s start by creating a simple table with a deferrable primary key:

SQL> create table def_bug(n number primary key deferrable initially deferred);

Table created

SQL> insert into def_bug values (1);

1 row inserted

SQL> insert into def_bug values (2);

1 row inserted

SQL> commit;

Commit complete

You can confirm that the primary key constraint is working fine by trying to insert a duplicate value:

SQL> insert into def_bug values (1);

1 row inserted

SQL> commit;

commit

ORA-02091: transaction rolled back
ORA-00001: unique constraint (SRC.SYS_C004070) violated

So far so good. Open a second session and execute the following update:

SQL> update def_bug set n=3 where n=2;

1 row updated

Do not commit yet and execute in your first session:

SQL> update def_bug set n=3 where n<=2;

The above update will block due to our second session holding a lock on the row where n=2. Now commit your second session…

SQL> update def_bug set n=3 where n=2;

1 row updated

SQL> commit;

Commit complete

…and then commit your first session:

SQL> update def_bug set n=3 where n<=2;

1 row updated

SQL> commit;

Commit complete

Take a look at the data now:

SQL> select * from def_bug;

         N
----------
         3
         3

Ouch! This was certainly unexpected. You can confirm that the primary key is still working by trying to insert a duplicate value again:

SQL> insert into def_bug values (3);

1 row inserted

SQL> commit;

commit

ORA-02091: transaction rolled back
ORA-00001: unique constraint (SRC.SYS_C004070) violated

It certainly looks like the update statement did not take into account deferrable constraint declared on the table during restart caused by the write consistency mechanism.

There were two things I have wanted to do for some time now. The first was to come up with a way to quickly and easily set up a DarkPAN mirror so that we would have more control over our dependency chain at work. The second was to make a portable CPAN proxy service, so that I can always have access to my favorite modules, even if the machine I’m working on has no Internet access. Last week, I finally had a few ‘rount tuits’ to spend on this type of background itch, and the result is dpanneur (for dépanneur, French Canadian for convenience store).

Installation

As it stands, dpanneur is a very thin Catalyst application gluing together the goodiness of CPAN::Cache and MyCPAN::App::DPAN, and throwing in Git as the archive manager.

To get it running, first fetch it from Github

$ git clone git://github.com/yanick/dpanneur.git

then check that you have all the dependencies

$ perl Makefile.PL

and run the script that will create the module repository

$ ./script/create_repo

For now, the module repository is hard-coded to be in the subdirectory cpan of dpanneur. A branch called proxy is created and checked out. Eventually, I’ll use GitStore to push newly fetched modules to the repository, but for the time being if dpanneur is to be used as a proxy, that branch must remain as the one being checked out.

All that is left is to fire up the server in whichever mode you prefer (single-thread test server would do nicely for now)

$ ./script/dpanneur_server.pl

and there you are, running your first dpanneur. Congrats! :-)

Using it as a caching proxy

You can use the server as a caching proxy, either for its own sake, or to seed the DarkPAN branches. To do that, you just have to configure your CPAN client to use http://yourmachine:3000/proxy:

$ cpan
cpan[1]> o conf urllist = http://localhost:3000/proxy
cpan[2]> reload index
cpan[3]> install Acme::EyeDrops
Running install for module 'Acme::EyeDrops'
Running make for A/AS/ASAVIGE/Acme-EyeDrops-1.55.tar.gz
Fetching with LWP:
    http://localhost:3000/proxy/authors/id/A/AS/ASAVIGE/Acme-EyeDrops-1.55.tar.gz
etc..

As the modules are downloaded, they are also saved and committed within the repo

[dpanneur]$ cd cpan

[cpan (proxy)]$ git log -n 3
commit d065ad152f2204295334c5475104a3da517b6ae1
Author: Yanick Champoux <yanick@babyl.dyndns.org>
Date:   Wed Mar 10 20:32:52 2010 -0500

    authors/id/A/AS/ASAVIGE/Acme-EyeDrops-1.55.tar.gz

commit e8d2e83d1b16e2e0713d125f9a4bd2742681f859
Author: Yanick Champoux <yanick@babyl.dyndns.org>

Date:   Wed Mar 10 20:31:42 2010 -0500

    authors/id/D/DC/DCONWAY/Acme-Bleach-1.12.tar.gz

commit 7e0b4b600bac8424c519199ee96dc56ffbb177eb
Author: Yanick Champoux <yanick@babyl.dyndns.org>
Date:   Wed Mar 10 20:30:47 2010 -0500

    modules/03modlist.data.gz

Using it as a DarkPAN server

There is not much more involved to enabling DarkPAN repos. All we have to do is to create a branch with the modules we want and have the ‘dpan’ utility bundled with MyCPAN::App::DPAN generate the right files for us.

To continue with the example of the previous section, let’s say that we want a DarkPAN branch containing Acme::EyeDrops, but not Acme::Bleach. Then we’d do

                        # only necessary if you are running
                        # the server while you work on the branch
[dpanneur]$ git clone cpan cpan-work   

[dpanneur]$ cd cpan-work

                        # branch just before we imported Acme::Bleach
[cpan-work (proxy)]$ git branch pictoral 7e0b4b600bac8424c519199ee96dc56ffbb177eb

[cpan-work (proxy)]$ git checkout pictoral
Switched to branch 'pictoral'

                        # cherry-pick the Acme::EyeDrops commit
[cpan-work (pictoral)]$ git cherry-pick d065ad152f2204295334c5475104a3da517b6ae1

                        # rebuild the module list
[cpan-work (pictoral)]$ dpan

                        # commit the new 02packages.details.txt.gz
[cpan-work (pictoral)]$ git add .
[cpan-work (pictoral)]$ git commit -m "dpan processing"

                        # push back to the mothership
[cpan-work (pictoral)]$ git push origin pictoral

And that’s it. Now point the cpan client to http://yourmachine:3000/pictoral, and you’ll get the limited mirror.

cpan[1]> o conf urllist http://localhost:3000/pictoral
cpan[2]> reload index

cpan[3]> i Acme::EyeDrops
Strange distribution name [Acme::EyeDrops]
Module id = Acme::EyeDrops
    CPAN_USERID  ASAVIGE (Andrew J. Savige <asavige@cpan.org>)
    CPAN_VERSION 1.55
    CPAN_FILE    A/AS/ASAVIGE/Acme-EyeDrops-1.55.tar.gz
    UPLOAD_DATE  2008-12-02
    MANPAGE      Acme::EyeDrops - Visual Programming in Perl
    INST_FILE    /usr/local/share/perl/5.10.0/Acme/EyeDrops.pm
    INST_VERSION 1.55

cpan[4]> i Acme::Bleach
Strange distribution name [Acme::Bleach]
No objects found of any type for argument Acme::Bleach

This week started out with me posting about International Women’s Day, and has me personally attending Confoo (Montreal) which is an excellent conference I hope to return to next year. I learned a lot from confoo, especially the blending nosql and sql session I attended.

This week was also the Hotsos Symposium. Doug’s Oracle Blog has a series of posts about Hotsos. If all this talk about conferences has gotten you excited, Joshua Drake notes that 14 days and the hotel is almost full for postgresql conference east which is March 25th-28th in Philadelphia. And the Oracle database insider notes that the Oracle OpenWorld call for papers is now open.

According to Susan Visser this week (ending tomorrow) is also read an e-book week. So if you have not already done so, read an e-book! She links a coupon for an e-book in the post.

Craig Mullins notes that the mainframe is a good career choice in Mainframes: The Safe IT Career Choice. He notes that the mainframe is still not dead:

People having been predicting the death of the mainframe since the advent of client/server in the late 1980s. That is more than 20 years! Think of all the things that have died in that timespan while the mainframe keeps on chugging away: IBM’s PC business, Circuit City, Koogle peanut butter, public pay phones, Johnny Cash… the list is endless.

In other career-related news, Antonio Cangiano is looking for [2] top-notch student hackers for a 16-month internship at IBM in Toronto starting in May. All the details, including how to apply, are in Cangiano’s blog post.

Willie Favero wants to know how you “solve the batch dilemma” for issues like “shrinking your batch window, designing your batch to play nicely with … OLTP” in how’s your batch workload doing? Perhaps Favero should read the updated batch best practices posted by Anthony Shorten.

Bryan Smith surveys a more personal question by asking if you go both ways and “manage both DB2 for Linux, UNIX, and Windows and DB2 for z/OS” in don’t ask, don’t tell, bi-platform DBAs. This week’s Log Buffer editor admits to being a tri-platform DBA — she has tried many platforms, and in fact, many databases (MySQL, Oracle, DB2, SQL Server, Sybase, Postgres and Ingres)!

Hari Prasanna Srinivasan promotes a patching survey in Oracle really wants to hear from you! Patching Survey.

Henrik Loeser explains what a deadlock and a hot spot are by using a real life analogy taken from a police report in deadlock and hot spot in real life.

Jamie Thomson asks why do you abbreviate schema names?. Shlomi Noach tries to solve the issue that “there is no consistent convention as for how to write [about table aliases in] an SQL query” in proper sql table alias use conventions. Noach also gives us a tip: faster than truncate.

Leons Petrazickis reminds us that “rulesets are chains” and it is important to have your rulesets in the proper order in iptables firewall pitfall.

Anyone interested in the history of MySQL AB will be informed after reading Dries Buytaert’s article.
Gavin Towey shares his software that helps centrally manage 120 MySQL servers in qsh.pl: distributed query tool For those who want to learn more about column-oriented databases, particularly in MySQL, Robin Schumacher of the InfiniDB blog announces that there is a MySQL University session recording on MySQL column databases now available. MySQL join-fu expert Jay Pipes has moved his blog to www.joinfu.com and starts with An SQL Puzzle and of course a follow up on the sql puzzle.

Ivan Zoratti is happy that finally, slides posted for the MySQL DW breakfast. Venu Anuganti gives you tips on one of the most common MySQL frustrations: optimizing subqueries in how to improve subqueries derived tables performance. Justin Swanhart posts the way in which he Gets Linux performance information from your MySQL database without shell access and emulates a ‘top’ CPU summary using /proc/stat and MySQL using the same method.

The Oracle Apps blog has an introduction to Oracle user productivity kit (UPK). Even though in this editor’s opinion the article is very sales-pitchy, it has valuable information, and does indeed live up to its promise:

UPK is a software tool that can capture all the steps in a system process. It records every keystroke, every click of the mouse, each menu option chosen and each button pressed. All this is done in the UPK Recorder by going through the transaction and pressing “printscreen” after every user action. From this, without any further effort from the developer, UPK builds a number of valuable outputs.

Allen White gives a great tip on how to optimize queries in keep your data clean.

Mike Dietrich reminds you to remove “old” parameters and events from your init.ora when upgrading, “as keeping them will definitely slow down the database performance in the new release.” He shows evidence of slowness when this is not done. Dietrich also shows how you can be gathering workload statistics “to give the optimizer some good knowledge about how powerful your IO-system might be”, especially “a few days after upgrading to the new release…while a real workload is running.”

Brian Aker shows the exciting features coming soon in Drizzle in Drizzle, Cherry, Roadmap for our Next Release.

Maybe you are thinking of migrating, not upgrading…..The O’Reilly Radar shows how to asses an Oracle to MySQL migration in MySQL migration and risk management. Actually, that article interviews Ronald Bradford on the subject — Bradford has been prolific lately, updating free my.cnf advice series and “Don’t Assume”: MySQL for the Oracle DBA series. Nick Quarmby also talks about migrating Oracle, but not to a new database, just to a new platform, in his primer on migrating Oracle Applications to new platforms. And the big news comes from Carlos of dataprix that Twitter will migrate from MySQL to Cassandra DB.

Paul S. Randal explains his way of benchmarking: 1 Tb table population on SQL Server.

Pete Finnigan shares his slides from a webinar on how to secure oracle, and Denis Pilipchuk shares his approaches for discovering security vulnerabilities in software applications.

Jeff Davis shares his thoughts about scalability and the relational model. Robert Treat responds actually, the relational model doesn’t scale and Baron Schwartz counters with NoSQL doesn’t mean non-relational.

Buck Woody explains “whenever you want to know something about SQL Server’s configuration, whether that’s the Instance itself or a database, you have a few options” — and of course what those options are — in system variables, stored procedures or functions for meta data.

This week’s T-SQL Tuesday topic was I/O. There are many links to great blog posts in the comments; three random posts I chose to highlight: Michael Zilberstein talks about IO capacity planning, while Kalen Delaney talks about using STATISTICS IO in I/O, you know, and Merrill Aldrich chimes in with information on real world SSD’s. Aldrich also begs folks not to waste resources and make more work for developers and DBAs in dear ISV, you’re keeping me awake nights with your VARCHAR() dates.

And we end with a bit of fin: Paul Nielsen wants us all to have a bit of fun; he has posted an SQL limerick and asks readers to create there own in there once was in Dublin a query.

Michael Bleigh from Intridea, high-end Ruby and Ruby on Rails consultants, build apps from start to finish, making it scalable. He’s written a lot of stuff, available at http://github.com/intridea. @mbleigh on twitter

NoSQL is a new way to think about persistence. Most NoSQL systems are not ACID compliant (Atomicity, Consistency, Isolation, Durability).

Generally, most NoSQL systems have:

• Denormalization
  
• Eventual Consistency
  
• Schema-Free
  
• Horizontal Scale
  

NoSQL tries to scale (more) simply, it is starting to go mainstream – NY Times, BBC, SourceForge, Digg, Sony, ShopWiki, Meebo, and more. But it’s not *entirely* mainstream, it’s still hard to sell due to compliance and other reasons.

NoSQL has gotten very popular, lots of blog posts about them, but they reach this hype peak and obviously it can’t do everything.

“NoSQL is a (growing) collection of tools, not a new way of life.”

What is NoSQL? Can be several things:

• Key-Value Stores
  
• Document Databases
  
• Column-oriented data stores
  
• Graph Databases
  

Key-Value Stores

Speaker’s favorite is Redis because it’s similar to memcached.

• key-value store + datatypes (list, sets, scored sets, soon hashes will be there)
  
• cache-like functions (like expiration)
  
• (Mostly) in-memory
  

Another interesting key-value store is Riak

• Combination of key-value store and document database
  
• heavy into HTTP REST
  
• You can create links between documents, and do “link walking” that you don’t normally get out of a key-value store
  
• built-in Map Reduce
  

Map Reduce:

• Massively parallel way to process large datasets
  
• First you scour data and “map” a new set of dataM
  
• Then you “reduce” the data down to a salient result — for example, map reduce function to make a tag cloud: map function makes an array with a tag name and a count of 1 for each instance of that tag, and the reduce tag goes through that array and counts them…
  
• http://en.wikipedia.org/wiki/MapReduce
  

Other key-value stores:

• Tokyo Cabinet
  
• Dynomite
  
• memcachedDB
  
• Voldemort
  

Document Databases

• MongoDB – Document store that speaks BSON (Binary JSON, which is compact). This is the speaker’s favorite because it has a rich query syntax that makes it close to SQL. Can’t do joins, but can embed objects in other objects, so it’s a tradeoff


• Also has GridFS that can store large files efficiently, can scale to petabytes of data
  
• does have MapReduce but it’s deliberate and you run it every so often.
  
• CouchDB
  
  • Pure JSON Document Store – can query directly with nearly pure javascript (there are auth issues) but it’s an interesting paradigm to be able to run your app almost entirely through javascript.
    
  • HTTP REST interface
    
  • MapReduce only to see items in CouchDB. Incremental MapReduce, every time you add or modify a document, it dynamically changes the functions you’ve written. You can do really powerful queries as easy as you can do simple queries. However, some things are really complex, ie, pagination is almost impossible to do.
    
  • Intelligent Replication – CouchDB is designed to work with offline integration. Could be used instead of SQLite as the HTML5 data store, but you need CouchDB running locally to be doing offline stuff w/CouchDB
    
  
  

Column-oriented store

• Cassandra – Built by Facebook, also used by Twitter
  
• BigTable
  
• Hypertable
  
• HBase
  

Graph Databases

When should I use this stuff?

Don’t look for a single tool that does every job. Use more than one if it’s appropriate, weigh the tradeoffs (ie, don’t have 7 different data stores either!)

NoSQL solves real scalability and data design issues. But financial transactions HAVE to be atomic, so don’t use NoSQL for those.

A good presentation is http://www.slideshare.net/bscofield/the-state-of-nosql.

Using SQL and NoSQL together

You can blend by hand, but the easy way is DataMapper:
Generic, relational ORM (adapters for many SQL dbs and many NoSQL stores)
Implements Identity Map
Module-based inclusion (instead of extending from a class, you just include into a class).

You can set up multiple data targets (default is MySQL, example sets up MongoDB too).
DataMapper is:

• Ultimate Polyglot ORM
  
• simple r’ships btween persistence engines are easy
  
• jack of all, master none
  
• Sometimes perpetuates false assumptions –
  
• If you’re in Ruby, your legacy stuff is in ActiveRecord, so you’re going to have to rewrite your code anyway.
  

Speaker’s idea to be less generic and better use of features of each data store – Gloo – “Gloo glues together different ORMs by providing relationship proxies.” this software is ALPHA ALPHA ALPHA.

The goal is to be able to define relationships on the terms of any ORM from any class, ORM or not
Right now – partially working activeRecord relationships
Is he doing it wrong? Is it a crazy/stupid idea? Maybe.

Example:

Need	Use
Assume you already have an auth system	it’s already in SQL, so leave it there.
Need users to be able to purchase items from the storefront – Can’t lose transactions, need full ACID compliance	use MySQL.
Social Graph – want to have activity streams and 1-way and 2-way relationships. Need speed, but not consistency	use Redis
Product Listings — selling moves and books, both have different properties, products are pretty much non-relational	use MongoDB

He wrote the example in about 3 hours, so integration of multiple data stores can be done quickly and work.

The session on joind.in, with user comments/feedback, is at http://joind.in/talk/view/1320.

Slides are at http://talks.php.net/show/confoo10

“My name is Rasmus, I’ve been around for a long time. I’ve been doing this web stuff since 1992/1993.”

“Generally performance is not a PHP problem.” Webservers not config’d, no expire headers on images, no favicon.

Tools: Firefox/Firebug extension called YSlow (developed by yahoo) gives you a grade on your site.

Google has developed the Firefox/Firebug pagespeed tool.

Today Rasmus will pick on wordpress. He checks out the code, then uses Siege to do a baseline benchmark — see the slide for the results.

Before you do anything else install an opcode cache like APC. Wordpress really likes this type of caching, see this slide for the results. Set the timezone, to make sure conversions aren’t being done all the time.

Make sure you are cpu-bound, NOT I/O bound. Otherwise, speed up the I/O.

Then strace your webserver processs. There are common config issues that you can spot in your strace code. grep for ENOENT which shows you “No such file or directory” errors.

AllowOverride None to turn off .htaccess for every directory, just read settings once from your config file….(unless you’re an ISP).

Make sure DirectoryIndex is set appropriately, watch your include_path. All this low-hanging fruit has examples on the common config issues slide.

Install pecl/inclued and generate a graph – here is the graph image (I have linked it because you really want to zoom in to the graph…)

In strace output check the open() calls. Conditional includes, function calls that include files, etc. need runtime context before knowing what to open. In the example, every request checks to see if we have the config file, once we have config’d we can get rid of that stuff. Get rid of all the conditionals and hard-code “include wp-config.php”. Examples are on the slide.

His tips to change:
Conditional config include in wp-load.php (as just mentioned)
Conditional did-header check in wp-blog-header.php
Don’t call require_wp_db() from wp-settings.php
Remove conditional require logic from wp_start_object_cache

Then check strace again, now all Rasmus sees is theming and translations, which he decided to keep, because that’s the good benefit of Wordpress – Performance is all about costs vs. flexibility. You don’t want to get rid of all of your flexibility, but you want to be fast.

Set error_reporting(-1) in wp-settings.php to catch all warnings — warnings slow you down, so get rid of all errors. PHP error handling is very slow, so getting rid of errors will make you faster.

The slide of warnings that wordpress throws.

Look at all C-level calls made, using callgrind, which sits under valgrind, a CPU emulator used for debugging. See the image of what callgrind shows.

Now dive into the PHP executor, by installing XDebug.

Check xhprof – Facebook open sourced this about a year ago, it’s a PECL extension. The output is pretty cool, try it on your own site, Rasmus does show you how to use it. It shows you functions sorted by the most expensive to the least expensive.

For example, use $_SERVER[REQUEST_TIME] instead of time(). Use pconnect() if MySQL can handle the amount of webserver connections that will be persistent, etc.

After you have changed a lot of the stuff above, benchmark again with siege to see how much faster you are. In this case there is not much gained so far.

So keep going….the blogroll is very slow — Rasmus gets rid of it by commenting out in the sidebar.php file. I’d like to see something to make it “semi-dynamic” — that is, make it a static file that can be re-generated, since you might want the blogroll but links are not changed every second…..

At this point we’re out of low-hanging fruit.

HipHop is a PHP to C++ converter & compiler, including a threaded, event-driven server that replaces apache. Rasmus’ slide says “Wordpress is well-suited for HipHop because it doesn’t have a lot of dynamic runtime code. This is using the standard Wordpress-svn checkout with a few tweaks.”

Then, of course, benchmark again.

The first time you compile Wordpress with HipHop, you give it a list of files to add to the binary, it will complain about php code that generate file names, so you do have to fix that kind of stuff. There’s a huge mess of errors the first time you run it (”pages and pages”), and Rasmus had to patch HipHop (and Wordpress) but the changes in HipHop have been put back into HipHop, so you should be good for the most part.

Check out the errors, lots of them show logical errors like $foo.”bar” instead of $foo.=”bar” and $foo=”bar” instead of $foo==”bar” in an if statement. Which of course is nice for your own code, to find those logical errors.

(Wordpress takes in a $user_ID argument and immediately initializes a global $user_ID variable, which overwrites the argument passed in, so you can change the name of the argument passed in….)

You can also get rid of some code, things that check for existence of the same thing more than once. So it will take a bit of tweaking, but it’s worth it.

There are limitations to HipHop, for example:

• It doesn’t support any of the new PHP 5.3 language features
  
• Private properties don’t really exist under HipHop. They are treated as if they are protected instead.
  
• You can’t unset variables. unset will clear the variable, but it will still be in the symbol table.
  
• eval and create_function are limited
  
• Variable variables $$var are not supported
  
• Dynamic defines won’t work: define($name,$value)
  
• get_loaded_extensions(), get_extension_funcs(), phpinfo(), debug_backtrace() don’t work
  
• Conditional and dynamically created include filenames don’t work as you might expect
  
• Default unix-domain socket filename isn’t set for MySQL so connecting to localhost doesn’t work

and HipHop does not support all extensions — see the list Rasmus has of extensions HipHop supports.

Then Rasmus showed an example using Twit (which he wrote) including the benchmarks. He shows that you can see what’s going on, like 5 MySQL calls on the home page and what happens when you don’t have a favicon.ico (in yellow).

In summary, “performance is all about architecture”, “know your costs”.

Be careful, because some tools (like valgrind and xdebug) you don’t want to put it on production systems, you could capture production traffic and replay it on a dev/testing box, but “you just have to minimize the differences and do your best”.

Why is social media and social networking essential to you and your business? (because it will drive sales, but there’s very few analytics for ROI on social networking and social media)

Relying on advertising is no longer working for print newspapers and television. So why do we think it will work on internet media?

Blogging — you must post 2-4 quality blog posts every week to maintain readership. This takes a lot of work! Content is king.

No matter how cool the technology/product/service is, people still buy more often and more easily from people they know and trust.

Social media is a way to show people that you are an industry expert, and that is how you should use them (not to spam and only say “buy my product”).

If you do not love your job and try to sell it (say, on social networking), you are going to fail, because you are not passionate about it.

Start small, do not promise a lot, it is better to have more to say than to have dead air time (radio analogy).

Social media is all about building good relationships by having good content that people trust.

Lots of people spend a lot of money on their website, but the website is just a vector to show people your content, and the content is the most important thing.

Cross-pollination – I think he means forward on information you learn (like, say, liveblogging!)

Get expert guest bloggers — he did not explain that you can leverage the relationships you form by asking them to blog. We do this with the Log Buffers….

How to make money:
sponsorships
white paper composition
paid articles
consulting
adjunct tie-ins to other related venues
branded blogs

I am personally disappointed because I wanted to learn more, and I feel as though Pythian already uses the knowledge presented — we have great exposure through our blog, and have started really using Twitter, Facebook and other social media sites, etc.

Hansen’s information was good, and absolutely 100% correct, but I felt that for me it was very basic. I would like to know some more advanced topics, like:
- How do you know when you have reached the tipping point?
- How do you convert anonymous readers/followers to people you know, without turning them away because they feel they’re being watched, spammed or don’t want to give out their info to you?
- When does copy/paste to send out your information start to bother people, how do you know how not to do too much?
- How do you convert readers/followers (anon or not) to paid customers without making them feel like you’re all about $$, what about if you have some free content and some paid content, how do you know how much to have?

And now, on to Mark Pilgrim of Google speaking on HTML5.

Timeline
1991 – HTML 1
1994 – HTML 2
1995 – Netscape discovers web, ruins it
1996 – CSS1 + JavaScript
1996 – Microsoft discovers web, ruins it
1997 – HTML4 + EMCAScript1
1998 – CSS2 + EMCAScript2 + DOM1
2000 – XHTML1 + EMCAScript3 + DOM2
2001 – XHTML 1.1
[long break!]
2009 – HTML 5 + ECMA5 + CSS 2.1

HTML5 is not a spec, it’s a marketing term. It’s really HTML5 + CSS3 + JavaScript.

IsHTML5ReadyYet.com and IsHTML5Ready.com are both real websites that give different answers to the question “is HTML 5 ready?”

Semantics
HTML started as a semantic language (until Netscape came along).

New elements (html tags) that do not do anything – they are for semantic use only:

<header> <footer>
<section>
<article>
<nav>
<aside> (pull quotes and such)
<time> (datetime markup)
<mark> (marking up runs of text)
<figure> <figcaption>

Instead of “div class=_____” use these tags….for example:

<body>
  <header>
    <hgroup>
    <h2>page title</h2>
    <h3>page subtitle</h3>
    </hgroup>
   </header>

<nav>
  <ul> Navigation......
     .....
  </ul>
</nav>

<section>
  <article>
   <header>
    <h2>Title</h2>
   </header>
</section>

Caveat: This doesn’t work in IE but there is a workaround…..

This can help blind people navigate better….and bots too!

“Google is just another blind web user with 7 million friends”

Forms
Web forms 2.0
To make a slider from 0-50:

<input type='range' mix='0' max='50' value='0'></input>

To use autofocus:

<input autofocus>

(works in 3 browsers)

Talking about blind users again: “Focus tracking is VERY important if you can’t see. You really need to know where on the page you are, if you start typing what will happen.”

Placeholder text — in a text box, that light text that goes away when you click:

<input type='text' placeholder='click here and this will disappear'>

(works in 2 browsers)

New input types
These are semantic types, do different things in different browsers

<input type='email'> (on the iphone you get a different keyboard, by default you just get a textfield, so these things degrade gracefully if the browser does not support the feature)
<input type='url'> (a browser like <A HREF="http://www.opera.com">Opera</A> can validate a URL for you instead of you doing it yourself!)
<input type='datetime'> (and more...date pickers are tedious)
<input type='file' multiple> (multiple files without using flash!)

For all the inputs HTML5 supports and which browsers support them (Opera is leading the way) search for “HTML5 input support”

Accessibility
ARIA = “accessible rich internet applications”. Alt-text is technology that’s long behind. ARIA does stuff like making tree views accessible. For example, right now with a tree view you have to tab through each item, which is a pain. With code like this:

<ul id='tree1' role='tree' tabindex='0' aria-labelledby='label_1'>
  <li role='treeitem' tabindex='-1' aria-expanded='true'>Fruits </li>
    <li role='group'>
    <ul>
      <li role='treeitem' tabindex='-1'>Oranges</li>
      <li role='treeitem' tabindex='-1'>Pineapples</li>
    </ul>
   </li>
</ul>

….keyboard users can tab to the treeview itself, then use arrow keys to navigate and spacebar to select. This makes selecting an item at the end of a tree view much easier, and also makes it easy to move beyond the tree view without having to press Tab a million times.

Use your favorite search engine for “ARIA accessibility” to learn more.

CSS
Mark threw this image up on the screen:

(image from http://www.zazzle.com/stevenfrank – on that site you can buy this coffee mug or a T-shirt with the design)

Web fonts finally work in CSS3 – you can use more than Times, Courier, Arial, and occasionally Helvetica. This works EVERYWHERE – Chrome, IE, Firefox, Opera, Safari, etc. Well, it’s true that they all use it, but they all have different fonts they support. Read Bulletproof font face for tips on how to get the font you want no matter what browser is used (yes, even IE).

Opacity is easy [author's note - it's just the "opacity" element, see examples at http://www.css3.info/preview/opacity/].

Rounded corners are EASY – Mark’s slide passed too fast for me, so I grabbed an example from http://24ways.org/2006/rounded-corner-boxes-the-css3-way:

.box {
  border-radius: 1.6em;
}

Gradients are easy [author's note -- looks like you need webkit, there's examples at http://gradients.glrzad.com/]

To test CSS3 stuff, use www.css3please.com – “This element will receive inline changes as you edit the CSS rules on the left.”

[Author's note -- while searching I found http://www.webappers.com/2009/08/10/70-must-have-css3-and-html5-tutorials-and-resources/ which is definitely a "must have".]

Canvas
A canvas is a blank slate where you can draw whatever you want, use the canvas tag and id, width and height attributes, everything else is javascript. Pretty awesome. [Author's note -- Mark had examples but I did not have time to capture them. I did find a nice tutorial at https://developer.mozilla.org/en/Canvas_tutorial.]

Multimedia
Video with no flash! YouTube has HTML5 integration. Here’s sample code of how to do movies in HTML5:

<video src='movie.ogv' controls></video>
<video src='movie.ogv' loop></video>
<video src='movie.ogv' preload='none'></video>  -- don't preload the movie
<video src='movie.ogv' preload='auto'></video>
<video src='movie.ogv' autoplay></video> -- if you don't have this you don't do evil autoplay....

Multimedia is in the DOM and responds to CSS effects, such as reflection:

<video src='movie.ogv' loop style='webkit-box-reflect: below 1px;'></video>

(this code might be wrong, the slide flipped fast)

Of course the problem — codecs. Right now, .ogv and .mp4 (h264).

Audio inline too, same problem — only .oga and .mp3:

<audio src ='podcast.oga' controls></audio>

Geolocation
IsGeolocationPartofHTML5.com is a real site, go to it to get the answer.
Geolocation demos — very much the same, find your location and display it. Simple but cool.

Cache manifest
Get everything you need for offline usage…

<html manifest='another-sky.manifest'>

CACHE MANIFEST
/avatars/zoe.png
/avatars/tamara.png
/scripts/holoband.jpg

search for “google for mobile HTML5 series” – good series of articles on using this stuff.

HTML 5 has much more
Local storage
Web workers
Web sockets (2way connections, like raw tcp/ip cxns over the web)
3D canvas (webgl)
Microdata (enhanced semantics)
Desktop notifications
Drag and Drop

Learn more:
whatwg.org/html5
diveintohtml5.org

I arrived at Omni Mandalay Hotel on Sunday evening with Dan Norris. I was flying through Chicago and it turned out that Dan was on the same flight and only few rows behind me. Small world.

Preparations for the conference were very chaotic on my part and, of course, I didn’t have either of my presentations ready. I was very stressed and getting sick as well — it looked like a complete disaster waiting to happen. I’d like to say that I was feeling like Doug Burns as he often managed to get sick just before a conference. Of course, I worked on my slides for the last few days as well as on the flight and presentation was slowly getting there but boy was I tired!

I quickly said hello to the crowd in the bar on the way to my room and rushed away to do some more damage to my slides. And then I had a brilliant idea — I could still see one of my best mates and do something good about my presentation! I asked Doug if he was interested in the preview (he probably wasn’t interested but he couldn’t say it to me) especially that my session wasn’t on his original agenda. Of course, that would mean that he had to leave a bunch of other good friends and spend some time tete-a-tete. Knowing Doug, this is some of the hardest thing to ask from him but it shows how good of a friend he is! (Plus, everyone thinks that he is anti-social anyway. Shhhh!)

Doug has made my day — while he provided lots of ideas and feedback on few things that I was lucking, he generally approved the idea and confirmed that it wasn’t totally crazy. I guess that was all I needed back then and Doug knew how nervous I was about it. (Thanks mate!)

So I called Sunday a day very early and went to bed before midnight. I really needed some sleep. Woken up by the alarm at 5AM (I woke up few times during the night looking at the clock — making sure I didn’t sleep through) and slides were ready just before lunch. I even managed to do a test run and it took 65 minutes — a wee bit too long for one hour session. But it was good test and I knew I had to be just a bit more concise in few parts.

Mi morning was very productive. Unfortunately, I missed the opening keynote from Tom Kyte. Such a pity! If what Doug wrote is true, Tom was talking about the mistakes we make *because* of our experience and our assumptions. This was exactly one of the points I was making in my Battle Against Any Guess — experience is danger. I wish I could see Tom’s example. Oh well, maybe another time.

I managed to attend half of the Richard Foote’s session on indexes but my mind was far away — with my own slides. Though, I did manage to focus on bitmap indexes part and the myth of bitmap indexes not working well for columns with high cardinality. Very interesting conclusions. I’m still wondering how much overhead updates will do to such bitmap index.

After lunch, it was my turn. I ordered few copies of the latest OakTable book — Expert Oracle Practices: Oracle Database Administration from the Oak Table — that I co-authored with the bunch of other Oakies. I contributed chapter 1 in the book titled just like my presentation — Battle Against Any Guess. The plan was to give a copy away during the presentation and do a draw for another one at the end of the session. I was so nervous that I forgot about it until the end of the session so I just did a draw for two copies. The lucky winners were Lynn-Georgia Tesch and Surendra Anchula. Congratulations! For the rest of you who left the contact details — please stay tuned and we’ll organize few things online.

Now the main topic of this post — my presentation. What’s unusual about this session is that it’s not some technical stuff that I usually do but a more conceptual and motivational talk. Could I pull it off? Well, I think it went fairly well in general even though I did identify few rough places and my lack of English language mastering. Might need to work a little bit more on the flow of the presentation.

We had quite a few good laughs. Later, people in the next hall were asking about it and Dan was making the jokes on the stage so it must have been loud. Anyway, I think nobody fell asleep and I managed to get people thinking about the topic. I received many “thank you” notes yesterday and compliments on a good session so by the end of the day I was more and more pleased. Thanks everyone for attending and especially big thanks to those of you who brought to my attention examples from their own battles. If you have more to discuss — contact me by email (my last name) {at} pythian.com.

Thanks to Marco Gralike for recording some fragments and sharing them. I think he has more to come.

This is the introductory couple minutes. You can definitely notice how nervous I am starting on the stage:

Solving the wrong problem example:

That’s all for now. Stay tuned — more to come.