Yes Luke has always done oustanding work for us as most of the guys do.

Much appreciated, Matt

John did a great job on CR310761 - he ws helpful and completed the task quickly.

yes great job, thanks.

Shakir is an amazing DBA

I'd just like to recognize Andre Araujo's amazing work for us in query optimizations. I like to refer to Andre as the "SQL Magician" because he is able to turn queries that take several hours, even days, into ones that a fraction of the time. Most recently, we needed to run an ETL job to get at some really important customer data on a table with near 1 billion records. He was able to extract the data for us in a couple hours and come up with some amazingly fast, albeit complex, queries to perform analysis on the data. I am continually amazed by Andre's deep understanding of how Oracle works and how he is able to apply it to help us out. Thank you Pythian for having such talented DBA's on your staff!

The restore of the elab database was able to be completed very quickly due to Pythian working on the Restore documents in conjunction with Working Instructions MDS was verifying. With the restore templates being made available, the restore happened within 2-3 hrs rather than a much longer downtime for the users, which was very beneficial for the BAS business unit.

I just wanted to thank Pythian for what they did today. You succesfully restored a broken Oracle server within a couple of hours and handled this case in a very professional way saving the day for a bunch of developers.

Special thanks go to Virgil Glavan who immediately answered my call and to Jan Polnicky who cured our server.

Thank you guys.

Gabor Sebestyen

EU Edge, LLC

Budapest, Hungary

Thanks Catherine, David and Sudeep for excellent job in making this maintenance go smoothly.

Susheel

We are very satisfied by the service given to us by Andre and Shakir in support of our recent data quality and reorganization initiative. Thanks!

Pythian did a very good job to fix our Oracle problem. I sent an e-mail to their support and got a call right away. Thanks to the sales team and the DBA, Karun, who fixed our very serious situation in a short timeframe.

I would like note the great work by Pythian on improving the performance and the stability of our production environment.

The Pythian team has helped us in identifying issues and worked with us to resolve them.

We had replaced our production servers and thought to be a long arduous task but Warner made that transition quick and simple. We were done several hours before the expected completion time which made our clients very happy. Another major task was when we added our new SAN array. Mohammed worked with us to make recommendations and outlined the steps needed to have the optimal configuration. He worked with us during several early hours and kept me up to date during the transition. Warner and Mohammed were very knowledgeable about the tasks and kept me up to date during the upgrade period which was very important since many parties were involved. They are both professional and great to work with.

I would highly recommend Warner, Mohammed and the Pythian team.

Thanks,

James Chong

IT Director

Don Seiler has done an tremendous job offering LIM support over the last few months. He seems to always be readily available to answer our questions...even after business hours. We really appreciate his efforts.

I just wanted to let you know how much I appreciate the work that was done for us last week by David, Vasu, Balraj and Murali. We were short-staffed, and they came through every time. We asked for their help on many issues, gave them little notice on when the work needed to be done, and they very promptly reacted and performed the work accurately and rapidly. They are a great team to work with, and we so appreciate their efforts. David also does a fantastic job on keeping us informed and communicating with us and the rest of the Pythian staff.

Hi,

I just wanted to let you know that we went live successfully for both internal and external websites.

I want to thank Pythian for the great work and excellent support. Your level of expertise and response time is outstanding.

Keep up the Great work!

Regards

Nithin Shetty

McGrath RentCorp Inc

Pythian did a very good job for cloning this environment for the first time.

Nicklas and Gerry did exceptional work for us, reacting swiftly to our production problems and restoring order to our databases.

I just wanted to compliment Gerry on his thoroughness and clarity in the documentation on his work in CR 254247. Thanks!

Vasu and Balraj did a great job especially with all the bugs they encountered while applying the RUP6 patch.

Last night I made a critical error at 4:30pm. I basically ran a command that destroyed a large number of files on a server that was 2 days out from go live. The back up server was still in the test phase and no full back up had been done on the system.

If you're wondering what I did I ran a simple command to give high level access to a sub directory of a web directory to allow a user to complete their job. This was to be a temporary solution until we could lock down the directory. As root I ran "chmod 777 mydirectory / -R". So basically I recursively reassigned 777 to the root directory not the directory I intended.

Okay so this was all bad things I did and I knew it, but it did not change where I was at. Less then 2 days from go live on a production product, that was new to my entire team, and we needed the final two days to complete tasks on these servers such as backups and redundancy testing.

So from 4:30-7:00 myself and another senior team member troubleshot the back up server for this application. It was a LAMP install, rh4 linux, apache 2.2, mysql 5, php 5. It went up originally very simply, but the on the back up server php would not talk to the database.

So that everyone knows we do not work with Pythian currently for any support on our mysql, php, apache servers. They support our oracle servers only.

At a little after 7pm we contacted Greg Leger at Pythian for support on the server, he assured me he had senior team members to support me if I needed them. A few minutes later Singer contacted us. At 7:30pm we were back online and ready to go home!

The issue was simple but neither myself or the other team members here could see the issue. Turned out there were two sets of mysql running, when we researched this originally we searched for "rpm -qa | grep MySql" which is the way the new release is spelled. It turns out the orignal release on the server was "mysql" or "rpm -qa | grep mysql".

Thanks, for bailing me out on such short notice on systems you had not worked with previously.

Rob Hamel:

If you have an employee-of-the-month program, I'd like to nominate Marc for it. His work this weekend went above and beyond the call of duty. It is difficult finding competent help, let alone great help. It is tough to find a vendor that will care about your systems as if they were their own. Pythian appears to be one of those shining stars. I know we pay you well for your services, but I'd like to note my thanks to Marc for doing his job so well in general and especially this weekend.

Keep up the great work.

As always the tasks and results are exceptional and show extreme dedication.

I want to extend my thank you to Augusto and Nick who helped us promptly to identify and fix the problem we had with one very weird behavior on version 4.1.11 where BOOLEAN type is not correctly mapped to TINYINT(1) and this broke the replication on our second most important server.

Once more - thank you guys.

Martin

Thanks, Marcelo, for taking care of two crucial services in our environment.

The task of upgrading our oracle instances on our production box was done in a timely and professional manner. Thank you.

I'd like to commend Sheeri in the strongest possible language for her support last week with our rapid rollout of mysql proxy. Her dedication is greatly appreciated and reflects well on Pythian.

I just wanted to let you know we have been hammering your team with requests hard, especially since last Thursday. This is primarily due to our GSS project implementation. In addition to these requests, your team has also had to deal with a few production issues. They have been working very well and keeping a great attitude. Please thank them for me.

Re: CR#207213

I was pleased with the response time

and the service given.

Thanks a lot!

We just got back from Pythian Day, an annual (for us) event in which we go visit the Pythian offices, meet the people on our team, and hear about the latest events in the Pythian universe. Once again, we had a good time and found it very useful.

Sometimes just sitting with your team in front of a whiteboard for an hour and getting to know them in-person and getting to share knowledge in-person can be a really useful exercise, and I think it definitely was in this case. It's always nice to put a face to a name, or faces to names in this case.

As usual, the Pythian folks were gracious hosts and the trip was a success.

One of the most blood pressure testing exercises we go though now is our quarterly security patch upgrades. Our web applications process 25,000+ new sessions an hour and every minute we're down you can feel our clients fidgeting. We have 30+ databases and it takes 3 weekends to complete the updates. Tonight we hit the final critical nine.

The work was assigned to Aleksey Sintyurev who I believe is one of the newer DBAs on staff. I knew he was very senior but we had never really worked with him before. He had also just started working this morning with a monthly process that Pythian helps us out with on several of our databases. It was new to him, fairly complicated and he was running into problems. Our SLA mandated window to start and complete the patching was rapidly approaching and since 3 of the databases to be updated are beyond critical I have to admit I freaked out enough to call Paul. Literally within minutes I had 3 additional senior DBAs on the line and team members familiar with that monthly process took it over and freed Aleksey up to concentrate on our patch process.

First, thanks to Paul and the other team leads (Alex Gorbachev and Greg Leger) and Andrey Goryunov in Australia for great handling of a panicking client who was having a meltdown!

Second I can only say excellent job to Aleksey! You were not only under the normal pressure of delivering the level of DBA support we all expect from Pythian, but the added Pythian senior corporate level scrutiny that my panic attack brought. Nicely done!

Paul Moen performed like a champion when our master databases were melting down off-hours. It really proved the value of having DBAs on-call around the world!

Keeping in mind that he's not one of our regular team and that our server setup is quite unusual -- he came up to speed very, very quickly. He was up with access on the right machines discerning the nature of the problem when we first got in touch over IM.

He was able to move quickly and get multiple restore options going in parallel, choosing the best one when it became clear which that was. Paul got the app back up as soon as reasonably possible (by using the old slave) while the new master was being made ready for production. Later he added some suggestions about ways to have higher availability for the master in EC2.

Thanks Pythian!

MDS has been live with Pythian DBAs for 3 months now on our Global ERP. Team Lead Sylvain Fontaine conducted a smooth transition from the former support vendor and within weeks has initiated proactive maintance. Vasu has implemented changes that have improved performance and reduced overhead bringing direct savings to MDS. Team 5 has also begun clearing outstanding issues that have posed risk to our system.

MDS Functional and Development Managers agree that the team is providing fast, 24x7 and easily engaged support.

I would like to state how pleased I am with the professionalism, knowledge and work ethic the Pythian Group analysts have shown in my dealings.

Whether it has been grappling with STREAMS issues, setting up environments monitoring or backup strategies. They have been excellent.

These views are my own and may or may not reflect those of Vale Inco.

Sheeri provided excellent support on an effort that needed very quick turn around. Sheeri was able to assist in very tangible ways in helping us to determine the database architecture all while workinging a an environment that was almost entirely new.

This past weekend was very busy. Our Penn Community environment was successfully upgraded to Oracle 10g. While this may sound like a commonplace task, when it comes to Penn Community, it certainly is not. This is one of our most critical environments. A multitude of applications rely on this database. Needless to say, we all had to have our act together when we upgraded PCOM from 9.2.0.6 to 10.2.0.3.

Shervin did an excellent job with planning, testing, and carrying out the production upgrade. He left no stone unturned and actually delivered the environment to us ahead of schedule. This upgrade consisted of about 25 post-upgrade testers alone who had to get up at 1:30 AM this past Sunday to certify all of those production apps that use Penn Community.

The result is that we have had NO reported problems since going live with 10g on Sunday. Sr. management all breathed a sigh of relief when I sent out the all clear on Sunday. It is a pleasure working with Shervin.

Well done Shervin!

I'd like to comment on the overall work that was done by several members of the team first as the preparation for the downtime and second during the maintenance.

Virgil and Alex were very helpful all the time and follow on the guidelines that were directed by me to make sure that all parties involved into the maintenance are in synch.

I also appreciate the fact that guys did extra analyses to make sure that the systems are working fine and all scheduled task that were temporarily disabled are back to normal.

Thank you.