I just wanted to thank all of the Pythian team for their professionalism and high standard of work. We resolved some big issues here at Casella with all their help.

I look forward to working with Pythian in the future, as we have identified some items that may require their help again.

Thanks again.

Gwen did an awesome job on restoring and recovering the database.

Andrey and Gleb did a great job investigating the issue and rebuilding the database in limited amount of time. I am totally happy with their service.

yes great job, thanks.

Yes Luke has always done oustanding work for us as most of the guys do.

Much appreciated, Matt

Thanks Catherine, David and Sudeep for excellent job in making this maintenance go smoothly.

Susheel

I just wanted to thank Pythian for what they did today. You succesfully restored a broken Oracle server within a couple of hours and handled this case in a very professional way saving the day for a bunch of developers.

Special thanks go to Virgil Glavan who immediately answered my call and to Jan Polnicky who cured our server.

Thank you guys.

Gabor Sebestyen

EU Edge, LLC

Budapest, Hungary

The restore of the elab database was able to be completed very quickly due to Pythian working on the Restore documents in conjunction with Working Instructions MDS was verifying. With the restore templates being made available, the restore happened within 2-3 hrs rather than a much longer downtime for the users, which was very beneficial for the BAS business unit.

I'd just like to recognize Andre Araujo's amazing work for us in query optimizations. I like to refer to Andre as the "SQL Magician" because he is able to turn queries that take several hours, even days, into ones that a fraction of the time. Most recently, we needed to run an ETL job to get at some really important customer data on a table with near 1 billion records. He was able to extract the data for us in a couple hours and come up with some amazingly fast, albeit complex, queries to perform analysis on the data. I am continually amazed by Andre's deep understanding of how Oracle works and how he is able to apply it to help us out. Thank you Pythian for having such talented DBA's on your staff!

John did a great job on CR310761 - he ws helpful and completed the task quickly.

We are very satisfied by the service given to us by Andre and Shakir in support of our recent data quality and reorganization initiative. Thanks!

Catherine did excellent on this CR. Her notes are very clear and shows her knowledge of the subject. She is prompt with communicating changes. I am very happy that she will be supporting the Team 5 and has excellent knowledge of Oracle Financials.

Andrew has done an excellent job on this project. He started working on this project late but able to manage it successfully. He has kept the team updated with any changes to the scope or time and made sure that we got all the action items taken care. I am glad that we have him manage the project by co-ordinating with resources from different continent.

Thanks Andrew for contribution.

Pythian did a very good job to fix our Oracle problem. I sent an e-mail to their support and got a call right away. Thanks to the sales team and the DBA, Karun, who fixed our very serious situation in a short timeframe.

Pythian were recently engaged to take a lead role in a high end infrastructure build project at Telstra. Our requirements were a combination of specific technical skills in clustered HA/DR environments and the ability to lead and communicate (verbally and documentation) across groups in a confident and clear manner. Alex's unique skill-set was an absolute must have on the project and he proved his worth repeatedly during the build process with fantastic troubleshooting skills and his ability to communicate complex requirements to demanding support teams. The communication component in today's marketplace is often overlooked and I believe Pythian's focus in this area coupled with their deep technical skill-base makes them a unique player in the professional services field. I would recommend Alex and Pythian highly and look forward to working with them again in the future.

I would like note the great work by Pythian on improving the performance and the stability of our production environment.

The Pythian team has helped us in identifying issues and worked with us to resolve them.

We had replaced our production servers and thought to be a long arduous task but Warner made that transition quick and simple. We were done several hours before the expected completion time which made our clients very happy. Another major task was when we added our new SAN array. Mohammed worked with us to make recommendations and outlined the steps needed to have the optimal configuration. He worked with us during several early hours and kept me up to date during the transition. Warner and Mohammed were very knowledgeable about the tasks and kept me up to date during the upgrade period which was very important since many parties were involved. They are both professional and great to work with.

I would highly recommend Warner, Mohammed and the Pythian team.

Thanks,

James Chong

IT Director

Pythian were recommended to me by a work colleague who had used their services previously. We started with a face to face meeting to discuss covering a month of leave, and signed up for 3 months. Within weeks it became obvious that they were not your average professional services organisation. As our relationship grew they understood what sort of work we were happy for them to do without our signoff, and what we wanted to have involvement with. This led to an immediate improvement in the state of health of our databases - Pythian were rapidly completing the sort of work that had previously been overlooked by other professional services organisations.

We have now engaged Pythian on an on-going basis and I would recommend them as highly as any other organisation that I've dealt with.

Dedicated and result driven, Lukas at Pythian consistently aims to deliver what our business requires. Overall Pythian have great flexibility and team spirit, which helps in our dynamic and changing environment.

Don Seiler has done an tremendous job offering LIM support over the last few months. He seems to always be readily available to answer our questions...even after business hours. We really appreciate his efforts.

I just wanted to let you know how much I appreciate the work that was done for us last week by David, Vasu, Balraj and Murali. We were short-staffed, and they came through every time. We asked for their help on many issues, gave them little notice on when the work needed to be done, and they very promptly reacted and performed the work accurately and rapidly. They are a great team to work with, and we so appreciate their efforts. David also does a fantastic job on keeping us informed and communicating with us and the rest of the Pythian staff.

Hi,

I just wanted to let you know that we went live successfully for both internal and external websites.

I want to thank Pythian for the great work and excellent support. Your level of expertise and response time is outstanding.

Keep up the Great work!

Regards

Nithin Shetty

McGrath RentCorp Inc

Pythian did a very good job for cloning this environment for the first time.

I just wanted to compliment Gerry on his thoroughness and clarity in the documentation on his work in CR 254247. Thanks!

Nicklas and Gerry did exceptional work for us, reacting swiftly to our production problems and restoring order to our databases.

One of the really nice things about how our hours work is the ability to use them for unexpected occurances. Our company was purchased awhile back and there was never any interest by the new owners in expanding the usage of Pythian to their databases. We just got off of a couple hour call with them. A really large table had been accidentially truncated 24 hours ago. They could get it back from tape, but ran into issues trying to get it restored. The import crashed and was in the process of rolling back and no one there had a clue how long it was going to take before they could try again or even if they were going to be able to successfully restore it.

As soon as they called us looking for any thoughts I contacted Shakir, our team lead, and after explaining the situation he brought in Christo. We had the person there establish a WebEx session and share out his desktop. It took Christo a few minutes to orient himself and then to figure out that it was going to take at least another 9 hours to complete the rollback. Long story short, Christo came up with two very workable options for them, tested them out and documented them. Because of auditing requirements they still need to do the physical work, but they were very happy and said they learned a lot. I'm suspecting that those couple hours could well generate some interest in having Pythian take over their DB's as well.

Thanks again Christo. Good job!

Kudos to Marc Fielding! Our tech group has been fighting a really weird data loss problem on one of our Linux database servers for several weeks now. It was related to Oracle writing a flat file out to a Samba mountpoint aimed at one of our Windows servers. It took Marc only a couple hours to diagnose the problem and have us running 100% again. These guys aren't just great DBA's!

Karun always does a good job for us. He is wonderful and everyone on our DBA team at LSS enjoys interacting with him. I trust him to follow up on any and all outstanding issues. Plus, I swear he is psychic and deals with erratic mood swings diplomatically.

Vasu and Balraj did a great job especially with all the bugs they encountered while applying the RUP6 patch.

We had been having intermittent indexing issues with one of our internal search applications. Our team lead Augusto Bott managed to pick out an obscure replication issue relatively quickly. His quick identification of the problem allowed us to bring a fast resolution to the application and bring things back onto the right track. This has been a reoccuring issue for us for about 6 months and Augusto figured it out in under 15 minutes. Thanks for being on the ball.

October is a stressful month at MDS as we gear up for our fiscal year end. Once again Pythian came through a with great game plan. Team Lead, David Ashlock, provided a proactive monitoring approach tailor made to suit our Finance team's needs. David's innovations continue to keep MDS's Global ERP running with the minimum downtime and maximum performance. His attention to detail and his ability to communicate make doing business with Pythian a pleasure.

Last night I made a critical error at 4:30pm. I basically ran a command that destroyed a large number of files on a server that was 2 days out from go live. The back up server was still in the test phase and no full back up had been done on the system.

If you're wondering what I did I ran a simple command to give high level access to a sub directory of a web directory to allow a user to complete their job. This was to be a temporary solution until we could lock down the directory. As root I ran "chmod 777 mydirectory / -R". So basically I recursively reassigned 777 to the root directory not the directory I intended.

Okay so this was all bad things I did and I knew it, but it did not change where I was at. Less then 2 days from go live on a production product, that was new to my entire team, and we needed the final two days to complete tasks on these servers such as backups and redundancy testing.

So from 4:30-7:00 myself and another senior team member troubleshot the back up server for this application. It was a LAMP install, rh4 linux, apache 2.2, mysql 5, php 5. It went up originally very simply, but the on the back up server php would not talk to the database.

So that everyone knows we do not work with Pythian currently for any support on our mysql, php, apache servers. They support our oracle servers only.

At a little after 7pm we contacted Greg Leger at Pythian for support on the server, he assured me he had senior team members to support me if I needed them. A few minutes later Singer contacted us. At 7:30pm we were back online and ready to go home!

The issue was simple but neither myself or the other team members here could see the issue. Turned out there were two sets of mysql running, when we researched this originally we searched for "rpm -qa | grep MySql" which is the way the new release is spelled. It turns out the orignal release on the server was "mysql" or "rpm -qa | grep mysql".

Thanks, for bailing me out on such short notice on systems you had not worked with previously.

Rob Hamel:

If you have an employee-of-the-month program, I'd like to nominate Marc for it. His work this weekend went above and beyond the call of duty. It is difficult finding competent help, let alone great help. It is tough to find a vendor that will care about your systems as if they were their own. Pythian appears to be one of those shining stars. I know we pay you well for your services, but I'd like to note my thanks to Marc for doing his job so well in general and especially this weekend.

Keep up the great work.

Just wanted to drop a quick note and say that I think that Isabel is likely the most cheerful/happy/fun to work with DBA i have come across! Its nice to work with someone who loves their jobs so much it shows at that level! I look forward to working with her in the future!

As always the tasks and results are exceptional and show extreme dedication.

I'm an experienced DBA, almost 10 years of senior experience. I had a major stumper today, and after banging my head for a few hours, Sheeri was able to help me in under 10 minutes. This just goes to prove the adage that everyone needs an extra pair of eyes. Pythian is proving to be a valuable partner to PalominoDB.

I want to extend my thank you to Augusto and Nick who helped us promptly to identify and fix the problem we had with one very weird behavior on version 4.1.11 where BOOLEAN type is not correctly mapped to TINYINT(1) and this broke the replication on our second most important server.

Once more - thank you guys.

Martin

July has been an extremely busy month for Register.com in terms of project work - both planned and unplanned. Pythian in its usual grace has supported us beautifully throughout, working with us under some difficult deadlines and coming out smelling like a rose. We also performed an extremely complex migration this past weekend from our old Solaris based Oracle Financials environment to a shiny new Linux one, and because of the detailed planning that Sylvain and his team of Balraj and Vasu produced, our project has literally gone off without a hitch - a truly incredible feat given the scope. We are all amazed (but we shouldn't be - this is just Pythian doing its thing).

Thanks!

With 30+ databases to patch and upgrade each quarter, it's nice to have a team who can reliably work on the 4th of July while we enjoy our holiday.

Thanks Virgil, Christo and Pythian team members who supported our upgrades this quarter.

Just wanted to leave a comment about the excellent work Nick Westerlund and Augusto Bott have done for us, particularly in the past two days when our application traffic has increased significantly. Augusto has always done good work and Nick, apparently a new addition to the team, has come on and hit the ground running.

Both of their excellent work, particularly in critical times on short notice, is very much appreciated.

Thanks, Marcelo, for taking care of two crucial services in our environment.

I wanted to pass along some feedback about David. I have heard nothing but good things about how he has been handing the work that Pandu was doing and he has by far exceeded any expectations we may have had about how this transition would go.

In fact, I would probably say that things are working even better than they were with him involved.

Please pass along our appreciation to David for the thoroughness of his knowledge transfer work along with his continuing work on our RT tickets.

thanks

The task of upgrading our oracle instances on our production box was done in a timely and professional manner. Thank you.

This agent deployment was done in the most *lazy dba* fashion (all scripted and automated ) as we love it here at CCI ... good job!

I'd like to commend Sheeri in the strongest possible language for her support last week with our rapid rollout of mysql proxy. Her dedication is greatly appreciated and reflects well on Pythian.

I just wanted to let you know we have been hammering your team with requests hard, especially since last Thursday. This is primarily due to our GSS project implementation. In addition to these requests, your team has also had to deal with a few production issues. They have been working very well and keeping a great attitude. Please thank them for me.

Re: CR#207213

I was pleased with the response time

and the service given.

Thanks a lot!

The MDS Infrastructure Consolidation Team has been working with Pythian for the past 3 months on site-specific consolidation efforts.

Karun Dutt, our dedicated Pythian DBA, has consistently performed over and above expectations to assist MDS with our hardware upgrade from Solaris to Linux, as well as an Oracle upgrade from 8i to 11g; trouble-shooting and finding the required solutions, often to fit a square peg in a round hole. He always makes himself available, even on very short notice, for team meetings and interfaces very well with the internal MDS technical team.

We have been very pleased with the work performed by Pythian, and Karun specifically, and look forward to continuing our engagement with them during our ongoing Global Consolidation efforts.

thanks to Alex, Vamsi and all members of Pythian team (including Singer/SA's) for the excellent handling of this critical event - thorough preparation, smooth execution and timely communications!

-- george

We just got back from Pythian Day, an annual (for us) event in which we go visit the Pythian offices, meet the people on our team, and hear about the latest events in the Pythian universe. Once again, we had a good time and found it very useful.

Sometimes just sitting with your team in front of a whiteboard for an hour and getting to know them in-person and getting to share knowledge in-person can be a really useful exercise, and I think it definitely was in this case. It's always nice to put a face to a name, or faces to names in this case.

As usual, the Pythian folks were gracious hosts and the trip was a success.

One of the most blood pressure testing exercises we go though now is our quarterly security patch upgrades. Our web applications process 25,000+ new sessions an hour and every minute we're down you can feel our clients fidgeting. We have 30+ databases and it takes 3 weekends to complete the updates. Tonight we hit the final critical nine.

The work was assigned to Aleksey Sintyurev who I believe is one of the newer DBAs on staff. I knew he was very senior but we had never really worked with him before. He had also just started working this morning with a monthly process that Pythian helps us out with on several of our databases. It was new to him, fairly complicated and he was running into problems. Our SLA mandated window to start and complete the patching was rapidly approaching and since 3 of the databases to be updated are beyond critical I have to admit I freaked out enough to call Paul. Literally within minutes I had 3 additional senior DBAs on the line and team members familiar with that monthly process took it over and freed Aleksey up to concentrate on our patch process.

First, thanks to Paul and the other team leads (Alex Gorbachev and Greg Leger) and Andrey Goryunov in Australia for great handling of a panicking client who was having a meltdown!

Second I can only say excellent job to Aleksey! You were not only under the normal pressure of delivering the level of DBA support we all expect from Pythian, but the added Pythian senior corporate level scrutiny that my panic attack brought. Nicely done!

Paul Moen performed like a champion when our master databases were melting down off-hours. It really proved the value of having DBAs on-call around the world!

Keeping in mind that he's not one of our regular team and that our server setup is quite unusual -- he came up to speed very, very quickly. He was up with access on the right machines discerning the nature of the problem when we first got in touch over IM.

He was able to move quickly and get multiple restore options going in parallel, choosing the best one when it became clear which that was. Paul got the app back up as soon as reasonably possible (by using the old slave) while the new master was being made ready for production. Later he added some suggestions about ways to have higher availability for the master in EC2.

Thanks Pythian!

MDS has been live with Pythian DBAs for 3 months now on our Global ERP. Team Lead Sylvain Fontaine conducted a smooth transition from the former support vendor and within weeks has initiated proactive maintance. Vasu has implemented changes that have improved performance and reduced overhead bringing direct savings to MDS. Team 5 has also begun clearing outstanding issues that have posed risk to our system.

MDS Functional and Development Managers agree that the team is providing fast, 24x7 and easily engaged support.

I would like to state how pleased I am with the professionalism, knowledge and work ethic the Pythian Group analysts have shown in my dealings.

Whether it has been grappling with STREAMS issues, setting up environments monitoring or backup strategies. They have been excellent.

These views are my own and may or may not reflect those of Vale Inco.

Sheeri provided excellent support on an effort that needed very quick turn around. Sheeri was able to assist in very tangible ways in helping us to determine the database architecture all while workinging a an environment that was almost entirely new.

This past weekend was very busy. Our Penn Community environment was successfully upgraded to Oracle 10g. While this may sound like a commonplace task, when it comes to Penn Community, it certainly is not. This is one of our most critical environments. A multitude of applications rely on this database. Needless to say, we all had to have our act together when we upgraded PCOM from 9.2.0.6 to 10.2.0.3.

Shervin did an excellent job with planning, testing, and carrying out the production upgrade. He left no stone unturned and actually delivered the environment to us ahead of schedule. This upgrade consisted of about 25 post-upgrade testers alone who had to get up at 1:30 AM this past Sunday to certify all of those production apps that use Penn Community.

The result is that we have had NO reported problems since going live with 10g on Sunday. Sr. management all breathed a sigh of relief when I sent out the all clear on Sunday. It is a pleasure working with Shervin.

Well done Shervin!

I'd like to comment on the overall work that was done by several members of the team first as the preparation for the downtime and second during the maintenance.

Virgil and Alex were very helpful all the time and follow on the guidelines that were directed by me to make sure that all parties involved into the maintenance are in synch.

I also appreciate the fact that guys did extra analyses to make sure that the systems are working fine and all scheduled task that were temporarily disabled are back to normal.

Thank you.