SandRidge deployed server virtualization over two years ago and has virtualized 60% of its application environment (including test and production applications). Today, over 140 machines are running on nine physical hosts supported by NetApp storage. The complex environment made it challenging to troubleshoot performance problems using the existing element management tools. When performance problems occurred, SandRidge consulted four different tools to resolve issues, which took a few systems engineers and up to a day to review data and find the “needle in the haystack.”

Cameron Han is the Infrastructure Operations Supervisor at SandRidge Energy, Inc.  I’m thrilled to share that he’s presenting his story at VMworld on how SandRidge’s IT department was able to improve their infrastructure performance using Akorri BalancePoint.

Here’s what he had to say about the product.

“BalancePoint is the only tool we found that can decipher our virtual machines, storage, and operating systems together, and we like that it will proactively notify us of potential performance bottlenecks.”

Check out the Session @ VMworld

Who: Cameron Han, Senior Systems Engineer at SandRidge Energy, Inc.

What: SandRidge Energy Increases IT Virtualization Efficiency

When: August 31, 2010 at 3:30 p.m.

Where: VMworld 2010, Moscone Center, San Francisco

Session ID: V18113, Track: Virtualization 101

If you can’t make the session, you can read the case study here.

See Us at VMworld, Booth #1331

• See a demo of BalancePoint v3.5 that Predicts Performance Issues.
• Get a cool t-shirt and wear it for a chance to win a new Apple iPad or GPS.
• Get a free drink (or two or three) during the VMworld “Hall Crawl”, Tuesday, August 31 from 4-6 p.m.

Stop by the Akorri booth and visit us!  Tell me you read my blog for an additional chance to win.

1. You bought more memory for your ESX server and it didn’t fix the performance problem.
2. VMware DRS causes you performance problems every time it moves a virtual machine.
3. VM performance problems come and go and you can’t figure out why.
4. You haven’t turned on VMware DRS because you’re afraid of what it will do to your performance.
5. The SSDs you purchased are not delivering the performance you expected.
6. You ran into poor performance due to running a VM on a LUN with the wrong RAID level.
7. The new array you purchased doesn’t perform as expected in your virtual environment
8. You spent days debugging a VMware related issue using data from several different tools.
9. Your application owners complain that their application performance on VMware is worse than when it was run on a dedicated physical server.
10. Application performance problems are experienced by end-users and you found it’s not the network or the application.

Do you have one we didn’t mention?   I’d love to hear about it and add it to the list.

Akorri BalancePoint: The Dynamic Data Center Optimization Tool

BalancePoint: A fresh new tool for your IT arsenal! This product is a lifesaver for those trying to actively manage their ever growing cloud infrastructure.

Over the past few days our Varrow engineers have been involved with training surrounding a product that the company is extremely excited about! Our engineers have been through introductions with the company and its product BalancePoint. BalancePoint “is a third generation software product that is the only analytics based solution for server to storage performance analysis and capacity planning for both physical and virtual environments. Think of Akorri as the GPS of your IT infrastructure. Akorri supplies critical information and guidance versus a monitoring tool which just reports on a single infrastructure element (physical or virtual servers or storage or network).” What I really found interesting with the product is its ease of use. You are really able to get this product installed and configured rather quickly and when used for troubleshooting issues between virtualization, physical servers and storage there may not be another player in the market as strong currently. The reporting features of the product are pretty amazing and are readily for IT administrators and their management teams to show performance within the virtual environment.

The amount of work being placed on administrators these days is tremendous and only gets heavier from there. Most companies have the philosophy to do more with less. These are just a few reasons why a solution must be found to keep up with network sprawl which grows at a weeds pace. Everything in the data center is evolving and if things aren’t setup immediately for monitoring sometimes things get lost in the shuffle. “BalancePoint agent-less software collects statistics from the entire infrastructure, creates a dynamic model and then intelligently analyzes the information to understand how an application workload’s server, SAN and storage resources are utilized and interact. BalancePoint uniquely enables IT administrators to visualize their infrastructure, proactively identify and troubleshoot performance problems within minutes, plan future capacity requirements and measure infrastructure service levels to demonstrate efficient resource utilization to management.” Below are some of the benefits gained by using Akorri BalancePoint in your workplace, especially those who have committed to the private or public cloud and the management challenges that it provides.

BalancePoint™ deploys quickly, providing immediate benefits including:

• Same-day non-intrusive implementation
  
• Rapid cross-domain troubleshooting
  
• Optimal IT performance delivery
  
• Effective Infrastructure capacity planning
  
• Justification for IT Virtualization and Consolidation projects
  

With BalancePoint™ in your environment you will:

• Identify Performance Bottlenecks — Find problems and contention points before they impact operations
  
• Troubleshoot Problems Faster — Identify root cause quickly so people can get back to work
  
• Improve Storage Utilization — Use storage and server resources more efficiently and eliminate or defer new storage purchases
  
• Manage VMware in Production — Improve VMware server and storage performance and utilization through “single pane of glass” management·
  
• Optimize and Plan — Identify opportunities for improvement and plan for future growth requirements
  
• Maintain Service Performance — Avoid application brownouts with cross-domain correlation of applications, servers, and storage
  
• Guarantee Future Service — Enable performance-based capacity planning across applications, servers, and storage
  
• Reduce Risk — Mitigate potential issues with what-if modeling for data-tier disaster recovery plans and data center optimization projects
  

The BalancePoint product has won many awards and if you are interested in the product, Varrow is a certified partner and reseller and can assist you with your project. Please feel free to contact us with any questions concerning the product.

In a nutshell, BalancePoint’s Performance Index was showing a rating of 600-700,  which is way out of normal range, for every monitored host for four days.  Over a seven day period, Performance Index had been spiking at the same time every morning.  In the actual servers however, the customer had seen no performance degradation or any complaints about the issue they saw in the logs.

For those not familar with Performance Index, ideally, you want to manage your infrastructure to a Performance Index of 100. Performance Index is BalancePoint’s data center level metric produced for each application that can be used to quickly and easily optimize usage of a system by balancing capacity utilization with good performance.  BalancePoint is the only solution on the market that has a metric of this kind.  Watch this video to learn more about Performance Index.

As it turns out, there was a change to the SAN backend that the admin was not aware of and this created a volume contention issue.  The customer’s resolution was to undo the change to the SAN. 
 
“Akorri saved the day by highlighting the issue,” the admin told our customer service team. 

 You can read how other companies have also had excellent results with BalancePoint here.

In the article, Dan asks what products were considered to help take virtualization to the next level and I thought Kevin’s response was great, so wanted to share it here.

No single product was compared which was a driver in selecting BalancePoint. We had several tools already, Microsoft MOM, VEEAM Report and other tools, Hardware Vendor tools and MS SQL tools.
IT was taking a full arsenal of tools and a team of people to try and get to a common conclusion on what was happening within the system. BalancePoint was picked because it was a common tool for diverse teams to use and get a collective answer.
The fingers could then point to the problem instead of each other.

In talking with our customers, a recurring theme is the need to tie together the virtual and physical worlds of their infrastructure and it’s great to hear that BalancePoint is being used as a tool to help teams better communicate, collaborate and manage service delivery.

You can read the entire post here. If you are a BalancePoint customer and would like to share your story, please contact me.

The first is an old standby – time to resolve problems. Your first line of defense is that the traditional element management tools you have for physically dedicated environments can be useful, but just don’t cut across all the devices and see the levels of virtual abstraction to give that integrated system view which is vital to understanding virtual environments. Those older tools aren’t designed to see through and track all the virtualized elements because they were designed to support one device type in a silo-like fashion. You know the endless hours that are wasted chasing data, going to meetings or sitting on conference calls trying to find the culprits involved in performance problems that are impacting critical application availability, could all be avoided. You suggest even a conservative assumption of reducing this wasted, always unplanned, activity by 50% would be worth the purchase alone. The average Akorri BalancePoint customer with 100 VMs can identify over $500,000 in staff productivity annually.

Now you’ve got the boss’ attention and move onto the next layer of getting to yes, which is having the right dashboard for today’s virtual and physical infrastructure. I’m a car guy and love all kinds of cars, old or new – it doesn’t matter. I love them all and my wife can attest to the fact that there isn’t an older car I wouldn’t want to drive and own. I also own a few and one of my favorites is a 1933 Chevy 3 window rumble seat coupe. That car is 77 years old and has a dashboard somewhat similar to the modern cars of today’s dashboards. Yes, today’s dashboards have improved a lot, which is the point, and provide more information and functionality, but the ’33 has all the basics like speedometer, gas gauge, battery, oil and temperature gauges and while I don’t stare at those gauges when I’m driving, I’m always checking them to see what is going on. So as virtual systems evolve and grow, the dashboards must also evolve to meet those needs. But the point is the same. Virtual infrastructure management provides you with today’s a dashboard to give you the true picture of your virtual and physical systems that you need to run your workloads. Given that new dashboard, you can covert more and more production applications onto shared infrastructure to drive down cost. For our average Akorri BalancePoint customer of 100 VMs, the ROI in reduced hardware costs for server and storage exceeds $300,000 annually and we easily triple the VM to admin ration.

The third reason is the clincher. How do we do more with what we have or “do more with less” as the management speak commonly refers to it. That needs a more comprehensive view of all the resources involved in all of the virtual machine I/O. Do I have enough storage capacity if I add more VM’s to the ESX server? How many VM’s can I add to optimally max out my servers and not introduce any performance problems to the applications on that server and the storage resources supporting the VM and which element of the data center resources is the problem when I’m trying to troubleshoot a difficult application I/O problem? When you walk into the meeting and the boss says, “Who knows how we leverage virtualization more?” or “Who can figure out why we are having problems managing what we just virtualized?” and you raise your hand and say, “Yes I do, I know how we can do all of that.” That’s when the boss is going to be ready to say – “YES.”

For information on how you can build your own compelling business case for virtual infrastructure management that typically shows a payback period of 2-4 months, download The Business Case for Virtualization Management: A New Approach to Meeting IT Goals.

My ’33 Chevy dashboard just won’t work in that New 2010 Chevy Camaro SS, although I wish I had one to try it out, or just to “test drive” it maybe.

BalancePoint can be used to monitor both virtual and physical servers, which is definitely a plus as we’re always trying to show return on investment to our superiors. In addition, BalancePoint is a Web-managed product that has loads of charts and features to maximize ROI. Installation is a breeze. The software is a VMware virtual appliance that can be run on an ESX host-and BalancePoint can additionally be installed as an add-on to VMware’s Infrastructure Client.

You can read the full review here. If you’d like to test drive BalancePoint, download the VM appliance demo at www.akorri.com/demo.

I found another story really interesting about a disagreement between Google and Microsoft at the Structure 09 conference back in June. As recounted by The Register, Gill from Google talks about how they view the Cloud as horizontal versus Microsoft’s tuning the applications and Cloud to better service the applications, Google’s “approach is a little more absolute than [Microsoft's],” Gill said.

Gill went on to say, “Not only does getting to the end user have to be fast, but the back-end has to be extremely fast too…[We are] virtualizing the entire fabric so you get maximum utilization and speed on a global basis as opposed to local fixes – putting one service in a data center”.

He later used this example, “if we (Google) make a minor change to, say, disk storage to get a three per cent gain, and we roll that out to the GFS library, suddenly the entire base of applications stored on GFS sees that gain.”

So why recount this discussion from The Register article? Because I think the missing part from both of these approaches is that a Cloud has multiple applications vying for finite resources. You really have two choices. The first is to keep increasing the infrastructure capacity to keep up and exceed demand. The second is to plan to optimize the “top talker applications” to play more nicely.

Most companies are looking to get away with the smallest amount of physical assets possible. They want to make sure they have enough to handle spikes and other anomalies. In other words, they’re trying to balance cost and efficiency.

Is adding 3% more infrastructure really going to get you a 3% performance gain? How do you know? What if the application that needs the 3% only gets 1%? How much more server or storage do you add to get that application workload serviced properly?

It feels much like the days before Big Bandwidth won over Managed Bandwidth (for the moment). Key Performance Indicators (KPIs) will have to smack people in the face and say – hey you’re nuts to put that application in the cloud. Right now we are seeing an increasing number of our customers using BalancePoint not only for “production virtuals” but aiming our software at the physicals to watch them over a quarter, to have quantitative performance metrics before virtualizing them to ensure service levels back to the business. This is the first step before taking them to the cloud. KPIs will need to be in vogue again or IT organizations run the risk of “shooting themselves in the foot” by moving too fast into the cloud.

Just like we learned in the 90’s, some applications will need to be rewritten prior to being “cloud-ized”. You know what I mean if you had PeopleSoft HR over a 56k DLCI Frame Relay connection using the first web client … hello 7MB-download-coffee-break.

But once you have them written, you still need to “Measure Twice, Cut Once” as my carpenter Brother-in-law Stephen would say. As we know in IT, the only thing that gets cut when an application goes slow is IT staff.

I am going to let you in on a little secret. The Hypervisor is a commodity. What you are paying for are the software services on top of them. Yes you can argue some have more bells and whistles in the hypervisor, scale differently, support different OS’s better than others, or are cheaper. What you are paying for is the right to buy the software add-ons.

It will be interesting to see how each differentiates from each other over the next year. Who will say performance matters and deliver on service level guarantees? Which one will be the most cloud-ish? Will you care?

I think you will care, but I believe the one that builds the most complete eco-system will win. I believe it is like email wars; exchange vs notes. Most companies use exchange. Not because it has more bells and whistles or is the best “all-in-one” product. But because it is a very effective email and calendaring system. Where Lotus Notes was trying to be everything collaborative … MS gave you the options to pick off the Chinese food menu at a reasonable price.

Same will happen with hypervisors. You’re going to have the “all in one” vs the one with features core to the hypervisor (performance / stability) …

I would say the one place MS and others have never done well is management beyond fault and event. So who will be the differentiator in this space? Will it be build or buy? Who will care that not all the workloads will live in virtualized / cloud environments … many with the label of “business critical”?

I believe you will be looking once again for “best of breed” products. Products that were not meant to be panaceas or “single-panes-of-glass” but never got implemented. In my time in the IT industry I have seen people buy five or more single-panes and still never achieve any benefit. Like in the storage industry you will probably buy your element/fault management from the vender and get the rest from innovative vendors outside those vendors.

Editorial Note:
Recently some of the aforementioned hypervisor vendors have been saying they will provide you everything for your data center an even cook you breakfast in the morning. Most call this the “cloud” strategy. Clouds where you use the “services” … think web services … available. Problem is that you will not be able to simply write to one API can move workloads between “clouds”. For clouds, cloud suppliers, and such to be successful this will need to become reality. Supplying underlying infrastructure services with no easy way to write to them is only part of what would be a dramatic game changer. This is not to say “private clouds” in our IT shops will work, but leveraging “public cloud providers” will be difficult.

I also agree with John Gavin, CEO of Akorri when he said “Virtualization used to be all about the low-hanging fruit, where server consolidation and cost management were the drivers. Now it’s really all about simplifying management and managing performance issues when in mission-critical applications.” I agree with him, full disclouser, not becuase he signs my pay checks … but he is dead right.

But for cloud/grid/dynamic IT to become real, been saying this since my stint with GGF/OGF, not only do we need to stop recreating standards, but we need the right standards. One in particular that would nice to see adopted and would jump-start would be Open Virtualization Format (OVF) really being used. We need standards; maybe not even new ones but right ones.

Another issue that will have to be addressed is performance and service level management in this environment. Right now the number one reason people do not virtualize is the strength of business units. What I mean here is if a business unit within a company had a bad experience with virtual so they went back to Physical in some cases; not a death nail but the role out slows dramatically.

Virtualization is the new whipping boy for IT. This is why understanding (baselining) pre-virtualization like our customers do, then comparing and managing to “all systems are green” from an infrastructure perspective is so important. Mature virtualized organizations, some at 90%+ virtualized have realized this. When I am talking to them they rank stability, availability and performance as the top concerns they battle from the FUD of business units.

This is not rocket science. Baseline what you have (physical), convert and test (test baseline), make the decision to virtualize, measure impact to the production virtual environment, and compare ongoing operations versus the baseline. Like the directions on a shampoo bottle … shampoo, rinse, and repeat as desired.