HP Chassis and Virtual Connect Firmware Upgrades – Don’t bother

So last night was a busy one thanks to the HP Chassis and “fantastic” Virtual Connect system.

Bare with this article, as its part what happens blah blah, and part, how to install firmware on Virtual Connect

The environment

So the environment consists of;

HP C7000 Chassis

6 x HP BL460c Blade Servers

2 x HP VC Flex-10 Enet Module

These VC Flex modules are connected by 2 x 10GB links in a LACP trunk to Cisco 3750X stack.

The plan of work

The customer wants to move to VMware 5.5, nice and simple, upgraded from VMware 5.0, to do this you have to also update your HP gear to the latest supported levels.

After some searching online and no thanks to the documentation of HP, it was deducted that you must do this in the following order.

1. Update your HP Chassis Onboard Administrator

2. Update all your blades using the SPP disc

3. Update your Virtual Connect Manager

4. Upgrade all ESXi hosts to 5.5, and your vCenter

Viola, and your done

The issue

So I updated the HP Onboard Administrator to 4.22 (Jun 13 2014) Firmware, this worked perfectly.

I updated all the blades using the SPP disc via the iLO and virtual media, nice and simple.

And then I update the Virtual Connect from 3.60 to 4.30. After this we lost all connectivity to the blades and all Virtual Machines died as the blades couldnt get to the iSCSI storage.

On the Virtual Connect Manager go to Hardware > Overview > Click on a VC Switch > Server Ports

And this showed all ports to servers as “Not Linked”2014-08-13_09-31-30

When looking at the Shared Uplink Port you can see that both of them say standby aswell, meaning neither switch is acting as active. And the view from the ESXi box

SharedUplinkSets

Bay 9 Server NIC Status

(Please note I've edited the screenshot to reflect what we were seeing at the time of the issue as I didnt get a screenshot of everything)

So this meant a phone call to HP support, of which there is no exact shortcut or option to press for “HP Virtual Connect Manager”, I dont even think HP want to admit it exists.

The Fix

The fix is quite simple, downgrade to where you have come from;

But I need to be on VMware 5.5, so I need to upgrade.

Yep, to downgrade its a pain, if you just try to downgrade it will fail,

You need to turn off all your blade servers first. (Yes so much for that High Availability blah blah, only exists if you have multiple chassis and the right VMware setup for failover)

Run the VCSU tool, as below, however you need to specify at the following line

Please enter Force Update options if any (eg: version,health):
version,health

2014-08-13_10-12-27

After this I was informed by HP support to update to 3.75 and then 4.10 and then 4.20 firmware in that order, and no further as there are issues with 4.30 as I experienced.

Rolling back the firmware to 3.60 worked straight away which connectivity restored, I just have to do the rest of the patches now.

Great eh?

How to update the Virtual Connect Manager

Note: Please plan downtime for this, as in my case it took everything offline whilst updating both modules.

So to update the Virtual Connect Manager, leave the web interface, as you don’t do it there, you need to download the “Virtual Connect Support Utility”, of which I downloaded the wrong version first, you need 1.10 which can be found here.

To do this, you open up the command interface,

2014-08-13_09-37-45

2014-08-13_09-39-43

As you can see this launches a command prompt in the install area of the utility, which is fantastic, because it also exits the program.

1. Type vcsu.exe and enter

2. Type update and enter

3. Enter the IP address of the Onboard Administrator, username and password.

4. Drag your firmware file from its location into the CMD prompt window and this will automatically out the file location in for you.

4. Click through the options until it says “Initializing, please wait… [Step 1 of 2]”

5. When prompted it will tell you its discovered the Virtual Connect domain, enter the username and password for the Virtual Connect Manager

6. Enter yes after confirming the version you are on and version you are going to.

7. Once done it will tell you its completed and where the log files are loaded.

Here is a full screenshot of the process, with the steps highlighted in yellow with green marks for easy identification.

2014-08-13_09-41-59

Things to know about the Virtual Connect Support Utility

it saves everything in your temp location, so where it tells you it’s saved the log, copy the location and open it up

2014-08-13_09-55-181

So it doesn’t just output a file which is the copy of your update session, which in this case is called “vcsu-15044.log”

It saves a copy of your Virtual Connect Manager settings before the upgrade, of which mine was called “vcConfig-8895406-BLE01_vc_domain-201408130822

And finally there is the Support Info output which can be used by HP support to determine the status of your hardware before and after the firmware update “vcSupportInfo-8895406-BLE01_vc_domain-201408130823

If you run the “collect” command, then it produces an output to the temp location for the following files

– Session output

– ShowAll of OA

– ShowAll of Virtual Connect Manager

2014-08-13_10-08-11

Summary

The cause of this is basically a faulty firmware, which the guy from HP wouldnt admit to actually being faulty, but basically said don’t go to 4.30 firmware after you have downgraded and start to upgrade again.

Who’s fault was this outage in the production network? Mine I guess, it was a scheduled maintainance window, and I’ve updated Blade chassis and switches and so forth hundreds of times, and I’ve never had and issue such as this. But its my fault because I could have dug deeper into the update etc. I read the guides on how to update everything, I read the release notes of the firmware, but I must admit HP dropped the ball here.

Going off previous experience when updating Dell devices, they usually state in their release notes and guides, supported upgrade paths, i.e how you get from ver 1.0 to ver 2.0 and any steps in between you need to take. For HP this wasnt the case, it just said, here’s the latest firmware, and here’s how to upgrade it. Hell I struggled to find the 3.6 firmware to roll back to, the HP support guy sent me the link!

End of the day, I wont be designing or offering HP Chassis and blades as a Solution any time soon, I’ll stick with Cisco UCS

 

 

7 thoughts on “HP Chassis and Virtual Connect Firmware Upgrades – Don’t bother

  1. Update Method is first update servers, then OA, them VC. Remember VC 3.60 > 4.30 cant be done in one step, you had first go with 3.70 before go to 4.30. see release notes pre-requirements page 12. Saying that, mostly its very wise to don’t do this VC updates online.

    http://h20566.www2.hp.com/portal/site/hpsc/template.BINARYPORTLET/public/kb/docDisplay/resource.process/?spf_p.tpst=kbDocDisplay_ws_BI&spf_p.rid_kbDocDisplay=docDisplayResURL&javax.portlet.begCacheTok=com.vignette.cachetoken&spf_p.rst_kbDocDisplay=wsrp-resourceState%3DdocId%253Demr_na-c04389285-1%257CdocLocale%253Den_US&javax.portlet.endCacheTok=com.vignette.cachetoken

    Questions about Blades, VC, Networking, you can always ask me.

    1. Hi, the only note I can see is

      A network outage may result when upgrading directly from VC 2.xx release version to VC 4.20 release
      in a Multi-Enclosure environment. Upgrade from VC 2.xx to VC 3.70 or VC 4.10 first, and then
      upgrade from VC 3.70 or VC 4.10 to VC 4.20 in a Multi-Enclosure setup.

      This environment was a not a multi-enclosure setup, and it was more than a network outage as such. Also HP shouldnt have buried this in at the end in my opinion. But I’ve learnt for next time,

      Thank you for your comment

  2. Very, very interesting read.
    We had a very very similar meltdown a few years back when taking our c7000 chassis (including VC) up a few firmware levels. Our infrastructure sounds very similar to yours bar the fact our storage is FC based.
    In our case, and oddly, our platform melted down a few days after installing the new VC firmware. We lost all network connectivity and the only way to fix it was to reboot the entire chassis, although we didn’t have to roll-back as you did.
    HP never ever got to the bottom of this. We even left our secondary cluster in this ‘bricked’ state for them to troubleshoot on. They basically faffed and batted it between support teams over a few days, got absolutely no-where as a result and even tried to blame VMware at one point.
    I have never had any confidence in applying a firmware update to VC since and haven’t as a result.
    I’m now in the exact same position as you at the start, in that I’ve been holding off upgrading from 5.0 to 5.5 due to the firmware update requirement.
    Massive thanks for sharing this though as you’ve potentially saved me from an enormous amount of pain.
    For what its worth, if we go down the blade route again, I don’t think we’ll go anywhere near HP. UCS looks more attractive by the day…

  3. Your Virtual Connect firmware should never be newer than your OA. Any compatibility matrix you look at will show you this. ie. OA 4.22 needs VC version 4.21 or older. HP did not drop the ball here, this has always been the case.

    1. Hi,

      Thanks for the reply, my issue was with the documentation that was at hand when downloading the firmware update. I’ve since been directed to some better documentation, which I’ll be using should I ever have to work on a C7000 Chassis again.

      1. Dean, thanks for sharing the story and steps here. After reading your story and heard from my other team member, it makes Cisco UCS Converged solution much more reliable than HP.

        1. I guess the whole Dell, HP and Cisco argument is subjective to the person who is writing it and their experiences, there is pro’s and cons for them all,

          I am just setting up a Cisco UCS B series solution, its more upfront work, but over all less management.

Leave a Reply