UCS Firmware upgrade high level steps

Here is the high level steps for UCS firmware upgrade for most the environment. You may need to check additional componenets depend on the environment but the below one covers mostof the common thing s we need to check prior to the upgrade


1. Checking the compatibility matrix for the esxi server with the version of UCS firmware
        http://www.cisco.com/web/techdoc/ucs/interoperability/matrix/matrix.html
2. Downloading the firmware and required hardware (enic and fnic) drivers
3. Disabling the phone home alert
4. Backup the ucs configuration
5. Check the space on fabric interconnect and if required delete the older version of bin files
6. Uploading the firmware to UCS
7. Upload the enic and fnic vib to the common datastore on the esx cluster
8. Checking the maintenance policies and make sure it is in user acknowledgement.
9. Upgrade the infrastructure firmware using auto install option
10. Reboot the secondary fabric interconnect
11. Reboot the primary fabric interconnect
12. Once ucs manager / fabric interconnect updated to the new version then using server firmware update the blade server components
13. Put each blade in maintenance mode ( vmotion the vm to other hosts)  and reboot
14. Once the blade is up, install the enic and fnic drivers and restart the blade
15. Exit from maintenance mode
16. Repeat the same steps ( 12 to 14 ) for rest of the blades
17. Check for any error once all the update is completed (during the update you may see increased number of errors that will resolve once update is completed)
18. Enable the phone home alert back.

How to change the web session limits in UCS manager

Here is the steps to limit the  maximum number of sessions per user and total maximum number of sessions for the UCS manager

Login to  UCS manager SSH session and type the below command

scope system

scope services

show web-session-limits

scope web-session-limits

set total 100 ( This limit to 100)

set per-user 10 (This limit to 10)

commit-buffer

How to change the maintenance policy for a service profile template

Problem : Default maintenance policy is set to "reboot immediate" for the service profile . If we do any changes on the template and if  that change requires server reboot , then we end up in rebooting all the server immediately because of this maintenance policy.

How to confirm the default maintenance policy for a template :

Navigate to UCS manager > Servers > Service profile templates > root > service profile template name 

On the Right pane  under properties  click Maintenance policy and see what is exactly configured .

How to change the Maintenance policy :

Navigate to UCS manager > Servers > Policies > root > Maintenance Policies > default

on the right hand side , select the maintenance policy as User Ack  and click save changes ( this option will ask for the confirmation to the user before rebooting the server)

Note : if this policy is attached to the service profile template and the service profile template is in updating mode then this will update on all service profile .


If you don't want to touch the default maintenance policy and you want to create  a new policies for the each service profile template , you will be able to create that under service profile template console itself

Navigate to UCS manager > Servers > Service profile templates > root > service profile template name 

click change maintenance policy on the right pane

click create maintenance policy

Provide the service policy name and description and select the reboot policy and click ok.

Drop down and select the new maintenance policy and click ok

Note: if you using updating template then all your service profile will get updated  for this immediately

How to add Vsphere Vcenter server to the UCS director cloud

Hare are the steps to add the vsphere Vcenter server to add  to the UCS director

1. on homepage of ucs director click  administrator -----> virtual account

2. Click +add  under virtual account

3. Under cloud Type Select the appropriate  cloud here I am selecting  vmware

   

4.Fill out the vcenter information     

5. Click Add and this will be listed under Virtual accounts

                                                                                                                                                                          

6. Click Convered tab on the home page and vmware will be displayed under the default pod if you choose default pod , if other pod is selected then it will display on that pod.

7. double click on the pod , under virtual vmware will be listed

8. double click on the vmware under virtual

9.after a few minutes UCS director will pull all the information from vcenter and  displays that in dashboard

How to create a POD in UCS director

Steps to create the POD in UCS director before adding  vsphere or hyperv environment into the UCS director

1. Click on converged on the ucs director home page

2. Click Add

3. select type first before selecting all other option and fill the detail depend on the environment

4.click Add  and the pod will be displayed under the converged tab now. If you want this pod to be as a hidden on then check the box hide pod before clicking add

How to Install UCS director on Vsphere Environment

Here are the steps to import the UCS director to the Vsphere Environment :

1.Right click on the vcenter and click deploy ovf template .If you are using native vsphere client then click file deploy ovf template .

2. Browse to the OVF file location and click open  and click next
3. Accept the end user license agreement
4. select the host or cluster and click next
5. select the datastore and select the disk format as thick if it is production ,thin if it is for lab
6. choose the proper port group for the network
7. In the ip address box check dhcp if you are suing dhcp if not leave that unchecked.
8. click finish
9. Once the vm is imported , power on the vm.
10.During the appliance start , it will prompt you for the  configuration of dhcp ip or static ip  ,if you want to use dhcp then enter  no , if you want to use static ip enter yes
11.provide the ip address, gateway  and netmask
12. enter yes to continue
13.once the server is up , open the browser with the ip configured during the install.
14.default user name and password for the GUI is admin /admin
15.default user name and password for the CLI is shelladmin / changeme

How to Add license to the UCS director

Here is the steps to add the License to the UCS Director

1. Save the .lic file on the desktop
2. open the UCS director using any of the browsers
3. default user name for the ucs director admin and default password is admin for GUI access
4. Navigate to administrator ----> license

5. Click update license
6.browse to the .lic file and click upload

7. once it is uploaded, then click submit

Changing the width of the device field in esxtop

Symptom : When running the esxtop for troubleshooting the issue some field width are too small and not able to see the full field name , for example when troubleshooting the storage issue , the device id will be displayed as truncated  like naa.xxxx but not the full device id.

How to fix that : for example when troubleshooting storage issue
1.run esxtop on the ssh session of the esxi server
2.press u for disk device view
3.shift l or simply L ( caps L)
4.press 36 ( to see 36 character)
 

Java error shows - Certificate has been revoked .The application will not be executed error on ucs KVM

When you try to access the KVM console on the ucs CIMC or UCS manager Java error shows

“ Certificate has been revoked . The application will not be executed”

How to fix this issue temporarily ( Workaround) :

1. Goto control panel on the machine where you are opening the KVM

2. double click on Java

3. Click on the advanced tab on the java control panel

4. under “Perform Certificate revocation checks on”

5. Select “do not check (not recommended).

Now you will be able to access the KVM . Once you are done with the work go back to the control panel and change back to the previous settings.

UCS Manager shows the Major error F0909 for the keyring certificate

Where to confirm this issue :
Check the ucs manager alerts it shows Error code F0909 with keyring default certificate expired.
Get into the SSH session of the ucs manager using putty then run the following command to check the certificate status.

UCS-A# scope security
UCS-A /security # scope keyring detail

Certificate status : expired ( this is what output screen shows)
How to fix this issue:
On the ssh session of the ucs manager run the following command to regenerate the default certificate

 UCS-A# scope security
 UCS-A /security # scope keyring default
 UCS-A /security/keyring* # set regenerate yes
 UCS-A /security/keyring* # commit-buffer
 UCS-A /security/keyring #
Goback to the ucs manager GUI screen and accept the new certificate. This will close and open the ucs manager gui session again .

If third party certificate is used instead of default.Then import the certificate using command

 UCS-A# scope security
 UCS-A /security # scope keyring XXXXX ( XXXXX keyring name for ke20)
 UCS-A /security/keyring # set trustpoint yyyyy ( yyyyy is the trustpoint name created during the certificate request)
 UCS-A /security/keyring* # set cert
 Enter lines one at a time. Enter ENDOFBUF to finish. Press ^C to abort.
 Keyring certificate:
 > -----BEGIN CERTIFICATE-----
               XXXXXXXXXXXXXXXXX
               XXXXXXXXXXXXXXXXX
               XXXXXXXXXXXXXXXXX
               XXXXXXXXXXXXXXXXX
> -----END CERTIFICATE-----
> ENDOFBUF
Commit-buffer
 Go back and accept the new certificate and the GUI session will close and reopen again.

Unable to communicate with Flex flash controller: Operation ffcardsGet, status Error Timeout /Bootbank cannot be found at the path /bootbank” error

Where to See this error :

1.CIMC ---> storage --->  Cisco Flexflash ---->controller info

2.If you are running ESXI server on this C series server you will be getting “Bootbank cannot be found at the path /bootbank” error

 Cause for this issue: this is a known bug if you are running firmware version less than 1.5(3d) on c series server. The bug id is CSCuh33982. It is resolved in firmware version 1.5(3d).

https://tools.cisco.com/bugsearch/bug/CSCuh33982

Workaround to fix this issue:
1. Reboot CIMC --- this will not cause any disruptions to the esxi server or the vm’s running on it. The management page of CIMC page need to be reloaded. I would recommend to clear the cache and try to access the management page after CIMC restart.

How to restart the CIMC:

Goto Admin tab on CIMC and click reboot CIMC.
If you want to do that on CLI, ssh into the management ip address of CIMC and run the below commands.
                         Scope cimc
                         Reboot

2. If CIMC shows the controller is healthy now and ESXI still shows the same error then ssh into the ESXI server and restart the service by using the below command
Services.sh restart ( this will not affect your VM running on it but this will disconnect the esxi server and connect it back automatically when the service comes up).