UCS Firmware upgrade high level steps

Here is the high level steps for UCS firmware upgrade for most the environment. You may need to check additional componenets depend on the environment but the below one covers mostof the common thing s we need to check prior to the upgrade


1. Checking the compatibility matrix for the esxi server with the version of UCS firmware
        http://www.cisco.com/web/techdoc/ucs/interoperability/matrix/matrix.html
2. Downloading the firmware and required hardware (enic and fnic) drivers
3. Disabling the phone home alert
4. Backup the ucs configuration
5. Check the space on fabric interconnect and if required delete the older version of bin files
6. Uploading the firmware to UCS
7. Upload the enic and fnic vib to the common datastore on the esx cluster
8. Checking the maintenance policies and make sure it is in user acknowledgement.
9. Upgrade the infrastructure firmware using auto install option
10. Reboot the secondary fabric interconnect
11. Reboot the primary fabric interconnect
12. Once ucs manager / fabric interconnect updated to the new version then using server firmware update the blade server components
13. Put each blade in maintenance mode ( vmotion the vm to other hosts)  and reboot
14. Once the blade is up, install the enic and fnic drivers and restart the blade
15. Exit from maintenance mode
16. Repeat the same steps ( 12 to 14 ) for rest of the blades
17. Check for any error once all the update is completed (during the update you may see increased number of errors that will resolve once update is completed)
18. Enable the phone home alert back.

Registering UCS manager with UCS Central

UCS Central is the centralized management tool for managing UCS domain.

Prerequisites : 

1. Make sure NTP server name and time zone matches exactly between the UCS central and UCS manager. Lot of time I  get into trouble in registering the UCS manager with UCS central due to this .

you can set this on UCS manager ----> admin -----> Time Zone Management 

 On UCS central Administration----->General-----> Date/time

2. Ucs central IP address and  Shared Secret password is required for this


Actual Steps :

Step 1:

Goto UCS Manager ----> Admin ----> UCS central   ( if you not seeing this option under Admin tab then check the filter under Admin tab and make sure either UCS central is selected or ALL is selected)

Step 2 :

click Register with UCS Central

Step 3:

update the UCS Central ip and Shared secret password


Step 4:

if you want to change some policy revolution control  to global  ( which means it will get the settings from UCS central ) you can select that as global or you can change this anytime after registration


Step 5:

click ok and accept 

Step 6:

once it is properly registered wtth UCS central then the repair state is showing as done and  Registration status will be showing as Registered . Check the FSM tab to see the progress

step 7:

On the UCS Central login  navigate to Equipment ---> UCS domain ------> Ungrouped domains and  UCS manager will be listing there if the registration is successful

 

Step 8: 

Right click on the UCS manager under Ungrouped Domain and click change group assignment

Step 9: 

 Select the proper domain Group , either put that on root domain group or you can create a sub domain group. I would prefer to create a sub domain group  and click yes for changing the group.

Step 10:

check the ucs manager is showing under the Domain group

How to expand or resize the netapp lun which is having snapmirror

Below are the simple steps to expand the or resize the snapmirror lun.

Step 1:  Note down the source volume name and destination volume name

Step 2: ssh into both source and destination netapp controller

Step 3: On the destination or target netapp ssh session  run the below command

            a. priv set advanced
            b. snapmirror break "destination volume name"
            c. vol options "destination volume name" fs_size_fixed off
            d. vol size "destination volume name"  1000g  ( here I am resizing the lun from 500 to 1000 GB)
                 Note: please change the size of the lun according to the requirement
           e.  vol options " destination volume name" fs_size_fixed on

step 4. On the source netapp ssh session run the below command

            a. priv set advanced
            b. vol options "source volume name" fs_size_fixed off
            c. vol size "source volume name"  1000g  ( here I am resizing the lun from 500 to 1000 GB)
                 Note: please change the size of the lun according to the requirement
           d. vol options " source volume name" fs_size_fixed on
       
step 5. On the destination netapp ssh session run the below command
     
           a.snapmirror resync -S sourcefiler:sourcevolumename targetfiler:targetvolumenane

Vcenter Inventory Service Failed to start in Vcenter appliance or accessing vsphere webclient shows the error " Client is not authenticated to Vmware inventory service"

Issue : Vcenter appliance inventory service failed to start or accessing the vsphere webclient shows that the client is not able to authenticated to the VMware inventory services.

Where to check the logs for this :  

1.Connect the vcenter appliance with putty (ssh session) or through the console

2.Navigate to the path /var/log/vmware/vpx/inventoryservice  and check the ds log file by running the below command.
                       cd  /var/log/vmware/vpx/inventoryservice
                      cat ds.log | more

3. check to see if the database is showing as corrupt , if it is then follow the solution. This steps is for version esxi 5.5 , you can refer this vmware  kb article 2037952 for rest of the version

Solution :

1.Connect the vcenter appliance with putty (ssh session) or through the console

2.Stop the inventory service by running the command
                   Service vmware-inventoryservice stop

3.remove the data
                   rm -rf /storage/db/inventoryservice/data

4. Navigate to the path /usr/lib/vmware-vpx/inventoryservice/scripts/

                   cd /usr/lib/vmware-vpx/inventoryservice/scripts

5. run the ds-invoke.sh with the default password
                     ./ds-invoke.sh -Dvim.logdir=/var/log/vmware/vpx/inventoryservice com.vmware.vim.dataservices.CreateDb /storage/db/inventoryservice/data changeme default changeme
6.start the inventory service
                     Service vmware-inventoryservice start

7. register the vcenter appliance
                      vcenterhost=vcneterservername.yourdomaname.com

8.navigate to the directory /usr/lib/vmware-vpx/inventoryservice-registration
                      cd /usr/lib/vmware-vpx/inventoryservice-registration

9. run the vcregtool.sh
                   ./vcregtool.sh -Dvim.logdir=/var/log/vmware/vpx/inventoryservice-registration com.vmware.vim.dataservices.vcregtool.RegisterVC -action register -lookupserviceurl https://$vcenterhost:7444/lookupservice/sdk -isurl https://$vcenterhost:10443 -vcurl https://$vcenterhost/sdk/vimService -vccert /etc/vmware-vpx/ssl/rui.crt -vcprivkey /etc/vmware-vpx/ssl/rui.key -vcinstancecfg /etc/vmware-vpx/instance.cfg -vcendpointsdir /usr/lib/vmware-vpx/endpoints -vcextensionsdir /usr/lib/vmware-vpx/extensions

10. restart the vmware-vpxd service
                   service vmware-vpxd restart

ESXI host 5.5 takes long time to boot and stays at VMW_Vaaip_Cx loaded successfully

Issue :  ESXI 5.5 boot take long time and stays at VMW_VAAIP_CX  loaded successfully , this may take an hour or so to bring the ESXI 5.5 completely.

Cause : This will be happening on the environment which have Microsoft cluster vm and RDM disk is presented to that Microsoft cluster vm . Especially the RDM lun is not showing as Perennially reserved =true. You can check whether the lun is perennially reserved or not by running the below command

esxcli storage core device list 

Solution: Will explain how to fix this on ESXi 5.5 version , for rest of the version please check the vmware kb article number 1016106.

1.We need to know the LUN identifier ( naa.id ) for all the rdm disk and  this identifier varies for different storage vendor.
2. There are multiple ways to get the identifier , simple way is to click edit settings on the vm and click the rdm lun and on the right hand side click manage path , note down the lun number and identifier.
3. We can also get the identifier from Configuration---storage adapter and click one of the hba .
4.ssh into the esxi server
5.Run the command for each RDM lun

esxcli storage core device setconfig -d naa.id --perennially-reserved=true

Do this for all the esxi host which is part of the same cluster.

How to add the spare disk on netapp filer

Here is the steps to add the disk as spare disk on netapp

1. SSH into the controller
2. turn off the auto assign option by running the command options disk.auto_assign off
3. Put the new hard drive on one of the empty slots on disk shelf
4. run the command disk show -n  , this will show the newly added disk as not owned by any controller
5. run this command disk assign oa.00.8  to assign the controller where 0a is the controller identification ,00 is the disk shelf number and 8 is the disk number
6. Turn on the auto assign option by running the command options disk.auto_assign on
7. run disk show , you will be seeing the new disk as spare
8. for some reason you want to remove that hard drive you can unassign the disk by running the command disk assign 0a.00.8 -s unowned -f


 

windows File server move using robocopy

Often I used to get request to move the file share server to a different server , There are several option available to finish this job, the easiest one would be the robocopy one.

Robocopy will take care of ntfs permission on the folder but will not copy the share permission. If robocopy is used for the file share move then we need to export the share permission from exisitng windows server then import that to the new fileshare server.

Here is the high level steps

1. copy the below command in a text file and save it as filename.bat

@ECHO OFF
SETLOCAL

SET _source=D:\
SET _dest=E:\
SET _what=/COPYALL /SEC /MIR /Z
 :: /COPYALL :: COPY ALL file info
 :: /B :: copy files in Backup mode.
 :: /SEC :: copy files with SECurity
 :: /MIR :: MIRror a directory tree
SET _options=/R:0 /W:0 /LOG:Logfile.txt /NFL /NDL
 :: /R:n :: number of Retries
 :: /W:n :: Wait time between retries
 :: /LOG :: Output log file
 :: /NFL :: No file logging
 :: /NDL :: No dir logging

ROBOCOPY %_source% %_dest% %_what% %_options%

Note : please replace d:\ with the source path and also I:\ with the destination path. Also keep in mind that this will excatly mirror the source to destination , if destination has any extra files or folders that will be deleted.

2.download robocopy and save that exe and filename.bat on same location

3. run the filename.bat on a administrative command prompt.

4. The log file will be generated on the same location which includes start time ,end time , how much size is copied or ignored or failed . If any files copy is failed it will be reported on the log file

5.From the HKEY_LOCAL_MACHINES\SYSTEM\CurrentControlSet\Services\LanmanServer\Shares

6. export the above key from source server and import on the destination server

Note: this will work if you are moving entire file share server .if you are moving only portion of file share server then after import delete the unwanted share key.

 

sim.vmdk was not found error while powering on netapp simulator on esxi host

Today I tried to deploy the netapp simulator 8.2 on my lab . When I tried to power on the simulator it failed with the error  " Mydataontap-sim.vmdk was not found "

Solution :

ESXI server is not loading the vmkernel multiextent module  , we have to load this manually.

Here is the steps to  load that manually

1. login to the ESXi SSH session
2. run the command  vmkload_mod multiextent

How to change the web session limits in UCS manager

Here is the steps to limit the  maximum number of sessions per user and total maximum number of sessions for the UCS manager

Login to  UCS manager SSH session and type the below command

scope system

scope services

show web-session-limits

scope web-session-limits

set total 100 ( This limit to 100)

set per-user 10 (This limit to 10)

commit-buffer

How to change the maintenance policy for a service profile template

Problem : Default maintenance policy is set to "reboot immediate" for the service profile . If we do any changes on the template and if  that change requires server reboot , then we end up in rebooting all the server immediately because of this maintenance policy.

How to confirm the default maintenance policy for a template :

Navigate to UCS manager > Servers > Service profile templates > root > service profile template name 

On the Right pane  under properties  click Maintenance policy and see what is exactly configured .

How to change the Maintenance policy :

Navigate to UCS manager > Servers > Policies > root > Maintenance Policies > default

on the right hand side , select the maintenance policy as User Ack  and click save changes ( this option will ask for the confirmation to the user before rebooting the server)

Note : if this policy is attached to the service profile template and the service profile template is in updating mode then this will update on all service profile .


If you don't want to touch the default maintenance policy and you want to create  a new policies for the each service profile template , you will be able to create that under service profile template console itself

Navigate to UCS manager > Servers > Service profile templates > root > service profile template name 

click change maintenance policy on the right pane

click create maintenance policy

Provide the service policy name and description and select the reboot policy and click ok.

Drop down and select the new maintenance policy and click ok

Note: if you using updating template then all your service profile will get updated  for this immediately

Unjoining Hyper-V server From Domain

Problem : I tried to remove the hyper-V host from the domain which is not there any more in my lab environment , but it fails with the error " The specified domain either does not exist or could not be contacted ".

Usual way of removing a Hyper-V server from domain :

Login into the Hyperv console and press option 1

Provided the password and username to remove . But it fails stating that it is not able to contact the domain.

Solution:

we need to remove the domain from that server forcefully , used the below command for that

Netdom remove /d:olddomainname.com hypervservername /Force

Registering Vcenter on Virtual storage console

Go to the virtual storage console registration url

               https://localhost:8143/Register.html

Please replace the  localhost with the server name which has virtual storage console .

update the host name  under plugin service information ( this host name should be the server name which has virtual storage console)

update the vcenter  information and click register.

How to add Vsphere Vcenter server to the UCS director cloud

Hare are the steps to add the vsphere Vcenter server to add  to the UCS director

1. on homepage of ucs director click  administrator -----> virtual account

2. Click +add  under virtual account

3. Under cloud Type Select the appropriate  cloud here I am selecting  vmware

   

4.Fill out the vcenter information     

5. Click Add and this will be listed under Virtual accounts

                                                                                                                                                                          

6. Click Convered tab on the home page and vmware will be displayed under the default pod if you choose default pod , if other pod is selected then it will display on that pod.

7. double click on the pod , under virtual vmware will be listed

8. double click on the vmware under virtual

9.after a few minutes UCS director will pull all the information from vcenter and  displays that in dashboard

How to create a POD in UCS director

Steps to create the POD in UCS director before adding  vsphere or hyperv environment into the UCS director

1. Click on converged on the ucs director home page

2. Click Add

3. select type first before selecting all other option and fill the detail depend on the environment

4.click Add  and the pod will be displayed under the converged tab now. If you want this pod to be as a hidden on then check the box hide pod before clicking add

How to Install UCS director on Vsphere Environment

Here are the steps to import the UCS director to the Vsphere Environment :

1.Right click on the vcenter and click deploy ovf template .If you are using native vsphere client then click file deploy ovf template .

2. Browse to the OVF file location and click open  and click next
3. Accept the end user license agreement
4. select the host or cluster and click next
5. select the datastore and select the disk format as thick if it is production ,thin if it is for lab
6. choose the proper port group for the network
7. In the ip address box check dhcp if you are suing dhcp if not leave that unchecked.
8. click finish
9. Once the vm is imported , power on the vm.
10.During the appliance start , it will prompt you for the  configuration of dhcp ip or static ip  ,if you want to use dhcp then enter  no , if you want to use static ip enter yes
11.provide the ip address, gateway  and netmask
12. enter yes to continue
13.once the server is up , open the browser with the ip configured during the install.
14.default user name and password for the GUI is admin /admin
15.default user name and password for the CLI is shelladmin / changeme

How to Add license to the UCS director

Here is the steps to add the License to the UCS Director

1. Save the .lic file on the desktop
2. open the UCS director using any of the browsers
3. default user name for the ucs director admin and default password is admin for GUI access
4. Navigate to administrator ----> license

5. Click update license
6.browse to the .lic file and click upload

7. once it is uploaded, then click submit

Enabling CDP neighbour on nexsus 1000v

How to enable the CDP neighbor on Cisco Nexsus 1000v

1. Login into the primary nexsus 1000V VSM

2.Conf t

3.interface mgmt0 

4.no cdp enable

5.cdp enable

6.sh cdp interface mgmt0

7.copy running-config startup-config

How to disable the CDP neighbor

1. Login into the primary nexsus 1000V VSM

2. Conf t

3. interface mgmt0

4. no cdp enable

5. sh cdp interface mgmt0

6. copy running-config startup-config

ESXI patch remediation fails with error " The Operation is not supported on the selected Inventory Objects"

Symptom :

Updating ESXI host ptach fails with the error
" The Operation is not supported on the selected inventory objects. Check the events for the objects selected for the operation."

Root cause: 

This error is due to the admission control policy restrictions set in the HA preventing the update manager from doing its job.

Workaround:
There are two options to resolve this issue

 Disable HA : (option 1)

1. Right click the ESXI cluster and click edit settings

2. Under cluster features , select Turn off Vmware HA

3. Click ok

 Disable Admission control : (option 2 )

1.Right Click the cluster and click edit settings

2. Under cluster features Click VMware HA

3.Under admission control, select “disable: power on vms that violate the availability constraints” and click ok


Note: Don’t forget to enable the options back once patching is completed.

SQL Server Installation Fails with the error access denied or access right issue

Symptom 

SQL 2012 or SQL 2008  installation failed with access denied or showing the error  "setup account privileges" failed


Root  cause :

the user account used for the sql installation is missing one of the below access

1. SeBackupPrivilege
2. SeDebugPrivilege
3. SeSecurityPrivilege

How to check if the user account have these access or not :

1. download accesschk.exe from http://technet.microsoft.com/en-us/sysinternals/bb664922.aspx
2. open the command prompt with elevated privilege
3. navigate to the accesschk.exe path in the command prompt
4. run the following command  accesschk.exe domain\username *
5. The output will show all the privilege the user have and see if the above three privilege are listed


How to fix the issue :

If any of the above privilege is missing then follow the below steps to add those.

1. click start --->Run ----> gpedit.msc
2. expand local security policy  ----> Local policies ----> User Rights Assignment 

Double click "Backup Files and Directories Properties" and add user or group if SeBackupPrivilege is missing

Double click "Debug Programs" and add user or group if SeDebugPrivilege is missing

Double click " Manage auditing and security log" and add user or group if SesecurityPrivilege is missing

Note: if  add user or group is grayed out then it  might be blocked by your group policy.

Changing the width of the device field in esxtop

Symptom : When running the esxtop for troubleshooting the issue some field width are too small and not able to see the full field name , for example when troubleshooting the storage issue , the device id will be displayed as truncated  like naa.xxxx but not the full device id.

How to fix that : for example when troubleshooting storage issue
1.run esxtop on the ssh session of the esxi server
2.press u for disk device view
3.shift l or simply L ( caps L)
4.press 36 ( to see 36 character)
 

Converting RDM disk to VMDK

ISSUE : need to convert the RDM disk (physical mode or virtual mode) to VMDK disk

symptoms : couple of vm might be having rdm disk ( physical or virtual) and need to convert those disk into vmdk.

Scenario A: Converting physical compatability mode  RDM disk to VMDK

                1.directly converting physical mode rdm to vmdk will not be possisble First this physical mode rdm disk need to be converted to virtual mode.
                2.shutdown the vm
                3. click edit settings and select the physical mode rdm disk
                4. click remove disk and delete from datastore ( this will not delete the data from the disk for physical compatability mode rdm disk)
                5.click edit settings
                6. Add hard disk
                7. rdm disk
                8.select the rdm disk and select the mode as virtual compatability mode
                9.power on the server
                10.click edit settings and select migrate
                11.select datastore migration option
                12.select advance under the disk
                13.change the disk format to thick or thin disk
                14.select the destionation datastore .

Scenario B: Converting Virtual compatability mode RDM disk to VMDK
      
                 1.click edit settings and select migrate
                 2.select datastore migration option
                 3.select advance under the disk
                 4.change the disk format to thick or thin disk
                 5.select the destination datastore .
 

"Media Is Write protected" error while initializing new disk in windows server 2008

Error Message : media is write protected

symptoms: added a new storage disk to the windows server  and try to initialize the disk under disk managment but it failed with the error "media is write protected"


Root cause: This is due to the san policy in windows 2008 server .If the san policy is VDS_SP_OFFLINE then the disk will be offline and read only disk.

How to fix the issue :

1. Open the command prompt as administrator and type diskpart and hit enter
2. list disk
3.select disk X ( where x is the disk number which is offline)
4.if the disk is offline then type online disk to bring that online
5.type detail disk and check for read only attribute is showing as yes
6.type attribute disk clear readonly to clear the read only attribute
7.exit
8.reinitialize the disk ,if reinitliaze popup doesn't come up then reboot the server and try to reintialize.
 

Java error shows - Certificate has been revoked .The application will not be executed error on ucs KVM

When you try to access the KVM console on the ucs CIMC or UCS manager Java error shows

“ Certificate has been revoked . The application will not be executed”

How to fix this issue temporarily ( Workaround) :

1. Goto control panel on the machine where you are opening the KVM

2. double click on Java

3. Click on the advanced tab on the java control panel

4. under “Perform Certificate revocation checks on”

5. Select “do not check (not recommended).

Now you will be able to access the KVM . Once you are done with the work go back to the control panel and change back to the previous settings.

UCS Manager shows the Major error F0909 for the keyring certificate

Where to confirm this issue :
Check the ucs manager alerts it shows Error code F0909 with keyring default certificate expired.
Get into the SSH session of the ucs manager using putty then run the following command to check the certificate status.

UCS-A# scope security
UCS-A /security # scope keyring detail

Certificate status : expired ( this is what output screen shows)
How to fix this issue:
On the ssh session of the ucs manager run the following command to regenerate the default certificate

 UCS-A# scope security
 UCS-A /security # scope keyring default
 UCS-A /security/keyring* # set regenerate yes
 UCS-A /security/keyring* # commit-buffer
 UCS-A /security/keyring #
Goback to the ucs manager GUI screen and accept the new certificate. This will close and open the ucs manager gui session again .

If third party certificate is used instead of default.Then import the certificate using command

 UCS-A# scope security
 UCS-A /security # scope keyring XXXXX ( XXXXX keyring name for ke20)
 UCS-A /security/keyring # set trustpoint yyyyy ( yyyyy is the trustpoint name created during the certificate request)
 UCS-A /security/keyring* # set cert
 Enter lines one at a time. Enter ENDOFBUF to finish. Press ^C to abort.
 Keyring certificate:
 > -----BEGIN CERTIFICATE-----
               XXXXXXXXXXXXXXXXX
               XXXXXXXXXXXXXXXXX
               XXXXXXXXXXXXXXXXX
               XXXXXXXXXXXXXXXXX
> -----END CERTIFICATE-----
> ENDOFBUF
Commit-buffer
 Go back and accept the new certificate and the GUI session will close and reopen again.

Reclaiming Thin provisioned Disk

Issue: esxi environment storage disk is showing the free space properly by from the netapp view it is not showing the proper free space. For example 100 Gb lun (named as lun1) is presented to the esxi server this lun is a thin provisioned lun from netapp. Created a 25 GB vm on the 100 Gb lun (lun1) , now the free space in that lun (lun1) is 75 GB which is good. Storage vmotioned that vm to the another lun (named lun2). Now the free space on the lun1 should be 100 GB from ESXI view as well as NetApp storage view but I am seeing 100 GB free from esxi view but NetApp storage view is still showing 75 is free and 25 GB in use.

 Root Cause: VAAI unmap is disabled by default in ESXI host due to the performance impact. This VAAI feature inform the storage array that the vm files is moved or deleted and allow the array to reclaim the blocks.

Workaround : to reclaim the free space , we have to use vmkfstools .Before running the reclaims command on the esxi host couple of things we need to check couple of things
1.verify the hardware acceleration status on the esxi storage tab which should show as supported or unknown if you disable the VAAI option from NetApp side.
2. not down the device id naa.XXXXXXXXXXXXXXXX , this can be noted from the esxi storage tab view or get into the ssh and run the below command

esxcli storage core device list | more

3.run this command on ssh session to see the lun is a thin provisioned lun or not

esxcli storage core device list –d naa.xxxxxxxxxxx 
at the end of the line check
                   Thin provisioning status : yes
                   Attached filters : VAAI_filter
                   VAAI status:supported (it may show unknown if it is disabled from Netapp)
1. Now we need to check if the delete is supported on that lun.run the below command on ssh to check this

esxcli storage core device vaai status get –d naa.XXXXXXXXXX

 last line of the result should show Delete status :supported

2. To reclaim the space run the below command on ssh .Navigate to the vmfs path by running the command.

Cd /vmfs/volumes/lun1/
Vmkfstools -y xx ( example vmkfstools –y 50 ------ to reclaim 50% of free space)

Note :don’t put reclaim percentage as 100 , because this will completely occupy your free space till the reclaim process get completed and if you have any other vm running on the same lun will go down if it doesn’t have enough space.

( where xx is the percentage of free space you want to reclaim) , for example consider a scenario that you have 100 GB lun name lun1. You have two vm each 25 GB disk , so you have 50 Gb free and 50 GB used. Now let vmotion one of the vm to another lun call this as lun2. Now lun 1 have 75 GB free space. If you want to put the reclaim percentage then put 50 which will create a balloon disk of size 37.5 GB ( 50 % of 75 GB) . putting 100% is not recommended as it may take the other vm down due to the space issue).

Virtual Storage Console: Host NetApp discovery hangs at terminating stale tasks @ 15%

Where this error shows:
1. In vsphere client it shows the task is hangs @ 15 % with terminating stale task.
2. Under Virtual storage console , any of the host is showing as unknown or disconnected.

 Cause for this issue:
1. Any of the ESXI host in the vsphere is in not responding or disconnected state.
2. Double check the account Permission used in adding the vsphere to the vsc.

 How to fix this issue:
1. Goto VSC and right click on the host which is showing as disconnected or not responding and click skip option and make sure you have check the small box in it. The host status should show as skipped if skip option is selected.
2. If the esxi host is permanently down then remove that from vsphere client.
3. Check the account permission used for the vsc console configuration having proper access to the vcenter.

     This is a bug in 4.1 version of VSC for vcenter version 5 and it is fixed in 4.2. the bug id is 592931. Here is the bug detail

http://support.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=592931

Unable to communicate with Flex flash controller: Operation ffcardsGet, status Error Timeout /Bootbank cannot be found at the path /bootbank” error

Where to See this error :

1.CIMC ---> storage --->  Cisco Flexflash ---->controller info

2.If you are running ESXI server on this C series server you will be getting “Bootbank cannot be found at the path /bootbank” error

 Cause for this issue: this is a known bug if you are running firmware version less than 1.5(3d) on c series server. The bug id is CSCuh33982. It is resolved in firmware version 1.5(3d).

https://tools.cisco.com/bugsearch/bug/CSCuh33982

Workaround to fix this issue:
1. Reboot CIMC --- this will not cause any disruptions to the esxi server or the vm’s running on it. The management page of CIMC page need to be reloaded. I would recommend to clear the cache and try to access the management page after CIMC restart.

How to restart the CIMC:

Goto Admin tab on CIMC and click reboot CIMC.
If you want to do that on CLI, ssh into the management ip address of CIMC and run the below commands.
                         Scope cimc
                         Reboot

2. If CIMC shows the controller is healthy now and ESXI still shows the same error then ssh into the ESXI server and restart the service by using the below command
Services.sh restart ( this will not affect your VM running on it but this will disconnect the esxi server and connect it back automatically when the service comes up).

Making Windows server 2012 as ISCSI target

Windows 2012 comes with the ISCSI target as a built in component, you have to just install the roles and make use of that.

How to install the ISCSI target role

1. Open Server Manager and Click Add Roles and Features
2. on Add Roles and features wizard Click next
3. Select Role based or feature based installation and click next
4. Click select a server from server pool option and select the server which is going to be running as iscsi target and click next
5. Expand file and storage services and select ISCSI Target server and click next
6. on Add roles and feature wizard , check the box include management tools and click Add features
7. Click next
8. Click next on the next window and click install

 How to configure the ISCI target
1. Open Server manager and click File and Storage Service
2. Click ISCSI Virtual Disks option
3. Click the “Launch the new virtual disk Wizard to create a virtual disk”
4. Select the option “type a custom path “
5. Browse to the path you want to create a target for example if you connect the external hard drive and the drive letter is F . then put the path F:\ISCSI-disk1 and click next.
6. On Specify ISCSI Virtual disk name you can give any name for example you can give it as VMdatastore1 or Hypervdatastore1
7. Click next
8. Specify the size of the disk for example 256 GB
9. Now select new ISCSI target option
10. Give any target name, if you using this for vmware cluster give that cluster name or if you using for hyperv cluster give that name
11. Click Add to add the initiators (Initiator will be your esxi host IQN or the iscsi ip address assigned to the esxi port group or if you are using hyperv then it should be the hyperv server ip address or the iqn)
12. Select the option “ Enter a value for the selected type” and select the type as IP address if you want to add ip or select IQN if you want to add ISCSI qualified name (IQN)
13. Click Next
14. Enable authentication, Enable the authentication based on your company policy , if you don’t have any idea then just click next without checking any option on this screen
15. Just go through the configuration overview which you select so far and click create
16. Click close

 How to add access the ISCSI disks now

1. Goto the server which you want to configure the ISCSI disk
2. Goto server manager click on TOOLS and select issi initiator
3. If the ISCSI service is not running then it will ask for the service start prompt click yes to start the ISCSI service
4. On the target tab put the ip address of your ISCSI target server name and click quick connect
5. Once it is able to connect to the target then the IQN of the target is listed under the quick connect and click done
 6. Goto disk management and rescan the disk.

Set as primary option is greyed out in ADFS certificate option

when we need to replace the token signing certificate or decryption certificate , after importing the new certificate , when we try to make the new certificate is primary , the primary option is greyed out

Cause : AutoCertificateRollover is enabled on the adfs properties.
How to fix that :

1.Open the powershell as administrator
2.Add-PSSnapin Microsoft.Adfs.PowerShell ---- this will load the powershell snapin module for the ADFS
3.get-adfsproperties --- this command will show you the Autocertificaterollover is $true which means it is enabled.
4. Set-ADFSProperties -AutoCertificateRollover $false ---- this sets the autocertificate rollover option to disable.
5.go back to your ADfs certificate console and right click on the new certificate and make that as primary certificate .
6. you can enable the autocertificaterollover back to enabled by running the Set-ADFSProperties -AutoCertificateRollover $true on powershell

Deleting a snapshots failed with the error: the virtual disk is either corrupted or not a supported format

 Error file location :

Two ways to see the vmware.log files

Method 1: using CLI

1.ssh into the ESXI host
2.navigate the virtual machine file path   Cd /vmfs/volumes/storage lun name/virtual machine name /
3.Cat vmware.log | more
4.hit space bar to see the log entries for the specific time
        
Note : the time might be in the UTC time if the esxi host is in UTC.

5.Look for the error on the specific time snapshot removal was initiated, the error will be “ SNAPSHOT: SnapshotConsolidateOpenDisks failed: Could not open/create change tracking file “

 Method 2 : Using GUI
1.In the vsphere client browse the datastore and navigate to the virtual machine file
2.Select the vmware.log file the current one and download to your desktop.
3. Open the file with wordpad or notepad and look for the error on the snapshot removal initiated time.
“ SNAPSHOT: SnapshotConsolidateOpenDisks failed: Could not open/create change tracking file”

 How to fix this issue :
1. Create a temp folder on the same datastore
2. Move all the files end with .CTK extension to the temporary folder
3. Go back to the vm and initiate the snapshot removal again.


Note : sometimes you may get the same error again but the snapshot will get deleted even though it is showing the same error.

Vcenter SQL transactional log full

 vcenter server service is getting stopped or terminated unexpectedly then one of the main reason for this problem is either database space or the transaction log file is full.

Where you will confirm the issue is due to the database transaction :

you have to view the vpxd.log files located under C:\ProgramData\VMware\VMware VirtualCenter\Logs\vpxd.log (for windows 2008 and above) . Open the vpxd file and go all the way down or check for the specific time where service get stopped.

For issue due to the database transaction log there will be error stating that

The transaction log for database 'databasename' is full. To find out why space in the log cannot be reused, see the log_reuse_wait_desc column in sys.databases" is returned when executing SQL statement.

How to fix this issue....

1. login to the sql server with the account having database admin access
2. Open the sql server management studio
3. Right click the vcenter database and click properties
4. Click options
5. set the recovery model to simple and click ok
6. Right click the database and click tasks --->shrink --- >files
7. select the file type as log
8. Confirm that the transaction log file name appears in the file name
9. select release unused space and click ok.
10. go back to vcenter and start the service.

Note : Selecting recovery model as simple has one drawback , for any reason if the database is corrupted you will be able to recover till the last full backup.