STP may cause temporary loss of network connectivity when a failover or failback event occurs (1003804)

Symptoms

In a switched network environment which uses Spanning Tree Protocol (STP), you experience these symptoms:

  • An ESXi or ESX host temporarily loses network connectivity when a failover or failback event occurs.
  • Virtual machines temporarily lose network connectivity when a failover or failback event occurs.
  • A VMware High Availability (HA) isolation event occurs after one of the teamed NICs of the COS is unplugged and plugged in to a different port.

Resolution

STP is used to accomplish a loop-free environment. Every time a port state goes up, STP calculation occurs. As the result of the calculation, the switch ports are either set to a forwarding or blocking state to prevent a traffic loop. STP topology convergence has four states:

  • Blocking
  • Listening
  • Learning
  • Forwarding

When STP convergence is initiated it forces all of the physical switches in the STP domain to dump their forwarding tables and relearn the STP topology and all MAC addresses. This process can take between 30-50 seconds. During this time, no user data passes through the port. Some user applications can time out during this period. Connectivity is restored when the STP domain completes this convergence.

To prevent the 30-50 second loss of connectivity during STP convergence, perform one of these options:

  • To set STP to Portfast on all switch ports that are connected to network adapters on an ESXi/ESX host
    Portfast allows the ports to immediately be set back to the forwarding state and prevents the link state changes that occur on ESX/ESXi hosts from affecting the STP topology. Setting STP to Portfast prevents the 30-50 second loss of network connectivity.
    The command to set STP to Portfast depends on the model of the switch. As the command is different from model to model and vendor to vendor, c ontact your physical switch vendor for more detailed information on how to configure the same.
    For example:
    To set STP to Portfast on a switch, run the below command based on the switch model:

    • CISCO-IOS
      spanning-tree portfast (for an access port)
      spanning-tree portfast trunk (for a trunk port)
    • NX-IOS
      spanning-tree port type edge (for an access port)
      spanning-tree port type edge trunk (for a trunk port)
    • To set STP to Portfast on a Dell switch, run the command:
      spanning-tree portfast
    • HP switches use a feature called admin-edge-port, which works the same way as Portfast or RSTP.
      To enable admin-edge-port, run the command:
      spanning-tree port-listadmin-edge-port
  • To disable STP
    VMware does not typically recommend that you disable STP. However, to prevent this issue from occurring, it may be necessary to disable STP. Before you disable STP, contact your switch vendor.
    The command to disable STP depends on the switch. Contact your switch vendor for more detailed information.
    For example:
    To disable STP on a Nortel switch, run the command:
    config ethernet stg stp disable

Taken from : http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003804

CentOS 7. Which installer need to Download: Minimal, DVD, or Everything?

Size of CentOS 7 iso. To determine which package that you want to install/download (DVD, Everything, or Minimal).

4.0G CentOS-7-x86_64-DVD-1503-01.iso
7.1G CentOS-7-x86_64-Everything-1503-01.iso
696M CentOS-7-x86_64-LiveCD-1503.iso
1.0G CentOS-7-x86_64-LiveGNOME-1503.iso
1.2G CentOS-7-x86_64-LiveKDE-1503.iso
636M CentOS-7-x86_64-Minimal-1503-01.iso
360M CentOS-7-x86_64-NetInstall-1503.iso

Better to start with minimal, and add package needed with yum install package later on after installation.

 

Kind Regards,
Doddi Priyambodo

Create your own vSphere 6 Private Lab

It is easy to create the vSphere Lab in your laptop!

These are the steps :

1. Prepare Storage :

EMC Appliance :

– http://techhead.co/webex-recordinginstall-run-the-emc-vnx-vsa-in-a-vsphere-lab/

– http://lab.piszki.pl/emc-vnx-for-file-simulator-instalacja-i-konfiguracja/

NetApp Appliance :

– http://vcommunique.blogspot.com/2013/09/netapp-simulator-on-vmware-vsphere.html

– http://vcommunique.blogspot.com/2013/09/netapp-simulator-on-vmware-vsphere.html

OR use open source NAS server (FreeNAS is easier, OpenFiler, etc)

2. Prepare Gateway, NTP, LDAP/AD, DNS

Create DNS and NTP easily with  this = http://www.virten.net/2013/12/howto-use-vcenter-server-appliance-vcsa-as-dns-server/

LDAP can be created easily with turnkeylinux openldap

3. Prepare vSphere ESXi

4. Prepare vCenter

DONE! 🙂

High Level Best Practice Configuration yang perlu dicek untuk VMware vSphere Production Environment

Berikut ini ada beberapa konfigurasi best practice yang perlu dicek untuk mengetahui apakah environment VMware yang anda miliki saat ini sudah appropriate atau  tidak untuk production level. Ini adalah guidance secara high level saja. Untuk detailnya perlu dijelaskan lebih lanjut, mudah2an dapat saya teruskan untuk beberapa komponen dibawah ini.

Component Recommended Action Item
Compute Configure firewall rules and ports according to best practices.
Compute VMware vSphere ESXi Shell and SSH access should be configured per the customer security and manageability requirements.
Datacenter Use vCenter Server roles, groups, and permissions to provide appropriate access and authorization to the VMware virtual infrastructure. Avoid using Windows built-in groups (Administrators).
Datacenter Tasks and Events Retention Policy set in the environment.
Datacenter Size with HA host failure considerations.
Datacenter Set up redundancy for the management port (either using a separate vmnic or a separate uplink) and an alternate isolation response gateway address (if appropriate) for more reliability in HA isolation detection.
Datacenter Maintain compatible and homogeneous (CPU and memory) hosts within a cluster to support the required functionality for vMotion, vSphere DRS, VMware vSphere Distributed Power Management (DPM), VMware vSphere HA, and vSphere FT.
Network Verify that there is redundancy in networking paths and components to avoid single points of failure. For example, provide at least two paths to each network.
Network Configure networking consistently across all hosts in a cluster.
Network If jumbo frames are enabled, verify that jumbo frame support is enabled on all intermediate devices and that there is no MTU mismatch.
Network Minimize differences in the number of active NICs across hosts within a cluster.
Network Configure networks so that there is separation of traffic (physical or logical using VLANs).
Network Use DV Port Groups to apply policies to traffic flow types and to provide Rx bandwidth controls through the use of Traffic Shaping.
Network Use Load-Based Teaming (LBT) to balance virtual machine network traffic across multiple uplinks.
Network Use Network I/O Control (NetIOC) to prioritize traffic on 10GbE network uplinks.
Network Adjust load balancing settings from the default virtual port ID only if necessary.
Storage Minimize differences in datastores visible across hosts within the same cluster or vMotion scope.
Storage NFS and iSCSI storage traffic should be separated physically (for performance) and logically (for security).
Virtual Machines Limit use of snapshots, and when using snapshots limit them to short-term use.
Virtual Machines Verify that VMware Tools is installed, running, and up to date for running virtual machines.
Virtual Machines Verify that virtual machines meet the requirements for vSphere vMotion.
Compute Avoid unnecessary changes to advanced parameter settings.
Datacenter Enable bidirectional CHAP authentication for iSCSI traffic so that CHAP authentication secrets are unique.
Datacenter Disconnect vSphere Clients from the vCenter Server when they are no longer needed.
Datacenter Maintain compatible virtual hardware versions for virtual machines to support vMotion.
Licensing Verify that adequate licenses are available for vCenter Server instances.
Licensing Verify that adequate CPU licenses are available for ESXi hosts.
Network Distribute vmnics for a port group across different PCI buses for greater redundancy.
Network Change port group security default settings for Forged Transmits, Promiscuous Mode, and MAC Address Changes to Reject unless the application requires the defaults.
Storage Use shared storage for virtual machines instead of local storage.
Storage Size datastores appropriately.
Storage Allocate space on shared datastores for templates and media/ISOs separately from datastores for virtual machines.
Storage Use Storage I/O Control (SIOC) to prioritize high importance virtual machine traffic.
Virtual Machines As a security enhancement initiative, disable certain unexposed features.
Virtual Machines Limit sharing console connections if there are security concerns.
Virtual Machines Allocate only as much virtual hardware as required for each virtual machine. Disable any unused or unnecessary or unauthorized virtual hardware devices.
Virtual Machines Consider using the latest virtual hardware version to take advantage of additional capabilities.
Virtual Machines Use the latest version of VMXNET that is supported by the guest operating system.
Virtual Machines Use reservations and limits selectively on virtual machines that need it. Don’t set reservations too high or limits too low.
Virtual Machines Select the correct guest operating system type in the virtual machine configuration to match the guest operating system.

Kind Regards,
Doddi Priyambodo

 

Bagaimana cara belajar VMware untuk Pemula? (VMware Tutorial Indonesia)

Jika ingin belajar megenai product VMware, berikut ini adalah link public yang bisa dibuka dan dapat menjadi reference :

1. Official Website VMware (http://www.vmware.com), ada banyak public material yang di-share disana.

2. VMwareTV di youtube channel (https://www.youtube.com/user/vmwaretv), referensi video-nya bagus dan silahkan lanjut  browsing ke beberapa channels disana

3. Website kumpulan video (http://www.vmwarelearning.com), kumpulan video-video yang sangat bagus

4. VMware Feature walk through (http://featurewalkthrough.vmware.com), tutorial step by step untuk VMware for newbie

5. Laboratorium Virtual di Cloud! (http://labs.hol.vmware.com/), one word from me: “WOW!”

6. Subscribe blog ini regularly 🙂

7. Join VMware Class di authorized training yang tersebar di Indonesia

8. Ada beberapa resources internal  (ex:vault portal) yang aksesnya hanya dimiliki oleh VMware Employee dan VMware Partner. Coba berkenalan dan tanyakan ke mereka, mungkin ada beberapa public material yang bisa di-share oleh mereka.

 

Kind Regards,
Doddi Priyambodo

Troubleshooting – Lokasi Log File dari VMware vRealize Automation 6.x

Berikut ini adalah informasi mengenai lokasi logs dari VMware vRealize Automation 6.x Suite (dahulu namanya adalah VMware vCloud Automation Center).

Troubleshooting dilakukan dengan membaca/menganalisa beberapa log yang terjadi dalam sebuah sistem. Lokasi dari file log ini tersebar di beberapa server berdasarkan keputusan arsitektur dari vRA pada saat instalasi/deployment, apakah menggunakan mekanisme distributed deployment atau simple deployment.

 

vRealize Automation Virtual Appliance Locations
Description
/var/log/vcac/catalina.out
tc Server Runtime logs, vRealize Automation webapp logs
/var/log/vco/app-server/catalina.out
vRealize Automation’s built-in vRealize Orchectrator logs
/var/log/apache2/access_log
Apache Access logs
/var/log/apache2/error_log
Apache GET/POST Error logs
/var/log/apache2/ssl_request_log
Apache SSL troubleshooting logs
vRealize Automation Infrastructure as a Service Locations
Description
C:Program Files (x86)VMwarevCACAgentsagent_namelogsfile
Plug-in logs example: CPI61, nsx, VC50, VC51Agent, VC51TPM, vc51withTPM, VC55Agent, vc55u, VDIAgent
C:Program Files (x86)VMwarevCACDistributed Execution ManagerDEMORLogsDEMOR_All Distributed Execution Manager logs
C:Program Files (x86)VMwarevCACDistributed Execution ManagerDEMWRLogsDEMWR_All Distributed Execution Worker logs
C:Program Files (x86)VMwarevCACServerLogs Manager Service logs
C:Program Files (x86)VMwarevCACServerConfigToolLogvCACConfiguration-date Repository Configuration logs
C:Program Files (x86)VMwarevCACServerModel Manager DataLogsnothing_today IIS Access logs (usually empty, but can be expected)
C:Program Files (x86)VMwarevCACServerModel Manager WebLogsRepository Repository logs
C:Program Files (x86)VMwarevCACServerWebsiteLogsWeb_Admin_All Web Admin logs
C:inetpublogs IIS logs
Identity Virtual Appliance Locations Description
/var/log/vmware/sso/catalina.out ID VA tc Server Runtime logs
/var/log/vmware/sso/ssoAdminServer.log
SSO Admin Server logsNote: Not applicable to vRealize Automation.
/var/log/vmware/sso/vmware-identity-sts-perf.log STS performance logs
/var/log/vmware/sso/vmware-identity-sts.log STS logs
/var/log/vmware/sso/vmware-sts-idmd-perf.log Identity service performance logs
/var/log/vmware/sso/vmware-sts-idmd.err Identity service error logs
/var/log/vmware/sso/vmware-sts-idmd.log Identity service logs
/var/log/vmware/vmafd/vmafdd.log Identity VA logs
/var/log/vmware/vmdir/vdcsetupldu.log Initial setup logs
/var/log/vmware/vmdir/vmafdvmdirclient.log VMware SSO LDAP initial configuration logs
/var/log/vmware/vmkdc/vmkdcd.log VMware SSO LDAP initial configuration logs
vRealize Application Services Location Description
/home/darwin/tcserver/darwin/logs/catalina.out Application Services tc Server Runtime logs
vMware vRealize Business Standard Description
/usr/local/tcserver/vfabric-tc-server-standard/tcinstance1/logs/catalina.out vRealize Business Advanced and Enterprise tc Server Runtime logs
/usr/local/tcserver/vfabric-tc-server-standard/tcinstance1/logs/auditFile.log REST API requests
/usr/local/tcserver/vfabric-tc-server-standard/tcinstance1/logs/itfm-external-api.log API logs
/usr/local/tcserver/vfabric-tc-server-standard/tcinstance1/logs/itfm-reflib-update.log vRealize Business standard reference library related changes.
/usr/local/tcserver/vfabric-tc-server-standard/tcinstance1/logs/itfm-vc-dc.log Data collector logs
/usr/local/tcserver/vfabric-tc-server-standard/tcinstance1/logs/itfm.log vRealize Business Advanced and Enterprise logs
vCenter Server Appliance (VCSA) 5.5.x Locations Description
/var/log/vmware/vpx/vpxd.log vCenter VPXD logs
/var/log/vmware/vpx/vpxd-alert.log vCenter VPXD alert logs
/var/log/vmware/vpx/vws.log Management Web Service logs
/var/log/vmware/vpx/vmware-vpxd.log vCenter VPXD status change logs
/var/log/vmware/vpx/inventoryservice/ds.log vCenter Inventory Service logs
/var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log vSphere Client logs
/var/log/vmware/vsphere-client/logs/virgo-server/log.log vSphere Client logs
/var/log/vmware/vsphere-client/eventlogs/eventlog.log vSphere Client event logs
vCenter SSO Locations Description
/var/log/vmware/sso/catalina.out SSO tc Server Runtime logs
/var/log/vmware/sso/ssoAdminServer.log SSO Admin Server logs (only in 5.5.x version)
/var/log/vmware/sso/vmware-identity-sts-perf.log STS performance logs
/var/log/vmware/sso/vmware-identity-sts.log STS logs
/var/log/vmware/sso/vmware-sts-idmd-perf.log Identity service performance logs
Kind Regards,
Doddi Priyambodo

(VMware Tutorial) Step by Step Installation for VMware vRealize Automation Distributed 6.1 (Part 1)

I would like to write a series on installation and configuration of VMware VRealize Automation (previously it was called vCloud Automation Center). I am writing these blog series based on my experience implementing it in one of big company, and they need a very big scalability infrastructure for their Cloud Management Platform. (> 50.000 users).

So, these are the steps to install, deploy & configure VRA 6.1 :

Step by Step Installation for VMware vRealize Automation Distributed 6.1 :

  1. Install & Configure F5 LTM and F5 GTM Load Balancer, DONE
  2. Create & Configure SAN Trusted Certificate (key, pem, pfx) from CA Server, DONE
  3. Install & Configure VMware Identity Appliance + certificate, DONE
  4. Install & Configure VCAC Appliance Primary + certificate, DONE
  5. Install & Configure Primary PostgreSQL. DONE
  6. Change DB for VCAC Appliance Primary to Stand Alone, DONE
  7. Install & Configure VCAC Appliance Secondary (Cluster), DONE
  8. Install & Configure VCAC Appliance Virtual Server (LB), DONE
  9. Install & Configure Secondary PostgreSQL (Cluster), DONE
  10. Test Cluster Functionality to VCAC from Load Balancer, DONE
  11. Test Cluster Functionality to PostgreSQL Database, DONE
  1. Configure vCenter and SQL Server to Join Domain,  DONE
  2. Install & Configure IAAS Web Primary + certificate, DONE
  3. Install & Configure IAAS Web Secondary + certificate, DONE
  4. Install & Configure IAAS Web Virtual Server (LB), DONE
  5. Install & Configure IAAS Mgr & DEM Orch Primary + certificate, DONE
  6. Install & Configure IAAS Mgr & DEM Orch Secondary + certificate, DONE
  7. Install & Configure IAAS Mgr & DEM Orch Virtual Server (LB), DONE
  8. Install & Configure DEM Worker @1st Site, DONE
  9. Install & Configure IAAS Agent @1st Site, DONE
  10. Install & Configure DEM Worker @2nd Site,  DONE
  11. Install & Configure IAAS Agent @2nd Site, DONE
  12. Install & Configure DEM Worker @3rd Site, DONE
  13. Install & Configure IAAS Agent @3rd Site, DONE
  1. Install & Configure VCO Server 1, DONE
  2. Install & Configure VCO Server 2, DONE
  3. Install & Configure VCO Virtual Server (LB), DONE
  1. Install & Configure ITBM, DONE

 

Step by Step for vCenter Upgrade from 5.1 to 5.5

I am taking this Tutorial from VMware Communitiies (VMTN). This is a very common questions to ask, so I re-post it again in this blog.

Make sure below 6 tasks has been completed before starting of vCenter Upgrade.

  1. 2 hours before the upgrade disable all Schedule VM backups for vCenter or 3rd Party Tools
  2. Once backup disabled request Database Team to backup below databases 30 mins before the vCenter upgrade –
    1. vCenter Server database
    2. SRM database
  3. On vCenter, make a copy of the SSL certificates, copy them backup folder
  4. Backup Inventory Service database. Follow below procedure
  • On the source machine, open the command prompt in the vCenter Server and change the directory to vCenter_Server_installation_directoryInfrastructureInventory Servicescripts.
  • Run the following command at the prompt to back up the Inventory Service database.
  • backup.bat -file gdciventoryvcenter.bak
  • When the backup operation finishes, the message Backup completed successfully appears, copy the file to backup folder
  1. Backup the SSO configuration
    1. Click on start/programs/vmware
    2. Select Generate vCenter Single Signon Backup bundle, this creates a zip file on the desktop
    3. Select Generate vCenter single Signon log bundle
    4. Copy these files to backup folder
  2. 5 minutes prior to upgrade of vCenter

    Create a snapshot of vCenter.

    ********** UPGRADATION **********

Once the jobs or tasks written in above completed, we will start with vCenter Server Upgrade from 5.1 to 5.5

We need to upgrade the vCenter Server in below sequence.

  1. Single Sign On – SSO
  2. vSphere Web Client
  3. vSphere Inventory Service
  4. vCenter Service
  5. vCenter Update Manager
  6. VMware Site Recovery Manager

     

  7. Single Sign On Upgrade:

     

  8. Mount the ISO and kick off the SSO installer under custom upgrade.
    1. All the required install files for all products we are upgrading are located on each VC server under “d:vCenter 5.5 and related”

     

Continue reading Step by Step for vCenter Upgrade from 5.1 to 5.5