Why Smaller vCPU is better than Bigger vCPU in a fully probable contention environment

In VMware vSphere environment, why Smaller vCPU is better than Bigger vCPU (if the workloads only require few vCPU) in a fully probable contention environment?

To explain this further let’s take an example of a four pCPU host that has four VMs, three with 1 vCPU and one with 4 vCPUs. At best only the three single vCPU VMs can be scheduled concurrently. In such an instance the 4 vCPU VM would have to wait for all four pCPUs to be idle. In this example the excess vCPUs actually impose scheduling constraints and consequently degrade the VM’s overall performance, typically indicated by low CPU utilization but a high CPU Ready figure.

So, always start with smaller vCPU and then add extra vCPU later on if needed based on your observation about the workload.

This reference post also share a very good description why too many vCPU will give poor performance to your Virtual Machine: http://www.gabesvirtualworld.com/how-too-many-vcpus-can-negatively-affect-your-performance/

Conclusion: “Right Size Your VMs!”

 

Kind Regards,
Doddi Priyambodo

Why do we need to Virtualize our Oracle Database

Usually customer would like to expand the benefits that they already achieved using virtualization (financial, business and operational benefits of virtualization within its operating environment) to another level. For example to Business Critical Applications such as Oracle Database, thereby reaping the many benefits and advantages through its adoption of this infrastructure.

Customer aims to achieve the following benefits:

  • Effectively utilise datacenter resources, as in traditional physical servers a lot of database server only utilize 30% of the resources.
  • Maximise availability of the Oracle environment at lower cost, as virtualization can give another layer of high availability.
  • Rapidly deploy Oracle database servers for development, testing & production, as virtualization can have templates and automation.
  • Maximise uptime during planned maintenance, as virtualization can give the ability to move database to another machine without any downtime for the workload.
  • Minimise planned and unplanned downtime, as virtualization can give better disaster recovery avoidance and disaster recovery actions.
  • Automated testing and failover of Oracle datacenter environments for disaster recovery and business continuity.
  • Achieve IT Compliance, as we have better monitoring systems, audit mechanism, policy enforcement, and asset managements.
  • Minimise Oracle datacenter costs for floor space, energy, cooling, hardware and labour, as some physical servers can be consolidated into just several physical servers. This will give customer a better TCO/ROI compare to physical servers approach.

 

Kind Regards,
Doddi Priyambodo

 

 

Update sequence for vSphere 6.0 and its compatible VMware products

Following our technical discussion regarding upgrade VMware environments, actually I already wrote about this topic in different thread  in this blog. But, I would like to emphasise again by using another KB from VMware. VMware has made available certain releases to address critical issues and architectural changes for several products to allow for continued interoperability:

  • vCloud Connector (vCC)
  • vCloud Director (vCD)
  • vCloud Networking and Security (VCNS, formerly vShield Manager)
  • VMware Horizon View
  • VMware NSX for vSphere (NSX Manager)
  • vCenter Operations Manager (vCOPs)
  • vCenter Server / vCenter Server Appliance
  • vCenter Infrastructure Navigator (VIN)
  • vCenter Site Recovery Manager (SRM)
  • vCenter Update Manager (VUM)
  • vRealize Automation Center (vRA, formerly known as vCloud Automation Center)
  • vRealize Automation Application Services (vRAS, formerly vSphere AppDirector)
  • vRealize Business, IT Cost Management (ITBM, formerly VMware IT Business Management)
  • vRealize Configuration Manager (VCM, formerly vCenter Configuration Manager)
  • vRealize Hyperic
  • vRealize Log Insight (vRLI)
  • vRealize Operations Manager (vROPs, formerly known as vCenter Operations Manager, vCOPs)
  • vRealize Orchestrator (vRO, formerly vCenter Orchestrator)
  • vSphere Big Data Extension (BDE)
  • vSphere Data Protection (VDP)
  • vSphere Replication (VR)
  • vSphere ESXi
  • vShield Edge / NSX Edge
  • vShield App / NSX Logical Firewall (NSX LFw)
  • vShield Endpoint / NSX Guest Introspection and Data Security (NSX Guest IDS)
This article only encompasses environments running vSphere and/or vCloud Suite 6.0 and VMware products compatible with vSphere 6.0.

In an environment with vSphere 6.0 and its compatible VMware products, perform the update sequence described in the Supported Update Sequence table.

Supported Update Sequence

Continue reading Update sequence for vSphere 6.0 and its compatible VMware products

VMware vSphere® Metro Storage Cluster Recommended Practices for VMware vSphere 6.0

Some of my customers ask about Metro Storage Cluster configuration for VMware Deployment to achieve better availability of their precious data. There is a very good resource from Duncan Epping (one of VMware most respectful technologist). One of the topic is the Requirement and Constraints from VMware technology perspective. Well, this is the explanation taken from the whitepaper.

Technical Requirements and Constraints

Due to the technical constraints of an online migration of VMs, the following specific requirements, which are listed in the VMware Compatibility Guide, must be met prior to consideration of a stretched cluster implementation:

  • Storage connectivity using Fibre Channel, iSCSI, NFS, and FCoE is supported.
  • The maximum supported network latency between sites for the VMware ESXiTM management networks is 10ms round-trip time (RTT).
  • vSphere vMotion, and vSphere Storage vMotion, supports a maximum of 150ms latency as of vSphere 6.0, but this is not intended for stretched clustering usage.
  • The maximum supported latency for synchronous storage replication links is 10ms RTT. Refer to documentation from the storage vendor because the maximum tolerated latency is lower in most cases. The most commonly supported maximum RTT is 5ms.
  • The ESXi vSphere vMotion network has a redundant network link minimum of 250Mbps.The storage requirements are slightly more complex. A vSphere Metro Storage Cluster requires what is in effect a single storage subsystem that spans both sites. In this design, a given datastore must be accessible—that is, be able to be read and be written to—simultaneously from both sites. Further, when problems occur, the ESXi hosts must be able to continue to access datastores from either array transparently and with no impact to ongoing storage operations.

Reference:
Download the complete document from here: vmware-vsphere-metro-storage-cluster-recommended-practices-white-paper (http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/vmware-vsphere-metro-storage-cluster-recommended-practices-white-paper.pdf)

 

Kind Regards,
Doddi Priyambodo

Kebutuhan Minimum dari VMware vCenter Appliance 6.x

I know that you can find this requirements in the Knowledge Based, I just want to write this again to remind me. Because I got a lot of this question from my customer.

Resource
Requirement
Disk storage on the host machine
Embedded Platform Services Controller:
  • Tiny: 120GB
  • Small: 150GB
  • Medium: 300GB
  • Large: 450GB
External Platform Services Controller:
  • Tiny: 86GB
  • Small: 108GB
  • Medium: 220GB
  • Large: 280GB
External Platform Services Controller Appliance:
  • Tiny: 30GB
  • Small: 30GB
  • Medium: 30GB
  • Large: 30GB
Memory in the vCenter Server Appliance

Platform Services Controller Only: 2GB Ram

All components on one Appliance.

  • Tiny: 8GB RAM
  • Small: 16GB RAM
  • Medium: 24GB RAM
  • Large: 32GB RAM
CPUs in the vCenter Server Appliance

Platform Services Controller Only: 2 CPUs

All components on one Appliance.

  • Tiny: 2 CPUs
  • Small: 4 CPUs
  • Medium: 8 CPUs
  • Large: 16 CPUs
Notes:
  • Tiny Environment (up to 10 Hosts, 100 Virtual Machines)
  • Small Environment (up to 100 Hosts, 1,000 Virtual Machines)
  • Medium Environment (up to 400 Hosts, 4,000 Virtual Machines)
  • Large Environment (up to 1,000 Hosts, 10,000 Virtual Machines)

 

 

VSAN Erasure Coding – Storage Based Policy Management

A new policy setting has been introduced to accommodate the new RAID-5/RAID-6 configurations in VSAN (only available in All-Flash configuration). Minimum 4 hosts will be required for RAID5, and minimum 6 hosts will be required for RAID6 configuration.

This new policy setting is called Failure Tolerance Method. This policy setting takes two values: performance and capacity. When it is left at the default value of performance, objects continue to be deployed with a RAID-1/mirror configuration for the best performance. When the setting is changed to capacity, objects are now deployed with either a RAID-5 or RAID-6 configuration.

The RAID-5 or RAID-6 configuration is determined by the number of failures to tolerate setting. If this is set to 1, the configuration is RAID-5. If this is set to 2, then the configuration is a RAID-6.

 

Kind Regards,
Doddi Priyambodo

What is Hadoop? Why do we need to virtualize it using VMware?

What is Hadoop?

Hadoop is an Apache open source project that provides scalable and distributed computing, originally built by Yahoo!. It provides a framework that can process large amounts of data by leveraging the parallel and distributed processing of many compute nodes arrayed in a cluster. These clusters can be configured as a single host or scaled up to utilize thousands of machines depending on the workload.

What are Hadoop Components?

These are the core modules of Hadoop, which build the capabilities to conduct distributed computing capabilities.

  • Hadoop Common – The utilities that support the other Hadoop modules.
  • Hadoop Distributed File System – The distributed file system used by most Hadoop distributions . Also known by its initials, HDFS.
  • Hadoop YARN – Used to manage cluster resources and schedule jobs.
  • Hadoop Map Reduce – YARN based system of processing large amounts of data.

In addition to the core modules, there are others that provide specific and specialized capabilities to this distributed processing framework. These are just some of the tools:

  • Ambari – A web-based tool for provisioning, management, and monitoring of Hadoop clusters.
  • HBase – Distributed database that supports structured data storage.
  • Hive – Data warehouse model with data summarization and ad hoc query capability.
  • Pig – Data flow language.
  • ZooKeeper – Coordination service for distributed applications.

These are modules available from the Apache open-source project, but there are also more than thirty  companies that provide Hadoop distributions that include the open-source code as well as adding competing management solutions, processing engines, and many other features.  Some of the best known and widest used are distributed from Cloudera, MapR, and Hortonworks.

Why do we need to Virtualize Hadoop workloads?

Now, after we know about Hadoop. We always discuss about virtualization in this blog. Is hadoop suitable to be virtualized? Yes, if you would like to have these additional values for Hadoop. Then you should consider to virtualize the workload.

  • Better resource utilization:
    Collocating virtual machines containing Hadoop roles with virtual machines containing different workloads on the same set of VMware ESXi™ server hosts can balance the use of the system. This leads to lower operating expenses and lower capital expenses as you can leverage the existing infrastructure and skills in the data center and you do not have to invest in bare-metal servers for your Hadoop deployment.
  • Alternative storage options:
    Originally, Hadoop was developed with local storage in mind, and this type of storage scheme can be used with vSphere also. The shared storage that is frequently used as a basis for vSphere can also be leveraged for Hadoop workloads. This re-enforces leveraging the existing investment in storage technologies for greater efficiencies in the enterprise.
  • Isolation:
    This includes running different versions of Hadoop itself on the same cluster or running Hadoop alongside other applications, forming an elastic environment, or different Hadoop tenant. Isolation can reduce your overall security risk, ensure you are meeting your SLA’s, and support Hadoop as a service back to the lines of business.
  • Availability and fault tolerance:
    The NameNode, the Resource Manager and other Hadoop components, such as Hive Metastore and HCatalog, can be single points of failure in a system. vSphere services such as VMware vSphere High Availability (vSphere HA) and VMware vSphere Fault Tolerance (vSphere FT) can protect these components from server failure and improve availability.
  • Balance the loads:
    Resource management tools such as VMware vSphere vMotion® and VMware vSphere Distributed Resource Scheduler™ (vSphere DRS) can provide availability during planned maintenance and can be used to balance the load across the vSphere cluster.
  • Business critical applications:
    Uptime consideration is just as important in a Hadoop environment, why would the enterprise want to go back in time to a place where the servers and server components were single points of failure. Leverage the existing investment in vSphere to drive meeting SLA’s and providing an excellent service back to the business.

VMware also have the component called VMWARE BIG DATA EXTENSIONS (https://www.vmware.com/products/big-data-extensions), to rapidly deploy High Available Hadoop components and easily manage the infrastructure workloads

vSphere Big Data Extensions enables rapid deployment, management, and scalability of Hadoop in virtual and cloud environments. It also has the functionality to do scale in/out capabilities built into Big Data Extensions tools enables on-demand Hadoop instances.

Simple cloning to sophisticated end-user provisioning products such as VMware vRealize Automation™ can speed up the deployment of Hadoop. This enables IT to be a service provider back to the business and provide Hadoop as a service back to the different lines of business, providing faster time to market. This will further enable today’s IT to be a value driver vs. seen as a cost center.

For more detail about VMware Big Data Extensions, please see this datasheet from VMware Inc. = https://www.vmware.com/files/pdf/products/vsphere/VMware-vSphere-Big-Data-Extensions-Datasheet.pdf

 

Kind Regards,
Doddi Priyambodo

Installation and Documentation Guide for VMware SDDC Proof of Concept

POC Installation and Documentation generally available online both in VMware website and in different blogs, but these are some recommendations:

Google.com and VMware.com of course…

 

Kind Regards,
Doddi Priyambodo

MICROSERVICES – What is Cloud Native Application?

DevOps, Containers, Docker, Mesos, Kubernetes, microservices, 12-factor applications, 3rd platform, oh my!   Modern application architecture and lifecycle is changing fast and that means even more demands on IT.  While some have argued that this new application approach calls for a whole new infrastructure,  actually these new business-driven demands head on, leveraging your existing investment while still delivering the highest SLAs – performance, availability, security, compliance, and disaster recovery.  This emerging 3rd Platform Application stack not only fits into existing SDDC infrastructure investments but is actually the best place to run containers and emerging 3rd platform applications.

Application Development and Delivery

 

If we look at the Outcomes Delivered from a new model of IT, Businesses are increasing their focus on App and Infrastructure Delivery Automation throughout the datacenter.

3RD PLATFORM – MICROSERVICES

3rd Platform! Microservices! What the heck are they? Put simply, the 3rd platform is this is a new paradigm for architecting applications to operate in a distributed fashion. While the 1st platform was designed around mainframes and the 2nd platform was designed around client-server, the 3rd platform is designed around the cloud. In other words, applications are designed and built to live in the cloud. We can effectively think of this as pushing many of the core infrastructure concepts (like availability and scale) into the architecture of the application itself with containers being a large part of this; they can be thought of as lightweight runtimes for these applications. With proper application architecture and a rock solid foundation either on-premise or in the cloud, applications can scale on demand, new versions can be pushed quickly, components can be rebuilt and replaced easily, as well as many other benefits discussed below.

History of Platforms

1st Platform systems were based around mainframes and traditional servers without virtualization. Consolidation was a serious issue and it was normal to run one application per physical server.

2nd Platform architectures have been the standard mode for quite a while. This is the traditional Client/Server/Database model with which you are likely very familiar, leveraging the virtualization of x86 hardware to increase consolidation ratios, add high availability and extremely flexible and powerful management of workloads.

3rd Platform moves up the stack, standardizing on Linux Operating Systems primarily, which allows developers to focus on the application exclusively. Portability, scalability and highly dynamic environments are valued highly in this space. We will focus on this for the rest of the module.

Does this mean you should immediately move all of your applications to this model? Not so fast! While 3rd Platform architectures are exciting and extremely useful, they will not be the answer for everyone. A thorough understanding of the benefits and, more importantly the complexities in this new world are extraordinarily important. VMware’s Cloud-Native Apps group is dedicated to ensuring our customers are well informed in this space and can adopt this technology confidently and securely when the time is right.

Microservices are growing in popularity, due in no small part to companies like Netflix and Paypal that have embraced this relatively new model. When we consider microservices, we need to understand both the benefits and the limitations inherent in the model, as well as ensure we fully understand the business drivers.

At its heart, microservice architecture is about doing one thing and doing it well. Each microservice has one job. This is clearly in stark contrast to the monolithic applications many of us are used to; using microservices, we can update components of the application quickly without forcing a full recompile of the entire application. But it is not a “free ride” – this model poses new challenges to application developers and operations teams as many assumptions no longer hold true.

The recent rise of containerization has directly contributed to the uptake of microservices, as it is now very easy to quickly spin up a new, lightweight run-time environments for the application.

The ability to provide single-purpose components with clean APIs between them is an essential design requirement for microservices architecture. At their core, microservices have two main characteristics; they are stateless and distributed. To achieve this, let’s take a closer look at the Twelve-Factor App methodology in more detail to help explain microservices architecture as a whole.

THE TWELVE FACTOR APP

To allow the developer maximum flexibility in their choice of programming languages and back-end services, Software-as-a-Service web applications should be designed with the following characteristics:

  • Use of a declarative format to attempt to minimize or eliminate side effects by describing what the program should accomplish, rather than describing how to go about it. At a high level it’s the variance between a section of code and a configuration file.
  • Clean Contract with the underlying Operating Systems which enables portability to run and execute on any infrastructure. API’s are commonly used to achieve this functionality.
  • Ability to be deployed into modern cloud platforms; removing the dependencies on underlying hardware and platform.
  • Keep development, staging, and production as similar as possible.  Minimize the deviation between the two environments for continuous development.
  • Ability to scale up (and down) as the application requires without needing to change the tool sets, architecture or development practices.

At a high level, the 12 Factors that are used to achieve these characteristics are:

  1. Codebase – One codebase tracked in revision control, many deploys
  2. Dependencies – Explicitly declare and isolate dependencies
  3. Config – Store config in the environment
  4. Backing Services – Treat backing services as attached resources
  5. Build, release, run – Strictly separate build and run stages
  6. Process – Execute the app as one or more stateless processes
  7. Port Binding – Export services via port binding
  8. Concurrency – Scale out via the process model
  9. Disposability – Maximize robustness with fast startup and graceful shutdown
  10. Dev/Pro Parity – Keep development, staging, and production as similar as possible
  11. Logs – Treat logs as event streams
  12. Admin Process – Run admin/management tasks as one-off processes

For additional detailed information on these factors, check out 12factor.net.

BENEFIT OF MICROSERVICES

Microservice architecture has benefits and challenges. If the development and operating models in the company do not change, or only partially change, things could get muddled very quickly. Decomposing an existing app into hundreds of independent services requires some choreography and a well thought-out plan. So why are teams considering this move? Because there are considerable benefits!

Resilience

 With a properly architected microservice-based application, the individual services will function similarly to a bulkhead in a ship. Individual components can fail, but this does not mean the ship will sink. The following tenet is held closely by many development teams – “Fail fast, fail often.” The quicker a team is able to identify a malfunctioning module, the faster they can repair it and return to full operation.

Consider an online music player application – as a user, I might only care about playing artists in my library. The loss of the search functionality may not bother me at all. In the event that the Search service goes down, it would be nice if the rest of the application stays functional. The dev team is then able to fix the misbehaving feature independently of the rest of the application.

Defining “Service Boundaries” is important when architecting a microservice-based application!

Scaling

If a particular service is causing latency in your application, it’s trivial to scale up instances of that specific service if the application is designed to take full advantage of microservices. This is a huge improvement over monolithic applications.

Similar to the Resilience topic, with a monolithic application, one poorly-performing component can slow down the entire application. With microservices, it is almost trivial to scale up the service that is causing the latency. Once again, this scalability must be built into the application’s DNA to function properly.

Deployment

Once again, microservices allow components to be upgraded and even changed out for entirely new, heterogeneous pieces of technology without bringing down the entire application. Netflix pushes updates constantly to production code in exactly this manner.

Misbehaving code can be isolated and rolled back immediately. Upgrades can be pushed out, tested, and either rolled back or pushed out further if they have been successful.

Organizational

“Organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations” –Melvin Conway

The underlying premise here is that the application should align to the business drivers, not to the fragmentation of the teams. Microservices allow for the creation of right-sized, more flexible teams that can more easily align to the business drivers behind the application. Hence, ideas like the “two pizza rule” in which teams should be limited to the number of people that can finish two pizzas in a sitting (conventional wisdom says this is eight or less…though my personal research has proved two pizzas do not feed more than four people.)

No Silver Bullet!

Microservices can be accompanied by additional operations overhead compared to the monolithic application provisioned to a application server cluster.  When each service is separately built out, they could each potentially require clustering for fail over and high availability.  When you add in load balancing, logging and messaging layers between these services, the real-estate starts to become sizable even in comparison to a large off the shelf application. Microservices also require a considerable amount of DevOps and Release Automation skills. The responsibility of ownership of the application does not end when the code is released into production, the Developer of the application essentially owns the application until it is retired. The natural evolution of the code and collaborative style in which it is developed can lend itself to challenges when making a major change to the components of the application.  This can be partially solved with backwards compatibility but it is not the panacea that some in the industry may claim.

Microservices can only be utilized in certain use cases and even then, Microservices open up a world of new possibilities that come with new challenges and operational hurdles. How do we handle stateful services? What about orchestration? What is the best way to store data in this model? How do we guarantee a data persistence model? Precisely how do I scale an application properly? What about “simple” things like DNS and content management?  Some of these questions do not have definitive solutions yet.  A distributed system can also introduce a new level of complexity that may not have been such a large concern like network latency, fault tolerance, versioning, and unpredictable loads in the application.  The operational cost of application developers needing to consider these potential issues in new scenarios can be high and should be expected throughout the development process.

When considering the adoption of a Microservices, ensure that the use case is sound, the team is aware of the potential challenges and above all, the benefits of this model outweigh the cost.

Recommended reading:  If you would like to learn more about the operational and feasibility considerations of Microservices, look up Benjamin Wootton and read some of his publications on the topic, specifically ‘Microservices – Not A Free Lunch!’.

Consideration to Deploy Edge Cluster in vCloud Director. Also discussing about VSAN Architecture.

If we are using vCloud Director, then Edge Cluster can not be combined with the management cluster. It can only be combined with the resource/payload/workload cluster.

If we are using VSAN, there are several consideration too if we want to use it as management cluster.

Please read these references to explain detail of the technical stuffs:

Once again, thanks Bayu for the discussion.

 

Kind Regards,
Doddi Priyambodo