vSphere Cluster Services (vCLS) is enabled by default and runs in all vSphere clusters. vCLS ensures that if vCenter Server becomes unavailable, cluster services remain available to maintain the resources and health of the workloads that run in the clusters. vCenter Server is still required to run DRS and HA.
vCLS is enabled when you upgrade to vSphere 7.0 U3 or when you have a new vSphere 7.0 U3 deployment. vCLS is upgraded as part of vCenter Server upgrade.
vCLS uses agent virtual machines to maintain cluster services health. The vCLS agent virtual machines (vCLS VMs) are created when you add hosts to clusters. Up to three vCLS VMs are required to run in each vSphere cluster, distributed within a cluster. vCLS is also enabled on clusters which contain only one or two hosts. In these clusters the number of vCLS VMs is one and two, respectively.
New anti-affinity rules are applied automatically. Every three minutes a check is performed, if multiple vCLS VMs are located on a single host they will be automatically redistributed to different hosts.
Table 1. Number of vCLS Agent VMs in Clusters
Number of Hosts in a Cluster
Number of vCLS Agent VMs
1
1
2
2
3 or more
3
vCLS VMs run in every cluster even if cluster services like vSphere DRS or vSphere HA are not enabled on the cluster. The lifecycle operations of vCLS VMs are managed by vCenter services like ESX Agent Manager and Workload Control Plane. vCLS VMs do not support NiC cards.
vSphere DRS
vSphere DRS is a critical feature of vSphere which is required to maintain the health of the workloads running inside vSphere Cluster. DRS depends on the availability of vCLS VMs.
If DRS is non-functional this does not mean that DRS is disabled. Existing DRS settings and resource pools survive across a lost vCLS VMs quorum. vCLS health turns Unhealthy only in a DRS enabled cluster when vCLS VMs are not running and the first instance of DRS is skipped because of this. vCLS health will stay Degraded on a non-DRS enabled cluster when at least one vCLS VM is not running.
Datastore selection for vCLS VMs
The datastore for vCLS VMs is automatically selected based on ranking all the datastores connected to the hosts inside the cluster. A datastore is more likely to be selected if there are hosts in the cluster with free reserved DRS slots connected to the datastore. The algorithm tries to place vCLS VMs in a shared datastore if possible before selecting a local datastore. A datastore with more free space is preferred and the algorithm tries not to place more than one vCLS VM on the same datastore. You can only change the datastore of vCLS VMs after they are deployed and powered on.
If you want to move the VMDKs for vCLS VMs to a different datastore or attach a different storage policy, you can reconfigure vCLS VMs. A warning message is displayed when you perform this operation.
You can perform a storage vMotion to migrate vCLS VMs to a different datastore. You can tag vCLS VMs or attach custom attributes if you want to group them separately from workload VMs, for instance if you have a specific meta-data strategy for all VMs that run in a datacenter.
Introduction to the vSphere Clustering Service (vCLS)
Overview
The vSphere Clustering Service (vCLS) is a new capability that is introduced in the vSphere 7 Update 1 release. It’s first release provides the foundation to work towards creating a decoupled and distributed control plane for clustering services in vSphere.
The challenge being that cluster services, like the vSphere Distributed Resource Scheduler (DRS), depend on vCenter Server availability for its configuration and operation. And while there’s ways to increase availability for vCenter Server, think about vSphere High Availability (HA) and vCenter Server High Availability (VCHA), its dependency is not ideal. Also, when thinking about vCenter Server scalability in large on-prem and public clouds, we need a better solution to support clustering services. That’s why vCLS is introduced. In the first release, a subset of DRS capabilities is already using the new vCLS feature.
Basic Architecture
The basic architecture for the vCLS control plane consists of maximum 3 virtual machines (VM), also referred to as system or agent VMs which are placed on separate hosts in a cluster. These are lightweight agent VMs that form a cluster quorum. On smaller clusters with less than 3 hosts, the number of agent VMs is equal to the numbers of ESXi hosts. The agent VMs are managed by vSphere Cluster Services. Users are not expected to maintain the lifecycle or state for the agent VMs, they should not be treated like the typical workload VMs.
Cluster Service Health
The agent VMs that form the cluster quorum state, are self correcting. This means that when the agent VMs are not available, vCLS will try to instantiate or power-on the VMs automatically.
There are 3 health states for the cluster services:
Agent VM Resources
The vCLS agent VMs are lightweight, meaning that resource consumption is kept to a minimum. vCLS automatically creates a max of 3 agent VMs per cluster in an existing deployment when vCenter Server is upgraded to vSphere 7 update 1. In a greenfield scenario, they are created when ESXi hosts are added to a new cluster. If no shared storage is available, the agent VMs are placed on local storage. If a cluster is formed before shared storage is configured on the ESXi hosts, as would be the case when using vSAN, it is strongly recommended to move the vCLS agent VMs to shared storage after.
The agent VMs run a customized Photon OS. The resource specification per agent VM is listed in the following table:
Memory
128 MB
Memory reservation
100 MB
Swap Size
256 MB
vCPU
1
vCPU reservation
100 MHz
Disk
2 GB
Ethernet Adapter
–
Guest VMDK Size
245 MB
Storage Space
The 2 GB virtual disk is thin provisioned. Also, there’s no networking involved, so there’s no network adapter configured. The agent VMs are not shown in the Hosts and Clusters overview in the vSphere Client. The VMs and Templates view now contains a new folder, vCLS, that contains all vCLS agent VMs. With multiple clusters, all vCLS agent VMs will show, numbered consecutively.
UI Overview
The vSphere Client contains messages and notes to show information about the vCLS agent VMs, also stating that the power state and resources of these VMs is handled by the vCLS.
Operations
As stated before, the agent VMs are maintained by vCLS. There’s no need for VI admins to power-off the VMs. In fact, the vSphere Client shows a warning when a agent VM is powered-off.
When a host is placed into maintenance mode, the vCLS agent VMs are migrated to other hosts within the cluster like regular VMs. Customers should refrain from removing, or renaming the agent VMs or its folder in order to keep the cluster services healthy.
The lifecycle for vCLS agent VMs is maintained by the vSphere ESX Agent Manager (EAM). The Agent Manager creates the VMs automatically, or re-creates/powers-on the VMs when users try to power-off or delete the VMs. Is the example below, you’ll see a power-off and a delete operation. Both from which the EAM recovers the agent VM automatically.
Automation and vCLS
For customer using scripts to automate tasks, it’s important to build in awareness to ignore the agent VMs in, for example clean-up scripts to delete stale VMs. Identifying the vCLS agent VMs is quickly done in the vSphere Client where the agent VMs are listed in the vCLS folder. Also, examining the VMs tab under Administration > vCenter Server Extensions > vSphere ESX Agent Manager lists the agent VMs from all clusters managed by that vCenter Server instance.
Every agent VM has additional properties so they can be ignored with specific automated tasks. These properties can also be found using the Managed Object Browser (MOB). The specific properties include:
vSphere Cluster Services (vCLS) is enabled by default and runs in all vSphere clusters. vCLS ensures that if vCenter Server becomes unavailable, cluster services remain available to maintain the resources and health of the workloads that run in the clusters. vCenter Server is still required to run DRS and HA.
vCLS is enabled when you upgrade to vSphere 7.0 U3 or when you have a new vSphere 7.0 U3 deployment. vCLS is upgraded as part of vCenter Server upgrade.
vCLS uses agent virtual machines to maintain cluster services health. The vCLS agent virtual machines (vCLS VMs) are created when you add hosts to clusters. Up to three vCLS VMs are required to run in each vSphere cluster, distributed within a cluster. vCLS is also enabled on clusters which contain only one or two hosts. In these clusters the number of vCLS VMs is one and two, respectively.
New anti-affinity rules are applied automatically. Every three minutes a check is performed, if multiple vCLS VMs are located on a single host they will be automatically redistributed to different hosts.
Table 1. Number of vCLS Agent VMs in Clusters
Number of Hosts in a Cluster
Number of vCLS Agent VMs
1
1
2
2
3 or more
3
vCLS VMs run in every cluster even if cluster services like vSphere DRS or vSphere HA are not enabled on the cluster. The lifecycle operations of vCLS VMs are managed by vCenter services like ESX Agent Manager and Workload Control Plane. vCLS VMs do not support NiC cards.
vSphere DRS
vSphere DRS is a critical feature of vSphere which is required to maintain the health of the workloads running inside vSphere Cluster. DRS depends on the availability of vCLS VMs.
If DRS is non-functional this does not mean that DRS is disabled. Existing DRS settings and resource pools survive across a lost vCLS VMs quorum. vCLS health turns Unhealthy only in a DRS enabled cluster when vCLS VMs are not running and the first instance of DRS is skipped because of this. vCLS health will stay Degraded on a non-DRS enabled cluster when at least one vCLS VM is not running.
Datastore selection for vCLS VMs
The datastore for vCLS VMs is automatically selected based on ranking all the datastores connected to the hosts inside the cluster. A datastore is more likely to be selected if there are hosts in the cluster with free reserved DRS slots connected to the datastore. The algorithm tries to place vCLS VMs in a shared datastore if possible before selecting a local datastore. A datastore with more free space is preferred and the algorithm tries not to place more than one vCLS VM on the same datastore. You can only change the datastore of vCLS VMs after they are deployed and powered on.
If you want to move the VMDKs for vCLS VMs to a different datastore or attach a different storage policy, you can reconfigure vCLS VMs. A warning message is displayed when you perform this operation.
You can perform a storage vMotion to migrate vCLS VMs to a different datastore. You can tag vCLS VMs or attach custom attributes if you want to group them separately from workload VMs, for instance if you have a specific meta-data strategy for all VMs that run in a datacenter.
В начале обозначим, что в рамках данной статьи мы будем понимать под кластером группу хостов (физических серверов) под управлением единого сервиса для совместного выполнения определенных функций как целостная система, связывающаяся через сеть.
На платформе виртуализации VMware vSphere можно построить 2 разновидности кластеров: High-availability кластер (HA) и Distributed Resource Scheduler кластер (DRS).
HA-кластер будет означать, что определенное количество физических серверов объединяется в кластер и на них запускаются виртуальные машины. В случае выхода из строя одного из хостов, виртуальные машины запускаются на других серверах из группы, на которых предварительно было выделено для этого место. В итоге время простоя равно времени загрузки операционной системы «виртуалки».
Если необходимо сократить время простоя до минимального времени рекомендуется использовать технологию VMware Fault Tolerance. Основную идею опции можно описать как создание синхронно работающей реплики виртуальной машины на другом сервере и мгновенное переключение на неё при выходе из строя основного хоста.
Fault Tolerance
Технология VMware DRS используется для выравнивания нагрузки в кластере. Для этого на первоначальном этапе ресурсы кластера объединяются в пул и затем происходит балансировка нагрузки между хостами путем перемещения виртуальных машин. DRS может рекомендовать перемещение с необходимым подтверждением от администратора или делать это в автоматическом режиме. Происходит это с использованием утилиты «живой миграции» vMotion, благодаря которой миграция не требует остановки ВМ. Пользователи продолжают работать с одним экземпляром ВМ до тех пор, пока данные не будут перенесены на другой хост. В последний момент копируются последние изменения из оперативной памяти, пользователь видит незначительное кратковременное снижение быстродействие системы и через мгновение уже работает с той же ВМ, которая по факту уже находится на другом физическом сервере.
Принцип работы VMware HA + DRS
В случае с кластером VMware группа из 2-х и более серверов ESXi находится под централизованным управлением VMware vCenter Server. Собственно, создавать виртуальные машины можно и на одном хосте с установленным гипервизором VMware ESXi, но возможностей HA, DRS и прочих у вас не будет. Вы просто сможете «нарезать» ваш физический сервер на несколько виртуальных, а его неработоспособность будет означать простой всех ВМ.
Чтобы пользоваться всеми кластерными возможностями необходимо использовать платформу VMware vSphere, которая включает в себя сервер управления ESXi-хостами и СХД, так называемый и упомянутый выше, vCenter Server. Также для построения кластера потребуется подключение системы хранения данных. В ней в особенной кластерной файловой системе VMFS хранятся разделы с файлами виртуальных машин, которые доступны для чтения и записи всем ESXi-хостам кластера. По причине хранения в одном месте и независимости виртуальной машины от физической платформы достигается быстрое перемещение и восстановление при помощи HA, DRS, FT, vMotion.
Платформа VMware vSphere
VMware vCenter Server, если говорить упрощенно, является набором служб и базой данных. Каждая из служб занимается своим конкретным списком задач и взаимодействует с другими службами и/или хостами ESXi. vCenter Server – это некий командный пункт, которому подчиняются гипервизоры ESXi на хостах. Общение между ними происходит через хостовых агентов VPXA. Из панели управления vCenter Server можно делать даже больше, чем подключившись напрямую к ESXi. Если в ESXi вы сможете создавать/удалять виртуальные машины, то с помощью vCenter Server вы можете дополнительно создать и настроить для них кластер и все необходимые кластерные опции, часть из которых описана выше.
VMware vCenter Server может работать как на отдельной физическом сервере, так и внутри виртуальной машины на том же хосте, которым сам же и управляет.
Тема безусловно интересная и обширная, однако для развертывания подобных инфраструктур требуются большие материальные затраты. Если мы хотим пользоваться всеми возможностями, которые повышают отказоустойчивость и надежность системы, необходимо приобрести минимум два сервера и СХД, купить лицензию на платформу VMware VSphere у одного из дистрибьюторов. Установка, настройка и администрирование кластера VMware также потребует от вас временных и финансовых вложений.
Что делать в случае, если от вашей IT-инфраструктуры требуется высокая надежность, которую предоставляет платформа VMware vSphere, но нет возможности понести значительные капитальные вложения? Ответом на этот вопрос для многих корпоративных клиентов стало использование облачных технологий, а именно услуга аренды инфраструктуры (IaaS). Облачный провайдер обладает необходимым сетевым и серверным оборудованием, которое расположено в безопасных дата-центрах. IT-специалисты провайдера оказывают техническую поддержку 24*7, а бизнес может воспользоваться всеми преимуществами виртуализации в кластере VMware.
Клиенты не используют VMware vCenter Server. За управление кластерами и физическим оборудованием отвечает провайдер. Клиенты получают значительное количество возможностей управления своим виртуальным ЦОДом с помощью удобного портала самообслуживания VMware vCloud Director. Создание vЦОДа для клиента происходит в кратчайшие сроки, при этом может быть создано необходимое количество виртуальных машин с нужными характеристиками и операционными системами, маршрутизируемые и изолированные сети с любой топологией, настроены гибкие правила Firewall и многое другое.
Покупку минимум двух собственных серверов, СХД и лицензий с необходимостью дальнейшей настройки можно заменить использованием облака. Соглашение с провайдером будет предусматривать уровень доступности услуг (SLA). В случае, с Cloud4Y SLA равен 99,982%. Это означает, что максимально допустимая по соглашению недоступность сервиса составляет не более 15 минут в месяц. Кроме того, Cloud4Y устанавливает минимально допустимые показатели производительности CPU и RAM системы. Количество MIPS на одно vCPU составляет не менее 2900, что гарантирует клиентам заявленное быстродействие процессора. Также не допускается «переподписка» физической оперативной памяти. Это означает, что выделенная при создании виртуальной машины Configured Virtual RAM, которую будет видеть гостевая ОС, является 100% выделенной физической памятью, которая доступна виртуальной машине в любой момент времени. Это создает условия, при которых облачные серверы по производительности в любой момент времени могут полноценно заменить для клиентов физический сервер с соответствующими характеристиками, а благодаря виртуализации в кластере надежность и отказоустойчивость может оказаться выше, чем при использовании собственного оборудования.
Если вашему бизнесу требуется надежное IT-решение на основе кластера VMware, для принятия решения рекомендуем воспользоваться тестовым доступом к облаку Cloud4Y.
VMware vSphere Cluster Services (vCLS) considerations, questions and answers.
In the vSphere 7.0 Update 1 release VMware introduced a new service called the VMware vSphere Cluster Services (vCLS). vCLS provides a mechanism that allows VMware to decouple both vSphere DRS and vSphere HA from vCenter Server. Niels Hagoort wrote a lengthy article on this topic here. You may wonder why VMware introduces this, well as Niels states. by decoupling the clustering services (DRS and HA) from vCenter Server via vCLS we ensure the availability of critical services even when vCenter Server is impacted by a failure.
vCLS is a collection of multiple VMs which, over time, will be the backbone for all clustering services. In the 7.0 U1 release a subset of DRS functionality is enabled through vCLS. Over the past week(s) I have seen many questions coming in and I wanted to create a blog with answers to these questions. When new questions or considerations come up, I will add these to the list below.
Share it:
Related
Reader Interactions
Comments
“Can I customize the name of the vCLS VMs? Today it is not possible to customize the name of the vCLS VM, this has been filed as a feature request and is considered for a future release”
In a lab environment, I was able to rename the vCLS VMs and DRS remained functional. Functionality also persisted after SvMotioning all vCLS VMs to another Datastore and after a complete shutdown/startup of the cluster.
Original vCLS VM names were vCLS (4), vCLS (5), vCLS (6). New vCLs VM names are now vCLS (1), vCLS (2), vCLS (3).
Does this question, pertain to customizing the names of newly generated vCLS VMs, existing vCLS VMs, or both? Functionality “seems” to be working even though I’ve renamed the generated vCLS VMs in my lab.
Yes you can rename the vCenter entity, but if this VM is for whatever reason deleted by EAM a new VM with a new name with the “vCLS” naming scheme will be created again. Also, you are not supposed to do any ops on the VMs, so renaming is also discouraged. In other words, that it works doesn’t mean VMware will support it 🙂
but it is a valid point, I just updated my post and clarified that section
Also, any impact to rename the vCLS folder?
Yes there could be an impact, I just asked the engineering team and they (right now) advise against renaming the folder!
Another question. Does vCLS currently manage vSAN functions when VC is unavailable?
Hope it is in the roadmap
This sounds like steps towards a cluster vCenter, with real cluster FT like availability. Would not be surprised if Corfu was down there 🙂 Nice to have DRS, HA was already distributed, but given the number of solutions that use vCenter as an endpoint, I hope to see full vCenter cluster soon.
how to put all hosts in maintenance mode in a vSAN cluster running vCLS VMs (without having a secondary storage)? The question is about my lab env (but can be interesting also for a prod env). I’ve 4 hosts with vSAN enabled and sometime i shutdown all the environment. Before the upgrade to 7U1, to shutdown my env, I turn off all the VMs, put all hosts in maintenance (without data migration) and finally power off the ESXi. Now i’ve some problems to perform a clean shutdown because the vCLS are running on vSAN (seems a chicken egg problem or i missing something)…
In my lab (3 clusters) I simply place all hosts in maintenance mode (one by one) and power them all off, I use the “no data migration” option after I power off all my VMs. It takes a bit of time for the last host to go into maintenance mode, but so far it just works. You are saying that this doesn’t work for you?
Hi Duncan, I just force shutdown the vCLS VM’s and then jump into Maintenance Mode. Quick and dirty. When I bring my lab back up the vCLS VM’s start automatically. What is the impact of doing this, even though the direction from VMware is to never interact with the vCLS VM’s.
I have just tested it 3 times, I do “maintenance mode” from the vCenter UI using “No Data Migration” and it just powers off the VMs running on the vSAN Datastore when I get to the last host.
Hi Duncan, well my vCenter Appliance is on that vSAN Cluster, so I have to shut it down. Trying to enter Maintenance Mode on the ESXi Host interface doesn’t allow you to stop the vCLS VM’s, as it might through the vCenter GUI. Kind of caught be-twixt & between perhaps.
In that situation you should use the “retreat” mode I discussed in this post. It will disable/delete the vCLS VMs, which will allow you to place it in maintenance mode.
yes this doesn’t work for me. Today i changed some configuration on vSAN to have a clean shutdown….
My vSAN is configured with a default storage policy as Erasure Coding with FTT=1, all the VMs and vCLS VMs was configured to using it and with this configuration I’m not able to put the third host in maintenance (i wait about 45 minutes to put the host in maintenance without success). Due to unavailability of some vCLS VMs vSAN storage objects when the second host is put in maintenance, I’m able to power off the vCLS VMs without they can be restarted automatically and then i’m able to put in maintenance the 3rd and 4th Hosts.
Now i changed the vCLS VMs vSAN storage policy with a “simple” FTT=1 (2 mirrors & a witness component), whit this change i can have a clean ESXi maintenance/shutdown. The 3rd host is very slow to reach the maintenance (about 19 Minutes), but the 4th work as expected.
I don’t know if the problem can be related to my env but if not, can be useful have a “maintenance mode” for the vCLS VMs to be able to shutdown it.
One more question, the 3 vCLS VMs are deployed for every vSphere Cluster?
Are you doing the maintenance mode from the vCenter Server or from the command line? Which “data evac” option did you select?
Via vCenter (deployed on another ESXi outside the cluster) and “data evac” = “No Data Migration”
😉 (@darthVikes) says
Is this a shift towards moving to microservices for vCenter or all vmware apps?
Mainly for clustering services right now, but I would expect decoupling to continue!
Doug McIntyre says
Why do I have four vCLS machines in my test lab instead of the max of 3 you state? (vCLS (1), vCLS (2), vCLS (3), vCLS (4))
How many clusters do you have?
Doug McIntyre says
There’s two clusters, one management of one host, and one of compute with four hosts.
So you’re implying I’m picking up the forth for the single host management cluster? If I left the single management host out of a cluster, I’d drop to three?
Correct, drop the cluster and that 4th vcls VM will disappear.
Gilles Le Ridou says
We have several two nodes clusters (FC SAN, not VSAN). Some of them have two vCLS VMs, some others three vCLS. Some reason? (vCenter 7.0 u1, ESX still 6.7)
Single host cluster gets 1 VCLS VM. Two node gets two, three node or more gets 3.
Gilles Le Ridou says
I can post an image of a two nodes cluster with three vCLS VMs on Twitter if you think this is interesting.
I’ve seen it in my lab as well, it is usually an issue with a clean up. Just enable “retreat mode” (False) and disable it again (True). This should clean up the environment.
Greg Merideth says
Bit of a nitpick but why does skyline health require CEIP? Are the health services of clusters sent back to vmare?
This is used for the “Skyline” part of the Healthcheck, which helps the support team during troubleshooting when you file a support request.
Thank you for this article. Had missing vCLS VMs after upgrade to vCenter 7.0.1. Tried Vmware support, and they basically told me to either restore vCenter from backup / snapshot or wait for a patch with no current ETA… Those solutions turned out to be unacceptable, as DRS was out of action and losing current vCenter was not an option. Resetting the STS certificate + restart all services absolutely did the trick and now the VMs were deployed – DRS working. This saved me a lot of trouble as a VI admin and I really appreciate it. 😀 😀
Thanks for the feedback, great to hear it solved your problem!
I have a 2 host cluster and as expected 2 vCLS VMs. I started rebooting my hosts to check my HA setup (which worked fine) however more vCLS VMs are appearing and the old ones aren’t being deleted.
Should the oldest VMs be automatically deleted?
Is this a lab environment or production? If you enable “retreat mode” and disable it again, all VMs should be cleaned up and new VMs should be created. However, if this is production, it may be useful to create a support request and upload the logs so it can be debugged.
Christoph Hochsticher says
Come on. This is a Beta release where you couldn’t specify the Datastores. It costs me hours to change all vCLS VMs away from some ISO, Logging and Repo Datastores. That is shiddy!
I’m shocked that this in the final U1.
Martin Schenker says
We tried to upgrade a 6.7 system to 7.0U1, first stage VCenter upgrade, then three hosts w. VSan. Two servers went OK, the vCLS VMs were migrated to the last 6.7 host. Enterning maintenance mode failed for that host, the vCLS VMs didn’t migrate off it or shut down. After a manual shutdown of the three vCLS VMs, the third system finally entered maintenance mode and could be upgraded to 7.01. The VCenter and all hosts are now up-to-date w. current patches and software. Now the vCLS VMs won’t start; we’re just getting the repetetive error (every 30s):
“Feature ‘bad_requirement.hv.capable’ was 0, but must be at least 1′. Failed to start the virtual machine. Module ‘FeatureCompatLate’ power on failed.
If it has been enabled, makes sure to disable the following: File on each host: /etc/vmware/config vhv.enable = “TRUE”
😉 (@darthVikes) says