How fast is fast?

Most use cases for vSphere Replication have a VM replicating between datacenters, whether it’s between company-owned facilities, to a colo, or a cloud like VMC on AWS. A big part of how fast your VMs will replicate is down to the WAN speed but how do you know where the bottlenecks are?

Handily the vSphere Replication appliance already comes with iperf3 https://iperf.fr/. Equally handily ESXi comes with iperf3 as well.

  Replication traffic will originate on the ESXi host where the VM is running. The destination will be the vSphere Replication Server on the destination site. With iperf3 on both sides, we can simulate the same path as the actual replication traffic takes.

Configuring the vSphere Replication Server as the IPerf3 Server

The first challenge is that the VR appliance has the firewall enabled and unsurprisingly IPerf3 isn’t allowed.

systemctl status iptables

iptables -nL

Rather than turning it off completely or making a permanent change we can open a port for IPerf3 temporarily.

iptables -A INPUT -p tcp --dport 5201 -j ACCEPT

We will setup the VR appliance as the IPerf3 server so it will listen for the client. In this setup I have vSphere Replication network traffic on a separate VLAN with a separate vmk on the ESXi hosts and the VR appliances. So to test properly I need to setup the IPerf3 server to also work on this network. I do that with the –bind option and specify the IP of the interface.

iperf3 --server --port 5201 --bind 172.18.0.101

ESXi host as the client

With the server side setup, we need to configure the ESXi hosts as IPerf3 clients. The ESXi firewall needs changed to allow outbound traffic. The easiest way to do this is just to turn it off.

esxcli network firewall get
esxcli network firewall set --enabled=false
esxcli network firewall get

Now we are ready to run iperf3

/usr/lib/vmware/vsan/bin/iperf3 --client 172.18.0.101 --port 5201

And on the server looks like this -

This gives a pretty good idea of what bandwidth you can expect for the VM replications. It only tests from 1 ESXi host and to 1 VR appliance. We can get closer to a realistic test by including more ESXi hosts as clients.

More ESXi hosts as clients

To simulate multiple VM replications we need to have multiple IPerf3 transfers running concurrently. Here I will add 2 more ESXi hosts as clients and setup 2 more IPerf3 servers on the VR appliance

On the VR appliance I open 2 more ports and start 2 more IPerf3 servers

iptables -A INPUT -p tcp --dport 5202 -j ACCEPT
iptables -A INPUT -p tcp --dport 5203 -j ACCEPT

iperf3 --server --port 5202 --bind 172.18.0.101
iperf3 --server --port 5203 --bind 172.18.0.101

On 2 more ESXi hosts I turn off the firewall and run IPerf3 clients Looking across the output of the IPerf3 servers on the VR appliance

I get 500 + 277 + 274 = 1,051 Mbits/sec

This compares pretty much with the result from only 1 ESXi host which shows that the network is the constraint with the available bandwidth being split across the 3 connections.

Finally let’s check that the VR appliance isn’t the bottleneck. From top I can see the 3 iperf3 processes are using roughly 20% CPU each. Its worth keeping an eye on the CPU but this isn’t the bottleneck.

Clean-up

We must remember to enable the ESXi firewall again as that change is persisted. The VR appliance changes won’t be kept after a reboot.