Terremark cost comparison with Amazon EC2

(Earlier posts in this series are: EC2 cost break down, GoGrid & EC2 cost comparison, Rackspace & EC2 cost comparison)

In this post, let us compare the VM cost between Terremark vCloud express and Amazon EC2. Terremark is one of the first cloud providers based on VMWare technology. Unlike EC2, Rackspace and GoGrid, which use Xen as the hypervisor, Terremark uses VMWare’s ESX hypervisor, which arguably is richer in functionality.

Following the methodology we have used so far, we need to first understand Terremark’s hardware infrastructure and its resource allocation policy. Using the same technique we used for EC2’s hardware analysis, we determine that Terremark runs on a platform with two sockets of Quad-core AMD Opteron 8389 processors. PassMark does not have a benchmark result for this processor, so we have to run the benchmark ourselves. We used the 16GB+8VPU configuration — its largest — to minimize interference from other VMs, and we run it multiple times late at night to ensure that we are indeed measuring the underlying hardware’s capacity. On average, the PassMark CPU mark result is 7100, which is roughly 18 ECU.

Terremark uses the ESX hypervisor’s default policy for scheduling CPU, i.e., a core shares the CPU equally with another core regardless of how much memory the VM has. This is different from GoGrid and Rackspace where the CPU is shared proportional to the amount of RAM a VM has. The scheduling policy can be verified by reading the GuestSDK API exposed by VMTools. By reading the API, we know that a VM not only has no minimum guaranteed CPU, but it also does not have a maximum burst limit. Each virtual core of a VM is assigned a CPU share of 1000, regardless of the memory it is allocated. Thus, the more cores a VM has, the more shares of the CPU it will get (e.g., 1VPU has 1000 shares, and 8VPU has 8000 shares).

It is difficult to determine how many VMs could be on a physical host, which determines the minimum guaranteed CPU. We are told in their forum that each physical host has 128GB of memory, which can accommodate at least 8 VMs, for example, each with 8 VPU+16GB RAM (its largest configuration). VMWare ESX hypervisor allows over-committing memory, so in theory, there could be many more VMs on a host. When we launched a vanilla 512MB VM, we learned from the Guest API that our VM only occupied 148MB RAM. Clearly, there is lots of room to over-commit, even though we see no evidence that they are doing so. Assuming there is no over-commitment, there still could be a lot of VMs competing for the CPU. In the worst case, all VMs on the host have 512MB RAM and 8VPU, which consume the least memory, but gain the maximum CPU weights. A physical host can host 256 such VMs, leaving a negligible CPU share for each VM. If a VM has only one core, it owns only 1/(8*256) share of the CPU, and an 8 VPU (8 virtual cores) VM owns only 1/256 share of the CPU.

Following what we did to get EC2’s unit cost, we can run regression analysis to estimate Terremark’s unit cost. We assume the Cost = c * CPU + m * RAM (Terremark charges storage separately from the VM cost at $0.25/GB/month). The regression determines the unit cost to be

c = 2.06 cents/VPU/hour
m = 6.46 cents/GB/hour

The regression result does not fit the real cost very well. The following table shows both the original cost (color green) and the cost as determined by the estimated parameters (color red) for the various VM configurations.

memory (GB)\CPU	1 VPU	2 VPU	4 VPU	8 VPU
0.5	3.5 / 5.29	4 / 7.36	4.5 / 11.48	5 / 19.72
1	6 / 8.53	7 / 10.6	8 / 14.7	10 / 23
1.5	9 / 11.8	10.5 / 13.8	12 / 17.9	13.6 / 26.2
2	12 / 15	14.1 / 17	16.1 / 21.2	20.1 / 29.4
4	24.1 / 27.9	28.1 / 30	30.1 / 34.1	40.2 / 42.4
8	40.2 / 53.8	48.2 / 55.8	60.2 / 60	80.3 / 68.2
12	60.2 / 79.6	72.3 / 81.7	90.3 / 85.8	120.5 / 94.1
16	80.3 / 105.5	96.4 / 107.5	120.5 / 111.7	160.6 / 112

The reason that the regression analysis does not work well here is that Terremark heavily discounts both CPU and RAM as you move up in the configuration. Our linear model does not capture the economy of scale very well. However, we can think of the linear regression as a trend line, and the trend line indicates that Terremark is likely more expensive than EC2. For example, it costs 6.46 cents/GB/hour for its RAM, which is much higher than the 2.01 cents Amazon values its RAM at.

Another way to compare cost is to use EC2’s unit cost to figure out what an equivalent configuration will cost in EC2. The following table shows the cost comparison where we assume you can only get the minimum CPU at the worst case, where all other VMs are busy and a physical host is fully loaded with 8VPU+0.5GB VMs (without over-commitment). Each row shows the RAM and CPU configuration, Terremark’s price, what it would cost in EC2, and the ratio between Terremark and EC2 cost.

memory (GB)	VPU	Terremark price (cents/hour)	Equivalent EC2 cost (cents/hour)	Terremark cost/EC2 cost
0.5	1	3.5	1.02	3.44
0.5	2	4	1.03	3.89
0.5	4	4.5	1.05	4.27
0.5	8	5	1.10	4.54
1	1	6	2.02	2.97
1	2	7	2.03	3.44
1	4	8	2.06	3.89
1	8	10	2.11	4.75
1.5	1	9	3.03	2.97
1.5	2	10.5	3.04	3.45
1.5	4	12	3.06	3.92
1.5	8	13.6	3.11	4.37
2	1	12	4.03	2.98
2	2	14.1	4.05	3.49
2	4	16.1	4.07	3.96
2	8	20.1	4.12	4.88
4	1	24.1	8.06	2.99
4	2	28.1	8.07	3.48
4	4	30.1	8.09	3.72
4	8	40.2	8.14	4.94
8	1	40.2	16.1	2.5
8	2	48.2	16.11	2.99
8	4	60.2	16.13	3.73
8	8	80.3	16.18	4.96
12	1	60.2	24.14	2.49
12	2	72.3	24.15	2.99
12	4	90.3	24.18	3.73
12	8	120.5	24.23	4.97
16	1	80.3	32.18	2.49
16	2	96.4	32.2	2.99
16	4	120.5	32.22	3.74
16	8	160.6	32.27	4.98

The table shows that Terremark is 2.49 to 4.98 times more expensive than an equivalent in EC2. This is mainly due to the way Terremark shares CPUs. A 0.5GB VM in Terremark shares the CPU equally with a 16GB VM; thus, in the worst case, a VM may get very little CPU. Since Terremark does not set a minimum guarantee on the CPU share in the hypervisor, we have to assume the worst case.

In reality, you are unlikely to encounter the worst case, and you are very likely to get the full attention of a physical core. The reason is not only because the majority of VMs have more than 0.5GB (so that you can pack fewer of them on a host), but also because Terremark uses VMWare’s DRS (Distributed Resource Scheduler). We have noticed that, when we drive up the load on our VMs, our VMs are often moved (through VMotion) to a different host, presumably to avoid contention. Thus, unless the whole cluster gets really busy, it is unlikely that your VM would have a lot of other busy VMs to contend with on the same host. The following table shows the EC2 equivalent cost assuming a virtual core can get the full power of the physical core.

memory (GB)	VPU	Terremark price (cents/hour)	Equivalent EC2 cost (cents/hour)	Terremark cost/EC2 cost
0.5	1	3.5	4.09	0.86
0.5	2	4	7.17	0.56
0.5	4	4.5	13.33	0.34
0.5	8	5	25.65	0.19
1	1	6	5.09	1.18
1	2	7	8.17	0.86
1	4	8	14.33	0.56
1	8	10	26.66	0.38
1.5	1	9	6.1	1.48
1.5	2	10.5	9.18	1.14
1.5	4	12	15.34	0.78
1.5	8	13.6	27.66	0.49
2	1	12	7.1	1.69
2	2	14.1	10.18	1.38
2	4	16.1	16.35	0.98
2	8	20.1	28.67	0.7
4	1	24.1	11.12	2.17
4	2	28.1	14.21	1.98
4	4	30.1	20.37	1.48
4	8	40.2	32.69	1.23
8	1	40.2	19.17	2.1
8	2	48.2	22.25	2.17
8	4	60.2	28.41	2.12
8	8	80.3	40.73	1.97
12	1	60.2	27.21	2.21
12	2	72.3	30.29	2.39
12	4	90.3	36.45	2.48
12	8	120.5	48.78	2.47
16	1	80.3	35.25	2.28
16	2	96.4	38.33	2.51
16	4	120.5	44.5	2.71
16	8	160.6	56.82	2.83

There are several configurations where Terremark is much cheaper than EC2. The 8VPU+0.5GB configuration is the cheapest at 19% of the equivalent EC2 cost. This is due to two reasons. First, the 8 VPU has more scheduling weight, and it can compete for the full power of the physical host. Second, the RAM is the smallest. As we have seen, Terremark values RAM more than EC2 does (m=6.46 cents/GB/hour vs. EC2 m=2.01 cents/GB/hour), so the less RAM a configuration has, the lower the cost. The cost savings go away as you add more RAM and more CPU to the configuration.