Terremark announces new lower price

Competition is a good thing. Continuous drop in the hardware price is a good thing. Improvement in efficient is also a good thing. All those should translate into a lower computing cost to you — the end user — over time. That is exactly what Terremark has done today — lowering its cloud offering price, in one case, up to 42% lower. We compared Terremark’s price to EC2 before, it is time to update the comparison based on the new pricing.

Following what we did to get EC2’s unit cost, we can run regression analysis to estimate Terremark’s unit cost. We only consider Linux VMs, in order not to factor is software license cost. We assume the Cost = c * CPU + m * RAM (Terremark charges storage separately from the VM cost at $0.25/GB/month). The regression determines the unit cost to be

c = 1.31 cents/VPU/hour
m = 5.38 cents/GB/hour

Not surprisingly, both unit costs are lower than their previous price (c used to be 2.06 cents/VPU/hour, and m used to be 6.46 cents/GB/hour).

The regression result still does not fit the real cost very well. Terremark offers economy-of-scale in its cost model, where it heavily discounts both CPU and RAM as you move up in the configuration. The following table shows both the newer reduced cost (color green) and the cost as determined by the estimated parameters (color red) for the various VM configurations.

memory (GB)\CPU	1 VPU	2 VPU	4 VPU	8 VPU
0.5	3.5 / 4	4 / 5.31	4.5 / 7.93	4.9/ 13.17
1	6 / 6.69	7 / 8	8 / 10.62	10 / 15.86
1.5	9 / 9.38	10.5 / 10.69	12 / 13.31	13.5 / 18.55
2	12 / 12.07	14.1 / 13.38	16.1 / 16	20 / 21.24
4	21.7 / 22.82	27.1 / 24.13	30.1 / 26.75	35.9 / 31.99
8	40.1 / 44.33	48.2 / 45.64	56.7 / 48.26	63.4 / 53.5
12	60.2 / 65.85	68.6 / 67.16	76.2 / 69.78	82.4 / 75.02
16	80.3 / 87.36	84.4 / 88.67	89.9 / 91.29	93.2 / 96.53

Again, we compare cost by comparing with a fictitious EC2 instance with the exact same spec. For simplicity, we assume a VPU in a Terremark’s VM can get the full attention of a physical core. This is a more common case because Terremark uses VMWare’s DRS (Distributed Resource Scheduler), which can dynamically reassign virtual cores to a different physical core to avoid contention.

The following table shows the EC2 equivalent cost assuming a virtual core can get the full power of the physical core.

memory (GB)	VPU	Terremark price (cents/hour)	Equivalent EC2 cost (cents/hour)	Terremark cost/EC2 cost
0.5	1	3.5	4.09	0.86
0.5	2	4	7.17	0.56
0.5	4	4.5	13.33	0.34
0.5	8	4.9	25.65	0.19
1	1	6	5.09	1.18
1	2	7	8.17	0.86
1	4	8	14.33	0.56
1	8	10	26.66	0.38
1.5	1	9	6.1	1.48
1.5	2	10.5	9.18	1.14
1.5	4	12	15.34	0.78
1.5	8	13.5	27.66	0.49
2	1	12	7.1	1.69
2	2	14.1	10.18	1.38
2	4	16.1	16.35	0.98
2	8	20	28.67	0.7
4	1	21.7	11.12	1.95
4	2	27.1	14.21	1.91
4	4	30.1	20.37	1.48
4	8	35.9	32.69	1.1
8	1	40.1	19.17	2.09
8	2	48.2	22.25	2.17
8	4	56.7	28.41	2.0
8	8	63.4	40.73	1.56
12	1	60.2	27.21	2.21
12	2	68.6	30.29	2.26
12	4	76.2	36.45	2.09
12	8	82.4	48.78	1.69
16	1	80.3	35.25	2.28
16	2	84.4	38.33	2.20
16	4	89.9	44.5	2.02
16	8	93.2	56.82	1.64

Like we observed before, there are several configurations where Terremark is much cheaper than EC2. The 8VPU+0.5GB configuration is still the cheapest at 19% of the equivalent EC2 cost. What is different from before is that the larger VM configurations are getting significantly cheaper. For example, the 16GB+8VPU configuration costs only 1.64 over its EC2 equivalent, compared to a ratio of 2.83 before. This means that it is getting more economical to run larger VMs in Terremark.

Let us hope the trend continues that cloud providers continue to reduce the cost of computing so that we can pay less for the same work or get more work done for the same budget.

Comparing cloud providers on VM cost

How do you compare two IaaS clouds? Is Amazon EC2’s small standard instance (1 ECU, 1.7GB RAM, 160GB storage) cheaper or is Rackspace cloud’s 256MB server (4 cores, 256MB RAM, 10GB storage) cheaper? It is obviously simpler to compare them if you focus only on one metric. For example, let us assume your application is CPU bound and it does not require much memory at all. Then you should focus solely on the CPU power a cloud VM gives you. We have translated GoGrid, Rackspace, and Terremark‘s VM configurations into their equivalent ECU, so you can simply take a ratio between the cost and the ECU rating and pick the lowest ratio. Unfortunately, real-life applications are never that simple. They demand CPU cycle, memory, as well as hard disk storage capacity. So, how do you compare apple-to-apple?

The methodology

Since no methodology exists yet, we will propose one. Since the comparison results depend highly on the methodology chosen, we first will spell out the methodology we use so that if you have a different one and you come up with a different result, you can trace the source of the difference. If you see areas where we can improve the methodology, please do leave a comment. The methodology works as follows:

1. We first break down the cost components in Amazon EC2. We assume Amazon has priced their instances using a linear model, i.e., the cost is equal to c * CPU + m * Mem + s * Storage, where c is the unit cost of CPU per ECU per hour, m is the unit cost of memory per GB per hour, and s is the unit cost of storage per GB per hour. Amazon provides several types of instances, each with a different combination of CPU, memory and storage, which is enough of a hint for us to use regression analysis to estimate c, m and s. The details are in our ECU cost breakdown analysis.
2. Once we have the unit cost in EC2, we can compare it with another cloud provider. We take one VM configuration from a cloud provider at a time, we then compute what Amazon EC2 would charge for an instance with the exact same specification if EC2 were to offer it. This can be easily done by multiplying the EC2 unit costs (c, m, and s) with the amount of CPU, RAM, and storage in the VM, and add them up. Of course, this is hypothetical, because EC2 does not offer an instance with an exact same spec. So even if the EC2 price is lower, you cannot just buy a substitute from Amazon. However, this gives us a good sense of the relative cost.

We have done the analysis with GoGrid, Rackspace, and Terremark.

We can compute a ratio between a cloud VM’s cost with its hypothetical equivalent in EC2. The following lists the top few VMs that have the lowest ratio. If you are curious about the ratio for other VM configurations, feel free to dig into the individual posts on each provider. The ratio listed is assuming that you will get the maximum CPU allowed under bursting, which is frequently the case in those cloud providers. Further, the ratio listed is comparing with EC2 N. Virginia data center. Other EC2 data centers have a higher cost.

Provider	RAM (GB)	CPU (cores)	storage (GB)	cost ratio with an equivalent in EC2
Rackspace	0.25	4	10	0.168
Terremark	0.5	8	charged separately at $0.25/month/GB	0.19
Rackspace	0.5	4	20	0.314
Terremark	0.5	4	charged separately at $0.25/month/GB	0.338
Terremark	1	8	charged separately at $0.25/month/GB	0.375
Terremark	1.5	8	charged separately at $0.25/month/GB	0.491

 

How to use this data?

Due to the limitations of this methodology (comparing with a hypothetical equivalent in EC2), it only makes sense if one of the cloud provider you are comparing is Amazon EC2. In other words, do not compare Rackspace with Terremark based on the ratio.

It also makes no sense to use our results if you know the exact specification for your server. In that case, you should find a minimum VM configuration that is just barely bigger than your requirement and compare price.

Our results are useful if your application is flexible. For example, instead of using one m1.small instance in EC2, you could use several Rackspace 256MB VMs to achieve a dramatic cost savings. Examples of a flexible application include a batch application, such as a MapReduce job, which could be chopped down to a finer granularity. Another example could be web servers in a web server farm, where the load balancer can divide up the work to take advantage of whatever computation capacity provisioned on the web server.

Our results are also useful if you want to get a high level overview. Consider an enterprise purchaser who wants to choose a cloud platform. There are many dimensions he has to consider, e.g., features, cost, SLA, contract terms….. Doing a deep analysis at the beginning is just going to be overwhelming. Since Amazon is a big player in cloud, it most likely will be part of the evaluation. Having a ratio would give a ten-thousand-feet view such that the decision maker would know whether an alternative cloud would save him money. Then, as the evaluation progresses, he can dig deeper into a finer comparison.

Caveats:

There are many caveats in using our results that we should spell out.

• This is only comparing a VM cost, including its CPU, memory and storage. But, it does not include other costs, such as bandwidth transfers. The bandwidth cost varies wildly, for example, GoGrid offers free inbound traffic, which can translate into a significant cost saving.

• When we compare CPUs, we are only comparing their processing power, not their IO capabilities (both disk and network IO). In Amazon, we sometimes observe degraded IO performance, possibly due to competing VMs on the same host. It is a sad side effect of using popular cloud offerings.

• As we mentioned, this only applies to fungible applications that can take full advantage of provisioned CPU, memory and storage resources. For example, if you cannot take advantage of the provisioned RAM, it does not matter if it is a good deal. You are wasting the memory, and you may be better off with a VM configuration from a different cloud provider with a smaller provisioned RAM.

• This is not a substitute for feature comparisons. For example, GoGrid offers free F5 hardware load balancer. If you need a hardware load balancer, you should consider that separately.

Terremark cost comparison with Amazon EC2

(Earlier posts in this series are: EC2 cost break down, GoGrid & EC2 cost comparison, Rackspace & EC2 cost comparison)

In this post, let us compare the VM cost between Terremark vCloud express and Amazon EC2. Terremark is one of the first cloud providers based on VMWare technology. Unlike EC2, Rackspace and GoGrid, which use Xen as the hypervisor, Terremark uses VMWare’s ESX hypervisor, which arguably is richer in functionality.

Following the methodology we have used so far, we need to first understand Terremark’s hardware infrastructure and its resource allocation policy. Using the same technique we used for EC2’s hardware analysis, we determine that Terremark runs on a platform with two sockets of Quad-core AMD Opteron 8389 processors. PassMark does not have a benchmark result for this processor, so we have to run the benchmark ourselves. We used the 16GB+8VPU configuration — its largest — to minimize interference from other VMs, and we run it multiple times late at night to ensure that we are indeed measuring the underlying hardware’s capacity. On average, the PassMark CPU mark result is 7100, which is roughly 18 ECU.

Terremark uses the ESX hypervisor’s default policy for scheduling CPU, i.e., a core shares the CPU equally with another core regardless of how much memory the VM has. This is different from GoGrid and Rackspace where the CPU is shared proportional to the amount of RAM a VM has. The scheduling policy can be verified by reading the GuestSDK API exposed by VMTools. By reading the API, we know that a VM not only has no minimum guaranteed CPU, but it also does not have a maximum burst limit. Each virtual core of a VM is assigned a CPU share of 1000, regardless of the memory it is allocated. Thus, the more cores a VM has, the more shares of the CPU it will get (e.g., 1VPU has 1000 shares, and 8VPU has 8000 shares).

It is difficult to determine how many VMs could be on a physical host, which determines the minimum guaranteed CPU. We are told in their forum that each physical host has 128GB of memory, which can accommodate at least 8 VMs, for example, each with 8 VPU+16GB RAM (its largest configuration). VMWare ESX hypervisor allows over-committing memory, so in theory, there could be many more VMs on a host. When we launched a vanilla 512MB VM, we learned from the Guest API that our VM only occupied 148MB RAM. Clearly, there is lots of room to over-commit, even though we see no evidence that they are doing so. Assuming there is no over-commitment, there still could be a lot of VMs competing for the CPU. In the worst case, all VMs on the host have 512MB RAM and 8VPU, which consume the least memory, but gain the maximum CPU weights. A physical host can host 256 such VMs, leaving a negligible CPU share for each VM. If a VM has only one core, it owns only 1/(8*256) share of the CPU, and an 8 VPU (8 virtual cores) VM owns only 1/256 share of the CPU.

Following what we did to get EC2’s unit cost, we can run regression analysis to estimate Terremark’s unit cost. We assume the Cost = c * CPU + m * RAM (Terremark charges storage separately from the VM cost at $0.25/GB/month). The regression determines the unit cost to be

c = 2.06 cents/VPU/hour
m = 6.46 cents/GB/hour

The regression result does not fit the real cost very well. The following table shows both the original cost (color green) and the cost as determined by the estimated parameters (color red) for the various VM configurations.

memory (GB)\CPU	1 VPU	2 VPU	4 VPU	8 VPU
0.5	3.5 / 5.29	4 / 7.36	4.5 / 11.48	5 / 19.72
1	6 / 8.53	7 / 10.6	8 / 14.7	10 / 23
1.5	9 / 11.8	10.5 / 13.8	12 / 17.9	13.6 / 26.2
2	12 / 15	14.1 / 17	16.1 / 21.2	20.1 / 29.4
4	24.1 / 27.9	28.1 / 30	30.1 / 34.1	40.2 / 42.4
8	40.2 / 53.8	48.2 / 55.8	60.2 / 60	80.3 / 68.2
12	60.2 / 79.6	72.3 / 81.7	90.3 / 85.8	120.5 / 94.1
16	80.3 / 105.5	96.4 / 107.5	120.5 / 111.7	160.6 / 112

The reason that the regression analysis does not work well here is that Terremark heavily discounts both CPU and RAM as you move up in the configuration. Our linear model does not capture the economy of scale very well. However, we can think of the linear regression as a trend line, and the trend line indicates that Terremark is likely more expensive than EC2. For example, it costs 6.46 cents/GB/hour for its RAM, which is much higher than the 2.01 cents Amazon values its RAM at.

Another way to compare cost is to use EC2’s unit cost to figure out what an equivalent configuration will cost in EC2. The following table shows the cost comparison where we assume you can only get the minimum CPU at the worst case, where all other VMs are busy and a physical host is fully loaded with 8VPU+0.5GB VMs (without over-commitment). Each row shows the RAM and CPU configuration, Terremark’s price, what it would cost in EC2, and the ratio between Terremark and EC2 cost.

memory (GB)	VPU	Terremark price (cents/hour)	Equivalent EC2 cost (cents/hour)	Terremark cost/EC2 cost
0.5	1	3.5	1.02	3.44
0.5	2	4	1.03	3.89
0.5	4	4.5	1.05	4.27
0.5	8	5	1.10	4.54
1	1	6	2.02	2.97
1	2	7	2.03	3.44
1	4	8	2.06	3.89
1	8	10	2.11	4.75
1.5	1	9	3.03	2.97
1.5	2	10.5	3.04	3.45
1.5	4	12	3.06	3.92
1.5	8	13.6	3.11	4.37
2	1	12	4.03	2.98
2	2	14.1	4.05	3.49
2	4	16.1	4.07	3.96
2	8	20.1	4.12	4.88
4	1	24.1	8.06	2.99
4	2	28.1	8.07	3.48
4	4	30.1	8.09	3.72
4	8	40.2	8.14	4.94
8	1	40.2	16.1	2.5
8	2	48.2	16.11	2.99
8	4	60.2	16.13	3.73
8	8	80.3	16.18	4.96
12	1	60.2	24.14	2.49
12	2	72.3	24.15	2.99
12	4	90.3	24.18	3.73
12	8	120.5	24.23	4.97
16	1	80.3	32.18	2.49
16	2	96.4	32.2	2.99
16	4	120.5	32.22	3.74
16	8	160.6	32.27	4.98

The table shows that Terremark is 2.49 to 4.98 times more expensive than an equivalent in EC2. This is mainly due to the way Terremark shares CPUs. A 0.5GB VM in Terremark shares the CPU equally with a 16GB VM; thus, in the worst case, a VM may get very little CPU. Since Terremark does not set a minimum guarantee on the CPU share in the hypervisor, we have to assume the worst case.

In reality, you are unlikely to encounter the worst case, and you are very likely to get the full attention of a physical core. The reason is not only because the majority of VMs have more than 0.5GB (so that you can pack fewer of them on a host), but also because Terremark uses VMWare’s DRS (Distributed Resource Scheduler). We have noticed that, when we drive up the load on our VMs, our VMs are often moved (through VMotion) to a different host, presumably to avoid contention. Thus, unless the whole cluster gets really busy, it is unlikely that your VM would have a lot of other busy VMs to contend with on the same host. The following table shows the EC2 equivalent cost assuming a virtual core can get the full power of the physical core.

memory (GB)	VPU	Terremark price (cents/hour)	Equivalent EC2 cost (cents/hour)	Terremark cost/EC2 cost
0.5	1	3.5	4.09	0.86
0.5	2	4	7.17	0.56
0.5	4	4.5	13.33	0.34
0.5	8	5	25.65	0.19
1	1	6	5.09	1.18
1	2	7	8.17	0.86
1	4	8	14.33	0.56
1	8	10	26.66	0.38
1.5	1	9	6.1	1.48
1.5	2	10.5	9.18	1.14
1.5	4	12	15.34	0.78
1.5	8	13.6	27.66	0.49
2	1	12	7.1	1.69
2	2	14.1	10.18	1.38
2	4	16.1	16.35	0.98
2	8	20.1	28.67	0.7
4	1	24.1	11.12	2.17
4	2	28.1	14.21	1.98
4	4	30.1	20.37	1.48
4	8	40.2	32.69	1.23
8	1	40.2	19.17	2.1
8	2	48.2	22.25	2.17
8	4	60.2	28.41	2.12
8	8	80.3	40.73	1.97
12	1	60.2	27.21	2.21
12	2	72.3	30.29	2.39
12	4	90.3	36.45	2.48
12	8	120.5	48.78	2.47
16	1	80.3	35.25	2.28
16	2	96.4	38.33	2.51
16	4	120.5	44.5	2.71
16	8	160.6	56.82	2.83

There are several configurations where Terremark is much cheaper than EC2. The 8VPU+0.5GB configuration is the cheapest at 19% of the equivalent EC2 cost. This is due to two reasons. First, the 8 VPU has more scheduling weight, and it can compete for the full power of the physical host. Second, the RAM is the smallest. As we have seen, Terremark values RAM more than EC2 does (m=6.46 cents/GB/hour vs. EC2 m=2.01 cents/GB/hour), so the less RAM a configuration has, the lower the cost. The cost savings go away as you add more RAM and more CPU to the configuration.