Comparing Windows Azure VM Performance, Part 2

Earlier this year, I wrote about Comparing Windows Azure VM Performance, using Geekbench 3.1.5 in 32-bit tryout mode to measure the processor and memory performance of a number of different sized Windows Azure VMs. These ranged from a Basic A0 VM all the way up to a Standard A7 VM, which was the largest available VM at that time. These machines were all in the US East Azure data center, and they all happened to be using the older Azure Gen 2 host hardware, which features the old and relatively slow AMD Opteron 4171 HE processor.

Consequently, the Geekbench results for these machines were quite low, both for the single-core and multi-core scores, as you can see in Table 1.

VM Size	CPU Cores	Memory	Monthly Cost	Single-Core Score	Multi-Core Score
Basic A0	1 (shared)	768 MB	$14	507	498
Basic A1	1	1.75 GB	$56	679	670
Basic A2	2	3.5 GB	$111	709	1,358
Basic A3	4	7 GB	$221	717	2,472
Basic A4	8	14 GB	$441	724	4,042
Standard A0	1 (shared)	768 MB	$15	492	502
Standard A1	1	1.75 GB	$67	1,068	1,083
Standard A2	2	3.5 GB	$134	1,069	2,002
Standard A3	4	7 GB	$268	1,070	3,593
Standard A4	8	14 GB	$536	1,094	6,446
Standard A5	2	14 GB	$246	1,080	2,026
Standard A6	4	28 GB	$492	1,080	3,686
Standard A7	8	56 GB	$983	1,056	6,185

Table 1: Selected Virtual Machine Specifications for Windows Azure in the East U.S. Data Center

Even the largest Standard A7 VM compared quite poorly with an average modern laptop, for both single-core and multi-core performance. In Geekbench, the single-core score measures the single-threaded performance of the processor, which is basically the raw speed of the processor. Single-threaded performance is very important for OLTP workloads, where most queries run on a single processor core. The multi-core score measures the overall processor capacity of the system, which equates to how much concurrent workload you can support. While many smaller SQL Server workloads might run perfectly well with this level of VM performance and capacity, as a DBA, I would not be very happy about using older Azure Gen 2 hardware for my SQL Server VMs.

More recently, Microsoft has made available larger and much faster Standard A8 and A9 Azure Compute Intensive virtual machines. These VMs feature much newer and faster 32nm Intel Xeon E5-2670 Sandy Bridge-EP processors, which were initially released in Q1 of 2012. This particular processor has eight physical cores (plus hyper-threading); with a base clock speed of 2.6GHz, and a Turbo Boost clock speed of 3.3GHz. This processor was two steps down from the "top of the line" eight-core Xeon E5-2690 processor, but it is actually a pretty decent processor. Another very important feature that comes with these new VM sizes is 40 Gbit/s InfiniBand networking, which gives you the potential of much better I/O performance.

VM Size	CPU Cores	Memory	Monthly Cost	Single-Core Score	Multi-Core Score
Standard A8	8	56 GB	$1,823	2,484	15,376
Standard A9	16	112 GB	$3,646	2,477	28,523

Table 2: Selected Virtual Machine Specifications for Windows Azure in the East U.S. Data Center

As you can see in Table 2, these new Compute Intensive Azure VMs have much better single-core and multi-core scores in Geekbench. They also have significantly higher monthly costs. Looking at these monthly costs from a straight hardware perspective, you could buy a brand new Dell PowerEdge R720 server with two newer 22nm, eight-core Intel Xeon E5-2667 v2 Ivy Bridge-EP processors and 192GB of RAM for roughly $10,000.00, which is about three months of Standard A9 time. With an on-premises, physical server, you would also have power and cooling costs, along with ongoing maintenance and administrative costs. You would also have a longer lead time to order and receive a physical server, and then get it racked, cabled, and configured. This process typically takes about three to four weeks at most companies, but can be much longer. Where the economic story changes is when you include your SQL Server 2012/2014 Enterprise Edition license costs, which would be about $110,000.00 for 16 processor core licenses, which would amount to roughly 30 months of Standard A9 time at current prices. If you use an Azure VM image from Microsoft’s standard gallery of VM images, then you do not have to pay for your OS or SQL Server licenses.

While I was doing this testing, I noticed a few other interesting differences between an Azure VM and a similar physical server. The first one was the fact that the Azure Standard A8 and A9 VMs appear to be using some sort of power management at the host or hypervisor level so that Intel Turbo Boost is not being used. When I run Geekbench on the system, and watch CPU-Z during the test run, the processor core speed stays at the rated base clock speed during the entire test run. If power management is properly configured at all levels, and if Turbo Boost is enabled in the host BIOS, then you should see the processor core speed jump up to the full 3.3GHz Turbo Boost clock speed periodically during the test run. This does not happen with the A8 and A9 Azure VMs, which hurts the single-core Geekbench 3.1.6 score by about 10%. The differences in processor core speeds are shown in Figures 1 and 2.

Figure 1: CPU-Z for Standard A9 Azure VM during Geekbench test run

Figure 2: CPU-Z for Dell PowerEdge R720 during Geekbench test run

So what are the main lessons learned from these experiments? First, it seems evident that you can get nearly equivalent VM performance from a Standard A9 Azure VM as you can from a similarly sized two-socket physical server (using the same processor), such as a Dell PowerEdge R720 or an HP ProLiant DL380 G8. You will be using a slightly older, middle-range Sandy Bridge-EP processor, without the benefit of Intel Turbo Boost being enabled, so you will be losing a total of about 40% of your single-threaded processor performance compared to a current vintage Intel Xeon E5-2667 v2 in a new physical, on-premises server.

Another potential issue is the memory limit of 112GB for the Standard A9 Azure VM, compared to 384GB in a physical, on-premises server (with 16GB DIMMs). If your database workload can run well with a roughly 95GB SQL Server buffer pool, then this should not be an issue. Otherwise, you might not be too happy with your performance. You would also want to do some I/O testing and benchmarking to determine how well the A9 Azure VM performs, and whether it can handle your workload.

Finally, you have to consider the economics of using a standard Azure gallery VM image, where the SQL Server 2014 license costs are included as part of your monthly Azure cost. Depending on how long you think your instance will be in service, using an Azure VM might be a very good deal or not such a good deal. One factor that might influence this calculation is the likelihood that Azure VM prices will go down over time.