Datacenter Benchmarking

Introduction

I've recently racked a server at Tier .NET and have been pretty disappointed by the network performance. Because of this, I've devise a sort of ad hoc way of measuring the quality of the network between this datacenter and two other datacenters I've been considering.

I'm not a network engineer so take these tests with a grain of salt. I'll try to provide the most accurate explanation I can of this but keep in mind I'm focusing more on documenting the journey vs anything else. Note that the tests were all ran for a period of a few days to a week.

I thought of making all three providers anonymous at first, but I guess if other people are curious about services form these providers, it would be good for me to disclose who they are. They are GigeNET, Tier NET, and SBA Edge. I won't disclose with provider I'm currently with, although I'm sure you can easily figure that out pretty easily.

Test 1 - Uptime Monitor Script

This is the simplest test I can think of. We use services like Uptime Robot along with the Uptime script to measure the uptime of all service. In response, it gives us the uptime percentage of all three providers. The SLA guarantee is listed along side the actual performance and the test result. The results of each are listed below.

Test Importance: Moderate

Test 2 - The continuous ping test

This is the second simplest test I can think of but helpful for considering UpTime. The actual ping is important, but I'm also looking standard deviation and packet loss. It's a bit unfair to compare ping directly across each location as one provider is in Dallas. All three providers are pretty reasonable, however Tier NET has the highest standard deviation by FAR in all locations. They also have the highest packet loss.

Test Importance: Moderate

Test 3 - SpeedTest

This script continuously runs a speedtest from the popular speedtest.net website.

So, there is something incredibly deceiving about this data though. First, SpeedTest thinks the GigeNet server is in New Jersey, so it really isn't the closest server. Second, the TierNET server is half a millisecond from the test server. It's possible they are in the same datacenter or extremely close physically. I can't force SpeedTest to choose a server in Chicago since it's too far away. Third, I lost the data for SBA Edge so it's not really a fair comparison. I'm pretty disappointed in SpeedTest to test the network in general, but I guess it's made just as a quick way to test network connection.

Test Importance: None

Test 4 - iperf3

I booted up iperf3 on a few virtual machines across Digitalocean, Azure, and AWS. I had a each datacenter node cycle connecting to each node every minute and running a 10 second test for both TCP and UDP in both regular and reverse mode. Note that the dataset count for Tier NET is lower because the Tier NET node had trouble connecting to some nodes at certain times of the day and particularly one node one Digital Ocean node on the West Coast. They were also the only node to have packet loss. GigeNet however never seemed to exceed 400Mbs upstream. Note that the SBA Edge only has a 100Mbit uplink while the other two have 1000Mbit uplinks.

Test Importance: Moderate

Test 5 - Real world applications

There's quite nothing like booting up a game server (Valheim and Minecraft to be exact!) and throwing a bunch of your friends at it! With SBA Edge and GigeNET, there were absolutely no complaints and everyone had a good time. With the Tier NET on all three days, at least half of my players experienced ping spikes and even random disconnects in both Minecraft and Valheim. It got better at times but took less than a hour to devolve into really bad. Switching back to one of the two other nodes instantly fixed it the network issues. No fancy tables on this one, so you'll just have to take my word for it.

Test Importance: High

SBA Edge

SBA Edge (Chicago Datacenter) provides 100mbit dedicated internet access and it shows as there was almost no instances were really any speed drops. Even though the uplink was sub-gigabit compared to the other two datacenters, the overall reliability exceeded the other two providers. They were also the most expensive of all three providers. They've also been the provider I've had a server with the longest and I've never had an issue with them.

Verdict:

+ Excellent Reliability and Network Quality
+ Actual dedicated internet access
+ Support staff is extremely quick answering tickets and very knowledgable
- Non-support staff have long response times.
- Expensive.
- Extremely expensive remote hands and service fees.
- A Gigabit Uplink is also expensive.
- Support staff

TierNET

TierNET (Dallas Datacenter) I've had for a few months and I've constantly had issues with clients using my applications or accessing game servers. They complain about lag spikes and disconnects. This shows on the random ping spikes and the by-far highest variance in ping. Along with the highest packet loss, they are also the only provider whose uptime was not within their SLA.

Verdict:

+ Affordable Prices
+ Friendly/Helpful Staff with decent ticket response time.
+ Pretty good speed
- AWFUL network reliability
- Network is EXTRA unreliable for certain people during real world tests
- Can't meet their SLA for network uptime

GigeNET

GigeNET (Chicago Datacenter) is the newest host to datacenters I've been with and sit comfortably between SBA Edge and Tier NET in both quality and price. The variance in ping is pretty good and uptime was on par with SBA Edge. Although a gigabit port is advertised, I've seen my upload speed hover around 350 Mbs but the downstream speed hovers close enough to a gigabit. The speed also has the a fairly high variance between all three providers.

Verdict

+ Fairly Reliable and Solid Network
+ Pretty Solid Bandwidth
+ Decent ticket response times.
- Have yet to see my upstream speed go about 350 Mbit
- Little bit on the pricey side

Conclusion

This was definitely an interesting experience. There is a wealth of services out there to assist with helping determine network quality, but many are either commercial or analyze network quality within an intranet versus testing against other locations over the internet. Using UptimeRobot and other services are a bit of a waste considering you can spin up multiple Azure/Amazon/DigitalOcean VMs for less in various locations and run both uptime and iperf3 on it. Using a combination of uptime and iperf3 on each node I believe provide a pretty solid measure of quality of the network.

Speedtest is a bit of a wash for testing server speed. The real world test using game servers is extremely ad hoc, but did a pretty good job reflecting what I've been seeing in the other tests.