We've become somewhat obsessed with squeezing every last bit of performance out of our HTML5 MMORPG, but up until recently, we hadn't put much thought outside of our own code. Since CasinoRPG is a rather large-scale HTML5 game, it has considerably more resources than your typical webpage. We are talking hundreds of images, javascript and audio files, that most of which must load before play begins.

Being a largely bootstrapped indie studio, we've been using CloudFlare's reasonable Pro plan to cut costs. CloudFlare offers some great features, but is it ideally suited to be a high performance content delivery network (CDN) like our game requires? This is what I set out to find, and I wrote and open-sourced a utility to help me out:

Node.js CDN Benchmark CLI on GitHub

Using this tool made it easy to setup a long-running benchmark to give us a good idea of performance -- say, over a 24-hour period. Our command looked something like:

node bench.js \
    --files https://cloudflare.example.com/large,https://cloudfront.example.com/large \
    --interval 120
    --limit 720 \
    --out results.csv

This downloads the listed files, one after the other, every 2 minutes for 24 hours. It then appends the download times in a CSV file for analysis. I used this to benchmark javascript and image files, but it can be used for entire sites just as easily.

Since the whole point of CDN's are for global speed, we ran two separate tests on different sides of the planet - one from our office in Oklahoma City and one on a Joyent server instance in Amsterdam. Each test benchmarked the download speed for a large javascript file (1.4 MB), a medium image file (167 KB) and a small image file (10 KB).

Oklahoma City Results

There isn't much that needs to be said after looking at those charts. CloudFlare performs admirably...for some of the day. The other half renders our game almost unusable on a fresh cache hit. I actually ran the test 3 different days because I thought we might have hit an anomaly, but the pattern is pretty consistent every day.

The large file was 428% faster on CloudFront, medium 131% and small 49%. The standard deviation is even more telling, at 1,285% higher on CloudFlare for the large, 167% for medium and 63% for small.

Amsterdam Results

Amsterdam shows a significantly different story than stateside, but still swings in the favor of CloudFront. 435% faster for the large file, 488% medium and a whopping 561% on the small. The standard deviations were much less troubling in this region, but still noticeable at a 44% higher rate on the large with CloudFlare, 63% medium and 5,389% on the small (somewhat skewed by a few abnormally long load times).

Conclusions

This is by no means an exhaustive test, but the results have made it pretty clear that if we want raw speed and overall consistency, we need to go with CloudFront. CloudFlare certainly has its benefits, but just like with any part of your stack, you need to select the tool that is best suited for the task at hand.

We only ran the test comparing CloudFlare to CloudFront as we already have a lot of resources in S3, but we'd love to see others use the utility to benchmark other networks (available on NPM as well) and share the results.