Tech Innovation Focus: The Path to 100Gb Market Connectivity

Tech Innovation Focus: The Path to 100Gb Market Connectivity

Tech Innovation Focus: The Path to 100Gb Market Connectivity

At Options, we regularly review the market to see what technologies can be adopted to improve our platform – and by extension, the service that we provide to our clients. We try to be early adopters where it makes sense, and we typically offer the benefits of these new technologies to our clients at no extra cost (e.g. Ixia TradeVision market data gap detection).

Until recently it had been cost prohibitive for us to do a wholesale migration of our WAN backbones to 40Gb. However, as market data rates have increased, we had found ourselves acquiring a large number of lit fiber 10Gb services, and doubling/tripling up to make 20/30Gb port channels where required. Very recently this took us to the tipping point where going with a majority dark fiber network made sense and we started the first step of a major backbone overhaul with a 40Gb upgrade across our US infrastructure. But the story doesn’t end there.

40Gb… and beyond?

Anyone familiar with the Options Managed Colocation platform will know that we’re an Arista shop (read our case study here). By making the choice to go with a dark fiber network, we were able to utilize our existing aggregation layer switches (Arista 7150s) to make the immediate jump to 40Gb, but we already had our sights set higher than that.

We recently started the migration from the 7150s to the 7060CX2 series switches at our aggregation layer. Layering this cutting-edge network technology on top of the dark fiber has future-proofed our platform for the next phase of upgrades:

  • Fulfilling the increasing client demand for 40Gb cross connects. While we can support these on the 7150s, we are limited by the number of physical QSFP ports and the agile ports (4x10Gb treated as one physical 40Gb). The new 7060CX2 switches will provide us with many more native 40Gb ports than we currently carry on the aggregation layer.
  • Laying the foundation for 100Gb. The 7060CX2 provides QSFP100 ports. This will facilitate a timely and seamless upgrade to a 100Gb dark fiber backbone across our NY Metro area in the near future.

Taking Care of Latency

When choosing our dark fiber paths, we obviously wanted to ensure that we were getting the best of breed latencies and paths between our sites. From a practical perspective, this led to us choosing the fastest paths from Equinix Secaucus (NY4) out to the 3 other major DCs in region (Carteret, Mahwah and Weehawken). From there, we secured backup paths via different carriers on geographically diverse lines to ensure that we weren’t sacrificing any resiliency in the new design. This resulted in our backbone looking something like the below:


With the move to a 40Gb backbone, we needed to up our monitoring and capturing capabilities. As standard, we had been using commodity RHEL 7 servers with Solarflare 8522-Plus cards for capturing at line rate. When traffic flows were bursting over 10Gb we would reconfigure the capture hosts in a port channel to support the higher rates.  With the jump to 40Gb, however, this became a bit more tricky to accommodate. As a result, we made the decision to upgrade to Solarflare 8542-Plus cards to ensure that we maintain the line rate capturing capabilities without issue.

The Final Steps

This hasn’t been a quick process. As with all major upgrades of production infrastructure we needed to undergo a comprehensive due diligence with a leading telecoms consultancy to ensure that there was no negative impact to our clients at any stage of the process, from the circuit selection to the actual cutover from our legacy WANs to the new 40Gb circuits.

In the world of low latency trading, optimizing speed is everything and yet bandwidth demands can counter that. And so, on to the next innovation for our clients… we are excited to be trialing cutting edge 25Gb server ports and 100Gb capable switches from a number of leading vendors. Watch this space for developments!

Gareth Mason, Managed Colocation Team

Leave a Reply

Your email address will not be published. Required fields are marked *