Technical fact 3: Connecting server and switch over 40GbE

19-mrt-2021 12:22:34

Incredible progress has been made by technology in the last couple of years. The future of technology is overwhelming for some, while it can be exciting for others. We have put together some interesting and surprising facts about tech! You might be surprised and amused by what we've discovered.

KristofKristof Van Elshocht: "How to connect server and switch over 40GbE?" 

 

What was the issue?

Are you still running on slow 10GbE legacy networks?

This seems like an odd question, considering that we are still migrating existing customers from 1GbE to 10GbE technology. However, the 10GbE ethernet standard made its first introduction in 2002! (Almost 20 years ago!)

Mellanox-40-GbE-SR4-MPO-Nutanix-SFP+-SFP28_1

This being said, recently we had a customer who wanted to set-up its new Nutanix configuration by using 40GbE standards instead of 10GbE. Not so strange, as the investment was made for 5years. In 5 years’ time, 40GbE might also be considered as a standard (who knows?).

Mellanox-40-GbE-SR4-MPO-Nutanix-SFP+-SFP28__7

There is something to say when setting up Nutanix environments, especially I/O intensive ones, to go for a high-speed (and low latency!!!) network as the storage traffic is also sent over the ethernet wires. Environments relying on flash storage sometimes find their new bottleneck on ethernet speeds. It is a never-ending exercise to evade bottlenecks, knowing that every time one is resolved, a new one will pop up.

 

Now, for me it was quite new to know how to connect these machines over 40GbE. I had a lot of questions about transceivers, cables, connections and after doing a lot of research it seemed quite difficult to find how to exactly connect these devices using this standard, hence this blog!

Kristof's approach:

Let me start on server-side. The Nutanix nodes (in this case, but you can threat it more general as “compute node”) had 40GbE Mellanox adapters configured.

Mellanox-40-GbE-SR4-MPO-Nutanix-SFP+-SFP28_2

As you might notice on the picture, these 40GbE adapters do not have a regular SFP+ or SFP28 connectors. The form factor of these ports (and transceivers) is QSFP28.

What the differences are between SFP, SFP+; SFP28, QSFP+ and QSFP28 can be found in this blog post.

As always, such adapters don’t come with transceiver modules. The reason behind this is that these transceivers can be configured in many different flavours (different connectivity, different distance support, etc…).

For this particular customer, the distance between switch and servers was less than 100m.

Looking at the datasheet of the adapter, we found out that for 40GbE and a distance between server and switch of less than 100m only 1 type of transceiver was suitable.

Mellanox-40-GbE-SR4-MPO-Nutanix-SFP+-SFP28_3This table definitely brought some questions with regards to connectivity. What exactly is this SR4 standard and what are the MPO connectors?

It became clear that this adapter would not support LC connectors as we all know (the typical fibre optic cables) but we needed “MPO cables”.

MPO (Multi-Fiber Push On) cables are capped with MPO connectors at either end. MPO connector is a connector for ribbon cables with at least 8 fibers, which is designed to provide multi-fiber connectivity in one connector to support high bandwidth and high-density cabling system applications.

Mellanox-40-GbE-SR4-MPO-Nutanix-SFP+-SFP28__8

Now we knew we needed MPO cables there was one more hurdle to take as there are multiple implementation methods possible, called “polarity”.

The three methods for proper polarity defined by TIA 568 standard are named as Method A, Method B and Method C. To match these standards, three types of MTP fibers with different structures named Type A, Type B and Type C are being used for the three different connectivity methods respectively.

MTP Trunk Cable Type A: Type A cable, also known as straight cable, is a straight through cable with a key up MTP connector on one end and a key down MTP connector on the opposite end. This makes the fibers at each end of the cable to have the same fiber position. For example, the fiber located at position 1 (P1) of the connector on one side will arrive at P1 at the other connector. The fiber sequence of a 12 fiber MTP Type A cable is showed as the following:

Mellanox-40-GbE-SR4-MPO-Nutanix-SFP+-SFP28_4

MTP Trunk Cable Type B: Type B cable (reversed cable) uses key up connector on both ends of the cable. This type of array mating results in an inversion, which means the fiber positions are reversed at each end. The fiber at P1 at one end is mated with fiber at P12 at the opposing end. The following picture shows the fiber sequences of a 12 fiber Type B cable.

Mellanox-40-GbE-SR4-MPO-Nutanix-SFP+-SFP28_5

MTP Trunk Cable Type C: Type C cable (pairs flipped cable) looks like Type A cable with one key up connector and one key down connector on each side. However, in Type C each adjacent pair of fibers at one end are flipped at the other end. For example, the fiber at position 1 on one end is shifted to position 2 at the other end of the cable. The fiber at position 2 at one end is shifted to position 1 at the opposite end etc. The fiber sequence of Type C cable is demonstrated in the following picture.

Mellanox-40-GbE-SR4-MPO-Nutanix-SFP+-SFP28_6

As we need to connect a switch and a server, we needed polarity type B to be implemented.

And guess what: a correct adapter, correct transceivers on both server and switch side and correct cabling with correct polarity method gave us a stable connection!

Do you need more information about this topic or maybe you can't solve another IT problem on your own? Please reach out, we will help you!
 

Contact Us

Aangeraden artikels

Op basis van Technical Facts

Schrijf u in op onze nieuwsbrief