I am trying to increase bandwidth for particular hosts by using lacp on a Cisco Catalyst 4500 switch.
However, it seems that I will always end up with the same flow (src/dst mac, ip, and port) being completely switched over one interface, no matter what port-channel load-balance option I choose.
Why is there no such thing as a round robin option for this? (It seems that some older models had this option)
From my understanding, LACP always requires that packets in a particular network "conversation" (a TCP session or a UDP stream between the same source and destination ports) traverse a single LACP path, in order to guarantee delivery order. LACP chooses a path based on a hashing algorithm. In Linux, the hashing algorithms are those described in the bonding driver's xmit_hash_policy setting:
layer2 - meaning the hash consists of the source and destination layer 2 (MAC) addresses;layer2+3 - meaning the hash consists of the layer2 information, plus the source and destination layer 3 (IP) addresses;layer3+4 - meaning the hash consists of the layer 3 (IP) addresses plus whatever layer 4 addressing information is available (e.g. TCP or UDP source and destination port numbers, if the packets are not fragmented).Cisco has a number of additional options.
The server and switch do not need to use the same hashing algorithm; all that is required is that the packets for a particular conversation choose the same path as long as that path is available.
LACP controls the aggregation of links, it doesn't control switching. The selection of a particular member of a bundle is via an algortithm decided upon and implemented by each switch vendor. It has to be something that can be programmed into hardware in order to operate at wire speeds relatively cheaply, and a hash of various fields of an Ethernet frame is suitable for this. To perform round-robin the ingress hardware would need to know all about the bundle members and keep state about previous frames switched. That problem is an order of magnitude more complex. I don't know of any switches that perform round-robin across a LAG.
Linux and ESX etc can perform round-robin in software, that's fairly straightforward in comparison since the entire interface selection process is already in software. Also notice that typically servers have a higher transmit requirement than receive and so for many applications the increased bandwidth that round-robin achieves is only required server-to-switch.
I tried using round-robin to connect two Linux machines together, as they both support this. While I did get more available bandwidth than a single link, the result was not spectacular.
I bonded together two 10G links, in theory allowing 20 Gbps between the hosts. Sometimes I would indeed get close to this, but a lot of the time I would only get 11-15 Gbps.
The problem is that for whatever reason, the packets don't always arrive in order. For TCP this can cause it to think the packet has been lost and request a retransmit (lowering overall bandwidth) and increasing latency, but for bulk transfers like file servers you still end up with an overall higher transfer rate so it's not the end of the world.
However for protocols that don't guarantee delivery like UDP and realtime video streaming it's a disaster, as they tend to process the latest packet available and discard anything earlier that hasn't arrived yet, in order to get the lowest latency possible - they don't want to wait for all the packets to arrive first as that can introduce a lot of delay. While a lost packet here and there is no problem, with round-robin you can get extremely high numbers of packets arriving out of order which can completely corrupt the data stream and for video, make it unwatchable!
Imagine the source machine is sending a single stream out of four bonded NICs in the order NIC1, NIC2, NIC3, NIC4, NIC1, etc. Now imagine the receiving machine is processing the packets in the opposite order, first checking NIC4, then NIC3, NIC2, NIC1 then NIC4 again. In this example the packet it picks up first will be from NIC4 which will be the latest, and the next three packets off NIC3, 2 and 1 will be older. That's 75% of the packets out of order, which could cause a realtime video stream to lose 75% of its packets easily making it unwatchable.
The only way this can work is if the receiving hardware is designed to store the packets until a complete set has arrived and then forward them on in order, which will increase latency. It would also mean a switch would have to decide how long to wait before deciding a packet has been lost and give up waiting for it, so before you know it you're implementing a cut-down TCP-style guaranteed delivery protocol on top of Ethernet. Then you have to consider what happens when someone tries to bond a 10G and four 1G links, hoping to get 14G. Your 10G link will send so many more packets it will look like all the ones coming in over the 1G link have gotten lost, even though they are just taking 10 times longer to transmit. So how do you tell the difference between a lost packet or one that is just taking ages to arrive across a slower link?
Long story short, it's complicated to make it work.
The 40G QSFP standard uses four 10G links in parallel to achieve 40 Gbps of bandwidth but this only works because it's not done at the packet level. Each switch is splitting the packet and sending it out across the four links in parallel, and the receiving switch is combining it at the far end, thus keeping every packet in order and the latency low. This is really the only way to do it properly, but it means coming up with a new standard like they did with QSFP because the data going over the wire is no longer standard Ethernet and not backwards compatible.
