10

I have limited knowledge of this and I'm still learning, so I hope someone can give me advice on this.

I have a 10GbE NAS, a 10GbE NIC in my PC and a switch with two 10GbE ports, which connect all together with fiber. Then there is also another 100Mbit cable attached which connects the switch to the router, which is also the standard gateway. This one also provides the internet connection for all connected devices.

I noticed that a direct connection to the NAS gives me speeds that 10GbE can handle, but when I put the switch in between (all standard VLAN, same subnet and standard gateway), the speeds to the NAS is limited to a max of 1GbE.

Is because the standard gateway is the central where all traffic comes together? How can I manage a peer to peer connection directly to the NAS via the switch, so the traffic goes through the 10GbE cable to the switch and from there through the 10GbE to the NAS?

Extra details:

IP NAS: 192.168.10.120
IP switch: 192.168.10.125
IP PC: 192.168.10.103
Standard gateway: 192.168.10.1
Subnet: 255.255.255.0
Switch: D-link 1510-52

Thank you

aardbol
  • 259
  • 4
  • 13
  • Which switch are you using? Is it a 10GbE switch? – Gerben Oct 20 '17 at 11:54
  • 1
    It's a switch with two sfp+ ports. The nas and PC are connected in these – aardbol Oct 20 '17 at 12:57
  • 1
    What type of SFP are you using? – MerlinTheMagic Oct 20 '17 at 16:31
  • 1
    Follow up. I see you state it’s a 10GBase-R in another comment, but if you provide the model of the SFP we could help you further. For example it’s possible you are driving a short jumper with a ZR SFP or the opposite a long cable with an SR module. – MerlinTheMagic Oct 20 '17 at 17:43
  • Thanks for the help MerlinTheMagic, but it seems my issue was solved without any change. Another test made the speeds go up to the previous 1GbE+ again – aardbol Oct 20 '17 at 18:33
  • What speed to do you get if only the two 10GbE devices are connected to the switch and nothing else is connected to the switch? – kasperd Oct 20 '17 at 21:36
  • @EarthMind computers don’t do random. You either get to the root cause or it comes back and bites you in the a.. :) – MerlinTheMagic Oct 20 '17 at 22:53
  • @MerlinTheMagic You're right. I'm keeping an eye on it while I do further tests. It could've been the NAS too for an unknown reason. I don't seem to reach full SSD speeds on it anyway, even though I do via a DAS and USB 3.1. Any case, I'm doing further tests – aardbol Oct 21 '17 at 12:06
  • 1
    @MerlinTheMagic See my last comment to Peter Green's post for the root of the cause :-) – aardbol Oct 22 '17 at 11:03

3 Answers3

16

I don't think the default gateway has anything to do with your problem. Local traffic between two devices on the same subnet doesn't flow through the default gateway and in any case the speeds you are reporting for your local traffic are 10 times higher than the speed you report for the link to your default gateway.

A couple of possibilities spring to mind.

Firstly are you in the correct ports? from some searching it seems only two of the four fibre ports on that switch are 10G.

Secondly I have heard reports of some early switches which have 10G ports but which can't handle a full 10G for a single flow because of their internal forwarding design.

Peter Green
  • 13,303
  • 2
  • 21
  • 47
  • 1
    My thoughts too. I'd check the netmasks and unplug the router to remove it from the problem. Also check you've no port mirror on the switch. – jonathanjo Oct 20 '17 at 12:56
  • 1
    I'm using the right ports yes, because they are sfp+ ports and I'm using the right transceivers to have a connection. These ports are advertised as uplink/stacking ports but no limitations have been described in the specs. Full duplex 10Gbase-R – aardbol Oct 20 '17 at 13:01
  • 1
    ... I'd also try locking the port speeds on the switch, NAS, PC to see if they might be changing speed for some reason. – jonathanjo Oct 20 '17 at 13:17
  • 1
    It might also be worth looking to see if there are any firmware updates available for the switch. – Peter Green Oct 20 '17 at 14:37
  • 2
    The max switching capacity is 140 Gbps from (http://us.dlink.com/products/business-solutions/52-port-gigabit-smartpro-switch/) so I believe it should easily be able to manage it... – djsmiley2kStaysInside Oct 20 '17 at 14:55
  • 2
    AIUI the issue on some early switches with 10G uplinks was that the switch engine was only designed for 1G ports. So the switch manufacturer used the link aggregation features of the switch engine to implement the 10G ports limiting the bandwidth of any one flow to 1G. This is not too much of a problem when using the ports for the intended purpose as uplinks but it is a problem in your usage scenario. – Peter Green Oct 20 '17 at 15:02
  • You're right Peter Green that the speed of the transfer was x10 than the 100Mbit. I miscalculated that. But for some reason, I was able to transfer files at the same speed as before, when the NAS was directly connected to the PC. I don't know what happened because I tested again without changing a thing. – aardbol Oct 20 '17 at 18:32
  • @PeterGreen I found what was causing the issue. Your point about the transfer speed being x10 than 100 Mbit helped me think towards that end. The reason was that I had two cables attached to the switch and my PC, one through SFP+ and one RJ-45 gigabit one, while the latter was attached second. For this reason my computer prioritized the gigabit connection to go before the 10 GbE one and therefore limiting the speed to that cable. – aardbol Oct 22 '17 at 11:02
3

When switching, throughput between two ports is completely independent of the link speed of another interface. With the non-blocking switches of today, it's even independent of the throughput on any other ports.

There are several possible reasons for the low throughput:

  • SFP+ transceivers: they need to support 10GBASE; SFPs/mini GBICs only support 1G rate - check link status in switch
  • interface link configuration: may be limited to 1G speed
  • other interface configuration: rate limiting, maximum frame size mismatch, ...
  • physical link problems: wrong fiber type (-SR = MM, -LR = SM), damaged fiber, dirty ports (FCS and - if indicated - FEC will show errors counting up)
Zac67
  • 84,333
  • 4
  • 69
  • 133
3

Looking at the Data Sheet for the D-Link 1510-52 switch:

It mentions switch being capable of "140Gbps" of throughput - using typical marketing "full-duplex" numbers this would mean that on a 50x1G and 2x10G switch like the 1510-52 that the switch is capable of running line-rate through every interface simultaneously. Eg: (1Gbps x 50) + (10Gbps x 2) = 70Gbps x 2 (Full Duplex) = 140Gbps.

Similarly the Packets-Per-Second (pps) numbers for the box are listed as 104.16Mpps, which equates nicely to line rate:

1Gbps = 1,000,000,000 bits/s = (1,000,000,000 bits/s) / (8 bits/byte)= 125,000,000 bytes/s

10Gbps = 10,000,000,000 bits/s = (10,000,000,000 bits/s) / (8 bits/byte)= 1,250,000,000 bytes/s

PPS on 1G port = (125,000,000 bytes/s) / (84 bytes/packet) = 1,488,095 pps

PPS on 10G port = (1,250,000,000 bytes/s) / (84 bytes/packet) = 14,880,952 pps

50 x 1,488,095 pps + 2 x 14,880,952 pps = 104,166,654 pps ~ 104.16Mpps

So, if the data sheet is to be believed, the switch should be capable of this easily.

The one thing you might want to confirm though is that there are two 1G SFP ports and two 10G SFP+ ports, so make sure that you are connected to ports 51 and 52 on the switch and that you are seeing Orange LED status on both ports (which means 10G). If you're seeing Green LEDs this means you're only getting a 1G link.

Benjamin Dale
  • 9,421
  • 17
  • 46
  • Thank you for the information. I shouldn't be able to use the SFP ports with the SFP+ plugs, because they are not backwards compatible. SFP can be used in SFP+ generally, but not vice versa. So this was why I was sure from the start that I didn't confuse the ports. For now I seem to be able to use the full 10 Gbit connection speed for some reason. I'll be keeping an eye on it whether it stays that way and if not, look for the root of the problem. – aardbol Oct 21 '17 at 12:04
  • @EarthMind that's correct - more just making sure your SFP+ modules are in fact SFP+ and not SFPs – Benjamin Dale Oct 22 '17 at 03:59