Do you have a dumb question that you're kind of embarrassed to ask in the main thread? Is there something you're just not sure about?
This is your opportunity to ask questions. No question too simple or too silly.
Culture war topics are accepted, and proposals for a better intro post are appreciated.
Jump in the discussion.
No email address required.
Notes -
it could also be you have a bunch of bad options for the TCP connection. tho, i suspect iperf would should have good defaults. a common problem with TCP application is not setting TCP_NODELAY and be a cause of extra latency. the golang language automatically sets this option but i'm sure a lot of languages/libraries do not set it. you can also have problems between userspace and kernelspace (but maybe not at this speed?). like if you can only shift 200 Mbps between the kernel and userspace because of syscall overhead on a single thread and in the multiple stream case you are using multiple threads then maybe that is why the performance improves. also, if you are using multiple streams you are going to have a much larger max receive window. there is some kind of receive buffer configuration (tcp_rmem?) that controls how large the receive buffer is and the thus the receive window. its possible this is not large enough and so using 10x connections means you effectively now have 10x the max receive window. also, there is tcp_wmem configuration that controls the write buffer in a similar way. cloudflare has an article on optimizing tcp_rmem https://blog.cloudflare.com/optimizing-tcp-for-high-throughput-and-low-latency/ which shows their production configuration.
More options
Context Copy link