c# - TCPIP Socket Listener Performance Issue -

i have straightforward windows service implements c# tcpip socket listener service. service proprietary simple protocol. works accurately , reliably under heavy load. however, performance inadequate purpose has serve, , have been working fix this.

i don't know architecture officially called. basic asynchronous implementation (iasyncresult state objects, acceptcallback(), readcallback(), etc.).

here basic performance issue: part of response request, service has make call out remote service (operated independent service provider). typically, responses operation take (approximately) 1 second. however, if load 20 test clients (running independent consumers multiple machines), 1 second responses remote service stretch out 5 seconds or more. have used tracing confirm of 5 seconds "within code conducts remote transactions". tracing confirms server handling requests concurrently expected , not stacking them up.

that said, have done independent testing proves remote service can handle stuff in 1 second time frame regardless of load (that service operated large well-known company knows doing , innumerable organizations - confident not problem). have altered load tests perform i/o directly rather utilizing service , achieves one-second turn-arounds.

finally, test, implemented multiple listeners within service listen on different ports , had load tester clients distribute requests amongst ports. doing increased performance in relation number of ports (two listeners == approx 2x performance)

now, know listening on multiple ports not correct solution , should not make difference. tell me there flaw in implementation interfering ability recycle listeners. or, indicates drivers remote service being starved of cycles , not able respond expediently (though inclined not believe scenario).

the throughput requirements don't seem high me - have handle around 2 incoming requests second. need able respond within around 2 seconds or on average. 5 7 second responses bring our consumer down (which, in itself, odd have no control on that).

i have listener.maxconnections set 2500. making number 5000 or 20 doesn't affect issue.

i hoping advice on how figure 1 out. suspect performing business-layer code within wrong context. i'm not seeing it.

regards,