Improve worker wakeup
The fact that we are worse than golang or tokio when the system is not saturated with work in our echo server benchmark bothers me. That's why I experimented with different ways of suspending and reactivating worker threads.
- Blocking IO: Read and write on an eventfd opened with EFD_SEMAPHORE
- Signals: sigwait and pthread_kill
I am not convinced that my implementations are sound. And not produce lost wakeups. If we decide to change the synchronization primitive they should be thoroughly checked.
I did 5 runs of the echo test but haven't done any real statistics. Here are the results: wakeup.pdf
connections 500 1000 5000 10000 25000
signal 220.5737 (+8%) 257.5492 (+14%) 325.7909 (+12%) 313.9826 (+4%) 288.9554 (+8%)
mutex + condvar 202.7931 (0%) 225.6365 (0%) 289.4035 (0%) 300.4161 (0%) 267.1373 (0%)
eventfd 162.5336 (-20%) 209.8668 (-7%) 319.8507 (+7%) 311.5511 (+3%) 286.3498 (+7%)
My interpretation
- BlockingIO with eventfd is the worst of the three variants.
- Using signal seams interesting.
- 5000 connections and eventfd has a really huge variance (not visible hear) and is not really trustworthy
- I can't really explain why both new variants perform better in a saturated system. I would expect that they do less in
onNewWork
which is executed in eachdispatchLoop
iteration. This would also explain why they behave very similarly.