Future cancellation is broken by design

The design decition of distributed independent io_urings per worker, which allows us to not synchronize request submission prevents us from having simple cancellation logic.

An io_uring can only cancel a request it actually knows about. But in the EMPER design it is totally possible that a Fiber submits a request, blocks, is continued on a different worker and then wants to cancel the original request. Which will fail because the io_uring of the current worker does not know the request submitted on the previous Worker.

The cancellation of the partially completed future chain in our CancelFutureTest is the perfect example:

cancelPartialCompletedChain();
004 1535.46106302149 PS    0x7ff904082460 constructed by fiber Fiber [ptr=0x562abe4b9c40 func=0 arg=0 aff=
nullptr]
004 1535.46106309403 PS    0x7ff9040824f0 constructed by fiber Fiber [ptr=0x562abe4b9c40 func=0 arg=0 aff=
nullptr]
IOC 1535.46106308601 IO    0x7ff904000900 Reaping completions for worker 4
004 1535.46106317007 PS    0x7ff904082580 constructed by fiber Fiber [ptr=0x562abe4b9c40 func=0 arg=0 aff=
nullptr]
004 1535.46106367300 IO    0x7ff904082570 submit read Future 0x7ff904082570 to IoContext 0x7ff904000900
004 1535.46106374043 IO    0x7ff904000900 submitting read Future 0x7ff904082570 and it's dependencies
004 1535.46106381056 IO    0x7ff904000900 Prepare read Future 0x7ff904082450 as a dependency

Worker 4 has prepared a chain of two read futures and submitted those.

004 1535.46106389892 IO    0x7ff904000900 Reaping completions for worker 4
004 1535.46106396855 IO    0x7ff9040824e0 submit write Future 0x7ff9040824e0 to IoContext 0x7ff904000900
004 1535.46106403487 IO    0x7ff904000900 submitting write Future 0x7ff9040824e0
004 1535.46106416892 IO    0x7ff904000900 Reaping completions for worker 4
004 1535.46106423745 IO    0x7ff9040824e0 Waiting on write Future 0x7ff9040824e0
004 1535.46106431088 PS    0x7ff9040824f0 block() blockedContext is 0x7ff904073a00
004 1535.46106443040 C     0x7ff904073a00 saving and switching to Context 0x7ff904083ac0 [tos: 0x7ff904093
b30 bos: 0x7ff904083b40]
004 1535.46106451175 IO    0x7ff904000900 Reaping completions for worker 4
004 1535.46106462526 SLEEP_S 0x7ffedd48c478 going to sleep

Worker 4 has submitted the write requests completing one of the reads and the Fiber blocks until the write is completed. Thus there is no more work in the system Worker 4 goes to sleep.

IOC 1535.46106575295 IO    0x7ff904000900 Reaping completions for worker 4
IOC 1535.46106629826 IO    0x7ff904000900 got 2 cqes from worker 4's io_uring
IOC 1535.46106651016 IO    0x7ff9040824e0 Complete write Future 0x7ff9040824e0 with result 8
IOC 1535.46106676874 PS    0x7ff9040824f0 unblock in fast path
IOC 1535.46106689126 IO    0x7ff904082450 Complete read Future 0x7ff904082450 with result 8
IOC 1535.46106698343 PS    0x7ff904082460 no unblock in slow path
IOC 1535.46106711087 SLEEP_S 0x7ffedd48c478 NotifyMany 1 from ANYWHERE
000 1535.46106848001 SLEEP_S 0x7ffedd48c478 awoken
000 1535.46106878818 IO    0x7ff91c000900 Reaping completions for worker 0
000 1535.46106890039 DISP  0x562abe4a0dd8 executing fiber 0x7ff8d0001140
000 1535.46106899837 F     0x7ff8d0001140 run() calling 0 (ZN5FiberC4ERKSt8functionIFvvEEPiEUlPvE_) with a
rg 0
000 1535.46106908683 C     0x7ff91c073a00 discarding and switching to 0x7ff904073a00
000 1535.46106916628 CM    0x562abe4a0c40 Freeing context 0x7ff91c073a00
000 1535.46106932126 PS    0x7ff9040821b0 constructed by fiber Fiber [ptr=0x562abe4b9c40 func=0 arg=0 aff=
nullptr]

The completer thread does its job and reaps the completions of the sleeping Worker 4 and notifies a sleeping Worker. Worker 0 is awoken and resumes the blocked Fiber.

000 1535.46106940171 IO    0x7ff9040821a0 submit cancel Future 0x7ff9040821a0 to IoContext 0x7ff91c000900
000 1535.46106947505 IO    0x7ff91c000900 submitting cancel Future 0x7ff9040821a0
000 1535.46106967592 IO    0x7ff91c000900 Reaping completions for worker 0
000 1535.46106975317 IO    0x7ff91c000900 got 1 cqes from worker 0's io_uring
000 1535.46106983652 IO    0x7ff9040821a0 Complete cancel Future 0x7ff9040821a0 with result -2

The cancellation fails with -EBADF because in the io_uring of Worker 0 the future submitted on Worker 4 is unknown.

000 1535.46106991056 PS    0x7ff9040821b0 no unblock in slow path
000 1535.46106998640 IO    0x7ff9040821a0 Waiting on cancel Future 0x7ff9040821a0
000 1535.46107005452 IO    0x7ff904082570 Waiting on read Future 0x7ff904082570
000 1535.46107012415 PS    0x7ff904082580 block() blockedContext is 0x7ff904073a00
000 1535.46107024698 C     0x7ff904073a00 saving and switching to Context 0x7ff91c073a00 [tos: 0x7ff91c083
a70 bos: 0x7ff91c073a80]
000 1535.46107032232 IO    0x7ff91c000900 Reaping completions for worker 0
000 1535.46107040748 SLEEP_S 0x7ffedd48c478 going to sleep
IOC 1535.46107059783 IO    0x7ff91c000900 Reaping completions for worker 0

The result is a sleeping emper.

Problem is this is not trivially fixable.