I can connect to TWRP (3.2.3-0) with
fastboot devices does not list any and
fastboot continue (while my phone is booted into TWRP) only prints:
$ fastboot continue < waiting for any device >
I can rule out a driver problem since I have used
fastboot successfully with the same combination of phone, cable and PC.
You are confused about
fastboot. It is not another
adb. There is no fastboot server (in TWRP) which accepts connections. Rather it is a step/mode in the boot process of Android. Think of it as GRUB. You need to reboot your phone while the
fastboot command is waiting for your device.