I had internalised that it was Windows Defender hooking every file operation and checking it against a blacklist? I've had it forced off for years.

Windows Defender is a file system filter which you cannot disable. You may have others (but they're fortunately rare, now).

All that said, you cannot disable the architecture, i.e. bypass the file system filter code.

You can with Dev Drives now apparently, which don't use NTFS and disable ALL the filter drivers (including the Defender one)

I stopped using Windows just as these were added so now I'm curious if there's any actual performance benefit to using the.

No, they don't disable the Windows Defender filter, they put it in async mode.

This guy gets it. Yes bingo. It's the VFS' filters/ACLs support afaik.