You can use SECCOMP_RET_TRACE to kick complicated cases back to the ptracer but handle the easy cases without the slowdown. So you can write a seccomp policy that does something like this pseudocode:

    if syscall == SYS_open:
        if flags == O_RDONLY:
            return (SECCOMP_RET_TRACE, 0)
        else:
            return (SECCOMP_RET_ERRNO, EPERM)
    else if syscall in (SYS_read, SYS_write, ...):
        return (SECCOMP_RET_ALLOW, 0)
    else:
        return (SECCOMP_RET_ERRNO, ENOSYS)

and it would be much much faster than tracing every system call, since most programs call open() rarely and read() and write() very often.

That said, the ptracer's job here is kind of hard, because the kernel still gets an untrusted userspace pointer, and another thread, another process, etc. can modify that memory in between the ptracer okaying it and the kernel getting to it. (See "Argument races" in Tal Garfinkel's 2003 "Traps and Pitfalls" paper.) So you either want the filtering to happen in the kernel after it's been copied to kernelspace (which is Landlock's approach), or do the open from a trusted process and send the fd over (which is I think gVisor's approach).