It's not different than the rest of the network stack, most of it consists in "inspecting packets" in one way or another, and security bugs can be (and some times are) introduced.

But nftables is actually a big win from a security perspective, because it simplifies the current code (lots of duplicated code goes away) and moves other parts to userspace.

Old netfilter system: 70.000 LoC in kernel + 50.000 in userspace

nftables: 7.000 LoC in kernel + 50.000 in userspace

(source: http://www.slideshare.net/ennael/2013-kernel-recipesnftables)

Also note that it's not really a "virtual machine" comparable with java, this is how the developers actually describe it

    In a nutshell, nftables provides a pseudo-state machine with 4 general
    purpose registers of 128 bits and 1 specific purpose register to store
    verdicts. This pseudo-machine comes with an extensible instruction set,
    a.k.a. "expressions" in the nftables jargon. The expressions included
    in this patch provide the basic functionality, they are:
    
    * bitwise: to perform bitwise operations.
    * byteorder: to change from host/network endianess.
    * cmp: to compare data with the content of the registers.
    * counter: to enable counters on rules.
    * ct: to store conntrack keys into register.
    * exthdr: to match IPv6 extension headers.
    * immediate: to load data into registers.
    * limit: to limit matching based on packet rate.
    * log: to log packets.
    * meta: to match metainformation that usually comes with the skbuff.
    * nat: to perform Network Address Translation.
    * payload: to fetch data from the packet payload and store it into
      registers.
    * reject (IPv4 only): to explicitly close connection, eg. TCP RST.
    
    Using this instruction-set, the userspace utility 'nft' can transform
    the rules expressed in human-readable text representation (using a
    new syntax, inspired by tcpdump) to nftables bytecode.

I don't even think this is the first vm in the networking part of the kernel.

No, BPF is very old.