While there isn't really a term (that I'm aware of) specifically for the this/self convention, both are idiomatic examples of dynamic scoping (compared to lexical scoping); that is, scope is determined at runtime by the most recent stack frame to define a given variable (ie, foo defines "this", then calls "bar", bar sees "this" as it was defined by foo), rather than at lex time.
I don't know if there is, but there should be.
The defining aspect seems to be that there aren't pure ADTs, just collections based an associative arrays ("dicts" / "tables"), and methods are just table fields that contain a function. Also, Python has classes where Lua really does just have pure tables with hooks ("metatables", which make it easy to implement prototypes, classes, and many other constructs).
I tend to think of them as "bag-of-things" languages, since they also all share the fact that objects are just tables/dictionaries, and the only thing that distinguishes an attribute reference from a method call is the "()" at the end. Certainly there's a more accepted nomenclature?