We could discover tomorrow that there's a "sad ganglia" in the human brain that can be set on or off with an electromagnetic field. Does that mean that humans are ultimately biological machines without rights?
I know what you're asking and the point you're trying to make, and I totally agree that it's an interesting problem. Perhaps one method is that non-humans will get rights when they evolve to the point to create their own rights and the means with which to stop other things from trampling on the rights they have assigned themselves.
My point was that understanding the nature of an emotion in a trivial way should be orthogonal to how we think about what rights that being should have. At some level, we're all machines. Just because one's software runs in silicon vs gray matter; just because one's hardware was deliberately built and is understandable in computing terms doesn't mean that we really understand what it is to be sentient with respect to rights to be free and exist.
It absolutely would require that. If you believe in determinism then part of the definition of intelligence, or an intelligent system, is that it exhibits behaviors that are just too complex for a human consciousness to intuitively follow the causal chain. Incidentally, biological intelligence is built on top of some other systems which, given our current levels of understanding, also meet that criterion themselves, so we're pretty clueless about how it works.
Part of this is a definition problem - the Turing Test was defined vaguely enough that everyone has different conceptions. When a person talks about a "good" or "strong" Turing Test, they are envisioning one that would pass all of their personal standards and all the ways they could think of to trick it. And when they talk about it with someone else, who likely envisions a somewhat different version of the test, there seems a tendency for that person to assume that their version would not be passed, so they start to talk past each other.
In other words, if an AI were to consistently surprise you with thoughtfulness, compassion, creativity, or whatever other constituents of "true intelligence" you assume the duck imposter would lack, would you then confer it those rights?
I think you would, because that's the point: it has thoroughly convinced you that it is "thinking", and you feel like it truly "understands" you - frankly a higher bar than many rights-granted humans would pass.
What separates animals that deserve human-level rights from those that don't? Would you argue that all animals do? If not, I would say that that distinction is no less arbitrary than the one you're drawing at the end.
That requirement by itself is probably the easiest of th requirements.
For example, if I say mean things to that algorithm, even one that can fool me into thinking it's another human for decades, is that morally wrong? Even if it's just setting some variable in memory to sad=true? If so, is it morally wrong for me to create a program consisting only of a singular red button, that when pushed, sets the sad flag?
There was another "game" created a couple years ago that featured a fictional human (a little girl, if memory serves) on life support. They programmed it so that unless somebody in the world clicked a button in their browser, the girl would die within 10 seconds and the game would basically delete itself, ending forever. (The response was so strong that people flooded the server, and I believe their hosting provider blocked access to their site, thereby killing the program untimely.) If the little girl was removed, and the goal was to just keep the program running as long as possible on life support, would that be morally wrong?
I ask these questions because I don't think a even a complex script which is doing nothing more than attempting to fool me should be considered worth assigning rights to, even if it's really good at it. To be honest, I'm not entirely sure how to define consciousness in this regard, but I suspect it would require surprising its creators to the point that they cannot fathom how it behaved the way it did. Or maybe I'm confusing consciousness with free will. Either way, if the only thing separating a program that deserves human-level rights from those that don't is some combination of power (enough to scan all the valid responses and deliver the best one) and a wide array of responses linked to conversations, then I'd argue that all programs deserve the same basic human rights.