This video demonstrates full coding proficiency with my Talon project, which still needs a lot of work (e.g. no auto-spacing yet) - https://youtu.be/ddFI63dgpaI?t=30 - this demo took me around 9min vs a 90wpm typist doing it in a little over 6min. I've also benchmarked inputting over 280 commands per minute with no false recognitions.
This is my project, with which I aim to change minds about voice input (even making it viable for real-time gaming): https://talonvoice.com/
The Perl video is embarrassing, but that voice engine is only designed for dictating english text. voicecode.io has some major architectural limitations (no dynamic grammars so it requires a Dragon restart to change commands or word lists, adds commands by adding them to the english vocabulary instead of a strict command tree so recognition seriously suffers), aenea, caster, dragonfly are built on a 90's Python project called natlink and have their own issues. "Continuous command recognition" is pretty much a must, which Tavis Rudd's video lacks (notice the pausing between words).
I had no idea eye tracking could work this well.
To my knowledge I am the only available high performance eye mousing project to not require a separate head tracker (I only require a $150 Tobii 4C eye tracker, and I do not use their integrated webcam head tracking system as it is not responsive enough).
I'm additionally planning to remove the ~$50-150 Dragon requirement for Mac users in the mid future.
I'm very committed to keeping the baseline software not just free, but a net negative cost over using other free solutions. As far as making it commercial, my current plan is to eventually charge a small subscription fee for extremely well integrated video game control systems, and a moderate subscription for professional input schemes like CADD, A/V products, and entire language models built around natural language programming that would ideally enable people who become injured to continue to work without much of any downtime.
These are some of the competition:
Voice:
- https://github.com/dictation-toolbox/aenea
- https://github.com/synkarius/caster
Eye/Head Tracking:
- http://precisiongazemouse.com/
- https://github.com/trishume/PolyMouse
- https://github.com/trishume/FusionMouse
(huge shout out to trishume, who was very helpful and a big inspiration for my tracking system)
One of my upcoming goals is to get a hands-free world-record game speedrun into the Awesome Games Done Quick stream.
I have very little practice at this point, especially considering I'm still changing my grammars regularly, so the editing video there is more a technical demo of the software's performance in spite of my own awkwardness while learning. That was my first and only take. I'm basically learning to type again, with cache misses as I try to remember commands and keep track of my position (like trying to remember where keys are, or the next note in a song as you pluck it out).
It's also possible the most accurate form of voice input does not sound like natural language. Finally, it's a different mental exercise to copy text that is already written versus writing and editing my own code, so I'm triply out of my element when recording these early videos.
Compare to Tavis Rudd's video from 2013, where he pauses between every command: https://youtu.be/8SkdfdXWYaI?t=1050
I've recorded my own version of his demo today for comparison: https://youtu.be/wt4PR5j7vBE
How would that work? Would your IDE play back the voice or display animations of the gestures to you.
It just feels as if text is the simplest solution, but I'm open to suggestions.
I would suppose that voice/gesture written code would be stored as text, when reading you could edit via keyboard or your voice/gesture interface. This comes back to languages written for voice - probably if this became feasible a lot of these languages would be compilers from voice to another traditional text target.
But like many people here I have a hard time actually believing this would ever be useful.
For people who spend most of their time clicking with a mouse or pointing at pictures on a phone screen, I could see voice working with things like IFTTT to create simple "programs" or actions. That's essentially what happens when I say "hey Google, turn on living room lights at 6pm".
Someone needed to program the framework for that to happen, but someone also needed to develop the Arduino IDE that I use when I engage in the very limited hobbyist programming that I do. I get to use a simpler interface because someone more specialized and experienced created it with lower level tools.
In a sense, that "if time=18:00 then set livingroom_lights=on" command is a very simple program entered via voice command.
It's worth a watch. Hilarious, despite the bad sound quality.
https://youtu.be/YRyYIIFKsdU?t=9m9s
Of course, there's plenty of room for improvement. Why not have the assistant understand keywords and perform high-level tasks?
"import"
"create function named main"
"return"
"Add two int arguments"
Oh please god no full stop imagine how hard it would be to even write a simple hello world program example ellipsis linebreak linebreak three backticks no not that delete word delete word backtick backtick backtick c linebreak void main open bracket int argc comma char asterisk argv close bracket space open curly bracket line break tab print-EFF open bracket double-quote percent d backslash n double-quote comma argc close bracket semi-colon line break backtick backtick backtick
Chance of success? Zero. Time taken? Three minutes.