kzakka

An open-source library for GPU-accelerated robot learning and sim-to-real transfer.

We have the sustain pedal implemented in the standalone MuJoCo simulation, e.g. see https://www.youtube.com/watch?v=VBFn_Gg0yD8. I just couldn't figure out how to do it with Tone.js :(

Indeed, browsers are pretty aggressive about not autoplaying sound, you need to interact with the screen (mouse click or finger tap + make sure your sound is up / ringer not on silent).

Yup, we're not trying to mimic human movements exactly but rather optimizing for the reward given the robot hardware. Fun fact, we do things like add an energy penalty to try and reduce jitteriness / un-human like movements and it does help enormously.

Thanks! We did indeed explore these additional degrees of freedom, you can find vestigial code for this here: https://github.com/google-research/robopianist/blob/main/rob...

We ended up picking a minimal subset of forearm DoFs that wouldn't impact training speed too much.

The fingering we use is taken directly from: https://beam.kisarazu.ac.jp/~saito/research/PianoFingeringDa...

From their whitepaper: "Fingerings in the dataset were provided by experienced pianists who graduated from a music college or who had played the piano for more than twenty years. The pianists were asked to choose pieces that they could play and provided the fingering that they had actually used for the performance."

We're using the MuJoCo simulator under the hood: https://mujoco.org/

The keys are actually implemented using a spring mechanism, but springs in MuJoCo are currently linear, which isn't the case in the real world.

Hi, one of the authors here :)

The demo you are watching is an agent trained from scratch with reinforcement learning. It has roughly 6 days of experience (10M steps at 20 Hz). The Javascript demo is replaying the policy open loop which is why it's not super robust to disturbances.

Re:fingering: we actually use fingering information to create a dense reward for the agent (otherwise it makes exploration super hard). It would be an exciting future direction to have the agent discover and optimize for fingering that best suits its kinematics :) And beyond that, having RL inform pianists about the difficulty of a piece or even more optimal fingering would be amazing.

We trained a bunch of these policies on roughly 150 songs (baroque, romantic, classical) and we did some analysis in the paper if you're interested: https://kzakka.com/robopianist/robopianist.pdf

Hi author here! The app should work on mobile/desktop and was tested on both Safari and Chrome. I've heard it's buggy for some people (unclear if it's an older hardware problem) but you can try this embedded demo which works better: https://kzakka.com/robopianist/#demo

Hey guys, author here. Thought I'd share this post on a classic algorithm called Neighbourhood Components Analysis. I almost never see it mentioned so this post is an attempt to rectify it. It's accompanied by a PyTorch implementation with a few modifications. There's also a neat discussion about its relationship to the contrastive loss function. Cheers