Modeling the social interaction of Shimi (an interactive audio dock) on a computer, to simulate Shimi’s head motions in response to audio, while imitating a user’s gestures and head motions.
The goal is to take a trajectory based on the user’s response to a song (kinect), and make Shimi traverse this trajectory in-sync with the playing music (phase locked loops.)
Head fake (2):
I’ll be able to work with macromolecular modeling, C, and Java – in one project!
Here’s the summary:
- Phase locked loops (for extracting embedded clocks from an audio signal, perhaps only the zeroth order term)
- C programming (talking to a kinect while running a Java applet)
- Why Java? Because I have been hoping to use it somewhere, so this might be a good start!
- Using a math model for controlling a simulation* (vector graphics…game engines…)
*depending on how interesting it seems, instead of animating some plots in Java, I could actually open up Unity and run some shaders…
Some initial thoughts:
Think of Shimi’s head movements as motions of a particle in a three dimensional space.
The simulated-Shimi-head dynamics would be controlled by this particle’s dynamics.
So, we need directional and timing information for this particle’s motion: for the tranjectory itself, and for timing the head’s movement (where physical constraints of Shimi’s hardware are built into the space itself.)
[ Michael Levitt ]: energy functions | Monte Carlo and dubstep
Shimi is an interactive dock, but for starters, we can just imitate a user’s response (who is listening to the same music as Shimi). Now, mimicry would work well for a presentation, but its also a good starting point, where we can extract a basic trajectory, for a part of a song, by observing a user (through kinect), and then add some variations on top of that!
[ Human computer interaction ]
I just learned about embedded clocks (8b/10b encoding!) Why not learn more about PLLs, and see if their equations can lock on to musical clocks too!
[ Phase locked loops ]
The best thing would be to just interface directly with a Shimi, but for now, I can simulate one, and it shouldn’t be hard, as I only need one head, and three degrees of freedom!
[ Applets for a demo ]
I have been thinking (day-dreaming) about the Shimi project for some days now. This morning, as I was talking myself through all the ideas (modules) I would like to present at our first group meeting this spring (they began this fall!), I thought it would just make more sense to show simulations of what I want Shimi to do, instead of just talking about that!