This thanksgiving: let’s connect some dots


Head fake:

Modeling the social interaction of Shimi (an interactive audio dock) on a computer, to simulate Shimi’s head motions in response to audio, while imitating a user’s gestures and head motions.

The goal is to take a trajectory based on the user’s response to a song (kinect), and make Shimi traverse this trajectory in-sync with the playing music (phase locked loops.)

Head fake (2):

I’ll be able to work with macromolecular modeling, C, and Java – in one project!

Here’s the summary:

  • Phase locked loops (for extracting embedded clocks from an audio signal, perhaps only the zeroth order term)
  • C programming (talking to a kinect while running a Java applet)
  • Why Java? Because I have been hoping to use it somewhere, so this might be a good start!
  • Using a math model for controlling a simulation* (vector graphics…game engines…)

*depending on how interesting it seems, instead of animating some plots in Java, I could actually open up Unity and run some shaders…

Some initial thoughts:

Dynamic Modeling

Think of Shimi’s head movements as motions of a particle in a three dimensional space.
The simulated-Shimi-head dynamics would be controlled by this particle’s dynamics.

So, we need directional and timing information for this particle’s motion: for the tranjectory itself, and for timing the head’s movement (where physical constraints of Shimi’s hardware are built into the space itself.)

[ Michael Levitt ]: energy functions | Monte Carlo and dubstep


Shimi is an interactive dock, but for starters, we can just imitate a user’s response (who is listening to the same music as Shimi). Now, mimicry would work well for a presentation, but its also a good starting point, where we can extract a basic trajectory, for a part of a song, by observing a user (through kinect), and then add some variations on top of that!

[ Human computer interaction ]

Timing Information

I just learned about embedded clocks (8b/10b encoding!) Why not learn more about PLLs, and see if their equations can lock on to musical clocks too!

[ Phase locked loops ]


The best thing would be to just interface directly with a Shimi, but for now, I can simulate one, and it shouldn’t be hard, as I only need one head, and three degrees of freedom!

[ Applets for a demo ]


I have been thinking (day-dreaming) about the Shimi project for some days now. This morning, as I was talking myself through all the ideas (modules) I would like to present at our first group meeting this spring (they began this fall!), I thought it would just make more sense to show simulations of what I want Shimi to do, instead of just talking about that!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s