Research

For a robot, choreography is a planning problem

A choreography can be written in one line: tinku 10, kullawada 10, breaking 10. For a robot that line is a planning problem: a sequence of movements, through space and time, that a body with 29 motors can hold. This note describes how Paqarina turns the line into movement, and the forms the same dance takes on the way: picture, points, words, sound and motor angles.

One dance, five forms

A dance in the archive is the same movement kept in five forms.

One kullawada passing from form to form: Lucía on video, her 26 points, the avatar, motor angles on the robot's model, and the robot in the lab. Images: Paqarina.
One kullawada passing from form to form: Lucía on video, her 26 points, the avatar, motor angles on the robot's model, and the robot in the lab. Images: Paqarina.
FormWhat it isWhere it comes from
PictureFour videos of the same take, 24 pictures a secondFour cameras around the dancer
Points26 points of the body in space, at every instantThe four views, joined
WordsThe name of the dance, and the score: which dance, for how many secondsThe catalogue and the choreographer
SoundThe music the dancer hearsThe session
Motor angles29 angles, 50 times a secondThe points, fitted to the robot's body

The robot adds a sixth form that no dancer gave us: what it feels. 50 times a second it reads 154 numbers about its own body and the dance it is following.

A system is called multimodal when it works with several of these forms and can pass from one to another. In Paqarina each passage is a separate piece, and not all of them exist.

FromToWhere it stands
PicturePointsWorking. Every take in the archive went through it
PointsMotor anglesWorking. It is how each dance reaches the robot
WordsA remix: recorded pieces joined in a new orderA first version works. It is the planner in this note
Points and the name of a danceNovel danceWorking. It wrote the suri sicuri of this series
Motor angles and what the robot feelsCommands to the motorsWorking in simulation. On the real robot, six seconds so far
SoundMovementNot built. The dancers keep the beat, the model does not
Picture and words togetherA reason why a phrase works for a bodyNot built. It is where the work is going

Choreography in a robot's terms

A choreographer and a robotics engineer name the same things with different words.

The choreographer saysThe engineer saysIn Paqarina
A phraseA motion segmentTwo seconds of a recorded take: 48 instants of 26 points
Where the dancer isThe stateWhere the hips are on the floor, which way they face, and the pose
What a phrase doesAn actionThe change of pose, the turn, the travel and the change of height in the next half second
The stageThe workspaceA circle around the starting point: 1.5 metres of radius for one dancer, 3 metres in the app
Staying on your feetA constraintThe hips do not go below 85 per cent of their standing height
The join between two phrasesA transitionThe next piece begins in a pose close to where the last one ended, and the two are blended over a quarter of a second
A partnerA second agentA second dancer who stays between 1.0 and 1.8 metres from the first and faces her
How it begins and endsThe first and last statesThree still seconds, in a pose the robot can stand in. It is asked of every take from the next recording on
TempoSpeed limitsEach motor has a top speed. The knee's is 20 radians a second

From one line to 26 seconds

Paqarina generates the dances it learned in two ways: novel dance, and remixes of what was danced, joined by how the pieces fit together. This note is about the second, asked for in words.

The planner begins with a small model of how a dancing body changes. It learned from 238 recorded takes: 137 of our own, and 101 short takes of street dance from a public collection.

It learned at two speeds. At the first it sees half a second of a body and an action, and predicts the body half a second later. At the second it does the same over two seconds. It saw 51,498 examples of the first kind and 23,311 of the second. Training took four minutes on a laptop.

It does not predict pictures or points. It predicts 64 numbers that sum up where the body will be. In robotics this is called a world model: it answers the question, if the body does this, where will it be.

Then comes the search. The score says tinku 10, kullawada 10, breaking 10, in seconds. The planner cuts it into 15 steps of two seconds. It tries 512 sequences of actions, keeps the best ones, and repeats 40 times. A sequence is good when it passes through the kind of movement each dance has, stays inside the circle and stays on its feet.

The last step is what makes the result safe to give to a robot. The planner does not invent movement. For each step of two seconds it takes the recorded piece closest to what the plan asks. The plan decides the order, and every instant was danced by a person. It is a remix.

The line tinku 10, kullawada 10, breaking 10 as movement, on an avatar and on the robot's model. Here the robot's model only copies the angles: it is not balancing. Render: Paqarina.

The line became 26.5 seconds: 15 pieces cut from 8 takes, all of them from our own recordings. All 15 are in the dance the score asked for.

Two limits were passed by a little. We asked the dancer to stay within 1.5 metres of the start, and the result reached 1.69. We asked the hips to stay above 85 per cent of standing height, and they went down to 83. The planner treats its limits as costs, not as walls. For a robot some of them will have to be walls.

Two bodies in one plan

A plan can hold two dancers. The second is planned around the first: where she starts, how far she stays, and that she faces her partner.

In the first planned duo the second dancer started 1.5 metres to the side. We asked for a distance between 1.0 and 1.8 metres. The plan kept it between 1.10 and 1.89.

An earlier version, from September, joined pieces by resemblance in an order we gave: breaking, tinku, suri sicuri, breaking, kullawada. We asked for more than one metre between the two, and the closest they came was 1.08 metres. Floor work was allowed for breaking and for nothing else, so the first dancer goes down between seconds 1.9 and 3.6 and nowhere else.

The September duo, 17.8 seconds, seen four ways: two avatars in Blender, two simulated bodies moved by motors, the points, and the plan itself. In the plan each dancer has a line coloured by dance, and the curve below is the distance between the two against the minimum asked. Render: Paqarina.

The stage in the app

The choreographer has her own way into the plan. In the app the floor is a stage, with a circle of 3 metres. Each dancer has a mark on the floor that says where she begins and which way she faces. The piece is divided into blocks with a length in seconds, and each dancer gets a take for each block.

The stage in the app: four marks on the floor, two blocks of six seconds, and two dancers placed. Screen recording: Paqarina.

In a robot's terms this is a plan for several agents, written by hand: who, where, facing what, for how long.

What we ask the robot to do

Four of the eight dances it has practised are remixes, joined from pieces of different dances by how they fit. Measured:

What is askedTinku, then kullawadaKullawada, then toprockTinku, then toprockTinku, kullawada, toprock
Length32 s32 s32 s31 s
Pieces2 of 15 s2 of 15 s2 of 15 s3 of 10 s
Motor angles sent45,96545,96545,96545,588
Farthest from where it starts148 cm64 cm90 cm122 cm
Turning of the hips, added up2,128 degrees2,123 degrees2,135 degrees2,449 degrees
Highest foot60 cm57 cm68 cm55 cm
Deepest knee bend134 degrees141 degrees152 degrees128 degrees
Highest hand144 cm139 cm142 cm161 cm

The planned line of this note has not been practised by the robot. Fitted to the robot's body, this is what it would ask:

What is askedValue
Length26.5 s
Pieces15 of 2 s
Distance between the robot's body and the dancers' points2.7 cm
Right ankle against its sideways limit of 15 degrees24 per cent of the time
Left ankle against the same limit20 per cent of the time
Fastest knee17.9 radians a second, of a top speed of 20
With a harness, in simulationIt stands for the whole sequence, 2.8 degrees from the plan at each joint

The ankle is the finding. A dancer rolls a foot onto its edge without thinking. The robot's ankle rolls 15 degrees to each side and no more, and this plan asks for that limit a fifth of the time. The planner did not know. It knows the stage and the height of the hips, and nothing yet about the robot's joints.

Where it is going

Today the forms are joined two at a time. The step we have not taken is a model that looks at the video and reads the score at once, and keeps a picture of the body in space. It would not only recognise a movement. It would reason about how each movement changes the body, and look for a sequence that is good dance and also possible: inside the stage, on its feet, within the motors, beside a partner.

That is what a multimodal robot means in this work: one for which a score in words, a dancer on video, the music and its own body are the same dance.

The next recording is made for it. The same dancers dance tinku, suri sicuri and kullawada to one medley of three minutes. There are duets between suri and breaking. The floor carries the marks of a stage, and every take begins and ends with three still seconds.

Thanks

Paqarina is a Mozilla Fellowship project.

Thanks to Dr Pan Hui, Professor Xin Tong and Jie Yu for allowing us to test our policies on the robot.

What the record shows

MeasureValue
Forms the same dance is kept in5, and a sixth that the robot adds
Recorded takes the planner's model learned from238: 137 of our own and 101 from a public collection
Examples of half a second51,498
Examples of two seconds23,311
Time to train the planner's model4 minutes, on a laptop
Sequences tried in each round of the search512, for 40 rounds
The scoretinku 10, kullawada 10, breaking 10
The result26.5 s, 15 pieces from 8 takes
Pieces in the dance the score asked for15 of 15
Stage radius, asked and reached1.5 m and 1.69 m
Hip height, asked and reached85 and 83 per cent of standing
Distance in the planned duo, asked and kept1.0 to 1.8 m, and 1.10 to 1.89 m
Robot's right ankle at its sideways limit in the planned line24 per cent of the time
Remixes among the dances the robot has practised4 of 8
Planned lines the robot has practised0

What remains open

Whether the planner can read the robot's limits the way it reads the stage: the 15 degrees of the ankle, the speed of each motor, balance. Whether sound can lead, so that a plan lands on the beat of the medley. Whether a duet between a robot and a person can be planned with the distance as a wall and not as a cost. Whether one model can hold picture, words and body together and say why a phrase works. The first planned line asked the robot's ankle for its limit a fifth of the time. The next plans read the robot's joints before they choose.