For a robot, choreography is a planning problem
A choreography can be written in one line: tinku 10, kullawada 10, breaking 10. For a robot that line is a planning problem: a sequence of movements, through space and time, that a body with 29 motors can hold. This note describes how Paqarina turns the line into movement, and the forms the same dance takes on the way: picture, points, words, sound and motor angles.
One dance, five forms
A dance in the archive is the same movement kept in five forms.

| Form | What it is | Where it comes from |
|---|---|---|
| Picture | Four videos of the same take, 24 pictures a second | Four cameras around the dancer |
| Points | 26 points of the body in space, at every instant | The four views, joined |
| Words | The name of the dance, and the score: which dance, for how many seconds | The catalogue and the choreographer |
| Sound | The music the dancer hears | The session |
| Motor angles | 29 angles, 50 times a second | The points, fitted to the robot's body |
The robot adds a sixth form that no dancer gave us: what it feels. 50 times a second it reads 154 numbers about its own body and the dance it is following.
A system is called multimodal when it works with several of these forms and can pass from one to another. In Paqarina each passage is a separate piece, and not all of them exist.
| From | To | Where it stands |
|---|---|---|
| Picture | Points | Working. Every take in the archive went through it |
| Points | Motor angles | Working. It is how each dance reaches the robot |
| Words | A remix: recorded pieces joined in a new order | A first version works. It is the planner in this note |
| Points and the name of a dance | Novel dance | Working. It wrote the suri sicuri of this series |
| Motor angles and what the robot feels | Commands to the motors | Working in simulation. On the real robot, six seconds so far |
| Sound | Movement | Not built. The dancers keep the beat, the model does not |
| Picture and words together | A reason why a phrase works for a body | Not built. It is where the work is going |
Choreography in a robot's terms
A choreographer and a robotics engineer name the same things with different words.
| The choreographer says | The engineer says | In Paqarina |
|---|---|---|
| A phrase | A motion segment | Two seconds of a recorded take: 48 instants of 26 points |
| Where the dancer is | The state | Where the hips are on the floor, which way they face, and the pose |
| What a phrase does | An action | The change of pose, the turn, the travel and the change of height in the next half second |
| The stage | The workspace | A circle around the starting point: 1.5 metres of radius for one dancer, 3 metres in the app |
| Staying on your feet | A constraint | The hips do not go below 85 per cent of their standing height |
| The join between two phrases | A transition | The next piece begins in a pose close to where the last one ended, and the two are blended over a quarter of a second |
| A partner | A second agent | A second dancer who stays between 1.0 and 1.8 metres from the first and faces her |
| How it begins and ends | The first and last states | Three still seconds, in a pose the robot can stand in. It is asked of every take from the next recording on |
| Tempo | Speed limits | Each motor has a top speed. The knee's is 20 radians a second |
From one line to 26 seconds
Paqarina generates the dances it learned in two ways: novel dance, and remixes of what was danced, joined by how the pieces fit together. This note is about the second, asked for in words.
The planner begins with a small model of how a dancing body changes. It learned from 238 recorded takes: 137 of our own, and 101 short takes of street dance from a public collection.
It learned at two speeds. At the first it sees half a second of a body and an action, and predicts the body half a second later. At the second it does the same over two seconds. It saw 51,498 examples of the first kind and 23,311 of the second. Training took four minutes on a laptop.
It does not predict pictures or points. It predicts 64 numbers that sum up where the body will be. In robotics this is called a world model: it answers the question, if the body does this, where will it be.
Then comes the search. The score says tinku 10, kullawada 10, breaking 10, in seconds. The planner cuts it into 15 steps of two seconds. It tries 512 sequences of actions, keeps the best ones, and repeats 40 times. A sequence is good when it passes through the kind of movement each dance has, stays inside the circle and stays on its feet.
The last step is what makes the result safe to give to a robot. The planner does not invent movement. For each step of two seconds it takes the recorded piece closest to what the plan asks. The plan decides the order, and every instant was danced by a person. It is a remix.
The line became 26.5 seconds: 15 pieces cut from 8 takes, all of them from our own recordings. All 15 are in the dance the score asked for.
Two limits were passed by a little. We asked the dancer to stay within 1.5 metres of the start, and the result reached 1.69. We asked the hips to stay above 85 per cent of standing height, and they went down to 83. The planner treats its limits as costs, not as walls. For a robot some of them will have to be walls.
Two bodies in one plan
A plan can hold two dancers. The second is planned around the first: where she starts, how far she stays, and that she faces her partner.
In the first planned duo the second dancer started 1.5 metres to the side. We asked for a distance between 1.0 and 1.8 metres. The plan kept it between 1.10 and 1.89.
An earlier version, from September, joined pieces by resemblance in an order we gave: breaking, tinku, suri sicuri, breaking, kullawada. We asked for more than one metre between the two, and the closest they came was 1.08 metres. Floor work was allowed for breaking and for nothing else, so the first dancer goes down between seconds 1.9 and 3.6 and nowhere else.
The stage in the app
The choreographer has her own way into the plan. In the app the floor is a stage, with a circle of 3 metres. Each dancer has a mark on the floor that says where she begins and which way she faces. The piece is divided into blocks with a length in seconds, and each dancer gets a take for each block.
In a robot's terms this is a plan for several agents, written by hand: who, where, facing what, for how long.
What we ask the robot to do
Four of the eight dances it has practised are remixes, joined from pieces of different dances by how they fit. Measured:
| What is asked | Tinku, then kullawada | Kullawada, then toprock | Tinku, then toprock | Tinku, kullawada, toprock |
|---|---|---|---|---|
| Length | 32 s | 32 s | 32 s | 31 s |
| Pieces | 2 of 15 s | 2 of 15 s | 2 of 15 s | 3 of 10 s |
| Motor angles sent | 45,965 | 45,965 | 45,965 | 45,588 |
| Farthest from where it starts | 148 cm | 64 cm | 90 cm | 122 cm |
| Turning of the hips, added up | 2,128 degrees | 2,123 degrees | 2,135 degrees | 2,449 degrees |
| Highest foot | 60 cm | 57 cm | 68 cm | 55 cm |
| Deepest knee bend | 134 degrees | 141 degrees | 152 degrees | 128 degrees |
| Highest hand | 144 cm | 139 cm | 142 cm | 161 cm |
The planned line of this note has not been practised by the robot. Fitted to the robot's body, this is what it would ask:
| What is asked | Value |
|---|---|
| Length | 26.5 s |
| Pieces | 15 of 2 s |
| Distance between the robot's body and the dancers' points | 2.7 cm |
| Right ankle against its sideways limit of 15 degrees | 24 per cent of the time |
| Left ankle against the same limit | 20 per cent of the time |
| Fastest knee | 17.9 radians a second, of a top speed of 20 |
| With a harness, in simulation | It stands for the whole sequence, 2.8 degrees from the plan at each joint |
The ankle is the finding. A dancer rolls a foot onto its edge without thinking. The robot's ankle rolls 15 degrees to each side and no more, and this plan asks for that limit a fifth of the time. The planner did not know. It knows the stage and the height of the hips, and nothing yet about the robot's joints.
Where it is going
Today the forms are joined two at a time. The step we have not taken is a model that looks at the video and reads the score at once, and keeps a picture of the body in space. It would not only recognise a movement. It would reason about how each movement changes the body, and look for a sequence that is good dance and also possible: inside the stage, on its feet, within the motors, beside a partner.
That is what a multimodal robot means in this work: one for which a score in words, a dancer on video, the music and its own body are the same dance.
The next recording is made for it. The same dancers dance tinku, suri sicuri and kullawada to one medley of three minutes. There are duets between suri and breaking. The floor carries the marks of a stage, and every take begins and ends with three still seconds.
Thanks
Paqarina is a Mozilla Fellowship project.
Thanks to Dr Pan Hui, Professor Xin Tong and Jie Yu for allowing us to test our policies on the robot.
What the record shows
| Measure | Value |
|---|---|
| Forms the same dance is kept in | 5, and a sixth that the robot adds |
| Recorded takes the planner's model learned from | 238: 137 of our own and 101 from a public collection |
| Examples of half a second | 51,498 |
| Examples of two seconds | 23,311 |
| Time to train the planner's model | 4 minutes, on a laptop |
| Sequences tried in each round of the search | 512, for 40 rounds |
| The score | tinku 10, kullawada 10, breaking 10 |
| The result | 26.5 s, 15 pieces from 8 takes |
| Pieces in the dance the score asked for | 15 of 15 |
| Stage radius, asked and reached | 1.5 m and 1.69 m |
| Hip height, asked and reached | 85 and 83 per cent of standing |
| Distance in the planned duo, asked and kept | 1.0 to 1.8 m, and 1.10 to 1.89 m |
| Robot's right ankle at its sideways limit in the planned line | 24 per cent of the time |
| Remixes among the dances the robot has practised | 4 of 8 |
| Planned lines the robot has practised | 0 |
What remains open
Whether the planner can read the robot's limits the way it reads the stage: the 15 degrees of the ankle, the speed of each motor, balance. Whether sound can lead, so that a plan lands on the beat of the medley. Whether a duet between a robot and a person can be planned with the distance as a wall and not as a cost. Whether one model can hold picture, words and body together and say why a phrase works. The first planned line asked the robot's ankle for its limit a fifth of the time. The next plans read the robot's joints before they choose.