Biol Cybern (2006) 94: 393–407 DOI 10.1007/s00422-006-0053-0

O R I G I NA L PA P E R

Gavin Simmons · Yiannis Demiris

Object grasping using the minimum variance model

Received: 5 September 2005 / Accepted: 11 January 2006 / Published online: 15 February 2006 © Springer-Verlag 2006

Abstract Reaching-to-grasp has generally been classified as the coordination of two separate visuomotor processes: transporting the hand to the target object and performing the grip. An alternative view has recently been formed that grasping can be explained as pointing movements performed by the digits of the hand to target positions on the object. We have previously implemented the minimum variance model of human movement as an optimal control scheme suitable for control of a robot arm reaching to a target. Here, we extend that scheme to perform grasping movements with a hand and arm model. Since the minimum variance model requires that signal-dependent noise be present on the motor commands to the actuators of the movement, our approach is to plan the reach and the grasp separately, in line with the classical view, but using the same computational model for pointing, in line with the alternative view. We show that our model successfully captures some of the key characteristics of human grasping movements, including the observations that maximum grip size increases with object size (with a slope of approximately 0.8) and that this maximum grip occurs at 60–80% of the movement time. We then use our model to analyse contributions to the digit end-point variance from the two components of the grasp (the transport and the grip). We also briefly discuss further areas of investigation that are prompted by our model.

1 Introduction Reaching to grasp an object is a complex motor task involving the movement of many joints and the coordination of several end-effectors. In much the same way as multi-joint reaching movements, studies of the kinematics of grasping (Jeannerod 1981, 1984; Kudoh et al. 1997; Kamper et al. 2003; Cuijpers et al. 2004) have revealed characteristic patterns of behaviour. G. Simmons · Y. Demiris (B) Department of Electrical and Electronic Engineering, Imperial College, London, UK E-mail: [email protected]

Among the most well established of these is the observation that maximum grip size (between the thumb and the finger) increases with the size of the object (Jeannerod 1981), with a slope of approximately 0.8. A further observation is that the maximum grip aperture occurs at around 60–80% of the movement time (Jeannerod 1984). The general description of grasping behaviour is based upon the separation of the grasp into two visuomotor channels: one for the transport component (moving the hand to the object) and one for the grip (moving the fingers to grip the object) (Jeannerod 1981). This separation implies that the two components are planned independently, but executed together to form a single coordinated movement. More recently, (Smeets and Brenner 1999) suggested the alternative view that grasping movements are carried out as smooth pointing movements of the thumb and finger to target positions on the object. They used the minimum jerk model of arm movement to successfully predict finger- and thumb-tip trajectories towards an object, however deliberately avoiding any consideration of the mechanics of limbs and joints (Smeets and Brenner 1999). In previous work (Simmons and Demiris 2004, 2005) we have implemented the minimum variance model of human reaching (Harris and Wolpert 1998) using an optimal control scheme suitable for controlling a robot arm. Following the general idea that grasping can be described as pointing movements performed by the finger and thumb, we show here that our implementation is suitable for examining the behaviour of grasping as well. Unlike other models of human reaching (Flash and Hogan 1985; Uno et al. 1989), the minimum variance model specifically allows for disturbances to the hand trajectory caused by noise on the motor command signals. The goal of the model is to minimise the variance in the hand position caused by this noise during some post-movement period (Harris and Wolpert 1998). This model provides an explanation for the speed-accuracy trade-off observed in human reaching movements (Fitts 1954) and has been shown to accurately predict both smooth arm movements and saccadic eye movements (Harris and Wolpert 1998; Miyamoto et al. 2003, 2004).

394

Grasping is an interesting task on which to apply the minimum variance model, as there are two sources of variance affecting the finger-tip position: that produced by the movement of the wrist and that produced by the movement of the fingers (Kudoh et al. 1997). Our aim with this paper is to produce an investigation into grasping with the minimum variance model, as a precursor to further computational and physiological studies. To correctly model the minimum variance trajectory, signal-dependent noise must be applied to the control signals of the arm (and digit) actuators. It is not possible therefore to eliminate the mechanics of the arm from the model. Instead, our approach builds on both the “classical” view of grasping and on the view of Smeets and Brenner (1999). We view reaching-to-grasp as two separate processes (the transport and the grip) that are nonetheless planned using the same computational model for pointing, under the same temporal constraints. In the following section we briefly describe our implementation of the minimum variance model. We then describe our model of the arm and hand and show how the system can be used to perform grasping. We perform an analysis of the effect of several different parameters on the end-point accuracy of the digits’ trajectory, identifying in each case the trend in that parameter that leads to the lowest level of inaccuracy. Using the identified values of the parameters, we examine whether the resulting trajectories capture the characteristic features of grasping. Our model allows us to study the nature of the variance over the whole course of the movement, for both the transport and grip components. We show how these are affected by changes in object size, movement time and transport distance. Finally, we will indicate some further areas of study that may be prompted by our work.

G. Simmons and Y. Demiris

This model has been shown to produce the roughly straight, smooth movements (Harris and Wolpert 1998; Miyamoto et al. 2003) observed in humans (Morasso 1981). Smooth movements are a clear consequence of the optimisation criteria, as non-smooth movements require larger motor commands which have higher amplitude noise, increasing the cumulative variance. The speed-accuracy trade-off observed in human movement also emerges as a property of the minimum variance optimisation: fast movements require large control signals while slower movements require small control signals. Example trajectories and demonstration of the speed accuracy trade-off for a simple reaching movement performed using our implementation are shown in Fig. 1.

2.1 Via-points The optimisation scheme can be extended to more complex trajectories through the inclusion of one or more via-points. Figure 2 shows trajectories, and velocity profiles for a reaching movement consisting of several via-points. The end-point standard deviation for repeated movements with different movement times is also shown, to demonstrate that the speedaccuracy trade-off still applies for complex trajectories. Each via-point has two parts: its spatial coordinates V (x, y) and its temporal location V (t) in the course of the movement. This is no different from the start and target points, except in those cases wherein the temporal information is unimportant (t = 0 for the start point and t = T for the target point, where T is the duration of the movement). Viapoints must be dealt with differently however, as their temporal position within the course of the movement affects the trajectory as much as their spatial coordinates.

3 Implementation 2 Minimum variance model for human-like movement In this section, we summarise the minimum variance model. Our implementation is briefly described in the Appendix, but for a full description of this system and comparisons to other neuroscientific models of human movement, see Simmons and Demiris (2004, 2005). Many theories that attempt to explain human movements are based on the assumption that some aspect of the movement is being optimised (Cruse et al. 1990; Loeb et al. 1990; Flash and Sejnowski 2001; Luo et al. 2004). Among the more successful theories is the minimum variance model (Harris and Wolpert 1998). This model is based on the fact that motor commands sent to the muscles are subject to neural noise, which is signal-dependent (i.e., the amplitude of the noise is proportional to the amplitude of the signal). This noise naturally causes variations, which build up over the course of the movement leading to inaccuracy in the final position. As such, Harris and Wolpert (1998) proposed that the motor system plans movements so as to minimise the variance caused by neuronal noise during some post-movement period.

Following our previous work (Simmons and Demiris 2004, 2005) and other motor control studies (Flash and Hogan 1985; Uno et al. 1989), we model the arm as a two-link planar device with two rotational degrees-of-freedom. We then model the digits as smaller two-link devices attached at the end of the arm, as shown in Fig. 3. We set the links of the arm to be 30 cm in length and the links of the digits to be 10 cm in length, values chosen as approximations to the proportions of a human arm. Overall the arm model has six joints which are controlled directly by the optimisation scheme. We focus on the kinematics of the movements, without explicitly accounting for the dynamics of the arm or modelling arm muscles. More complex models of the hand have been examined for grasping (for example, Meulenbroek et al. 2001), but our model is sufficient to demonstrate the required principles, and strikes a balance between these approaches and the simple model of Smeets and Brenner (1999). From the given position of the object, the wrist target position is specified as being 6 cm vertically below the centre

Object grasping using the minimum variance model

0.55

a

0.45

0.5

0.4

0.45

0.35

0.4 Y (m)

Y (m)

0.5

395

0.3

a

0.35 0.3

0.25

0.25 0.2

0.2 0.15

0.2

0.3 X (m)

0.35

0.4

0.15 –0.45 –0.4 –0.35 –0.3 –0.25 –0.2 –0.15 X (m)

b

0.6

0.7

0.5

0.6

0.4

0.5 Speed (ms–1)

Speed (ms–1)

0.7

0.25

0.3 0.2 0.1 0

b

0.4 0.3 0.2

0

0.2

0.4

0.6 0.8 Relative time

1

0.1

1.2

0 0

c

0.01

0.005

0 500

0.2

0.4

0.6 0.8 Relative time

1

1.2

c

0.015

750 1000 1250 1500 1750 Instructed movement time (ms)

2000

Fig. 1 A movement between (0.2,0.2) and (0.4,0.5) was carried out several times: (a) Trajectories produced by the model – signal-dependent noise on the control signal results in slightly different trajectories for repeated movements, but each trajectory is still roughly straight and smooth; (b) The velocity profiles for the movements. The velocity follows the characteristic bell-shape of reaching movements, and exhibits the asymmetry predicted by the minimum variance model (Harris and Wolpert 1998); (c) The model exhibits a speed-accuracy trade-off, a well-documented feature of human movement

of mass of the object. In this paper, we kept the distance of the wrist target position from the centre of the object constant. Examining how this variable affects the digit trajectories will

End–point standard deviation (m)

End–point standard deviation (m)

0.02

0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 500

1000 1500 Movement time (ms)

2000

Fig. 2 A movement between (−0.2, 0.5) and (−0.4, 0.2), with viapoints at (−0.4, 0.4) and (−0.2, 0.3) to form an ‘s’ shape. The first via-point was set to occur at 25% of the movement time and the second was set to occur at 75% of the movement time. (a) Via-point trajectories produced by the model. (b) The velocity profiles for the movements. The velocities of individual segments of the movement are clearly visible, and are bell-shaped. (c) Even when complex movements are carried out, the model exhibits a speed-accuracy trade-off

be considered for future studies (see Sect. 5.1). The Cartesian wrist target is converted into target joint angles using the inverse kinematics for a two-link planar arm [Eqs. (1), (2)].

396

G. Simmons and Y. Demiris

0.4

the Cartesian positions of the wrist and the tips of the digits, given the appropriate joint angles.

0.3

3.1 Experiments 0.2

Y (m)

0.1

0

–0.1

–0.2

–0.3

0

0.1

0.2 X (m)

0.3

0.4

Fig. 3 The initial setup of the arm, hand and object. The digit target positions on the surface of the object are shown as dots, while the required via-points for the digits (relative to the object) are shown as crosses (see Sect. 3.1). The dashed line between the digits represents the distance between the wrist and the mid-point of the grip


θ2 = arccos θ1 = arctan

x 2 + y 2 − L 21 − L 22 2L 1 L 2

y x

 − arctan



L 2 sin θ2 L 1 + L 2 cos θ2

(1)  (2)

where θ1 and θ2 are the shoulder and elbow joints respectively, x and y are the wrist target coordinates, and L 1 and L 2 are the link lengths of the upper and lower arms, respectively. According to our optimal control scheme, the “costto-go” (see appendix) is then calculated for the reaching part of the movement. We then specify target points on the object, using a disk for simplicity. The digit target positions are specified as being on the surface of the object, connected by a line that is perpendicular to the surface and passes through the centre of mass of the object. These target points are translated into the hand frame of reference by subtracting the wrist target position. Again we use the inverse kinematics, with appropriately link lengths, to get target joint angles for the digits. These are also used to create a “cost-to-go” for each digit’s movement. From the starting point (with the digits touching) we use the “cost-to-go” to generate motor commands for each joint, adding signal-dependent noise with a 1% coefficient of variation and updating the state at each time step. As the movement is executed the forward kinematics are used to specify

The model of Smeets and Brenner (1999), which uses the minimum jerk trajectory, does not account for the natural inaccuracies of human movement introduced by neural noise. Instead, they empirically discuss why finger-tip trajectories that approach the surface of the object perpendicularly result in more accurate grasping than those that approach tangentially. Accordingly, they introduce an “approach parameter” which constrains the minimum jerk trajectory to approach the surface of the object perpendicularly. The minimum variance model has no equivalent parameter to model this perpendicular approach to the object. As described above, our implementation of the model does, however, allow the introduction of one or more via-points into the trajectory. We therefore introduce via-points into the planning of the digit movements to cause the trajectory to approach the target positions perpendicularly. As the digit planning takes place from the target position of the wrist, close to the object, these via-points are specified relative to the object. When the movement begins the digits follow the joint trajectory specified by the target and via-points. However, since the reach and grip components are executed together, the digit end-point trajectories do not pass through the via-point spatial positions relative to the object. Given this separation of planning and execution, as mentioned in Sect. 1, our first set of results determine where these via-points should be located both relative to the target points on the object and within the time course of the movement. In line with the minimum variance principle this is done by selecting the values of these parameters that result in the lowest variance of the digit end-point – that is, the highest accuracy when making contact with the object. Following this, we analyse whether the minimum variance model of pointing, with the introduction of these viapoints, can actually reproduce the characteristic features of grasping as previously identified: a maximum grip aperture proportional to the size of the object, decreasing with a slope of approximately 0.8 as the object size increases; and a time for that maximum aperture at between 60–80% of the movement time (Jeannerod 1981, 1984). Specifically, we try to match predictions three and four of Smeets and Brenner (1999): – The maximum grip increases and occurs later for larger disk sizes, – The maximum grip size increases and occurs earlier if the via-point is located further away from the object. Here, our first prediction is the same as prediction three of Smeets and Brenner (1999), while our second prediction matches prediction four of Smeets and Brenner (1999), but with a modification to account for our via-point implementation.

Object grasping using the minimum variance model

a 0.35 0.3

Y (m)

0.25 0.2 0.15 0.1 0.05 0 –0.05

0.35

0.2

0.3 X (m)

0.28

0.3 0.32 X (m)

0.4

b

0.3

Y (m)

We then look at aspects of the task that may affect the contributions of the different components to the end-point variance of the digits in the reach-to-grasp movement. Specifically, we show how the movement time, the object size and the distance of the object from the start position of the hand affect the variances of the wrist and grip aperture. Experimental work in this area has been carried out by Kudoh et al. (1997), who recorded the spatiotemporal variability of the two segments when both distance to the object and the object size were manipulated. Their results indicate that object size had a significant effect on the variability of both the transport and the grasp components, while a change in starting distance mostly affected the transport component. They also found that the peak wrist variability depended on the distance but not object size, while the peak aperture variability depended on both distance and object size. Variability in grasping has also been studied by Girgenrath et al. (2004), who observed a clear speed-accuracy trade-off in prehension as well as reaching.

397

0.25 0.2 0.15 0.1 0.26

4.1 Via-point position

1.0

0.34

c

0.8 0.6 0.4 0.2 0 –2.0 –1.5 –1.0 –0.5 0.0 0.5 1.0 1.5 2.0 Via–point vertical distance from target positions (cm)

1.0 End–point standard deviation (cm)

Each via-point consists of three parameters: its spatial coordinates on the plane of the arm, and its temporal location within the time course of the movement. To confirm whether a perpendicular approach to the surface of the object is in line with the minimum variance principle, we examine the effects of changing the vertical position of the via-points relative to the object and their temporal position, as these parameters have the greatest affect on the angle of approach. To do this, we change one parameter at a time and perform 25 repeated movements for each value, plotting the end-point standard deviation for each movement. We set up our model to perform a reach-to-grasp movement on a disk located 20 cm directly in front of the starting position of the arm and hand, with via-points located 1 cm horizontally from the object, as shown in Fig. 3. Each movement was performed on an object of diameter 4 cm and took place over 2,000 ms, with a post-movement duration (as required for the minimum variance model) of 500 ms. We first varied the vertical distance of the via-points, defined as the distance between the line connecting the target positions and the line connecting the via-points, as shown in Fig. 4a. For these movements, the via-points occurred after 80% of the movement time, or 1,600 ms into the movement. The accuracy of the digit trajectories was measured as the root-mean-square standard deviation of the digits movement in the x and y axes. In Fig. 4b, the thumb and finger trajectories are different from each other. This is due to the slight curvature introduced by the reach component. This curvature is a feature of human movement that is captured by the minimum variance model but that is not present in the minimum jerk model used by Smeets and Brenner (1999).

End–point standard deviation (cm)

4 Results

d

0.8 0.6 0.4 0.2 0 –2.0 –1.5 –1.0 –0.5 0.0 0.5 1.0 1.5 2.0 Via–point vertical distance from target positions (cm)

Fig. 4 Plots showing: (a) Plot illustrating the starting position of the hand and relationship between the line connecting the via-points and the line passing through the target positions, in the wrist target frame of reference; (b) The average digit trajectories for each value of the via-point y-coordinate. To reduce clutter, the changing positions of the via-points are not shown. Thumb trajectories are shown as dashed lines, finger trajectories as dotted lines; (c) The thumb end-point standard deviations and average for each value; (d) The finger end-point standard deviations and average for each value

398

0.35

a

Y (m)

0.3 94% 90% 86% 82% 78%

0.25 0.2 0.15 0.1 0.26

0.01 End–point standard deviation (cm)

From the results shown in Fig. 4c and d, it is clear that the lowest end-point standard deviation in digit end-point position is found when the via-points are located at roughly the same level as the target positions to the object in the wrist target frame of reference. This is in line with the argument given in Smeets and Brenner (1999) relating a greater perpendicular approach of the digits to increased accuracy of the final grip position. We then varied the position of the via-point within the time course of the movement. The same movement parameters were used as for the previous experiment. Following those experiments, the via-points were placed along the line passing through the target positions. The results shown in Fig. 5 indicate that a via-point late in the movement increases the inaccuracy of the digit trajectory end-point. However, via-points at times below 80% of the movement time have little affect on the movement accuracy. From the results of Figs. 4c,d and 5b,c, we conclude that a via-point located perpendicularly along the axis of the target positions on the object, with the trajectory required to pass through that point at approximately 80% of the movement time, results in trajectories that maximise the accuracy of the digit trajectories end-points – i.e., their contact with the surface of the object. Typical digit trajectories, and their velocity profiles, produced using these parameters are shown in Fig. 6. We now look at whether these digit trajectories actually match the features of human grasping, and the predictions discussed in the previous section, in a quantitative manner.

G. Simmons and Y. Demiris

0.28

0.3 0.32 X (m)

0.34

b

0.008 0.006

0.004 0.002

0 75

80

85

90

95

Via–point position within time course (%)

Using the parameters identified in the previous set of results, we set up our model to test the predictions given above (see Sect. 3.1). These predictions examine the characteristic form of grasping movements, specifically the maximum grip aperture and the time at which it occurs within the course of the movement. We initially performed two sets of movements: in one we varied the via-point horizontal distance from the object, while in the other we varied the object size. For both sets of experiments, each movement again took place over 2,000 ms, with the same post-movement duration of 500 ms. The via-point was located along the orientation of the grip and occurred at 80% of the movement time, or 1,600 ms into the movement, in line with the results above. The first set of results, shown in Figs. 7 and 8, show five grasping movements to an object of diameter 4 cm. Viapoint distances from the object were set at 0.0, 0.25, 0.5, 0.75 and 1.0 cm. Figure 7 shows the digit paths, the digit velocity profiles and the time course of the grip aperture for each via-point. From these movements we were able to plot the via-point distance against both the maximum grip aperture and the time at which that maximum occurred, as shown in Fig. 8. The results shown in the figure clearly confirm the second prediction given in the previous section: that grip size

End–point standard deviation (cm)

0.012

4.2 Grasping predictions

c

0.01 0.008 0.006 0.004 0.002 0 75

80

85

90

95

Via–point position within time course (%)

Fig. 5 Plots showing: (a) The average digit trajectories for a representative selection of via-point temporal positions; (b) The thumb end-point standard deviations and average for each value; (c) The finger end-point standard deviations and average for each value

increases and occurs earlier if the via-point is located further away from the object. The second set of movements were performed to objects of varying sizes, with the via-point distance fixed at 1.0 cm from the object. All other parameters were kept the same as for the first set of movements. The grasps were performed on objects of size 0.0, 2.0, 4.0, 6.0 and 8.0 cm. Figure 9 shows

Object grasping using the minimum variance model

0.35

399

0.35

a

0.3

0.3

0.25

Y (m)

Y (m)

a

Finger Thumb

0.25 Finger Thumb

0.2

0.2

0.15

0.15

0.1 0.25

0.3 X (m)

0.1 0.25

0.35

0.25

0.25

b

Finger Thumb

0.3 0.35 X (m)

b

Finger Thumb

0.2 Speed (ms–1)

Speed (ms–1)

0.2

0.15

0.1

0.15

0.1

0.05

0.05 0

0.2

0.4

0.6 0.8 Relative time

1

0

0.2

0.4

1.2

Fig. 6 Plots showing: (a) Typical trajectories for the finger and thumb with via-points level with the axis of the target positions and occurring at 80% of the movement time. The via-points were located 1 cm horizontally from the object and the movement took place over 2,000 ms; (b) Velocity profiles for the digit trajectories

the digit paths, the digit velocity profiles and the time course of the grip aperture for each object size. In the same way as the first set of movements, we used these trajectories to examine the relationship between object size and maximum grip aperture, and between object size and the time at which that grip aperture occurs, as shown in Fig. 10. From these plots, it is clear that our model confirms our first prediction from Sect. 3, that grip size increases and occurs later for increasing object sizes. In addition, Fig. 10a shows the equation for the line of regression between the points. This line has a slope of 0.92, which is larger than the average value reported in Figs. 6a and 7a of Smeets and Brenner (1999), but still well within the range of maximum grip slopes from the numerous experimental studies of grasping shown in those figures. The second plot, Fig. 10b, shows a range of relative times for the maximum grip aperture, with values between 60 and 80% of the movement time for the object sizes examined here. These values and the form of the regression line are

10

0.6 0.8 Relative time

1

0.6 0.8 Relative time

1

1.2

c 0.0cm 0.5cm 1.0cm 1.5cm 2.0cm

8 Grip aperture (cm)

0 0

6

4

2

0

0

0.2

0.4

1.2

Fig. 7 Grasping movements performed to a disk of diameter 4cm, with via-points at horizontal distances 0.0, 0.25, 0.5, 0.7 and 1.0 cm from the object. (a) Movement paths for the thumb (dashed line) and finger (dotted line). (b) Velocity profiles for the digits. (c) Time course of the grip aperture

also well within the range of values reported in Figs. 6b and 7b of Smeets and Brenner (1999). 4.3 Transport and grip variability Having demonstrated that our grasping model captures the experimentally observed features of human movement, we

400

G. Simmons and Y. Demiris

10

0.35

a

0.3

8 7

Y (m)

Maximum grip aperture (cm)

9

a

6

0.25 Finger Thumb 0.2

5

0.15 4 3 0.0

0.5

1.0

1.5

0.1 0.25

2.0

Via–point horizontal distance (cm)

0.25

b

0.8

0.75

Finger Thumb

0.15 0.1 0.05

0.7

0.65 0.0

b

0.2

0.85

Speed (ms–1)

Relative time to maximum grip aperture

0.9

0.3 0.35 X (m)

0 0.5

1.0

1.5

2.0

0

0.2

0.4

0.6 0.8 Relative time

1

0.4

0.6 0.8 Relative time

1

Via–point horizontal distance (cm)

can now use it to study the contributions to the end-point digit variance made by the two components of the reach-tograsp movement. We looked at the affect of movement time, object size and distance to the object from the start position of the hand. As in the previous sets of experiments we performed movements with a via-point located 1 cm from the object along the line joining the target positions on the surface of the object, and occurring at 80% of the movement time. We analysed the variability of the movement by performing 50 repeated grasps for each parameter change, plotting both the wrist variance and the grip aperture variance from the noiseless trajectories. As well as looking at the overall variance in this way, we also used the peak variance, and the time within the course of the movement that it occurred, as measurements of the variance. We plotted these values against the parameter values and used a one-way analysis of variance (ANOVA) for each parameter to determine whether that parameter had an affect on movement variability.

12

c 0cm 2cm 4cm 6cm 8cm

10 Grip aperture (cm)

Fig. 8 Plots showing: (a) The relationship between via-point horizontal distance from the object and maximum grip aperture; (b) The relationship between via-point horizontal distance and the relative time at which the maximum grip aperture occurs. These show the results for ten grasping movements to an object of 4 cm, including the five shown in Fig. 7. These plots confirm the prediction that the grip size increases and occurs earlier if the via-point is located further away from the object

1.2

8 6 4 2 0

0

0.2

1.2

Fig. 9 Grasping movements performed to disks of diameter 0.0, 2.0, 4.0, 6.0 and 8.0 cm, with via-points at horizontal distance 1.0cm from the object. (a) Paths for the thumb (dashed line) and finger (dotted line). (b) Velocity profiles for the digits. (c) Time course of the grip aperture

We first looked at the effect of movement time. As shown in Fig. 1d we expect the variability of both components to decrease as the movement time increases, for movements over the same distance. We performed grasping movements on an object of 4 cm diameter, located 20 cm in front of the hand, with via-points placed as described above. The movement times were 1,200, 1,400, 1,600, 1,800 and 2,000 ms.

Object grasping using the minimum variance model

1.2 a

a

10

Standard deviation (cm)

Maximum grip aperture (cm)

12

401

y = 0.92*x + 0.028

8

6 4

1200ms 1400ms 1600ms 1800ms 2000ms

1.0 0.8 0.6 0.4 0.2

2 0

2

4

6

0

8

Object diameter (cm)

b

0.2

0.4

0.6 0.8 Relative time

1.2 b

0.74

Standard deviation (cm)

Relative time to max grip aperture

0.76

0

0.72 0.7 0.68 0.66 0.64

1

1.2

1200ms 1400ms 1600ms 1800ms 2000ms

1.0 0.8 0.6 0.4 0.2

0.62

0

0.02

0.04

0.06

0.08

Object diameter (cm)

0

0

0.2

0.4

0.6 0.8 Relative time

1

1.2

Fig. 10 Plots showing: (a) The relationship between object size and maximum grip aperture, along with the equation for the line of regression for nine different object sizes, from 0 to 8 cm; (b) The relationship between object size and the relative time at which the maximum grip aperture occurs. These plots confirm that the prediction that the grip size increases and occurs later for larger object sizes

Fig. 11 Plots showing the time course of the (a) wrist variability, (b) grip aperture variability for different movement times. These plots show the averages of 50 repeated movements for each value of the movement time

The results shown in Fig. 11 confirm that increasing the movement time decreases the overall variability of the movement for both the wrist (transport component) and the grip aperture (grasp component). The variability of the grip aperture (Fig. 11b) shows an increase in variability just after the via-point, and another at the point of contact. This can also be seen in Figs. 13b and Fig. 15b. The increase after the via-point is caused by a reduction in constraints acting on the trajectory. After the via-point has been reached, the dominant term acting on the trajectory is the target position, which must be reached in the remaining movement time. This leads to an increase in velocity (and hence variability) just after the via-point. The increase in variability at the point of contact is caused by the noisy trajectory under- or over-shooting the target. Since at this point the movements has stopped, no corrective movements are made (see the Appendix for a further description of variability in the post-movement period). From Fig. 12a, and c it can be seen clearly that peak variance decreases as movement time increases. Our one-

way ANOVA with the movement time and found that the decrease shown on the plots was significant for both the wrist (F = 27.96, P < 0.01) and the grip aperture (F = 35.79, P < 0.01). Figure 12b and d shows that the movement time had no significant affect on the timing of the peak variance, indicating that the pattern of the variance was unaffected by changes in movement time. This can also be seen in Fig. 11. These results are largely expected, as according to Fitts’ Law (Fitts 1954; MacKenzie 1995) an increased movement time for a movement over a fixed distance will result in lower velocities. Since the variance of the movements are signal-dependent, lower velocities (which require lower control signals) result in lower variances for both the digits and the wrist. These results also follow those of Girgenrath et al. (2004), who specifically showed that prehension is subject to the same speed-accuracy trade-off as reaching. We then examined the affect of object size on the variability. As before we performed grasping movements on an object located 20 cm in front of the hand, with the movement time set to be 2,000 ms. The object size was varied as 0, 2, 4, 6 and 8 cm.

402

G. Simmons and Y. Demiris

1.0

0.5

Relative time of peak deviation

1

1400 1600 1800 Movement time (ms)

2000

0cm 2cm 4cm 6cm 8cm

0.8 0.6 0.4 0.2 0 0

b

0.2

0.4

0.6 0.8 Relative time

1

1.2

0.8

1.0 b

0cm 2cm 4cm 6cm 8cm

0.6 0.4 0.2 0 1200

1.0 Peak deviation from mean (cm)

Standard deviation (cm)

1.5

0 1200

1400 1600 1800 Movement time (ms)

2000

c

0.8

0.8 0.6 0.4 0.2 0 0

0.6

0.2

0.4

0.6 0.8 Relative time

1

1.2

Fig. 13 Plots showing the time course of the (a) wrist variability, (b) grip aperture variability for different object sizes. These plots show the averages of 50 repeated movements for each object size

0.4 0.2 0 1200

Relative time of peak deviation

1.0 a

a

Standard deviation (cm)

Peak deviation from mean (cm)

2.0

1

1400 1600 1800 Movement time (ms)

2000

1400 1600 1800 Movement time (ms)

2000

d

0.8 0.6 0.4 0.2 0 1200

Fig. 12 Plots showing the change in peak variability and relative time of peak variability for 50 movements for each value of the movement time. For these plots, each cross represents the peak deviation of each movement from the mean (i.e., noiseless) trajectory. For each movement time, the 50 values are averaged to show how the peak deviation changes. (a) Wrist peak variability and (b) Time of peak variability; (c) Grip aperture peak variability and (d) Time of peak variability

The results shown in Fig. 13a indicate that increasing object size has no clear effect on the overall variability of the wrist movement. This is confirmed by Fig. 14a, and b, which shows no significant changes in the peak standard deviation or the timing of the peak. By contrast, Fig. 14c, and d shows that peak standard deviation for the grip aperture increases as object sizes increases (F = 9.93, P < 0.01), and that this peak occurs earlier in the movement (F = 6.42, P < 0.01). This can also be seen in Fig. 13b. This is largely in line with the results of Kudoh et al. (1997) which showed similar affects of increasing object size on the grip aperture at a significant level, but no affect on the wrist movement. Finally, we examined the effect of starting distance between the hand and the object on the variability. Movements were performed to an object of 4 cm diameter, over 2,000 ms. The distance to be moved was set as 8, 12, 16, 20 and 24 cm. As can be clearly seen in Fig. 15, increasing the movement distance increases the overall variance of wrist movement, but has no affect on the grip movement. This is also clear from Fig. 16a, and c. The effect of increasing move-

Object grasping using the minimum variance model

a

Standard deviation (cm)

1.0 1.2 1.0 0.8 0.6 0.4 0

Relative time of peak deviation

1

2

4 Object size (cm)

6

0.8 0.6 0.4 0.2

8

0

b

0

0.2

0.4

0.6 0.8 Relative time

1

1.2

0.8

1.2

b

8cm 12cm 16cm 20cm 24cm

0.6

1.0 0.4 0.2 0 0

0.8

2

4 Object size (cm)

6

8

c

0.8 0.6 0.4 0.2

0.6 Peak deviation from mean (cm)

8cm 12cm 16cm 20cm 24cm

1.4

0.2

0.4

0

0.2

0

0.2

0.4

0.6 0.8 Relative time

1

1.2

1.0

Fig. 15 Plots showing the time course of the (a) wrist variability, (b) grip aperture variability for different movement distances. These plots show the averages of 50 repeated movements for each distance

0.8 0.6 0.4 0.2 0

1 Relative time of peak deviation

a

1.2

Standard deviation (cm)

Peak deviation from mean (cm)

1.6

403

2

4 Object size (cm)

6

8

d

0.8 0.6

ment distance on peak standard deviation of the wrist was highly significant (F = 61.04, P < 0.01), while the effects on timing and on the grip aperture were negligible. These results were also largely in line with those of Kudoh et al. (1997), in that the effect of increasing movement distance on wrist variability was highly significant. However, their study also showed a significant effect on grip aperture variability which is not replicated using our model.

0.4

5 Discussion

0.2 0 0

2

4 Object size (cm)

6

8

Fig. 14 Plots showing the change in peak variability and relative time of peak variability for 50 movements for each object size. As in Fig. 12, each cross represents the peak deviation from the noiseless trajectory, and the lines show the averages of these for each object size. (a) Wrist peak variability. (b) Time of peak variability. (c) Grip aperture peak variability. (d) Time of peak variability

Together, the two experiments shown in Sects. 4.1 and 4.2 confirm that our model of reaching-to-grasp captures the experimentally determined characteristics of human grasping. As stated before, our approach was to consider grasping to be two separate processes, planned using the same motor control principle. Through the addition of a single via-point into each digit’s trajectory we have been able to replicate the perpendicular

404

G. Simmons and Y. Demiris

a

Peak standard deviation (cm)

2.0

1.5

1.0

0.5

0

relative time of peak standard deviation

8 1

10

12 14 16 18 20 Movement distance (cm)

22

24

10

12 14 16 18 20 Movement distance (cm)

22

24

10

12 14 16 18 20 Movement distance (cm)

22

24

10

12 14 16 18 20 Movement distance (cm)

22

24

b

0.8 0.6 0.4 0.2 0 8

Peak standard deviation (cm)

0.8

c

0.6 0.4 0.2 1.0 0.8 0.6 0.4 0.2

Relative time of peak standard deviation

8

1

d

0.8 0.6 0.4 0.2 0

8

Fig. 16 Plots showing the change in peak variability and relative time of peak variability for 50 movements for each movement distance. (a) Wrist peak variability and (b) Time of peak variability; (c) Grip aperture peak variability and (d) Time of peak variability

approach of the digits to the object target positions observed in grasping. Following the minimum variance principle, the spatiotemporal positions of the via-points were determined as those that resulted in the lowest end-point standard deviation of the digit trajectories, quantitatively confirming that a perpendicular approach to the target positions of the object results in a more accurate grasp. The via-point constrains the trajectory to pass through a set location (or close to it, as the addition of signal-dependent noise means the trajectory is unlikely to pass through the exact spatial location) at a set time during the movement. The via-point imposes no other demands, such as velocity or acceleration constraints, on the system. Our results show that increasing the horizontal distance of the via-points from the object increases the maximum grip aperture of the hand during grasping and causes that maximum to occur earlier in the movement. This follows as a fairly logical consequence, as a via-point further from the object will require a larger grip, which will have to be performed earlier if the digits are to successfully reach the target positions on the object in the required movement time. We are also able to confirm that our model matches the characteristics observed in numerous experiments on grasping. Both the relationship between maximum grip aperture and object size, and between time of maximum grip and object size were successfully captured by our model. These results compare favourably with the literature summaries reported in Smeets and Brenner (1999). Our model allows us to study the contributions to the endpoint variance of both components of grasping. By varying the movement time for a grasping movement over a fixed distance, we demonstrated that both the transport component and the grip component exhibit reduced variance as movement time increases (Girgenrath et al. 2004). Since our model of reaching has been shown to obey the speed-accuracy tradeoff of Fitts’ Law (see Simmons and Demiris 2005 and Fig. 1), this result is not unexpected. We were also able to carry out experiments to check whether our model matched the findings of an experimental study into the spatiotemporal variability of grasping. Following Kudoh et al. (1997), we varied both object size and movement distance and observed the affects on the variability of the movement. Our model showed that object size has a significant effect on the variability of the grip aperture, but not on the wrist movement. This follows from the separation of the grasping movement into two separately planned components, as the planning of the reaching component takes no account of the object properties. With regards to changing the movement distance, the results from our model differed slightly from those of Kudoh et al. (1997). We observed the same affect on the transport component, that an increase in movement distance results in an increase in variability. Again, this is a logical consequence of the speed-accuracy trade-off in that a shorter movement distance over the same movement time will require lower velocities and therefore be more accurate.

Object grasping using the minimum variance model

However, we did not observe any significant effect of movement distance on the grip aperture variability. The separation of planning for the two components is again the reason for this result from our model. This would generally be the case for models that follow the dual-channel view of grasping, since it proposes that the information processed by the grip channel is intrinsic to the object and not affected by properties that relate to the object and its environment (Jeannerod 1981; Smeets and Brenner 1999). In turn, these extrinsic properties are processed solely by the transport channel.

405

An interesting area of study would be to examine how grasping trajectories produced from pointing models are affected by the presence of obstacles, both in the path of hand and close to the object. Experimental work has already been performed looking at how obstacles influence the speed of grasping movements (Biegstraaten et al. 2003) and how obstacle position in the workspace can change grasping trajectories (Mon-Williams et al. 2001). The affects of obstacles on minimum variance reaching movements have also been examined (Hamilton and Wolpert 2002), allowing obstacle avoidance to be incorporated into a minimum variance grasping model.

5.1 Further investigations Our work clearly prompts further investigation into the behaviour of grasping. It supports the view that grasping could be planned as pointing movements of the digits, but this remains to be physiologically or neurologically confirmed. The results presented in this paper have been achieved even with a relatively simple model for the arm and hand. A more realistic model of the hand, such as that used by Meulenbroek et al. (2001), could certainly be introduced to our system without changing its fundamental characteristics. However, different types of grip have been modelled using two “virtual” fingers (Oztop and Arbib 2001) that are similar to our arrangement for the hand. A further extension of our model would be to move away from planar grasping and model full three-dimensional grasping movements, with all the challenges involved in the potential link configurations. One of the assumptions of our model and that of Smeets and Brenner (1999) is that the digits begin the movement touching each other. An area for further modelling is to relax this assumption and observe the effect on the digit trajectories of a starting grip aperture that is not zero. Timmann et al. (1996) showed that when subjects started the movement with maximum grip aperture, they initially closed the grip before reopening to second maximum and then closing again to grip the object. Current models, including those of Smeets and Brenner (1999) and Meulenbroek et al. (2001) are unable to account for this, although Smeets and Brenner (1999) suggest that additional constraints on their model would conform to the results of Timmann et al. (1996). In this work we have not touched on the effects of hand orientation or object-to-wrist distance on grasping. Clearly these will have a large influence on the movement of the digits, and will also change the shape of the grasp. The biggest impact of changing the object-to-wrist distance will be to restrict the maximum size of object that can be grasped – the largest object can only have a radius equal to the object-towrist distance. A second factor will be a limitation on the size of the maximum possible grip aperture. We have also looked at only a single object shape, while there is evidence that object shape has a large effect on the grasping kinematics (Cuijpers et al. 2004; Schettino et al. 2003). Furthermore, we have only explored precision grip between the finger and thumb – the principle could equally be applied to other types of grip.

6 Summary We have put forward a model of grasping based on both the “classical” dual visuomotor channel view, and on a new view that attempts to explain reaching-to-grasp as pointing movements with the digits. Using an implementation of the minimum variance model of human movement, modified to allow grasping movements, we have shown that our model captures many of the characteristic features of human grasping. In particular, we demonstrated how our model exhibits an increase in maximum grip aperture for increasing object size, with a slope of approximately 0.8. The maximum grip aperture produced by our model was also shown to occur at around 60–80% of the movement. Both of these results follow observations made by a large body of experimental studies. Furthermore, we studied the contribution of each component (the transport and the grip) to the variability of grasping movements, by analysing the affects of changing three task-related parameters: the movement time, the movement distance and the object size. The results produced by our model again follow the experimental studies carried out in this area. Overall then, the approach presented here seems to support the view that reaching and grasping are planned using the same motor principle. However, as it is still not known whether the human motor system acts in this way, further biological and neuroscientific studies are needed to clarify this. As a condition of our movement model we found it necessary to retain elements of the “classical” view of grasping as two separate visuomotor processes. The paper analysed the contribution of each of the components in a unified manner through the application of the minimum variance principle, computationally confirming human experimental studies. Acknowledgements The authors would like to thank an anonymous reviewer for the helpful comments. We would also like to thank Anthony Dearden, Bassam Khadhouri, Matthew Johnson and Paschalis Veskos, from the BioART team at Imperial College London, for their advice and suggestions. The first author has been supported by a Doctoral Training Award from the UK’s EPSRC.

406

G. Simmons and Y. Demiris

u n = K n sn , n = 0, . . . , T + N

Appendix Our implementation of the minimum variance model (Simmons and Demiris 2004, 2005 is designed for the control of a robot arm, using the discrete-time linear quadratic regulator (DLQR) optimal control scheme. We first formulate a cost function Cv that defines the criteria to be optimised: Cv =

T +N



σx2 + σ y2

 (3)

n=T +1

where T is the movement time, N is the length of the postmovement period, and σx2 and σ y2 are the estimated variances of the Cartesian hand position in the x- and y-axis respectively. No costs are associated with the movement itself from n = 0, . . . , T , unless a via-point is included, as explained below. From this the optimal control scheme requires a matrix cost function J that describes the costs associated with each time step of the movement: J=

T +N n=0

u
Ru + sT
Q T sT +

T +N

sn
Q pm sn

(4)

n=T +1

where u are the control signals, R is the control signal cost matrix, si is the state vector at time step i, Q T is the state cost matrix at time step T and Q pm is the state cost matrix for every time step during the post-movement period. Q T ensures the target is reached, while Q T +1,...,T +N = Q pm penalises the hand variance. Since there are no state costs associated with the movement for n = 0, . . . , T , Q 0,...,T = 0. The only exception to this occurs when via-points are included. In that case, the state cost Q n for n = i (where i is the time step at which the trajectory must pass through the via-point) takes a form similar to that of Q T , ensuring the via-point is reached. This results in a set of state cost matrices Q = {Q 0 , . . . , Q T , . . . , Q T +N }. The cost function matrices Q are used to calculate a “costto-go” for each time step. This is an estimate of the total remaining cost from the next state. It is calculated recursively, starting from the fact that the remaining “cost-to-go” at n = T + N is simply the final state cost Q T +N . This results in another set of matrices P = {P1 , . . . , PT +N }. Because costs incurred during the movement are only taken into account during the post-movement period, the length of the post-movement period has a direct affect on the resulting trajectory. The longer the post-movement period, the greater the cost of any deviation at the time the movement comes to a halt at n = T . Because the optimisation algorithm works on a predicted “cost-to-go”, a longer post-movement period therefore results in a more accurate trajectory, even though there is no movement during the post-movement period (as is the case when grasping an object). The optimal control scheme uses state feedback to produce a motor command; the current state is multiplied by a gain term to produce the control signals (Eq. (5)). The set of state feedback gains K are calculated using the “cost-to-go” matrices, giving a set of matrices K = {K 0 , . . . , K T +N }.

(5)

Following the minimum variance model, this control signal is then subject to white noise with zero mean and variance proportional to the amplitude of the signal:  u˜ n = u n + wn , wn ∼ N 0, ku 2n (6) Finally, the noisy motor command is used to update the state vector: sn+1 = A.sn + B.u˜ n

(7)

which has state dynamics matrices A and B.

References Biegstraaten M, Smeets JBJ, Brenner E (2003) The influence of obstacles on the speed of grasping. Exp Brain Res 149:530–534 Cruse H, Wischmeyer E, Brüwer M, Brockfield P, Dress A (1990) On the cost functions for the control of the human arm movement. Biol Cybern 62:519–528 Cuijpers RH, Smeets JBJ, Brenner E (2004) On the relation between object shape and grasping kinematics. J Neurophysiol 91:2598–2606 Fitts PM (1954) The information capacity of the human motor system in controlling the amplitude of movement. J Exp Psychol 47(6):381– 391 Flash T, Hogan N (1985) The coordination of arm movements: An experimentally confirmed mathematical model. J Neurosci 5(7):1688–1703 Flash T, Sejnowski TJ (2001) Computational approaches to motor control. Curr Opin Neurobiol 11:655–662 Girgenrath M, Bock O, Jüngling S (2004) Validity of the speed-accuracy tradeoff for prehension movements. Exp Brain Res 158:415–420 Hamilton AFDC, Wolpert DM (2002) Controlling the statistics of action: obstacle avoidance. J Neurophysiol 87:2434–2440 Harris CM, Wolpert DM (1998) Signal-dependent noise determines motor planning. Nature 394:780–784 Jeannerod M (1981) Intersegmental coordination during reaching at natural visual objects. In: Long J, Baddeley A (eds) Attention and performance IX, pp 153–169 Jeannerod M (1984) The timing of natural prehension movements. J Motor Behav 16:235–254 Kamper DG, Cruz EG, Siegel MP (2003) Stereotypical fingertip trajectories during grasp. J Neurophys 90:3702–3710 Kudoh N, Hattori M, Numata N, Maruyama K (1997) An analysis of spatiotemporal variability during prehension movements: effects of object size and distance. Exp Brain Res 117:457–464 Loeb GE, Levine W, He J (1990) Understanding sensorimotor feedback through optimal control. Symp Quant Biol 55:791–803 Luo Z, Svinin M, Ohta K, Odashima T, Hosoe S (2004) On optimality of human arm movements. In: Proceedings of the IEEE international conference on robotics and biomimetics 2004, pp 447–452 MacKenzie IS (1995) Movement time prediction in human-computer interfaces. In: Baecker RM, Buxton WAS, Grudin J, Greenberg S (eds) 2nd edn. Readings in human-computer interaction, pp 483–493 Meulenbroek RGJ, Rosenbaum DA, Jansen C, Vaughan J, Vogt S (2001) Multijoint grasping movements – simulated and observed effects of object location, object size, and initial aperture. Exp Brain Res 138:219–234 Miyamoto H, Wolpert DM, Kawato M (2003) Computing the optimal trajectory of arm movement: The TOPS (Task optimization in the presence of signal-dependent noise) model. In: Dura RJ, Santos J, Grana M (eds) Biologically inspired robot behaviour engineering, vol 109. Studies in fuzziness and soft computing, Chapt 14 Springer, Berlin Heidelberg New York, pp 395–416

Object grasping using the minimum variance model

Miyamoto H, Nakano E, Wolpert DM, Kawato M (2004) TOPS (Task optimization in the presence of signal-dependent noise) model. Syst Comput Japan 35(11):48–58 Mon-Williams M, Tresilian JR, Coppard VL, Carson RG (2001) The effect of obstacle position on reach-to-grasp movements. Exp Brain Res 137:497–501 Morasso P (1981) Spatial control of arm movements. Exp Brain Res 42:223–227 Oztop E, Arbib MA (2001) A biologically inspired learning to grasp system. In: Proceedings of the 23rd annual EMBS international conference, IEEE, pp 857–860 Schettino LF, Adamovich SV, Poizner H (2003) Effects of object shape and visual feedback on hand configuration during grasping. Exp Brain Res 151:158–166

407

Simmons G, Demiris Y (2004) Biologically inspired optimal robot arm control with signal-dependent noise. In: Proceedings of the 2004 IEEE/RSJ international conference on intelligent robots and systems, vol 1. pp 491–496 Simmons G, Demiris Y (2005) Optimal robot arm control using the minimum variance model. J Robot Syst (11):677–690 Smeets JBJ, Brenner E (1999) A new view on grasping. Motor Control 3:237–271 Timmann D, Stelmach GE, Bloedel JR (1996). Grasping component alterations and limb transport. Exp Brain Res 108:486–492 Uno Y, Kawato M, Suzuki R (1989) Formation and control of optimal trajectory in human multijoint arm movement: Minimum torquechange model. Biol Cybern 61:89–101