Biomed Eng Lett (2015) 5:98-108 DOI 10.1007/s13534-015-0179-x

ORIGINAL ARTICLE

Lower Body Motion Analysis to Detect Falls and Near Falls on Stairs Gemma S. Parra-Dominguez, Jasper Snoek, Babak Taati and Alex Mihailidis

Received: 10 November 2014 / Revised: 16 March 2015 / Accepted: 2 April 2015 © The Korean Society of Medical & Biological Engineering and Springer 2015

Abstract Purpose We present a methodology to automatically detect falls on stairs, an application of computer vision and machine learning techniques with major real-world importance. Falls on the stairs, in particular, are a common cause of injury among older adults. Comprehending the conditions under which accidents take place could significantly aid in the prevention of falls, support independent living, and improve quality of life. Methods We extract a set of features, composed of Fourier coefficients and entropy metrics of instantaneous velocities from 3D motion sensor data, to encode lower body motion during stair navigation. A supervised learning algorithm is then trained on manually annotated data simulated in a home laboratory. Results In our empirical analysis, the algorithm obtains high fall detection accuracy (> 92%) and a low false positive rate (5-7%). In contrast with previous research, we identify that motion of the hips, rather than that of the feet, is the best indicator of dangerous activity given the 3D trajectory of various lower body joints. It is also shown that entropy measures alone provide sufficient information to detect abnormal events on stairs. Conclusions The study of falls is difficult due to their

Gemma S. Parra-Dominguez Intelligent Assistive Technology and Systems Lab at the University of Toronto Jasper Snoek Center for Research on Computation and Society at Harvard University Babak Taati Toronto Rehabilitation Institute-University Health Network Alex Mihailidis ( ) Toronto Rehabilitation Institute-University, Health Network and the Department of Occupational Science and Occupational Therapy and the Institute of Biomaterials and Biomedical Engineering at the University of Toronto Tel : +416-946-8565 / Fax : +416-946-8573 E-mail : [email protected]

exceedingly sparse nature; but an automatic non-contact fall detection system can assist in data collection by sieving through large quantities of data, e.g., from public stairways. Keywords Fall detection, Stairs, Ambient intelligence, Safety monitoring, RGB-D camera, Aging-in-place

INTRODUCTION As the world’s population is aging and the global proportion of elderly adults is rapidly increasing, the burden of care is growing beyond what healthcare systems can support. The most practical solution available today is to encourage independent living or aging in place [1]. However, a major hurdle to independent living is the danger of falling in the home, as falls in the home are the leading cause of both fatal and non-fatal accidental injury. Falls on the stairs are among the most common and serious causes of trauma and incur major and often long-term healthcare costs [2]. In this work we explore the application of computer vision and machine learning to develop an automated methodology to detect falls on the stairs. Older adults are especially exposed to dangerous situations on stairways because of their weakened visual and musculoskeletal systems. Comprehending the conditions, both environmental and behavioral, under which an accident may occur can have a significant effect in fall prevention and in the design of improved staircases and handrails. The ability to automatically detect and predict falls can also help provide prompt assistance when an accident does occur. The currently accepted technique to study falls and their causes is to record and observe a large amount of dangerous events in real-life (e.g., [3]) and later visually inspect them in terms of the physical environment and other factors influencing each fall. However, it is estimated that on public stairways, a slip, stumble, trip, or other loss of balance not resulting in a fall occurs only once in every 2,222 stair uses

Biomed Eng Lett (2015) 5:98-108

and studies show that falls are observed only once in 63,000 stair traversals [4]. Consequently, creating a sufficiently large database of real stair accidents in order to produce statistically meaningful analysis involves collecting vast amounts of video over extended periods of time. Moreover, it requires the annotation, usually by hand, of recorded videos with times and locations at which falls are observed. Manual annotation can be a laborious, time consuming, and expensive task. Automating the partition of recorded data into informative segments and identifying potentially dangerous situations could be an efficient tool when creating a meaningful database of real falls. The automation of video annotation has been used in other applications such as [5, 6], which involved the analysis of human-environment interaction. A major goal of this work is to develop a computer vision system to automatically detect unusual events during stair traversal that can generalize to new people and stairways. Detecting unusual events on stairs is difficult because people exhibit highly idiosyncratic behavior while traversing stairs. Thus, generalizing to new subjects is a major challenge as it requires distinguishing potentially dangerous behavior from the idiosyncrasies of individual gait [4, 7, 8]. In this work, abnormal events are considered to be those which are potentially dangerous, i.e., falls or near falls. While idiosyncratic behavior, such as a person waving their arms irregularly while otherwise normally traversing stairs, can be considered abnormal, in this work they are considered a nuisance to which the system is desired to be invariant. We develop a methodology to automatically detect abnormal events during stair traversal and experimentally validate it on a ground truth dataset collected in a smart home environment. A Microsoft Kinect sensor is used to track relevant joint positions (i.e., the lower limb joints: hips, knees, ankles and feet) in real-time. From this data, a number of features are extracted which encode the joints' trajectory during a sliding window of time equal to the gait cycles. A binary classifier is then trained on these features to discriminate between normal and abnormal traversal of stairs. An empirical analysis was performed, comparing different sets of features extracted from various combinations of lower body joints in order to identify the best performing combination of skeletal joints and features.

RELATED WORK Three dimensional (3D) cameras or depth sensors such as the Microsoft Kinect, along with 3D data processing algorithms are now widely used to track and analyse human pose and motion [9]. The use of inexpensive depth sensors has been explored to recognize human activities such as cooking, drinking water, brushing teeth, performing rehab exercises,

99

walking, standing up, and washing hands [6, 10-14]. Activity recognition can aid the development of smart home systems which allow older adults to age-in-place by providing different services, such as automated reminders for daily tasks [15]. Safety at home is naturally also a major concern and a priority for aging-in-place [8]. The use of depth maps in fall detection systems provides advantages over 2D cameras, such as 3D reconstruction of scenes, faster and more accurate segmentation of cluttered backgrounds and objects, and real-time detection and tracking of human joints. Moreover, infrared sensing allows Kinectbased systems to work at day and night, and the privacy of the person being monitored can readily be preserved via the processing of depth maps rather than color (RGB) data. Fall detection systems using the Kinect have been proposed and examined by a number of researchers, including [16-24]. Despite the large body of work exploring depth sensors, and the Kinect in particular, for fall detection applications, virtually all published work examines falls on flat surfaces. To the best of our knowledge, our work is the first to employ the Kinect to detect falls on stairs. A systematic review of Kinect applications to rehabilitation and home monitoring, including fall detection applications is provided in [25]. Skeletal tracking using the Kinect has also been used to analyze gait [13, 26, 27]. Stone and Skubic [13, 26] developed an in-home vision based monitoring system to capture strideto-stride gait variability from a 3D point cloud using a Microsoft Kinect. The authors noted that changes in gait have been shown to be predictive of future falls in older adults. Some techniques to analyze human gait are based on entropy metrics, as described in [28-30]. In particular, variability in the elderly during walking was evaluated using entropy measures to identify balance impairments and prevent fall injuries [31, 32]. In this work, the motion of the lower limb joints of a subject is analyzed using entropy measures to detect unusual events in a time series. Using monocular images, Snoek et al. [4] detected dangerous situations on stairways by tracking the feet of a subject. The authors noted that 75% of stairs falls causing injury occurred during descents, and they also argued that computationally efficient algorithms were necessary to perform real-time tracking of the feet and to achieve real-time classification of falls. The motivation of this work is to explore the use of 3D information to perform real-time tracking of the joints while walking on the stairs, to improve detection accuracy, and to achieve a faster detection of unusual events. In previous work and preliminary results [33], we recorded and analyzed stair descents. It was found that monitoring just the motion of the hips was sufficient to achieve high accuracy results when detecting abnormal events. Using a random forest classifier on features derived from the motion of the

100

hips yielded the highest accuracy results for this application (~92.0%). This study expands our earlier work [33]. The specific contributions here are 1) the collection of a more comprehensive dataset including both stair descent and ascents from a larger number of participants; 2) demonstrating the generalization of the proposed methodology to detect abnormal events in both ascents and descents; and 3) a deeper analysis of the features' influence on the obtained results to determine the most informative body joints and the most discriminating set of features extracted from motion trajectories.

DATASET The experimental dataset was collected in the “HomeLab” of

Biomed Eng Lett (2015) 5:98-108

the Toronto Rehabilitation Institute and using a Microsoft Kinect sensor mounted at the top of a staircase with 9 steps. In total, 330 stair descents and 330 stair ascents were recorded from 11 participants (5 women and 6 men): 165 with no abnormal events and 165 with a dangerous situations occurring. For this study, data was recorded at the rate of fs ≅ 18.6 frames per second (fps). In Fig. 1 a few sample frames from the recorded videos are illustrated. As a quality improvement study, i.e., a project to improve the accuracy of fall detection algorithms on stairs, this project was exempt from ethics. All abnormal events were manually identified and annotated in each sequence by marking the frames in which the event started and ended. After the recording, it was noted that in 3.03% of the recorded descents and 7.27% of the recorded

Fig. 1. Example of recorded stairway ascents and descents, simulating: (a)-(b) normal, (c)-(d) fall, and (e)-(f) a trip and an overstep. These data were collected only from research team members and were used only to improve the operation of this technology. As such, this project was categorized as a quality improvement project and was exempt from a research ethics review.

Biomed Eng Lett (2015) 5:98-108

ascents, the tracking of the joints had failed due to the sensor limitations. The collected tracking data was converted from the sensor coordinate frame to a world coordinate frame, using the methodology described in [33]. In short, a 4 × 4 homogenous

101

rigid transformation matrix was formed after capturing a single “background” depth map of the scene. This homogeneous transformation was then applied to all 3D sensed data such that the resulting values were represented in world coordinates. In this world coordinate frame, the y axis was

Fig. 2. This figure demonstrates example trajectories of the hips, knees, feet and ankles as tracked by the 3D sensor during various forms of stair descent. The first column shows the trajectories of the hips and knees and the second column shows that of the feet and ankles. Figures (a-b), (c-d), and (e-f) show the trajectories during a normal stair descent, a fall and an overstep respectively. The deviation from the natural periodic gait is clear when comparing the trajectories of the knees and hips in a normal descent (a) to the abnormal descents (c, e). The estimated trajectories of the ankles and feet, however, are extremely noisy due to their partial occlusion by the rise of the stairs, which makes distinguishing abnormal events challenging.

102

chosen to be perpendicular to the ground (up and down direction) and y = 0 lied on the floor, the x axis was chosen to be perpendicular to a wall and to the y axis, and the z pointed towards the sensor. An example of the retrieved lower body tracking is shown

Biomed Eng Lett (2015) 5:98-108

in Fig. 2 and in Fig. 3 for a stairway descent and ascent respectively. Only the y and z coordinates are depicted in these figures. The position signals of the hips and knees are illustrated separately from the ankles and feet for a better visualization. Note that the temporal information is implicit

Fig. 3. This figure demonstrates example trajectories of the hips, knees, feet and ankles as tracked by the 3D sensor during various forms of stair ascent. Similarly to Fig. 2, the first column shows the trajectories of the hips and knees and the second column shows that of the feet and ankles. Figures (a-b), (c-d), and (e-f) show the trajectories during a normal stair descent, a fall and a trip respectively. Again, the deviation from the natural periodic gait is clear when comparing the trajectories of the knees and hips in a normal ascent (a) to the abnormal ascents (c, e). The occlusion of the ankles and feet by the rise of the stairs again results in unreliable tracking results, which makes distinguishing abnormal events more difficult.

Biomed Eng Lett (2015) 5:98-108

Fig. 4. Example of lower body tracking for a normal walk at ground level. A better tracking of the feet is performed when the joints are not occluded as in stair navigation.

in these graphs. That is, while, for instance, Figs. 2a and 3a appear similar to each other, one represents a descent with decreasing y and z values (when walking away from the sensor and downwards), and another represents an ascent with increasing y and z values (when walking up towards the sensor). In both normal descent and ascent, a similar structure is observed for the hips and knees and some periodicity is observable (Figs. 2a and 3a). By contrast, the trajectories of the ankles and feet appear to be noisier signals (Figs. 2b and 3b). Even in the presence of unusual events, the same behavior is observed for the hips and knees, except in the time segment where the abnormal event occurred. Again, such structure is hard to identify for the ankles and feet with visually inspection. Fig. 4 illustrates the position signal of the hips and feet of a subject normally walking at ground level towards the sensor. There, it is possible to observe a good tracking of the hips and feet. Hence, some inconsistencies in the tracking of the feet and ankles during stair navigation are attributed to the rise of the stairs.

METHODOLOGY The analysis of gait on the stairs begins with the estimation of the subject's walking frequency, a measure of the periodicity of the subject's gait. A sliding window equal to the length of a single gait cycle in time is then used to inspect the motion of lower body joints over time. Feature vectors composed of Fourier coefficients and entropy metrics are extracted and used to train a binary classifier on manually annotated data to distinguish between normal and abnormal gait cycles. The walking frequency (ω) is estimated as the periodic motion that best approximates the motion of all tracked body

103

parts, as proposed by Livne et al. [34]. Two methods two estimate the walking frequency were explored in [33] that are used here again in this study. The first method (ωˆ 1) is an exhaustive search and is therefore more accurate at the expense of being time consuming, while the second approach (ωˆ 2) is slightly less accurate but more computationally efficient. Shannon Entropy (ShEn) [35] and the Approximate Entropy (ApEn) [36] were used to quantify the structure of the motion during a normal gait. Fourier coefficients ( Fc ) were used as indicators of the periodic nature of human gait. The estimated walking frequency of the subject was used to set the length of the sliding window to one full gait. Features were computed from the velocity (time derivative) of each joint for each time segment. Similar to [33], several feature vectors Θ were composed and evaluated after trying 15 different combinations of the lower body joints, e.g., hips alone, hips and knees, hips and ankles and knees, etc. For each tracked body part, e.g., the left ankle, the features consisted of 3 Fourier coefficients, and 2 entropy values, extracted from 3 time series along the x, y, and z axes. This amounted to a total 30 features (left and right joints) per symmetric body part, i.e., for feet, the ankles, the knees, and the two points tracked on the hips. In reported results, the subscript denotes the tracked joints and Θf , Θa , Θk , and Θf refer to the 30-D feature vectors of the feet, the ankles, the knees, and the feet respectively. A well-defined supervised machine learning problem is sketched by the set of vectors for each time window and for various subjects, with the manually annotated ground truth labels. The training phase of this learning process is intended to obtain a mapping from a given vector to a binary label: normal vs. abnormal gait cycle. Ambiguous time windows, i.e., those during which the manual annotation switched from one label to another, were excluded from both the training and the test datasets. This conservative approach ensured that highly transient events, e.g., the exact moment of loss of balance, were not included in the dataset as those moments are typically highly idiosyncratic. Various binary classification algorithms were experimented with in [33] and the best performing one for the purpose of identifying falls on stairs, namely Random Forests (RF) [37], is used here in all reported results. All reported results are based on leave-onesubject-out cross validation.

RESULTS AND DISCUSSION In order to identify the best features, the classification process was first performed using only one type of feature, i.e., Fourier coefficients ( Fc = [ Fc1 , Fc2 , Fc3 ]), Shannon's entropy (ShEn) or Approximate entropy (ApEn), and then

104

Biomed Eng Lett (2015) 5:98-108

Table 1. Classification results in detecting abnormal events during stairway descent. ShEn Θ Θ h : hips features Θ k : knees features Θ a : ankles features Θ f : feet features [ Θh , Θk ] [ Θh , Θa ] [ Θh , Θf ] [ Θk , Θa ] [ Θk , Θf ] [ Θa , Θf ] [ Θh , Θk , Θa ] [ Θh , Θk , Θf ] [ Θh , Θa , Θf ] [ Θk , Θa , Θf ] [ Θh , Θk , Θa , Θf ]

ωˆ ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2

Accuracy (%) 92.99 90.63 90.60 88.25 85.13 84.31 80.04 76.77 74.62 81.68 77.77 76.49 93.36 91.32 91.26 93.49 91.56 91.18 93.58 91.55 91.72 89.40 87.15 87.14 89.44 87.74 87.77 82.52 79.11 77.52 93.41 91.71 91.52 93.67 91.92 91.82 93.52 91.77 91.53 89.62 87.72 87.77 93.73 91.95 91.86

[Fc, ApEn, ShEn]

False -ive rate

False +ve rate

8.38 11.08 9.91 14.66 18.32 18.68 29.98 34.46 33.80 27.61 32.58 31.50 6.98 10.07 9.03 6.79 9.31 9.05 6.90 9.82 8.83 13.43 17.15 16.27 13.32 15.92 15.16 27.13 31.60 30.76 6.74 9.19 9.08 6.69 9.07 8.65 6.61 9.25 9.01 12.98 16.93 15.61 6.50 9.09 8.67

5.99 7.99 8.93 9.61 12.08 12.92 12.55 14.14 17.57 11.44 13.87 16.11 6.38 7.55 8.47 6.31 7.73 8.60 6.07 7.34 7.76 8.51 9.37 9.71 8.52 9.30 9.52 10.33 12.22 14.80 6.48 7.55 7.93 6.07 7.28 7.74 6.39 7.41 7.98 8.45 8.52 9.09 6.10 7.20 7.65

using a combinations of two features (e.g., [ Fc , ShEn]), and finally using all features (i.e., [ Fc , ApEn, ShEn]). It was found that ShEn provided the most discriminative features. For comparison, in all experiments, three walking frequencies (ωˆ 1 or ωˆ 2 estimates, or the true value, ω, annotated manually) were used to set the size of the sliding window, their performance is compared later in this section. Analysis of the hips’ motion resulted in the highest

Accuracy (%) 93.12 91.46 91.53 88.60 86.47 85.89 81.23 77.92 76.11 82.24 79.27 78.10 93.28 91.76 91.73 93.44 91.93 92.07 93.55 91.72 92.03 89.05 87.74 87.40 89.46 88.31 87.70 82.40 79.92 77.88 93.67 92.08 92.18 93.57 92.06 92.26 93.42 91.81 91.45 89.31 87.77 87.65 93.49 91.64 92.06

False -ive rate

False +ve rate

6.54 9.97 8.93 14.15 16.62 15.30 30.73 34.81 34.10 29.01 32.75 30.61 6.24 8.92 8.28 6.30 8.71 8.27 6.36 9.33 8.28 14.12 16.67 15.37 13.82 15.30 14.76 28.95 32.45 32.05 6.12 8.67 8.15 5.96 9.04 8.08 6.56 9.16 8.84 14.43 16.80 15.40 6.18 9.62 8.44

7.13 7.38 8.05 9.36 11.02 13.00 9.92 11.78 14.44 9.43 11.01 13.84 7.08 7.69 8.26 6.75 7.54 7.62 6.51 7.42 7.68 8.61 8.69 10.03 8.12 8.77 10.01 9.19 10.07 12.92 6.48 7.32 7.51 6.78 7.06 7.43 6.59 7.40 8.28 7.93 8.54 9.51 6.76 7.34 7.48

classification performance, as compared with the knees, the ankles, or the feet. The knees were the second most discriminative joint. Better results were obtained using a combination of the features from the hip and knee joints, with a small improvement over using the hip joints alone. This behavior is expected after observing Figs. 2a and 3a, where a more accurate and stable tracking of those joints was performed compared to the ankles and feet.

Biomed Eng Lett (2015) 5:98-108

105

Table 2. Classification results in detecting abnormal events during stairway ascent. ShEn Θ Θ h : hips features Θk : knees features Θ a : ankles features Θ f : feet features [ Θh , Θk ] [ Θh , Θa ] [ Θh , Θf ] [ Θk , Θa ] [ Θk , Θf ] [ Θa , Θf ] [ Θh , Θk , Θa ] [ Θh , Θk , Θf ] [ Θh , Θa , Θf ] [ Θk , Θa , Θf ] [ Θh , Θk , Θa , Θf ]

ωˆ ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2 ω ωˆ 1 ωˆ 2

Accuracy (%) 91.95 89.84 88.83 90.78 87.98 86.90 83.04 80.21 79.26 82.10 78.97 78.05 93.49 91.25 90.56 93.30 90.96 90.11 93.69 90.89 89.98 92.48 90.30 89.33 94.03 92.11 91.14 94.24 92.14 91.27 94.03 92.11 91.14 94.24 92.14 91.27 93.54 91.48 90.24 92.59 90.61 89.56 94.30 92.34 91.28

[Fc, ApEn, ShEn]

False -ive rate

False +ve rate

14.34 16.84 17.83 15.64 19.07 20.55 29.68 33.09 34.43 34.07 40.42 41.39 10.35 14.62 15.08 10.44 14.70 15.18 9.75 15.01 15.69 12.43 15.49 16.37 9.41 13.27 14.12 8.97 13.33 14.18 9.41 13.27 14.12 8.97 13.33 14.18 10.29 14.08 15.54 12.01 15.22 16.38 9.01 12.80 14.08

5.27 7.12 8.07 6.40 8.81 9.64 11.35 13.74 14.38 10.77 12.21 12.90 4.81 6.08 6.82 5.06 6.46 7.43 4.79 6.43 7.37 5.36 7.07 8.02 4.45 5.44 6.41 4.34 5.37 6.20 4.45 5.44 6.41 4.34 5.37 6.20 4.77 5.99 7.07 5.38 6.74 7.68 4.24 5.32 6.22

The validation results, i.e., the accuracy levels, the percentage of missed events (a.k.a. false negative rate or type I error), and the percentage of false alarms (a.k.a. false positive rate or type II error) in detecting unusual events are presented in Tables 1 and 2 for stairway descents and ascents respectively. Results show good accuracy values in all the evaluated combinations, with a slightly better performance

Accuracy (%) 92.43 90.86 89.78 91.72 89.24 88.25 87.20 83.62 82.77 86.94 82.90 82.41 93.28 91.41 90.61 93.38 91.29 90.40 93.33 91.40 90.37 92.46 90.07 89.61 93.70 91.77 91.05 93.56 91.81 90.91 93.70 91.77 91.05 93.56 91.81 90.91 93.45 91.44 90.73 92.79 90.47 89.70 93.70 91.81 91.37

False -ive rate

False +ve rate

14.39 16.73 18.05 14.38 16.97 18.05 22.21 27.08 28.53 23.57 30.90 31.33 12.73 15.47 15.72 11.58 15.52 16.78 11.66 15.67 16.48 13.18 15.93 16.54 11.09 14.81 15.47 11.53 14.88 15.93 11.09 14.81 15.47 11.53 14.88 15.93 11.35 15.13 16.34 12.39 15.47 16.33 11.52 14.76 15.10

4.52 5.65 6.53 5.56 7.90 8.78 8.58 11.45 11.90 8.35 10.74 11.12 4.03 5.41 6.41 4.40 5.57 6.22 4.43 5.34 6.40 5.02 7.17 7.49 4.15 5.20 5.88 4.15 5.12 5.87 4.15 5.20 5.88 4.15 5.12 5.87 4.40 5.53 5.94 4.89 6.79 7.46 3.96 5.16 5.58

for stair ascents over stair descents. This was expected because, as illustrated in Figs. 2b, 3b and 4, a the feet were tracked better when the subject faced the sensor, i.e., when ascending the stairs. A high accuracy of 94.30% with ω, 92.34% with ωˆ 1 and 91.37% with ωˆ 2 was achieved for stair ascents, over the accuracy values of 93.73% with ω, 92.08% with ωˆ 1 and

106

92.26% with ωˆ 2 for stair descents. Results show that constructing the length of the sliding window based on the true walking speed resulted in the highest accuracy levels, however, by a small increment over the estimated walking frequency. Moreover, setting the gait cycle based on ωˆ 1 typically outperformed that of ωˆ 2, but only by a very small margin. For instance, on the last row of Table 1, the drop in accuracy levels from using ω to ωˆ 1 and ωˆ 2 is 1.85 and 1.43

Biomed Eng Lett (2015) 5:98-108

percentage points difference respectively. In the last row of Table 2 a similar output is observed if comparing ω to ωˆ 1 and ωˆ 2, with 1.89 and 2.33 percentage points respectively. This shows that only a small fraction of the overall classification error could be attributed to the inaccuracies in estimating the walking frequency. Results also show that estimating the walking frequency via the more computationally efficient approach (i.e., ωˆ 2) has little negative effect on the

Fig. 5. The ROC curves illustrate the true positive rate (i.e., the percentage of abnormal events correctly identified) plotted vs. the false positive rate (i.e., the false alarm rate) for different settings of the decision parameter on the classification result for all subjects during: (a) descents, and (b) ascents.

Fig. 6. Validation results of using all joints for each combination of features: (a) descents and (b) ascents. The classification process demonstrated that ShEn, or any combination including these features, obtained the best performance (i.e., high accuracy levels and low rates of false positives and missed abnormal events) in detecting abnormal events on stair navigation.

Biomed Eng Lett (2015) 5:98-108

overall classification accuracy, e.g., 0.09 percentage points in the last row of Table 1. A low rate of false positives was obtained for most of the combinations as shown in Tables 1 and 2. The lowest rate obtained by ω was 5.99%, 7.06% by ωˆ 1 and 7.43% by ωˆ 2 for stair descents. Even lower levels were achieved for stair ascents, where the lowest rate obtained by ω was 4.03%, 5.12% by ωˆ 1 and 5.58% by ωˆ 2. The validation results also show a low rate of false negatives for most of the combinations. The lowest rate during descent achieved by ω was 6.12%, 8.67% by ωˆ 1 and 8.44% by ωˆ 2 for stair descents. Surprisingly, higher false negative rates were obtained for stair ascents, where the lowest rate for ω was 8.97%, 12.80% for ωˆ 1 and 14.08% for ωˆ 2. As discussed earlier, both estimated frequencies reached similar results because only a small portion of the overall classification error can be attributed to the inaccuracies in estimating the walking frequency. In Fig. 5, the Receiver Operating Characteristic (ROC) curve for the binary classifier performance using a combination of all joints and all features is depicted for ωˆ 1 and ωˆ 2. The validation results for all experiments also showed that ShEn provides enough information to achieve high accuracy levels, together with low rates of false positives and missed events. In Fig. 6, the validation results of using all joints (i.e., Θh , Θk , Θa , Θf ]) for various combinations of features is depicted. The lowest accuracy was obtained when using ApEn, where lower than 75.0% was obtained for stair descents and ascents. The results in Fig. 6 also demonstrated that ShEn, or any combination of features containing these, obtained the lowest rate of false positives and false negatives. As described earlier, a more accurate tracking of the lower limbs is obtained for stair ascents, therefore, it is not surprising that good results were achieved when using Fc , more than 88.0% of correct classification compared to less than 82.0% for stair descents. Consequently, better results were obtained when using [ApEn, ShEn] over [ Fc , ShEn] for stair descents, and better results were achieved when using [ Fc , ShEn] over [ApEn, ShEn] for stair ascents.

CONCLUSIONS AND FUTURE WORK In this work a method to detect abnormal events during stair traversals, via the 3D motion analysis of lower limb joints, was proposed and experimentally validated. Overall accuracy levels of > 92% confirmed the effectiveness of the approach at identifying a diverse set of simulated abnormal events during stair navigation. Also, small false positive rates (~5.0%) were obtained together with low rates of missed abnormal events (~8.0%). Two approaches to estimate gait frequency based on [34] were also experimentally evaluated. Entropy metrics and Fourier coefficients were used to extract

107

features during a sliding window equal to the gait cycle. When the gait cycle length was estimated according to a more efficient approach, the features extracted resulted in only slightly lower accuracy levels in detecting unusual events. Validation results showed that features based on Shannon's entropy provide enough information to achieve high accuracy levels and therefore, they outperformed features based on approximate entropy or Fourier coefficients. When considering the motion of a single joint, the hips provided the best discrimination of abnormal events as compared to the knees, the ankles, and the feet. This observation differs with previous results reported by Snoek et al. [4], which had identified the visual tracking of the feet as the most discriminating motion in identifying dangerous situations in stairs. The discrepancy can be explained by the variation in the viewing angle. Because the sensor in our study was mounted at the top of the flight of stairs, the feet and the ankles were partially occluded from the viewing angle of the Kinect. Fortunately, the use of 3D information facilitates the accurate tracking of the hips, which is challenging using a monocular camera. Still, better results were obtained for the feet and ankles when the subject faced the sensor (stair ascent), as compared to when the subject walked away from it (stair descent). Ongoing work involves the installation of the system in a busy public stairway at the Toronto Rehabilitation Institute and the use of the system to automatically tag a small subset of traversals as real falls or potential abnormal events for further study over an extended period of time. This involves overcoming the engineering challenges to prepare a robust system for real-world deployment and evaluating and extending the results obtained in the home laboratory to the real-world data.

HUMAN RIGHTS STATEMENTS AND INFORMED CONSENT This study and all of its procedures were deemed to be exempt from a research ethics review by the authors’ institution as it was considered to fall under the category of “quality improvement”. All data used were collected from the research team members and were used for the primary purpose of improving the technology described in this paper.

ACKNOWLEDGEMENTS This work was supported by Mobility in Aging Emerging Team Grant #MAT-91865 from the Canadian Institutes of Health Research (CIHR) and MITACS (mitacs.ca).

108

Biomed Eng Lett (2015) 5:98-108

CONFLICT OF INTEREST STATEMENTS Parra-Dominguez GS declares that she has no conflict of interest in relation to the work in this article. Snoek J declares that he has no conflict of interest in relation to the work in this article. Taati B declares that he has no conflict of interest in relation to the work in this article. Mihailidis A declares that he has no conflict of interest in relation to the work in this article.

[16]

[17] [18] [19]

REFERENCES

[20]

[1] Population aging and development, Report by the Population Division, Department of Economic and Social Affairs, United Nations. 2013. [2] Marshall SW, Runyan CW, Yang J, Coyne-Beasley T, Waller AE, Johnson RM, Perkis D. Prevalence of selected risk and protective factors for falls in the home. Am J Prev Med. 2005; 28(1):95-101. [3] Robinovitch S, Feldman F, Wan D, Aziz O, Sarraf T. Videobased recording of the cause of falls among older adults in long term care. Int Conf Soc Posture Gait Res. 2009. [4] Snoek J, Hoey J, Stewart L, Zemel RS, Mihailidis A. Automated detection of unusual events on stairs. Image Vision Comput. 2009; 27(1-2):153-66. [5] Taati B, Snoek J, Mihailidis A. Video analysis for identifying human operation difficulties and faucet usability assessment. Neurocomputing. 2012; 100:163-9. [6] Taati B, Wang R, Huq R, Snoek J, Mihailidis A. Vision-based posture assessment to detect and categorize compensation during robotic rehabilitation therapy. IEEE Conf Biomed Robotic Biomechatronics. IEEE. 2012; 1607-13. [7] Belshaw M, Taati B, Giesbrecht D, Mihailidis A. Intelligent vision-based fall detection system: preliminary results from a real-world deployment. RESNA/ICTA: Rehabil Eng Technol Soc N Am - Int Conf Technol Aging. 2011. [8] Belshaw M, Taati B, Snoek J, Mihailidis A. Towards a single sensor passive solution for automated fall detection. Conf Proc IEEE Eng Med Biol Soc. 2011; 1773-6. [9] Shotton J, Fitzgibbon A, Cook M, Sharp T, Finocchio M, Moore R, Kipman A, Blake A. Real-time human pose recognition in parts from single depth images. PROC CVPR IEEE. 2011: 1297-304. [10] Ni B, Wang G, Moulin P. RGBD-HuDaAct: A color-depth video database for human daily activity recognition. IEEE Int Conf Comp Vision W. 2011; 1147-53. [11] Sung J, Ponce C, Selman B, Saxena A. Human activity detection from RGBD images. AAAI W Pattern Activity Intent Recog. 2011; 47-55. [12] Dolatabadi E, Taati B, Parra-Dominguez GS, Mihailidis A. A markerless motion tracking approach to understand changes in gait and balance: A case study. RESNA/ICTA: Rehabil Eng Technol Soc N Am - Int Conf Technol Aging. 2013. [13] Stone EE, Skubic M. Passive in-home measurement of stride-tostride gait variability comparing vision and kinect sensing. Conf Proc IEEE Eng Med Biol Soc. 2011; 6491-94. [14] Taati B, Campos J, Griffiths J, Gridseth M, Mihailidis A. Vision-based categorization of upper body motion impairments and post-stroke motion synergies. Int J Disabil Hum De. 2014. [15] Czarnuch S, Cohen S, Parameswaran V, Mihailidis A. A real-

[21] [22] [23] [24] [25] [26] [27] [28] [29]

[30]

[31]

[32]

[33] [34] [35] [36] [37]

world deployment of the coach prompting system. Amb Int Sm Env. 2013; 5(5):463-78. Kepski M, Kwolek B, Austvoll I. Fuzzy inference-based reliable fall detection using kinect and accelerometer. Artif Int Soft Comput, ser. Lect Notes Comput Sc. Springer Berlin/ Heidelberg. 2012; 7267:266-73. Mastorakis G, Makris D. Fall detection system using kinect’s infrared sensor. J Real-Time Imag Process. 2012; 1-12. Rougier C, Auvinet E, Rousseau J, Mignotte M, Meunier J. Fall detection from depth map video sequences. Lect Notes Comput Sc. 2011; 6719:121-8. Stone E, Skubic M. Fall detection in homes of older adults using the microsoft kinect. IEEE J. Biomed. Health Inform. 2014; (99):1. Marzahl C, Penndorf P, Bruder I, Staemmler M. Unobtrusive fall detection using 3d images of a gaming console: concept and first results. Amb Assist Living. 2012; 135-46. Gasparrini S, Cippitelli E, Spinsante S, Gambi E. A depth-based fall detection system using a Kinect sensor. Sensors. 2014; 14(2):2756-75. Planinc R, Kampel M. Introducing the use of depth data for fall detection. Pers Ubiquit Comput. 2013; 17(6):1063-72. Bian Z, Hou J, Chau L, Magnenat-Thalmann N. Fall detection based on body part tracking using a depth camera. IEEE J Biomed Health Inform. 2015; 19(2):430-9. Kepski M, Kwolek B. Human fall detection using kinect sensor, ser. Adv Soft Comp. 2013; 743-52. Webster D, Celik O. Systematic review of kinect applications in elderly care and stroke rehabilitation. J Neuroeng Rehabil. 2014; 11(1):108. Stone E, Skubic M. Evaluation of an inexpensive depth camera for in-home gait assessment. Amb Int Sm Env. 2011; 3(4):349-61. Dolatabadi E, Taati B, Mihailidis A. Vision-based approach for long-term mobility monitoring: Single case study following total hip replacement. J Rehabil Res Dev. 2014; 51(7):1165-76. Costa M, Peng CK, Goldberger AL, Hausdorff JM. Multiscale entropy analysis of human gait dynamics. Phys A Statistical Mech Appl. 2003; 330(1-2):53-60. Georgoulis A, Moraiti C, Ristanis S, Stergiou N. A novel approach to measure variability in the anterior cruciate ligament deficient knee during walking: the use of the approximate entropy in orthopaedics. J Clin Monitor Comput. 2006; 20:11-8. Pansera M, Estrada J, Pastor L, Cancela J, Greenlaw R, Arredondo M. Multi-parametric system for the continuous assessment and monitoring of motor status in parkinson’s disease: An entropy-based gait comparison. Conf Proc IEEE Eng Med Biol Soc. 2009; 1242-45. Karmakar C, Khandoker A, Begg R, Palaniswami M, Taylor S. Understanding ageing effects by approximate entropy analysis of gait variability. Conf Proc IEEE Eng Med Biol Soc. 2007; 1965-68. Khandoker A, Palaniswami M, Begg R. A comparative study on approximate entropy measure and poincare plot indexes of minimum foot clearance variability in the elderly during walking. J NeuroEng Rehabil. 2008; 5(1):4. Parra-Dominguez GS, Taati B, Mihailidis A. 3d human motion analysis to detect abnormal events on stairs. Int Conf 3D Imag Model Process. Visual Transm. 2012. Livne M, Sigal L, Troje NF, Fleet DJ. Human attributes from 3d pose tracking. Comput Vis Imag Und. 2012; 116(5):648-60. Shannon C, Weaver W. The mathematical theory of communication. University of Illinois Press. 1975. Pincus SM. Approximate entropy as a measure of system complexity. P Natl Acad Sci. 1991; 88(6):2297-301. Breiman L. Random forests. Mach Learn. 2001; 45:5-32.