J Ambient Intell Human Comput DOI 10.1007/s12652-017-0571-8
ORIGINAL RESEARCH
Fuzzy association rule mining for recognising daily activities using Kinect sensors and a single power meter Hossein Pazhoumand‑Dar1
Received: 26 April 2017 / Accepted: 24 August 2017 © Springer-Verlag GmbH Germany 2017
Abstract The recognition of activities of daily living (ADLs) by home monitoring systems can be helpful in order to objectively assess the health-related living behaviour and functional ability of older adults. Many ADLs involve human interactions with household electrical appliances (HEAs) such as toasters and hair dryers. Advances in sensor technology have prompted the development of intelligent algorithms to recognise ADLs via inferential information provided from the use of HEAs. The use of robust unsupervised machine learning techniques with inexpensive and retrofittable sensors is an ongoing focus in the ADL recognition research. This paper presents a novel unsupervised activity recognition method for elderly people living alone. This approach exploits a fuzzy-based association rule-mining algorithm to identify the home occupant’s interactions with HEAs using a power sensor, retrofitted at the house electricity panel, and a few Kinect sensors deployed at various locations within the home. A set of fuzzy rules is learned automatically from unlabelled sensor data to map the occupant’s locations during ADLs to the power signatures of HEAs. The fuzzy rules are then used to classify ADLs in new sensor data. Evaluations in real-world settings in this study demonstrated the potential of using Kinect sensors in conjunction with a power meter for the recognition of ADLs. This method was found to be significantly more accurate than just using power consumption data. In addition, the evaluation results confirmed that, owing to the use of fuzzy logic, the proposed method tolerates real-life variations in
* Hossein Pazhoumand‑Dar
[email protected] 1
Edith Cowan University Faculty of Health Engineering and Science, Perth, WA, Australia
ADLs where the feature values in new sensor data differ slightly from those in the learning patterns. Keywords Activities of daily living · Kinect depth maps · Fuzzy association rule mining · 2D fuzzy sets · Activity classification
1 Introduction The latest statistics on world population show that population ageing has become a global phenomenon (World Health Organization 2015). Also, older adults are reported to prefer to live independently in their own homes and maintain control of their lives as long as possible (Claes et al. 2015). This requires them to be capable of independently performing activities of daily living (ADLs) such as walking or cooking- i.e. routine activities that are part of an individual’s everyday life (Xiang et al. 2015). While the advantages of enabling older people to stay in their own homes are apparent, there are also associated risks. In particular, gradual cognitive decline can leave an elderly person more vulnerable to household accidents. The deployment of sensor-based systems that can recognise the performance of ADLs has become prominent as these systems can provide the longterm profile of the monitored person’s ADLs and facilitate early detection of their cognitive decline. The use of household electrical appliances (HEAs) such as microwaves and toasters provides inferential information that can facilitate the recognition of elderly people’s ADLs (Yang and Hsu 2012). For example, turning on a hair dryer implies that the house occupant is grooming, or turning on an electrical stove indicates a cooking activity. ADL recognition based on the use of HEAs involves fitting sensors into the house and mapping low-level features
13
Vol.:(0123456789)
H. Pazhoumand‑Dar
from sensor data to human activities. However, existing algorithms face several practical challenges. Many of them [e.g. the methods presented by Clement et al. (2014) and Wilson et al. (2015)] cannot address real-life variations in the occupant’s habitual performance of ADLs and require a labelled dataset or prior knowledge about the characteristics of appliances. Another challenge is equipping an existing house with numerous sensors. Finally, it is difficult to differentiate between the uses of devices that have similar electrical consumption patterns. This paper presents a novel, person-tailored fuzzy method for the recognition of daily activities (FMR-ADL) based on combining the occupant’s locations with the power consumption of HEAs, without any prior knowledge of the appliances. The depth maps obtained from a few Microsoft Kinect sensors provide the occupant’s physical locations while the house composite power consumption is captured by one power sensor from the house electricity panel box. An unlabelled dataset from the sensors is processed by a fuzzy association rule-mining algorithm to map power signatures generated by HEAs to the physical locations of the occupant. This fuzzy algorithm integrates fuzzy logic with association rule-mining to address the problem of the uncertainty in sensor measurements and variations during real-life ADLs. The mapping is used in the classification stage of FMR-ADL to verify whether a power signature observed on the power line has resulted from the occupant interacting with HEAs and to classify it as an ADL. To the researcher’s knowledge, this is the first ADL recognition method that combines data from Kinect sensors with a single power meter. The novelty of this paper is that it describes a new fuzzy-based method to combine a person’s locations with the power signatures of HEAs for detecting human activities, where real-life variations in the person’s ADL habits are taken into consideration. Unlike existing supervised ADL recognition approaches that require the user to label all instances of ADLs in a training dataset, after the training, FMR-ADL only needs the user to label each association rule as one activity. Furthermore, a novel technique is presented in this paper to generate 2D fuzzy membership functions over ADL attributes, with the number of required membership functions learned from the training data. Parameters associated with these membership functions are learned automatically from the distribution of attributes. The rest of the paper is organised as follows Sect. 2 comprises a review of the relevant literature on ADL recognition in healthcare. Section 3 explains the stages of the FMR-ADL approach. The experimental evaluation results of this method are presented in Sect. 4, followed by conclusions and future directions in Sect. 5.
13
2 Related work Researchers have employed different sensor data in supervised and unsupervised machine learning techniques in order to recognise ADLs. Several studies have reported the use of wearable sensors and video cameras. For example, Gu et al. (2009) asked participants to wear a radio frequency identification (RFID) tag reader on their wrists to identify when the individual used a tagged object during activities. Zhan et al. (2015) asked people to wear both an accelerometer and a video camera to recognise ADLs based on their vision and head motion. A graphical model was trained using a labelled dataset of video data, while support vector machines were trained in a supervised manner for the acceleration features. Another study by Chung and Liu (2008) segmented the image of the person in video frames and classified activities by statistically analysing body postures. A labelled dataset combining the person’s postures with the location and temporal duration of each activity was employed to develop a hierarchical behaviour model, capable of detecting abnormal activities. Video cameras are, however, intrusive and the problem with wearable sensors is that the elderly may forget to put them on. ADL recognition techniques have proposed instrumenting the house with different environmental sensors such as passive infrared motion detectors (PIRs) and magnetic switches to detect changes to the objects caused by human interactions. Krishnan and Cook (2014) employed a wireless network of PIR sensors in a home environment to obtain sequences of sensor events resulting from human movements in the house. Those sequences were used to infer appliance usage, resident activities, and the mobility of the person. Data obtained from PIR sensors, however, only indicate the occupancy of rooms and using them to infer ADLs is not generally accurate in real-life settings. Another study by Mehr et al. (2016) investigated state-change sensors attached to objects in an apartment. The study used labelled data obtained from sensors and compared the ADL classification performance of different training algorithms of artificial neural networks (ANNs). The ANNs trained by the Levenberg Marquardt training algorithm achieved 92.81% activity recognition accuracy. Several studies investigated the use of environmental sensor data for ontological activity recognition and the prediction of a person’s intention (Gayathri et al. 2017; Noor et al. 2016). Ontological activity recognition employs an explicit representation of activities through structuring them into a hierarchy of related sub-activities (actions) and concepts. For example, using kettle is a subclass of make tea. Sub-activities can have relationships and concepts can have properties with restrictions. Chen et al. (2012) introduced an ontology-driven approach for activity recognition based on occupant-object interaction
Fuzzy association rule mining for recognising daily activities using Kinect sensors and a single…
data in home settings. A library of behaviours was defined manually to represent the knowledge of ADLs within a hierarchy of tree levels. The first level indicated information related to the time, location and the actor of actions. The second level represented the functional relations of actions involved in an activity and the third level encoded the interrelationship between activities. Evaluation of this approach based on a dataset of several activities collected from 40 environmental sensors achieved an average activity recognition rate of 94.44%. Rafferty et al. (2017) proposed an intention recognition mechanism in which environmental sensors were used in the monitored home to indicate the intention of the occupant based on the use of objects. A library of goal hierarchy was encoded through the use of ontologies, where sensor activations represented atomic actions being performed by the occupant. A predictive reasoning technique subsequently determined the most expected goal of the occupant by considering recent atomic actions and related actions carried out within a specific interval. An average of 82.5% intension recognition accuracy was achieved when this approach was evaluated across different homes retrofitted with sensors. The use of computational state space models (CSSMs) has shown promising results for identifying activities and intensions of home occupants (Krüger et al. 2014). CSSMs are probabilistic models in which the underlying dynamic system is encoded based on a symbolic representation of activities. The method presented by Röhlig et al. (2015) employed CSSMs in order to segment and label the stream of sensor data with the monitored person’s executed actions and intensions. ADLs in this method were encoded through an ontology, and CMMs determined the occupant’s intensions based on the list of observed actions. Whitehouse et al. (2016) also explored the application of CSSM to intension recognition of unscripted kitchen activities. A set of symbolic rules reasoned in a probabilistic manner about the occupant’s intension based on the set of identified actions and initial state of objects involved in the actions. The evaluation of the approach showed an overall intension recognition accuracy of 80% when a specified model of meal preparation was used. However, although techniques based on environmental sensors are accurate, they generally entail installing numerous sensors in the house. Electrical utility sensors have been employed in ADL classification techniques to identify the use of HEAs through their consumption patterns (Berenguer et al. 2008; Gaddam et al. 2011; Noury et al. 2011). Studies by Suryadevara et al. (2012) and Cho et al. (2010) proposed the installation of separate power sensors for each HEA with data related to the operating status of the appliance transmitted wirelessly to a computer. ADLs were recognised based on the function and location of the appliance connected to the sensor and time of use. Installing and maintaining separate sensors for
each device, however, increases the cost and complexity of the system. Another alternative approach for monitoring ADLs is through nonintrusive load monitoring (NILM) from the electricity panel outside the house. A power meter is installed in the panel and a training dataset of the house power consumption is collected to identify power signatures associated with the use of each HEA. Several studies have used this approach for the classification of ADLs. For example, Rahimi et al. (2011) demonstrated the application of a supervised NILM system that was trained to map the power signatures of various electrical devices into ADLs based on an annotated dataset. The advantage of this method was that the environment in the home was not altered as monitoring occurred through the external power meter box. A similar study by Noury et al. (2011) defined relationships between ADLs and electrical appliances in a monitored house. The method mapped power signatures detected on the power line to the use of HEAs in the home using an annotated training dataset of house power consumption. It then identified the performance of ADLs from the relationships between ADLs and appliances used. Belley et al. (2014) also proposed the recognition of ADLs by measuring the power signatures of appliances from the external power meter box. A k-nearest neighbourhood classifier identified the used appliances based on an annotated dataset of power signatures. The ADLs were estimated based on the function and location of the appliances with which users interacted. A more recent study by Clement et al. (2014) presented another supervised approach, based on hidden Markov models, to detect the performance of ADLs from analysing annotated power meter data. The occupant’s ADL was inferred based on the similarity of the sensor data to the learned model of each target activity. Existing activity recognition methods via NILM face several challenges. Many of these methods are based on supervised learning and cannot address variations in reallife ADLs. The classifiers in these methods also need to be trained with long traces of data (ranging from months to years) to ensure high recognition accuracy and labelling a dataset for such a long period is difficult. Many of the existing methods are also not accurate since they do not filter out power signatures generated automatically by self-regulated HEAs. Two or more HEAs being used for different ADLs may generate similar power signatures and become virtually indistinguishable when only their power signatures are processed.
3 Proposed approach The FMR-ADL method is comprised of three stages (1) data acquisition and feature extraction, (2) extracting fuzzy
13
H. Pazhoumand‑Dar
classification rules, and (3) classifying new data into ADLs. The first stage takes a training dataset from a combination of a single power meter and a few Kinect sensors, each covering a different functional subarea in the house. This stage then extracts features from the data including the physical location of the occupant and the power consumption of appliances. The second stage converts the extracted features into fuzzy linguistic terms. A fuzzy association rule-mining algorithm is then employed to generate fuzzy rules that associate frequently repeating power signatures of HEAs with the occupant’s locations during ADLs. These fuzzy rules are labelled with ADLs and are employed in the third stage to recognise ADLs based on the use of appliances. In the rest of this section, the steps of the FMR-ADL approach are detailed. 3.1 Data acquisition and feature extraction This stage is initiated by the collection of an unlabelled dataset of the occupant’s locations and the home’s composite power consumption. Features are extracted from each source of data after pre-processing operations. 3.1.1 Power consumption data A single power meter is installed in the main electricity panel of the house to measure the voltage (V), current (I) and phase angle (𝜑) from the power line at 1-s intervals. The phase angle is caused by the operation of inductive and capacitive loads and indicates the angle by which the voltage leads the current. In contrast to conventional intrusive load-monitoring techniques that require putting sensors on each appliance, this configuration is achievable: most of the inexpensive power sensors can provide the specified measurements at 1-s intervals and installing the sensor does not entail access to each electrical appliance in the house. The real and reactive powers are calculated from each power sensor measurement t using Eqs. 1 and 2, respectively. These are typical features used by many existing NILM methods (Rahimi et al. 2011) to characterise the electricity consumption of HEAs. ( ) P(t) = V(t) ⋅ I(t) Cos 𝜑(t) , (1) ( ) Q(t) = V(t) ⋅ I(t) Sin 𝜑(t) . (2) Real power (P), expressed in watts (W), provides load characteristics as every usage instance of an HEA results in a step change in the composite real power consumption of the house. Reactive power (Q) is expressed in volt-ampere reactive (VAR) and is associated with capacitive and inductive loads. FMR-ADL extracts power signatures from only the turnon events of HEAs using a technique proposed by Rahimi
13
et al. (2011). The reason to consider only the turn-on events of HEAs is that many HEAs, such as toasters and washing machines, automatically stop operating after a certain period of time and generate a turn-off event. The real power signal is processed and every positive step-change in two consecutive values that exceeds a threshold α is detected as a usage event. For each usage event, the amount of step change in both real dP and reactive dQ power signals along with their timestamp are stored in a list named Usage_Events. An example of seven records in Usage_Events is shown in Table 1. 3.1.2 Depth maps Acquiring the occupant’s physical locations in the monitored environment is an essential task in FMR-ADL. This is carried out using the latest version of Microsoft Kinect v2 sensors associated with the Xbox One gaming consoles. The home is divided into several functional subareas, based on the location of HEAs used in activities, and a Kinect sensor is installed in each subarea to provide the occupant’s location. Each Kinect v2 sensor must be hosted on a separate computer with at least 4 Gigabyte RAM, a 64-bit processor and a USB 3 port (Kinect for Windows SDK 2.0). The Kinect v2 sensor contains a 70° horizontal by 60° vertical field of view wide-angle depth sensor. The skeletal tracking feature of Kinect SDK uses the acquired depth maps to track as many as six individuals, and provides 3D positions of 25 skeleton joints per tracked person at a frame rate of 30 Hz (Kinect for Windows SDK 2.0). Colour images showing a person sitting on a sofa in a living room and using an electric cooktop in a kitchen are shown in Fig. 1a, c, respectively. Figure 1b, d show respectively the corresponding depth maps of Fig. 1a, c, with the locations of the person’s skeleton joints obtained from the Kinect sensor depicted on the depth map of the person. The location of the head joint in Fig. 1b, d is shown via the red rectangle. Note that the higher depth values are displayed through brighter pixels in these depth maps. To represent the 3D coordinates of skeleton joints, the Kinect sensor uses a Cartesian coordinate system Table 1 An example of records in Usage_Events dP
dQ
Timestamp
45 1700 115 1752 915 453 42
0 0 44 651 0 0 0
15-Dec-16, 09:15:04 15-Dec-16, 11:35:47 15-Dec-16, 12:25:14 15-Dec-16, 18:42:43 15-Dec-16, 21:11:18 15-Dec-16, 09:23:09 15-Dec-16, 11:40:34
Fuzzy association rule mining for recognising daily activities using Kinect sensors and a single…
Fig. 1 Examples of Kinect sensor data a, c show the colour images of a person sitting on a sofa in a living room and using an electric cooktop in a kitchen; b, d respectively show the corresponding depth map of part a and part c including the skeleton of the detected person
y
x
z Fig. 2 Kinect sensor’s x, y, and z coordinates for representing the position of skeleton joints
centred at the sensor, as shown in Fig. 2. The positive y axis extends upward, the positive z axis points along the viewing direction, and the positive x axis extends to the left. The values of joint positions in the x and y axes range from approximately −2.2 to +2.2 and −1.6 to +1.6, respectively. The values of positions in the z axis range from 0.0 to 4.5 indicating the range of tracking in metres. This allows a single Kinect v2 sensor to effectively monitor most of a regular-size room. Person localisation using this type of skeletal tracking method is nonintrusive, in comparison with wearable sensors that the home occupant must remember to put on. Each Kinect sensor is given a unique ID to represent the room being monitored by the sensor. Observations from each Kinect sensor are taken, and those in which a person is detected are stored. Each stored observation includes 3D positions of 25 skeletal joints for the tracked occupant, a timestamp, and the respective Kinect ID. Only the positions
13
H. Pazhoumand‑Dar
the data. A fuzzy association rule-mining algorithm is then employed to establish associations between the occupant’s locations and the power signatures. This results in a set of rules which have a location and a power signature as their antecedent and consequent, respectively. This rule set is used to recognise the occupant’s ADLs in case the occupant’s current location is associated with a detected power signature on the power line. Fig. 3 An example of the original signal of the occupant’s head joint in the x axis along with its smoothed values Table 2 Example of Occupant_Locations consisting of seven records Kinect ID
x
z
Timestamp
K K K L L B K
0.4973 −0.4760 −0.5653 −0.4829 0.4480 0.3475 0.1322
2.4705 1.9812 1.9355 2.0520 2.3061 2.2758 2.4330
15-Dec-16, 08:47:04 15-Dec-16, 11:35:47 15-Dec-16, 15:25:11 15-Dec-16, 18:42:43 15-Dec-16, 21:11:18 15-Dec-16, 09:23:09 15-Dec-16, 11:40:34
of the occupant’s head joint in the x and y axes are stored, and the information from other joints is discarded. The sequence of raw joint positions is found to contain high frequency jitters and temporary spikes. The noise in the obtained head positions is removed using a variant of the Holt Double Exponential Smoothing method (Kalekar 2004). This method provides smoothing with less latency than in other signal smoothing techniques. The red line in Fig. 3 shows an example of the original head positions of the occupant in the x axis and the blue diagram shows their smoothed values. This figure shows that significant jitters are eliminated—in particular those occurred after Frame 130 in the original signal. The positions of the occupant’s head joint from all frames acquired in 1-s are averaged along each of the axes to obtain the occupant’s location at 1-s intervals. The calculated occupant’s locations in the scene along with their timestamps and Kinect IDs are stored in a list named Occupant_Locations. An example of Occupant_Locations consisting of seven records is shown in Table 2. Note that the timestamp of some records can be matched to those in Table 1. 3.2 Extracting fuzzy classification rules This section details techniques employed in Stage 2 of FMR-ADL to extract ADL classification rules. First, features extracted from each source of data are transformed into fuzzy linguistic labels, parameters of which are learnt from
13
3.2.1 Fuzzification of power signatures The power signatures of an HEA may have variations in the P–Q space due to the noise on the power line and errors in sensor measurements. This step involves defining 2D fuzzy sets to model fine variations in the power signatures of HEAs in the P-Q space. A variant of the DBSCAN clustering technique (Ester et al. 1996) is used to group the power signatures resulting from the operation of each HEA. This clustering algorithm does not need the number of clusters in advance and can detect outlier data while clustering. A 2D fuzzy set is then defined for each detected cluster to represent the variation of power signatures in the cluster. The DBSCAN clustering algorithm assumes power signatures to be data points in the P–Q space and groups those that are density-reachable from each other into a cluster. It has two parameters, namely MinPnts (the minimum number of power signatures required to form a cluster) and radius (the radius in which two points in a cluster are reachable). The clustering algorithm selects an arbitrary unprocessed power signature in Usage_Events and calculates the neighbourhood radius for the point. In FMR-ADL, two radius values are calculated for each power signature to define neighbourhood distances of that power signature in both real and reactive powers. The selected power signature is labelled as processed and all neighbours from that power signature that fall within the neighbourhood distances are selected and grouped into the same cluster. This process is repeated until no unprocessed power signature is left. At the end of the process, those clusters that have fewer than or equal to a specific number of data points (i.e. MinPnts) are labelled as noise. MinPnts is set to one to tag data points for infrequent situations where two or more turn-on events occur at the same time. Such occasions tend to generate a single unusual data point in the P–Q space and therefore a new cluster should not be created based on them. Once all the power signatures in Usage_Events are labelled with cluster IDs, every power signature labelled as noise is eliminated from the dataset. A fuzzy set with a 2D Gaussian membership function is then defined for each remaining cluster to represent the variations of the cluster members. The parameters of the fuzzy set for each cluster are calculated based on the statistical characteristics of data points associated with the cluster.
Fuzzy association rule mining for recognising daily activities using Kinect sensors and a single…
Fig. 4 An illustration of a 2D membership function defined based on mP, mQ, 𝜎P, 𝜎Q
{ } Let PS = ps1 , … , psn be a set of power signatures grouped as the same cluster and mP, mQ be the average of the real and reactive powers of the cluster members, respectively. ( ) Each power signature in PS is specified by pair dP , dQ where dP and dQ represent the real and reactive powers of the power signature. Also, let 𝜎P and 𝜎Q denote the variance of real and reactive powers of the cluster members, respectively. The 2D membership function of the fuzzy set that represents the cluster of these power signatures is defined using Eq. 3. ( (( ) ( ) )) dQ − mQ 2 ( ) dP − mP 2 . 𝜇PS dP , dQ = exp − + 𝜎P 𝜎Q (3) Figure 4 shows an example of a 2D membership function for a group of power signatures. The impact of parameters (i.e. mP, mQ, 𝜎P, 𝜎Q) on the location and shape of the membership function is shown on the figure. Figure 5a illustrates an example of three groups of power signatures for operation modes of HEAs; two groups of power signatures are located between 20 and 50 watts close to each other, although their power signatures are generated by two different HEAs. The other cluster, shown in red, is scattered vertically along the P axis at nearly 70 VAR. The fuzzy sets generated using Eq. 3 to represent these groups of power signatures are shown in Fig. 5b. The distribution of the fuzzy set to the right is skewed along the P axis to represent the cluster in red. Note that the generated membership functions cover slightly greater areas in the P–Q space than their respective clusters of training data points. This caused the membership functions associated with the green and blue clusters in Fig. 5 to overlap, whereas those clusters are separated in the P–Q space. This means that the generated membership functions can tolerate slight variations in unseen data without labelling them as noise. Once fuzzy sets are defined over the P–Q space, each set is labelled with an arbitrary linguistic term. The membership degrees of each power signature in Usage_Events
Fig. 5 a Example of three groups of power signatures in the P–Q space. b The respective 2D fuzzy Gaussian membership functions are defined to represent the groups of power signatures. The z axis in b represents the degree of membership
to different fuzzy sets are then calculated and the fuzzy term for the set with the maximum membership degree is added to the power signature’s entry in Usage_Events. 3.2.2 Fuzzification of the occupant’s locations A home occupant usually visits specific physical locations to interact with HEAs in the house. The physical locations that are visited in order to interact with a specific HEA are usually close to each other (Huang et al. 2009). This step of FMR-ADL involves two operations: to define 2D fuzzy sets representing the occupant’s locations when interacting with HEAs, and to process Usage_Events so each entry in that list includes the respective fuzzy term for the occupant’s location. The DBSCAN clustering algorithm is again employed here to define 2D fuzzy sets representing the occupant’s locations. The input to this algorithm comprises those data points in Occupant_Locations whose timestamps correspond to those of power signatures in the processed Usage_Events. The DBSCAN clustering algorithm groups the occupant’s locations based on their Euclidean distances and outputs a cluster ID for each data point. Radius is measured directly from the monitored house to indicate the radius of space normally used by the occupant during interactions with an HEA (the “usage zone”). Two data points whose Euclidean distance from each other is less than radius are grouped into the same cluster. A regular choice for this parameter would
13
H. Pazhoumand‑Dar
be 50 cm. The distance between two physical locations of the occupant is measured through performing a calibration procedure on the Kinect sensor (Webb and Ashley 2012). HEAs occupying a larger space cause larger clusters to be created, as clusters in the DBSCAN algorithm can have arbitrary sizes and shapes. A fuzzy set is defined for each cluster of the occupant’s locations to represent members of the cluster. The statistical characteristics of the data points of each cluster are calculated to define a 2D Gaussian membership function for the { } fuzzy set. Let L = l1 , … , ln be a set of the occupant’s locations grouped as the same cluster and mx, mz be the average of these locations in x and z axes, respectively. ( )Each of the occupant’s locations is specified by pair xl , zl where xl and zl indicate the location in the Kinect sensor’s x and z coordinates. Also, let 𝜎x and 𝜎z denote the variance of these locations in x and z axes, respectively. The 2D membership function of the fuzzy set that represents the cluster of these locations is defined using Eq. 4. ( (( ) ( ) )) ( ) zl − mz 2 xl − mx 2 . 𝜇L xl , zl = exp − + 𝜎x 𝜎z (4) The fuzzy sets defined to represent visited locations are labelled with arbitrary linguistic terms. For each occupant’s location in the dataset, the membership degrees of different fuzzy sets are calculated and the fuzzy term for the set with the maximum membership degree is assigned to the occupant’s location. For example, L01 and L02 can be two fuzzy labels given to the groups of locations in the kitchen. Similarly, L03 and L04 can correspond to groups of locations which the occupant visits most often in order to manipulate a TV and a computer in a living room, respectively. The second operation in this step processes each data point in Usage_Events to include the corresponding fuzzy term of the occupant’s location using their timestamps. Those data points in Usage_Events that do not have an occupant location are eliminated from the list as they cannot help to associate power signatures with the occupant’s locations. The corresponding label of the occupant’s location is then added to each of the remaining data points in Usage_Events. After this operation, each record in Usage_Events has a power signature label, a label for the occupant’s location, and a timestamp. 3.2.3 Generating fuzzy classification rules This stage takes Usage_Events as input and employs a fuzzy association rule-mining algorithm (Kuok et al. 1998) to establish associations between the occupant’s locations and power signatures that co-occur frequently
13
enough. These association rules are then labelled with ADLs by the user and are used for recognition of ADLs in new sensor data. { } Given a transaction database D with T = t1 , … , tM transactions and a set of numer ical attr ibutes { } I = a1 , … , aK , assume the value of attr ibute ak (1 ⩽ k[ ⩽]K) can be retrieved from the m-th transaction using tm ak . Also, let each { attribute}ak be associated with a j set of fuzzy sets Fak = fa1 , … , faJ . Each fuzzy set, fak in k
k
Fak, represents the j-th fuzzy set in Fak and has an associated linguistic term as well as a membership function 𝜇faj (x) such k ( ) that 𝜇faj (x) : dom ak → [0 , 1]. A fuzzy association rule k
can be mined in the form:
X is A → Y is B, { } { } I where X = x1 , … , xp{ and Y = y}1 , … , yp are { subsets of} and X ∩ Y = �. A = fx1 , … , fxp and B = fy1 , … , fyp are fuzzy sets associated with X and Y. Each fuzzy association rule is interpreted as when “X is A” is satisfied, it can be inferred that “Y is B” is also satisfied. The word satisfied here means that there are enough records that support attribute fuzzy set pairs and the sum of these votes is greater than a user-specified threshold. Confidence of a rule is the ratio of transactions that contain both the consequent and antecedent parts of the rule, divided by the total number of transactions that contain only the antecedent of the rule. It is calculated using Eq. 5.
� � �� 𝛼ck ti zk Conf (X, A, Y, B) = ∑ � � �� , ∏ 𝛼 t xj a i ti ∈T xj ∈X j ∑
∏
ti ∈T
zj ∈Z
(5)
[ ] where Z = X ∪ Y , C = A ∪ B, and 𝛼aj ti xj is obtained from Eq. 6.
[ ] 𝛼ck ti zk =
{
𝜇fc ∈C (ti [zk ]), if 𝜇fc ≥ 𝜔 k k 0, otherwise.
(6)
The output of the fuzzy association rule-mining algorithm is a set of fuzzy rules having the fuzzy terms of a visited location and a power signature as their antecedent and consequent, respectively, or vice versa. Given a {set of fuzzy } terms to represent the occupant’s location, L1 , … , Ls , labels to represent power signatures, {and a set of fuzzy } PS1 , … , PSm , the fuzzy association rules are divided into two categories based on whether a rule has a fuzzy term for the occupant’s location or a power signature as its antecedent with TYPE 1: “location is Li → the power signature is PSj” and TYPE 2: “power signature isPSj → the location is Li.”
Fuzzy association rule mining for recognising daily activities using Kinect sensors and a single…
Note that each rule in TYPE 1 has a corresponding rule in TYPE 2 with the antecedent and consequent parts in the reverse order. Rules in the form of TYPE 1 indicate locations that are associated with specific power signatures. The confidence of such rules is the probability of the power signature in the rule being a result of the occupant’s interaction with an HEA at the location specified by the rule. If during the collection of the training dataset, a location is mostly visited for manipulating one particular HEA, the rule associating that location with the power signature of the HEA carries a high degree of confidence. For example, the rule associating the occupant’s location with the power signature of a refrigerator light, which would only be generated when the occupant was in front of the refrigerator, has almost 100% confidence. Rules in the form of TYPE 2 indicate power signatures that are associated with specific locations. The confidence of the rules in this category specifies the confidence with which a power signature observed on the power line can be linked to a specific location. If a power signature is mostly detected when the occupant is visiting a specific location, the corresponding rule associating the power signature with the location would carry a high degree of confidence. This means that the confidence of rules associating the power signatures of an automatic HEA to the occupant’s locations is low because those power signatures would be detected when the occupant is in different locations. Since the approach in this paper seeks to recognise ADLs based on the occupant’s location, each rule in the final rule set should be in the form of TYPE 1. Those rules in TYPE 1 with a high enough confidence can be used to confirm whether, based on the current occupant’s location, an observed power signature has resulted from
performing an ADL. To achieve a reliable and accurate set of fuzzy rules, the rules in the form of TYPE 1 are pruned by considering the maximum confidence of those rules and their respective rules in the TYPE 2 category. This maximisation takes place to improve the confidence of rules for adjacent HEAs. For example, if a toaster and its adjacent kettle share the same group of the occupant’s locations (denoted as L01) and the number of their interaction events are similar, the confidence of rules associating L01 to the power signatures of the toaster and the kettle would be low (around 50%). Such rules will be eliminated during the pruning process. The confidence of those rules with antecedent and consequent in the reverse order in the TYPE 2 category, however, would be high because the power signatures of those adjacent HEAs are observed only when the occupant was visiting L01. Once the rules in TYPE 1 with low confidence are removed, the remaining rules are regarded as the final rule set. Each rule is labelled by the user as one activity, a technique typically used by existing approaches (Noury et al. 2011). Users need to label only a small number of rules instead of labelling all the instances of activities. For each rule, a sample skeleton frame of the occupant at the time of using the appliance is shown on the depth map of the room to help the user label the rule. To do this, the timestamp of a usage event in Usage_Events with the same fuzzy labels of the occupant’s location and power signature in the rule is used to retrieve the skeleton data from the training dataset. Figure 6a, b show examples of this for rules associated with a living room and a kitchen area, respectively. The skeleton shown in each part is obtained from a training dataset of Kinect data. It can be observed that the occupant is interacting with a computer in Fig. 6a and a refrigerator
Fig. 6 Examples of the skeleton frame of the occupant shown on the depth map of the scene supplied by Kinect v2 sensors
13
H. Pazhoumand‑Dar
in Fig. 6b and hence, the corresponding rules for these two images can be labelled per activities shown in the images. Once the rules are labelled with ADLs, the final fuzzy rule set is in the form of: Ruler: IF the occupant’s location is Li AND power signature is PSj THEN activity is ar where the occupant’s location and power signature are crisp variables indicating the location of the occupant in the x–z and P–Q spaces, respectively. Li and PSj are two fuzzy sets determined for a group of the visited locations and power signatures, respectively. ar is the activity label assigned by the user to the rule. As an example, if two groups of visited locations are identified for the kitchen labelled with fuzzy terms L1 and L2, and two groups of power signatures are observed whenever the occupant uses an HEA at those locations, the list of fuzzy rules for the kitchen can be: Rule1: IF the occupant’s location is L1 AND the power signature is PS1 THEN activity is making tea Rule2: IF the occupant’s location is L2 AND the power signature is PS2 THEN activity is using the toaster As observed in the example of fuzzy rules above, statements in the antecedent of the rules involve a fuzzy logical conjunction (fuzzy AND) which is referred to as T-norm. This operator aggregates membership functions in the input variables using Eq. 7. [ ] 𝜇Li AND PSj = min 𝜇Li , 𝜇PSj , (7)
𝜇A1 and 𝜇A2 in Eq. 7 are the membership functions of fuzzy sets that represent clusters Li and PSj, respectively. 3.3 Recognising ADLs The resulting fuzzy rule set from the previous stage is used to detect interactions with HEAs and to recognise ADLs. Each detected power signature, along with the corresponding location of the occupant, is compared against the fuzzy rule set using a fuzzy concept called firing strength (Kukolj 2002). The firing strength of a rule is the degree of satisfaction of the antecedent { } of the rule by the input crisp variables. Let V = v1 , v2 be{ the set}of variables in the antecedent of rule Rulep and A = f1 , f2 , the { set of}fuzzy sets associated with those variables. Also, let 𝜇f1 , 𝜇f2 be the set of membership functions of A, such that 𝜇fw (w = 1, 2) represents the membership function of fw. Given a detected power signature in the P–Q space along with the respective location of the occupant on the Kinect’s x–z coordinates, the following formula is used to calculate the firing strength of Rulel in the rule set.
f⟨Om ,rulep ⟩ =
2 �
� � 𝜇fw vw ,
The best match to V is given by the rule with the maximum firing strength (see Eq. 9)
Activity = ar such that r = arg max f⟨Om ,Ruler ⟩ . 1⩽r⩽R
(9)
4 Results and discussion In this section, the results of experimentally evaluating the effectiveness of FMR-ADL are presented. First, the experimental environment used to evaluate this approach is explained. Next, the results of associating the occupant’s locations with the power signature of HEAs are detailed. This is followed by reporting the accuracy of FMR-ADL in classifying detected interactions into specific activities. Finally, the results of comparing the accuracy of FMR-ADL with that of alternative methods are presented. 4.1 Dataset Since no public dataset supplies a combination of continuous power consumption and Kinect data for ADLs inside a residential house, a dataset was collected in December 2016 to evaluate the effectiveness of FMR-ADL. The experimental data were captured using a single-bedroom apartment consisting of a living room, a kitchen, a dining room, a bedroom, and a combined toilet and bathroom. In this apartment, data related to ADLs performed by a volunteer were collected over 28 days (4 weeks). The occupant was asked to log the time and names of the HEAs interacted with to capture the ground truth of activities. To check the results of clustering power signatures against the ground truth, information was collected about HEAs present in the house, namely their names, the rooms where they were located, and
(8)
w=1
where ∏ is the standard fuzzy intersection operator defined as
13
∏( ) [ ] 𝜇f1 , 𝜇f2 = min 𝜇f1 , 𝜇f2 .
Fig. 7 An example for measuring the power consumption of individual appliances using Power-Mate 10AHD Serial
Fuzzy association rule mining for recognising daily activities using Kinect sensors and a single…
their power consumption. To measure the power consumption of HEAs, a power sensor (Power-Mate 10AHD Serial 2016) was placed between each appliance and its power outlet. Figure 7 shows an example of measuring the TV power consumption. For appliances hardwired to power lines, such as ceiling lights, the rated power consumption obtained from their power consumption label was used. The ground truth was used solely to evaluate the performance of the FMR-ADL approach and is not generally needed. Table 3 shows the names and locations of the monitored HEAs regularly used by the house’s occupant. To capture the occupant’s locations, the house was equipped with four Kinect v2 sensors placed in the kitchen, living room, dining room, and bedroom. All Kinect v2 sensors were set up to cover the location of HEAs in Table 3 except for the washing machine in the bathroom. Observations were recorded from each Kinect every time the occupant was observed. This resulted in more than 3 million Kinect observations of normal ADLs involving the use of HEAs. Figure 8 shows the setup of the sensors and the location of the HEAs used during the data recording: furniture in this house included 11 typical HEAs and some Table 3 HEAs and their locations in the experimental setting HEA
Location
HEA
Location
Washing machine Toaster Refrigerator (light) Electric cooktop Kettle
Bathroom Kitchen Kitchen Kitchen Kitchen
Computer TV Floor lamp Hair dryer Microwave
Living room Living room Living room Bedroom Kitchen
other items frequently used by the occupant during ADLs. Some examples are the dining table in the dining room (mostly used at mealtimes) and the sofa in the living room (used for reading and watching TV). Figure 9a shows an example of using the computer in the living room and Fig. 9b shows an example frame for using the refrigerator in the kitchen. It should be noted that the locations of the occupant in this dataset were obtained from the occupant’s head joint. A power meter was installed in the house electricity panel to capture the aggregated power consumption of the house. Figure 10 shows an example of the aggregated real power signal with the positive step changes indicating the turn-on events of HEAs. The x axis indicates time of day. 4.2 Experimental results The collected dataset was partitioned into a training set and an unseen test set. The training set consisted of the occupant’s ADLs for 21 days randomly selected out of the 28 days. The unseen test set held data for the remaining 7 days. The training dataset was employed to obtain fuzzy association rules necessary for classifying new ADLs. The rules were labelled with ADLs based on the occupant’s location and the HEA that was associated with the rule. This information was then used to recognise ADLs in the unseen testing dataset in order to evaluate the performance of FMR-ADL. The following sections explain the details of these experiments.
Fig. 8 Furniture location layout of the experimental house
Kinect sensor Bedroom Bathroom
Kitchen
Dining room
Electricity panel box
Living room
livin
13
H. Pazhoumand‑Dar
Fig. 9 Example of activities carried out in the experimental setting. a Using the toaster in the kitchen, b having a meal in the dining room, and c using the hair dryer in the bedroom
4.2.1 Extracting fuzzy classification rules
Fig. 10 An example of the composite active power consumption signal of the experimental setting
13
The training dataset, consisting of data from Kinect sensors and the power meter, was processed by following the procedure described in Sect. 3. Detecting the turn-on events of HEAs in the training dataset and extracting the power signatures associated with those events resulted in obtaining Usage_Events with 1727 entries. These were grouped into clusters, resulting in 11 clusters, and fuzzy sets were defined to represent these clusters. These fuzzy sets were labelled arbitrarily with linguistic terms as shown in Table 4. Note that the name of the HEA associated with each detected group of power signatures is identified using the ground truth data for the power consumption of HEAs. All the groups of power signatures that resulted from the interaction of the occupant in a monitored area are shown in bold
Fuzzy association rule mining for recognising daily activities using Kinect sensors and a single… Table 4 The name of the HEAs in the home along with the fuzzy label assigned to their group of power signatures HEA
Fuzzy label for power signatures
Washing machine Kettle Refrigerator light Electric cooktop Toaster Microwave Computer TV Living room light Hair dryer Refrigerator compressor
PS05 PS07 PS11 PS10 PS06 PS09 PS02 PS08 PS11 PS06 PS01
PS04
PS03
PS07
Fuzzy labels in bold are generated upon the interaction of the occupant. The labels are generated arbitrarily
Fig. 11 The membership functions were learned from the training dataset to represent the groups of power signatures. The x and y axes are cut at 220 watts and 200 VARs in order to represent details of the membership functions associated with the low power-consuming HEAs
in Table 4. Since no Kinect was placed in the bathroom, none of the fuzzy sets for the washing machine are in bold. Figure 11 shows the learned membership functions representing the power signatures of low power consuming HEAs in the experimental setting. Only the power signatures from the living room lamp and the refrigerator light were grouped into the same cluster of power signatures (i.e. PS11) and other HEAs with low power consumption were assigned different fuzzy labels. This figure also shows that the membership functions associated with some groups of power signatures overlap (e.g. PS04 and PS05). In regard to high power consumption HEAs, evaluations confirmed that the power signatures from one of the operating modes of the electric cooktop and the kettle shared the same group of power signatures, labelled PS07, as they both performed a heating operation with similar consumption. This was also the case for the hair dryer and the toaster. Table 4 shows that some HEAs generated multiple clusters. For example, the washing machine produced different
groups of power signatures due to its components and operation program. In addition, the electric cooktop produced two groups of power signatures in the data, generated upon the interaction of the occupant. When the occupant’s locations co-occurred with a power signature on the power line, the locations were grouped together and each cluster was modelled using a fuzzy set. Figure 12 shows the physical locations of these clusters and the fuzzy label given to them. The physical location of each detected cluster in this figure was manually determined based on the ID of the corresponding Kinect and coordinates of the cluster centre. Note that most of the turn-on events of the washing machine took place when the occupant was performing activities elsewhere (e.g. at the dining table or in bed). As a result, the occupant’s locations at the dining table and bed were associated with a cluster ID. These two detected locations (i.e. L01 and L08) are referred to as false locations. Some clusters in Fig. 12 represent the location of more than one group of power signatures. For example, L03 was the location of the occupant’s interactions with both modes of operation of the electric cooktop. Similarly, L04 included the occupant’s locations while interacting with both the kettle and the toaster. Table 5a shows the initial TYPE1 fuzzy rules that associated the fuzzy labels of the occupant’s locations with the fuzzy labels of the power signatures. The confidence of each rule is shown next to the rule. Table 5b shows the corresponding TYPE 2 fuzzy rules that associated the fuzzy labels of power signatures with the fuzzy labels of the occupant’s locations along with the confidence of rules. As shown in Table 5a, the confidence of the fuzzy rules associating false locations (i.e. L03 and L10) with the groups of power signatures is relatively low. This is also the case with their corresponding rules in Table 5b, with the consequence and antecedent in the reverse order. This is because the power signatures of automatically operating HEAs (i.e. the refrigerator compressor and the washing machine) in these rules were observed when the occupant was visiting different places. The precision metric in the context of generating rules to associate visited locations with the groups of power signatures is the ratio of the number of location labels correctly associated with their power signature labels to the total number of location labels associated with power signature labels. The recall metric is the ratio of the number of location labels correctly associated with power signature labels to the number of location labels that should have been associated with a power signature label. An accurate set of rules results in a high recall, which means interactions with HEAs are detected in most locations. An accurate set of rules also results in high precision, meaning false locations are not included in the reported ones.
13
H. Pazhoumand‑Dar Fig. 12 Fuzzy labels for the locations of the occupant while using each monitored HEA
L02
L03
L10
L09
L04
L05
L08
L07 L06 L01
Table 5 (a) TYPE 1 fuzzy rules associating fuzzy sets for the location of the occupant to those for power signatures, (b) TYPE 1 fuzzy rules associating fuzzy sets for power signatures to locations of the occupant (a) TYPE 1 Rules Location is Location is Location is Location is Location is Location is Location is Location is Location is Location is Location is Location is Location is Location is Location is Location is Location is Location is Location is Location is
L01 L01 L01 L02 L03 L03 L03 L04 L04 L05 L05 L06 L06 L07 L08 L08 L08 L09 L10 L10
→ → → → → → → → → → → → → → → → → → → →
Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is
PS08 PS09 PS10 PS04 PS03 PS06 PS09 PS06 PS07 PS02 PS08 PS05 PS09 PS11 PS08 PS09 PS10 PS07 PS08 PS02
Conf (%)
(b) TYPE 2 Rules
38.3 40.8 20.9 100 45.2 33.6 21.2 78.8 21.2 98.1 1.9 88.7 11.3 100 34.2 45 20.8 100 6 94
Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is Power signature is
It was observed that when rules in the TYPE 1 category were pruned based on a low threshold, rules associating false locations with power signatures were not eliminated. This decreased the precision of the output rule set. Increasing the threshold, on the other hand, caused the
13
Conf (%) PS08 PS09 PS10 PS04 PS03 PS06 PS09 PS06 PS07 PS02 PS08 PS05 PS09 PS11 PS08 PS09 PS10 PS07 PS08 PS02
→ → → → → → → → → → → → → → → → → → → →
location is location is location is location is location is location is location is location is location is location is location is location is location is location is location is location is location is location is location is location is
L01 L01 L01 L02 L03 L03 L03 L04 L04 L05 L05 L06 L06 L07 L08 L08 L08 L09 L10 L10
32.8 39.9 52.9 100 100 80 10 20 77.2 75.2 5.7 100 8.7 100 31.5 41.3 47.3 22.8 30 24
rules associating locations with less frequently observed power signatures to be ignored and a lower level of recall. This threshold was experimentally set to 75%, which was considered as a good compromise between high precision and high recall.
Fuzzy association rule mining for recognising daily activities using Kinect sensors and a single… Table 6 The output rule set of the training phase associating locations in the house to power signatures Classification rules IF location is IF location is IF location is IF location is IF location is IF location is IF location is IF location is IF location is IF location is
L02 L03 L03 L04 L04 L05 L06 L07 L09 L10
AND power signature is AND power signature is AND power signature is AND power signature is AND power signature is AND power signature is AND power signature is AND power signature is AND power signature is AND power signature is
PS04 PS03 PS06 PS06 PS07 PS02 PS05 PS11 PS07 PS02
THEN Activity is THEN Activity is THEN Activity is THEN Activity is THEN Activity is THEN Activity is THEN Activity is THEN Activity is THEN Activity is THEN Activity is
The final set of association rules is shown in Table 6. This table shows that the occupant’s locations during interactions with all the monitored HEAs have been associated with the power signatures of the respective HEAs. HEAs that had more than one mode of operation (e.g., the electric cooktop) caused multiple rules to be generated, with each rule associating the occupant’s location with one of their power signatures. Each rule in the final rule set was labelled with an activity based on the example skeleton frames shown to the user. The labels assigned to the rules are shown in Table 6. Each label is given an ID in the last column of this table. 4.2.2 Evaluating the performance of the method The unseen testing dataset was used to evaluate the performance of the FMR-ADL approach in recognising activities from sensor data. FMR-ADL labelled the occupant’s interactions with HEAs as ADLs based on the labels given to the learned fuzzy rules in Table 6. The label given to each detected interaction was verified by comparing it with the ground truth provided by the occupant. When the system accurately identified an activity that had occurred, it Table 7 The confusion matrix for classification of each ADL
Using the microwave Using the cooktop mode #1 Using the cooktop mode #2 Using the kettle Using the toaster Using the refrigerator Watching TV Using the computer Using the hair dryer Using the floor lamp
Conf (%)
Activity ID
100 100 80 78.8 77.2 98.1 100 100 100 94
1 2 3 4 5 6 7 8 9 10
recorded a true positive (TP). False positive (FP) cases included incorrectly classified events (e.g. ‘using the kettle’ was classified as ‘using the toaster’) and cases where the system incorrectly identified ADLs because of automatically generated power signatures. A case where the system did not detect the occupant’s activity was considered as a false negative (FN). This happened when the occupant’s location or the power signature of the used HEA was not within the boundaries of their respective clusters. The confusion matrix for all classification instances is shown in Table 7. There are extra columns in the table to show the number of FN classifications and FP cases. FP1 indicates the number of incorrectly classified events for each ADL and FP2 shows the number of cases where the system incorrectly identified an ADL due to automatically generated power signatures. Each of the values shown in bold in Table 7 indicates the number of times the FMR-ADL method accurately identified the performance of a specific ADL. Many FN classifications were associated with the use of the cooktop and the electric kettle, as indicated in Table 7. The table also shows that most instances of the misclassified activities belong to ‘using the kettle’ and ‘using the cooktop
Activity ID
1
2
3
4
5
6
7
8
9
10
FN
FP1
FP2
FP (FP1 + FP2)
1 2 3 4 5 6 7 8 9 10
80 0 0 0 0 0 0 0 0 0
0 54 0 0 0 0 0 0 0 0
0 0 37 10 0 0 0 0 0 0
0 0 0 80 0 0 0 0 0 0
0 0 5 0 51 0 0 0 0 0
0 0 0 0 0 258 0 0 0 0
0 0 0 0 0 0 219 0 0 0
0 0 0 0 0 0 0 81 0 0
0 0 0 0 0 0 0 0 57 0
0 0 0 0 0 0 0 0 0 90
3 11 5 13 3 24 0 12 0 15
3 2 3 0 0 16 0 0 0 0
0 0 5 10 0 0 0 0 0 0
3 2 8 10 0 16 0 0 0 0
Column labels (1 to 10) are the activity ID of the identified ADLs and the row labels represent the activity ID of the actual ADLs
13
H. Pazhoumand‑Dar
mode #2’, which were misclassified as each other. The power signatures generated by these two activities shared the same cluster. Since the locations of these activities were close to each other, the FMR-ADL method confused these two activities in cases where the occupant was making tea but was located more within the boundary of ‘using the cooktop mode #2’ or vice versa. In those situations, the membership of the occupant’s location in the fuzzy set associated with ‘using the cooktop mode #2’ was more than the fuzzy set for ‘using the kettle’. Therefore, the fuzzy rule for ‘using the cooktop mode #2’ generated a higher firing strength. Three standard metrics were used to summarise the classification performance of the FMR-ADL approach, with results shown in Table 8. Accuracy is the proportion of correct classifications to the total number of classifications, as shown in Eq. 10. Specificity indicates the probability of having the correct classification for ADLs (see Eq. 11), and sensitivity is the probability that the FMR-ADL approach correctly identifies the correct ADL, as indicated in Eq. 12.
Accuracy =
TP , TP + FN + FP
(10)
Specificity =
TP , TP + FP
(11)
Sensitivity =
TP . TP + FN
(12)
FMR-ADL was modified in different ways to compare its performance with that of other alternatives. Equation 10 was used to measure the accuracy of other versions of FMRADL in recognising ADLs, and the results are shown in Table 9. The first modified version of FMR-ADL used only the data captured from the power meter. This was carried out to evaluate the impact of combining the occupant’s locations with power consumption data on the accuracy of recognising
ADLs. The occupant’s activities were recognised in this version based on observing specific power signatures on the power line. After grouping the power signatures of HEAs, each detected cluster of the power signatures was given an activity label. The labelling of the power signatures in this version of FMR-ADL required prior knowledge of the power consumption of appliances in use. The classification stage in this version checked whether a detected power signature fell within the boundaries of a cluster associated with an activity. In this case the activity label associated with the cluster was assigned to the power signature, hence the recognition of ADLs. The evaluations showed that this version of FMR-ADL has an average classification accuracy of 77.46%. This rate was much lower than the average accuracy of 89.17% obtained when the occupant’s locations as established by Kinect sensors were combined with power consumption data. This version of FMR-ADL could not correctly distinguish activities involving different HEAs with similar power signatures. For example, interactions with the hair dryer were labelled ‘using the toaster’ since the hair dryer and the toaster had similar power signatures and therefore grouped in the same cluster. This was also the case for the kettle and the electric cooktop. It was observed that this version of FMR-ADL could successfully filter out power signatures generated automatically by self-regulated HEAs in the experimental setting (i.e. the refrigerator and the washing machine). However, it required the user to indicate groups of power signatures belonging to those appliances, which was not an easy task. The FMR-ADL method was also modified to use data from the power meter and PIR sensors (instead of Kinect sensors). This combination of sensors was used in a similar way to the method developed by Srinivasan et al. (2013) to identify the use of appliances in residential environments. PIR sensors provide binary data indicating only the presence or absence of a person in their field of view. To simulate the data that would have been collected from PIR sensors in the testing environment, the occupant’s locations inside each room were replaced with the Kinect camera ID. This converted the occupant’s 3D locations in the room to binary data that only indicated the presence of the occupant in that room.
Table 8 Results of evaluating the classification performance of FMR-ADL
Performance metric
Value (%)
Accuracy Specificity Sensitivity
89.17 96.27 92.28
Table 9 Comparisons between the proposed and alternative methods in regards to recognising ADLs
Method
Accuracy
Requires the knowledge of HEAs in use?
FMR-ADL without the use of Kinect sensors’ data FMR-ADL with the use of PIR sensors instead of Kinect sensors FMR-ADL without the use of fuzzy logic FMR-ADL (the proposed method)
77.46% 64.63%
Yes No
81.31% 89.17%
No No
13
Fuzzy association rule mining for recognising daily activities using Kinect sensors and a single…
This version of the FMR-ADL approach accurately recognised 64.63% of the ADLs as it could distinguish between the use of HEAs in different rooms with similar power consumption (e.g., the hair dryer and the toaster). However, the PIR sensors’ data were not sufficient to differentiate between ADLs inside the same room that involved the use of HEAs with similar power consumption. For example, the use of the electric cooktop and the kettle were labelled as belonging to the same activity because both HEAs were in the kitchen and generated similar power signatures. In addition, this version of the FMR-ADL approach could not distinguish the automatically generated power signatures of self-regulating appliances from those that resulted from the occupant’s interactions with HEAs. It associated the occupant’s locations in the kitchen, dining room, living room, and bedroom areas with the power signatures of the refrigerator since those power signatures were frequently generated when the occupant was in those areas. The accuracy of FMR-ADL without the use of fuzzy logic was also evaluated, as shown in Table 9. Once the occupant’s locations and power signatures were grouped into clusters, a non-fuzzy association rule-mining algorithm (Agrawal et al. 1993) was employed to establish associations between the occupant’s locations and power signatures. This version of the approach achieved an accuracy of 81.31%, which was lower than that of the proposed FMR-ADL. The accuracy was lower because in some appliance usage events, the occupant’s locations and/or the power signatures in the unseen testing dataset were slightly outside the boundaries of the learned clusters. In those situations, the occupant’s location and the detected power signature did not trigger any rules in the learned rule set and as such, the approach did not consider those events as the occupant’s ADL. The fuzzy version of FMR-ADL, however, could detect and recognise those instances of ADLs correctly, as the definition of fuzzy sets allowed slight variations in the occupant’s locations and power signatures.
5 Conclusion and future work This paper presented the FMR-ADL approach, a fuzzy method of recognising ADLs which uses a combination of inexpensive and low-intrusive sensors to recognise ADLs. A few sensors needed for this method can be retrofitted to existing homes and the method does not need any prior sensor data annotation. This addresses two important issues associated with many existing ADL recognition approaches. The first stage of FMR-ADL involved extraction of features from an unlabelled sensor dataset. The features included the physical location of the occupant in the room and the power consumption of the interacted HEAs. The second stage grouped the extracted features from each data
source into clusters and defined 2D fuzzy sets to represent these clusters. Once the extracted features were converted into fuzzy terms, a fuzzy association rule-mining algorithm was employed to generate fuzzy rules that associated frequently co-occurring fuzzy labels of the power signatures with the occupant’s locations. These fuzzy rules were then labelled with ADLs by the user and were employed in the third stage to recognise the occupant’s ADLs. The results in Table 9 showed that FMR-ADL with an accuracy of 89.17% significantly outperformed alternative ADL recognition approaches as it could more effectively recognise ADLs involving appliances with similar power consumption and tolerate fine variations in performing ADLs. The promising results reported in this paper will serve as an impetus for further research on the use of a power sensor together with Kinect depth sensors to monitor ADLs in an unsupervised situation. Although the performance of FMR-ADL was evaluated in a one-bedroom apartment as an experimental setting, it can be scaled to monitor larger homes using additional Kinect sensors. In this case, each additional Kinect sensor simply adds its own dataset of observations to the training data. If fields of view of multiple Kinect sensors in the same room interfere, each Kinect adds its own fuzzy rules to the final rule set. Some rules in this case might model the same activity carried out in the overlapping field of view. It is worth mentioning that one can use only the contextual information for the room associated with each rule and select more general activity labels. For example, knowing that the kitchen is usually visited for preparing meals, each rule associating a location in the kitchen with a power signature can be labelled as a meal-preparation activity. Thus, each interaction with an HEA in the kitchen is recognised as a meal-preparation activity. While the training stage in FMR-ADL needs computational resources and processing time to generate fuzzy rules for classification, the classification stage identifies the performance of ADLs from new sensor data almost in real-time. This stage only needs to obtain the current location of the occupant and the change in the home power consumption once every second and compare the information against the fuzzy classification rules. The occupant’s location is obtained from the skeletal tracking feature of the Kinect for Windows SDK, which provides real-time detection and user tracking. The calculations of the power consumption features have very low computational complexity. The average processing time of a MATLAB implementation of classifying new sensor data on an Intel i5 Windows 8.1 and 4 GB RAM laptop is 0.15 s. Future work in this study will involve investigating techniques to update an existing rule set in an incremental fashion to accommodate new HEAs and modifications in the monitored home. It is not uncommon for a person
13
H. Pazhoumand‑Dar
to change the placement of furniture in their home or the location where they perform ADLs. Data relevant to new behaviour patterns might become available well after the training period. An incremental learning technique can be incorporated to the FMR-ADL method in order to update an existing fuzzy rule set once a new normal behaviour pattern is identified. In this paper, the occupant was located based on the use of Kinect v2 depth sensor and the Kinect for Windows SDK. A further future direction of this research could investigate the use of alternative sensing technologies for this purpose. One possible alternative is the recently introduced Intel RealSense camera. This camera has a smaller size compared to Kinect v2 sensors and does not require an external power supply. The person- tracking algorithm in the SDK of the Intel RealSense camera (SR300) has an operating range of 0.5 to 5 metres, which covers a larger area of a room than the Kinect v2 sensor SDK. This algorithm also performs face recognition, which facilitates the monitoring of dwellings occupied by more than one person. Other future directions of this research are extending the approach to monitor dwellings occupied by more than one person, through identifying people from sensor data, and incorporating techniques to automatically determine the settings of the data mining algorithm (e.g. the value of min_conf). A data-driven approach could be investigated to derive this value automatically from the frequency distribution of behaviour patterns.
References Agrawal R, Imieliński T, Swami A (1993) Mining association rules between sets of items in large databases. ACM SIGMOD Record 22:207–216. doi:10.1145/170036.170072 Belley C, Gaboury S, Bouchard B, Bouzouane A (2014) An efficient and inexpensive method for activity recognition within a smart home based on load signatures of appliances. Pervasive Mob Comput 12:58–78 Berenguer M, Giordani M, Giraud-By F, Noury N Automatic detection of activities of daily living from detecting and classifying electrical events on the residential power line. e-health Networking, Applications and Services, 2008. HealthCom 2008. 10th International Conference on, 7–9 July 2008 2008. pp 29–32. doi:10.1109/ health.2008.4600104 Chen L, Nugent CD, Wang H (2012) A knowledge-driven approach to activity recognition in smart homes. IEEE Trans Knowl Data Eng 24:961–974. doi:10.1109/TKDE.2011.51 Cho HS, Yamazaki T, Hahn M (2010) AERO: extraction of user’s activities from electric power consumption data. IEEE Trans Consum Electron 56:2011–2018. doi:10.1109/TCE.2010.5606359 Chung P-C, Liu C-D (2008) A daily behavior enabled hidden Markov model for human behavior understanding. Pattern Recogn 41:1572–1580. doi:10.1016/j.patcog.2007.10.022 Claes V, Devriendt E, Tournoy J, Milisen K (2015) Attitudes and perceptions of adults of 60 years and older towards in-home monitoring of the activities of daily living with contactless sensors: an explorative study. Int J Nurs Stud 52:134–148
13
Clement J, Ploennigs J, Kabitzsch K (2014) Detecting activities of daily living with smart meters. Ambient assisted living. Springer, Berlin, pp 143–160 Ester M, Kriegel H-P, Sander J Xu X (1996) A density-based algorithm for discovering clusters in large spatial databases with noise. Kdd 34:226–231 Gaddam A, Mukhopadhyay SC, Sen Gupta G (2011) Elder care based on cognitive sensor network. Sens J IEEE 11:574–581 Gayathri KS, Easwarakumar KS, Elias S (2017) Probabilistic ontology based activity recognition in smart homes using Markov Logic Network. Knowl Based Syst 121:173–184. doi:10.1016/j. knosys.2017.01.025 Gu T, Wu Z, Tao X, Pung HK, Lu J (2009) EPSICAR: An Emerging Patterns based approach to sequential, interleaved and Concurrent Activity Recognition. In: 2009 IEEE International Conference on Pervasive Computing and Communications, pp 1–9. doi:10.1109/ PERCOM.2009.4912776 Huang Y-M, Hsieh M-Y, Chao H-C, Hung S-H, Park JH (2009) Pervasive, secure access to a hierarchical sensor-based healthcare monitoring architecture in wireless heterogeneous networks. IEEE J Sel Areas Commun 27:400–411 Kalekar PS (2004) Time series forecasting using Holt-Winters exponential smoothing. Kanwal Rekhi Sch Inf Technol 4329008:1–13 Kinect for Windows SDK 2.0 (2017) http://www.microsoft.com/en-au/ download/details.aspx?id=44561. Accessed 15 June 2017 Krishnan NC, Cook DJ (2014) Activity recognition on streaming sensor data. Pervasive Mobile Comput 10:138–154 Krüger F, Nyolt M, Yordanova K, Hein A, Kirste T (2014) Computational state space models for activity and intention recognition. A Feasibility Study. PLOS One 9:e109381. doi:10.1371/journal. pone.0109381 Kukolj D (2002) Design of adaptive Takagi–Sugeno–Kang fuzzy models. Appl Soft Comput 2:89–103 Kuok CM, Fu A, Wong MH (1998) Mining fuzzy association rules in databases. ACM Sigmod Record 27:41–46 Mehr HD, Polat H, Cetin A (2016) Resident activity recognition in smart homes by using artificial neural networks. 20–21 April 2016 2016. 4th International Istanbul Smart Grid Congress and Fair (ICSG), pp 1–5. doi:10.1109/SGCF.2016.7492428 Noor MHM, Salcic Z, Wang KIK (2016) Enhancing ontological reasoning with uncertainty handling for activity recognition. Knowl Based Syst 114:47–60. doi:10.1016/j.knosys.2016.09.028 Noury N, Berenguer M, Teyssier H, Bouzid MJ, Giordani M (2011) Building an index of activity of inhabitants from their activity on the residential electrical power line. IEEE Trans Inf Technol Biomed 15:758–766. doi:10.1109/TITB.2011.2138149 Power-Mate 10AHD Serial (2016) POWER-MATE™ 10A Power Meter Serial. http://www.cabac.com.au/products/electrical-testand-measurement/power-meters/PM10AHDS. Accessed 25 Dec 2016 Rafferty J, Nugent CD, Liu J, Chen L (2017) From activity recognition to intention recognition for assisted living within smart homes. IEEE Trans Hum Mach Syst 47:368–379. doi:10.1109/ THMS.2016.2641388 Rahimi S, Chan AD, Goubran RA (2011) Usage monitoring of electrical devices in a smart home. In: 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp 5307–5310. doi:10.1109/IEMBS.2011.6091313 Röhlig M et al (2015) Supporting activity recognition by visual analytics. In: 2015 IEEE Conference on Visual Analytics Science and Technology (VAST), 25–30 Oct 2015, pp 41–48. doi:10.1109/ VAST.2015.7347629 Srinivasan V, Stankovic J, Whitehouse K (2013) FixtureFinder: discovering the existence of electrical and water fixtures. In: Proceedings of the 12th international conference on Information processing in sensor networks, 2013. ACM, pp 115–128
Fuzzy association rule mining for recognising daily activities using Kinect sensors and a single… Suryadevara NK, Quazi M, Mukhopadhyay SC (2012) Intelligent sensing systems for measuring wellness indices of the daily activities for the elderly. In: 2012 8th International Conference on Intelligent Environments (IE), pp 347–350. doi:10.1109/IE.2012.49 Webb J, Ashley J (2012) Beginning Kinect programming with the Microsoft Kinect SDK. Apress, New York Whitehouse S, Yordanova K, Paiement A, Mirmehdi M (2016) Recognition of unscripted kitchen activities and eating behaviour for health monitoring. In: Proceedings of 2nd IET International Conference on Technologies for Active and Assisted Living (TechAAL 2016), 2016. Institution of Engineering and Technology, pp 1–6 Wilson C et al (2015) Identifying the time profile of everyday activities in the home using smart meter data. Paper presented at the the
European Council for an Energy Efficient Economy (ECEEE), Toulon/Hyères, France World Health Organization (2015) World report on ageing and health Xiang Y, Tang Y-p, Ma B-q, Yan H-c, Jiang J, Tian X-y (2015) Remote Safety Monitoring for Elderly Persons Based on Omni-Vision Analysis. PloS One 10:e0124068 Yang C-C, Hsu Y-L (2012) Remote monitoring and assessment of daily activities in the home environment. J Clin Gerontol Geriatr 3:97–104. doi:10.1016/j.jcgg.2012.06.002 Zhan K, Faux S, Ramos F (2015) Multi-scale conditional random fields for first-person activity recognition on elders and disabled patients. Pervasive Mobile Comput 16 (Part B):251–267. doi:10.1016/j.pmcj.2014.11.004
13