A searchable list of some of my publications is below. You can also access my publications from the following sites.
My ORCID is
https://orcid.org/0000-0002-6236-2969
Publications:
1.
Harish Haresamudram, Irfan Essa, Thomas Plötz
A Washing Machine is All You Need? On the Feasibility of Machine Data for Self-Supervised Human Activity Recognition Proceedings Article
In: International Conference on Activity and Behavior Computing (ABC) 2024 , 2024.
Abstract | Links | BibTeX | Tags: activity recognition, behavioral imaging, wearable computing
@inproceedings{2024-Haresamudram-WMNFMDSHAR,
title = {A Washing Machine is All You Need? On the Feasibility of Machine Data for Self-Supervised Human Activity Recognition},
author = {Harish Haresamudram and Irfan Essa and Thomas Plötz
},
url = {https://ieeexplore.ieee.org/abstract/document/10651688},
doi = {10.1109/ABC61795.2024.10651688},
year = {2024},
date = {2024-05-24},
booktitle = {International Conference on Activity and Behavior Computing (ABC) 2024 },
abstract = {Learning representations via self-supervision has emerged as a powerful framework for deriving features for automatically recognizing activities using wearables. The current de-facto protocol involves performing pre-training on (large-scale) data recorded from human participants. This requires effort as recruiting participants and subsequently collecting data is both expensive and time-consuming. In this paper, we investigate the feasibility of an alternate source of data for its suitability to lead to useful representations, one that requires substantially lower effort for data collection. Specifically, we examine whether data collected by affixing sensors on running machinery, i.e., recording non-human movements/vibrations can also be utilized for self-supervised human activity recognition. We perform an extensive evaluation of utilizing data collected on a washing machine as the source and observe that state-of-the-art methods perform surprisingly well relative to when utilizing large-scale human movement data, obtaining within 5-6 % Fl-score on some target datasets, and exceeding on others. In scenarios with limited access to annotations, models trained on the washing-machine data perform comparably or better than end-to-end training, thereby indicating their feasibility and potential for recognizing activities. These results are significant and promising because they have the potential to substantially lower the efforts necessary for deriving effective wearables-based human activity recognition systems.
},
keywords = {activity recognition, behavioral imaging, wearable computing},
pubstate = {published},
tppubtype = {inproceedings}
}
Learning representations via self-supervision has emerged as a powerful framework for deriving features for automatically recognizing activities using wearables. The current de-facto protocol involves performing pre-training on (large-scale) data recorded from human participants. This requires effort as recruiting participants and subsequently collecting data is both expensive and time-consuming. In this paper, we investigate the feasibility of an alternate source of data for its suitability to lead to useful representations, one that requires substantially lower effort for data collection. Specifically, we examine whether data collected by affixing sensors on running machinery, i.e., recording non-human movements/vibrations can also be utilized for self-supervised human activity recognition. We perform an extensive evaluation of utilizing data collected on a washing machine as the source and observe that state-of-the-art methods perform surprisingly well relative to when utilizing large-scale human movement data, obtaining within 5-6 % Fl-score on some target datasets, and exceeding on others. In scenarios with limited access to annotations, models trained on the washing-machine data perform comparably or better than end-to-end training, thereby indicating their feasibility and potential for recognizing activities. These results are significant and promising because they have the potential to substantially lower the efforts necessary for deriving effective wearables-based human activity recognition systems.
2.
Harish Haresamudram, Irfan Essa, Thomas Ploetz
Towards Learning Discrete Representations via Self-Supervision for Wearables-Based Human Activity Recognition Journal Article
In: Sensors, vol. 24, no. 4, 2024.
Abstract | Links | BibTeX | Tags: activity recognition, arXiv, wearable computing
@article{2023-Haresamudram-TLDRSWHAR,
title = {Towards Learning Discrete Representations via Self-Supervision for Wearables-Based Human Activity Recognition},
author = {Harish Haresamudram and Irfan Essa and Thomas Ploetz},
url = {https://arxiv.org/abs/2306.01108
https://www.mdpi.com/1424-8220/24/4/1238},
doi = {10.48550/arXiv.2306.01108},
year = {2024},
date = {2024-02-24},
urldate = {2023-06-01},
journal = {Sensors},
volume = {24},
number = {4},
abstract = {Human activity recognition (HAR) in wearable computing is typically based on direct processing of sensor data. Sensor readings are translated into representations, either derived through dedicated preprocessing, or integrated into end-to-end learning. Independent of their origin, for the vast majority of contemporary HAR, those representations are typically continuous in nature. That has not always been the case. In the early days of HAR, discretization approaches have been explored - primarily motivated by the desire to minimize computational requirements, but also with a view on applications beyond mere recognition, such as, activity discovery, fingerprinting, or large-scale search. Those traditional discretization approaches, however, suffer from substantial loss in precision and resolution in the resulting representations with detrimental effects on downstream tasks. Times have changed and in this paper we propose a return to discretized representations. We adopt and apply recent advancements in Vector Quantization (VQ) to wearables applications, which enables us to directly learn a mapping between short spans of sensor data and a codebook of vectors, resulting in recognition performance that is generally on par with their contemporary, continuous counterparts - sometimes surpassing them. Therefore, this work presents a proof-of-concept for demonstrating how effective discrete representations can be derived, enabling applications beyond mere activity classification but also opening up the field to advanced tools for the analysis of symbolic sequences, as they are known, for example, from domains such as natural language processing. Based on an extensive experimental evaluation on a suite of wearables-based benchmark HAR tasks, we demonstrate the potential of our learned discretization scheme and discuss how discretized sensor data analysis can lead to substantial changes in HAR.},
howpublished = {arXiv:2306.01108},
keywords = {activity recognition, arXiv, wearable computing},
pubstate = {published},
tppubtype = {article}
}
Human activity recognition (HAR) in wearable computing is typically based on direct processing of sensor data. Sensor readings are translated into representations, either derived through dedicated preprocessing, or integrated into end-to-end learning. Independent of their origin, for the vast majority of contemporary HAR, those representations are typically continuous in nature. That has not always been the case. In the early days of HAR, discretization approaches have been explored - primarily motivated by the desire to minimize computational requirements, but also with a view on applications beyond mere recognition, such as, activity discovery, fingerprinting, or large-scale search. Those traditional discretization approaches, however, suffer from substantial loss in precision and resolution in the resulting representations with detrimental effects on downstream tasks. Times have changed and in this paper we propose a return to discretized representations. We adopt and apply recent advancements in Vector Quantization (VQ) to wearables applications, which enables us to directly learn a mapping between short spans of sensor data and a codebook of vectors, resulting in recognition performance that is generally on par with their contemporary, continuous counterparts - sometimes surpassing them. Therefore, this work presents a proof-of-concept for demonstrating how effective discrete representations can be derived, enabling applications beyond mere activity classification but also opening up the field to advanced tools for the analysis of symbolic sequences, as they are known, for example, from domains such as natural language processing. Based on an extensive experimental evaluation on a suite of wearables-based benchmark HAR tasks, we demonstrate the potential of our learned discretization scheme and discuss how discretized sensor data analysis can lead to substantial changes in HAR.
3.
Harish Haresamudram, Irfan Essa, Thomas Ploetz
Assessing the State of Self-Supervised Human Activity Recognition using Wearables Journal Article
In: Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), vol. 6, iss. 3, no. 116, pp. 1–47, 2022.
Abstract | Links | BibTeX | Tags: activity recognition, IMWUT, ubiquitous computing, wearable computing
@article{2022-Haresamudram-ASSHARUW,
title = {Assessing the State of Self-Supervised Human Activity Recognition using Wearables},
author = {Harish Haresamudram and Irfan Essa and Thomas Ploetz},
url = {https://dl.acm.org/doi/10.1145/3550299
https://arxiv.org/abs/2202.12938
https://arxiv.org/pdf/2202.12938
},
doi = {doi.org/10.1145/3550299},
year = {2022},
date = {2022-09-07},
urldate = {2022-09-07},
booktitle = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)},
journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)},
volume = {6},
number = {116},
issue = {3},
pages = {1–47},
publisher = {ACM},
abstract = {The emergence of self-supervised learning in the field of wearables-based human activity recognition (HAR) has opened up opportunities to tackle the most pressing challenges in the field, namely to exploit unlabeled data to derive reliable recognition systems for scenarios where only small amounts of labeled training samples can be collected. As such, self-supervision, i.e., the paradigm of 'pretrain-then-finetune' has the potential to become a strong alternative to the predominant end-to-end training approaches, let alone hand-crafted features for the classic activity recognition chain. Recently a number of contributions have been made that introduced self-supervised learning into the field of HAR, including, Multi-task self-supervision, Masked Reconstruction, CPC, and SimCLR, to name but a few. With the initial success of these methods, the time has come for a systematic inventory and analysis of the potential self-supervised learning has for the field. This paper provides exactly that. We assess the progress of self-supervised HAR research by introducing a framework that performs a multi-faceted exploration of model performance. We organize the framework into three dimensions, each containing three constituent criteria, such that each dimension captures specific aspects of performance, including the robustness to differing source and target conditions, the influence of dataset characteristics, and the feature space characteristics. We utilize this framework to assess seven state-of-the-art self-supervised methods for HAR, leading to the formulation of insights into the properties of these techniques and to establish their value towards learning representations for diverse scenarios.
},
keywords = {activity recognition, IMWUT, ubiquitous computing, wearable computing},
pubstate = {published},
tppubtype = {article}
}
The emergence of self-supervised learning in the field of wearables-based human activity recognition (HAR) has opened up opportunities to tackle the most pressing challenges in the field, namely to exploit unlabeled data to derive reliable recognition systems for scenarios where only small amounts of labeled training samples can be collected. As such, self-supervision, i.e., the paradigm of 'pretrain-then-finetune' has the potential to become a strong alternative to the predominant end-to-end training approaches, let alone hand-crafted features for the classic activity recognition chain. Recently a number of contributions have been made that introduced self-supervised learning into the field of HAR, including, Multi-task self-supervision, Masked Reconstruction, CPC, and SimCLR, to name but a few. With the initial success of these methods, the time has come for a systematic inventory and analysis of the potential self-supervised learning has for the field. This paper provides exactly that. We assess the progress of self-supervised HAR research by introducing a framework that performs a multi-faceted exploration of model performance. We organize the framework into three dimensions, each containing three constituent criteria, such that each dimension captures specific aspects of performance, including the robustness to differing source and target conditions, the influence of dataset characteristics, and the feature space characteristics. We utilize this framework to assess seven state-of-the-art self-supervised methods for HAR, leading to the formulation of insights into the properties of these techniques and to establish their value towards learning representations for diverse scenarios.
4.
Harish Haresamudram, Apoorva Beedu, Varun Agrawal, Patrick L Grady, Irfan Essa, Judy Hoffman, Thomas Plötz
Masked reconstruction based self-supervision for human activity recognition Proceedings Article
In: Proceedings of the International Symposium on Wearable Computers (ISWC), pp. 45–49, 2020.
Abstract | Links | BibTeX | Tags: activity recognition, ISWC, machine learning, wearable computing
@inproceedings{2020-Haresamudram-MRBSHAR,
title = {Masked reconstruction based self-supervision for human activity recognition},
author = {Harish Haresamudram and Apoorva Beedu and Varun Agrawal and Patrick L Grady and Irfan Essa and Judy Hoffman and Thomas Plötz},
url = {https://dl.acm.org/doi/10.1145/3410531.3414306
https://harkash.github.io/publication/masked-reconstruction
https://arxiv.org/abs/2202.12938},
doi = {10.1145/3410531.3414306},
year = {2020},
date = {2020-09-01},
urldate = {2020-09-01},
booktitle = {Proceedings of the International Symposium on Wearable Computers (ISWC)},
pages = {45--49},
abstract = {The ubiquitous availability of wearable sensing devices has rendered large scale collection of movement data a straightforward endeavor. Yet, annotation of these data remains a challenge and as such, publicly available datasets for human activity recognition (HAR) are typically limited in size as well as in variability, which constrains HAR model training and effectiveness. We introduce masked reconstruction as a viable self-supervised pre-training objective for human activity recognition and explore its effectiveness in comparison to state-of-the-art unsupervised learning techniques. In scenarios with small labeled datasets, the pre-training results in improvements over end-to-end learning on two of the four benchmark datasets. This is promising because the pre-training objective can be integrated "as is" into state-of-the-art recognition pipelines to effectively facilitate improved model robustness, and thus, ultimately, leading to better recognition performance.
},
keywords = {activity recognition, ISWC, machine learning, wearable computing},
pubstate = {published},
tppubtype = {inproceedings}
}
The ubiquitous availability of wearable sensing devices has rendered large scale collection of movement data a straightforward endeavor. Yet, annotation of these data remains a challenge and as such, publicly available datasets for human activity recognition (HAR) are typically limited in size as well as in variability, which constrains HAR model training and effectiveness. We introduce masked reconstruction as a viable self-supervised pre-training objective for human activity recognition and explore its effectiveness in comparison to state-of-the-art unsupervised learning techniques. In scenarios with small labeled datasets, the pre-training results in improvements over end-to-end learning on two of the four benchmark datasets. This is promising because the pre-training objective can be integrated "as is" into state-of-the-art recognition pipelines to effectively facilitate improved model robustness, and thus, ultimately, leading to better recognition performance.
5.
Daniel Castro, Steven Hickson, Vinay Bettadapura, Edison Thomaz, Gregory Abowd, Henrik Christensen, Irfan Essa
Predicting Daily Activities from Egocentric Images Using Deep Learning Proceedings Article
In: Proceedings of International Symposium on Wearable Computers (ISWC), 2015.
Abstract | Links | BibTeX | Tags: activity recognition, computer vision, ISWC, machine learning, wearable computing
@inproceedings{2015-Castro-PDAFEIUDL,
title = {Predicting Daily Activities from Egocentric Images Using Deep Learning},
author = {Daniel Castro and Steven Hickson and Vinay Bettadapura and Edison Thomaz and Gregory Abowd and Henrik Christensen and Irfan Essa},
url = {https://dl.acm.org/doi/10.1145/2802083.2808398
https://arxiv.org/abs/1510.01576
http://www.cc.gatech.edu/cpl/projects/dailyactivities/
},
doi = {10.1145/2802083.2808398},
year = {2015},
date = {2015-09-01},
urldate = {2015-09-01},
booktitle = {Proceedings of International Symposium on Wearable Computers (ISWC)},
abstract = {We present a method to analyze images taken from a passive egocentric wearable camera along with contextual information, such as time and day of the week, to learn and predict the everyday activities of an individual. We collected a dataset of 40,103 egocentric images over 6 months with 19 activity classes and demonstrate the benefit of state-of-the-art deep learning techniques for learning and predicting daily activities. Classification is conducted using a Convolutional Neural Network (CNN) with a classification method we introduce called a late fusion ensemble. This late fusion ensemble incorporates relevant contextual information and increases our classification accuracy. Our technique achieves an overall accuracy of 83.07% in predicting a person's activity across the 19 activity classes. We also demonstrate some promising results from two additional users by fine-tuning the classifier with one day of training data.},
keywords = {activity recognition, computer vision, ISWC, machine learning, wearable computing},
pubstate = {published},
tppubtype = {inproceedings}
}
We present a method to analyze images taken from a passive egocentric wearable camera along with contextual information, such as time and day of the week, to learn and predict the everyday activities of an individual. We collected a dataset of 40,103 egocentric images over 6 months with 19 activity classes and demonstrate the benefit of state-of-the-art deep learning techniques for learning and predicting daily activities. Classification is conducted using a Convolutional Neural Network (CNN) with a classification method we introduce called a late fusion ensemble. This late fusion ensemble incorporates relevant contextual information and increases our classification accuracy. Our technique achieves an overall accuracy of 83.07% in predicting a person's activity across the 19 activity classes. We also demonstrate some promising results from two additional users by fine-tuning the classifier with one day of training data.
6.
Vinay Bettadapura, Irfan Essa, Caroline Pantofaru
Egocentric Field-of-View Localization Using First-Person Point-of-View Devices Honorable Mention Proceedings Article
In: IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE Computer Society, 2015.
Abstract | Links | BibTeX | Tags: awards, best paper award, computer vision, WACV, wearable computing
@inproceedings{2015-Bettadapura-EFLUFPD,
title = {Egocentric Field-of-View Localization Using First-Person Point-of-View Devices},
author = {Vinay Bettadapura and Irfan Essa and Caroline Pantofaru},
url = {https://ieeexplore.ieee.org/document/7045943
http://www.vbettadapura.com/egocentric/localization/},
doi = {10.1109/WACV.2015.89},
year = {2015},
date = {2015-01-01},
urldate = {2015-01-01},
booktitle = {IEEE Winter Conference on Applications of Computer Vision (WACV)},
publisher = {IEEE Computer Society},
abstract = {We present a technique that uses images, videos and sensor data taken from first-person point-of-view devices to perform egocentric field-of-view (FOV) localization. We define egocentric FOV localization as capturing the visual information from a person's field-of-view in a given environment and transferring this information onto a reference corpus of images and videos of the same space, hence determining what a person is attending to. Our method matches images and video taken from the first-person perspective with the reference corpus and refines the results using the first-person's head orientation information obtained using the device sensors. We demonstrate single and multi-user egocentric FOV localization in different indoor and outdoor environments with applications in augmented reality, event understanding and studying social interactions.
},
keywords = {awards, best paper award, computer vision, WACV, wearable computing},
pubstate = {published},
tppubtype = {inproceedings}
}
We present a technique that uses images, videos and sensor data taken from first-person point-of-view devices to perform egocentric field-of-view (FOV) localization. We define egocentric FOV localization as capturing the visual information from a person's field-of-view in a given environment and transferring this information onto a reference corpus of images and videos of the same space, hence determining what a person is attending to. Our method matches images and video taken from the first-person perspective with the reference corpus and refines the results using the first-person's head orientation information obtained using the device sensors. We demonstrate single and multi-user egocentric FOV localization in different indoor and outdoor environments with applications in augmented reality, event understanding and studying social interactions.
7.
Edison Thomaz, Aman Parnami, Irfan Essa, Gregory Abowd
Feasibility of Identifying Eating Moments from First-Person Images Leveraging Human Computation Proceedings Article
In: Proceedings of ACM International SenseCam and Pervasive Imaging (SenseCam '13), 2013.
Links | BibTeX | Tags: activity recognition, behavioral imaging, computational health, ubiquitous computing, wearable computing
@inproceedings{2013-Thomaz-FIEMFFILHC,
title = {Feasibility of Identifying Eating Moments from First-Person Images Leveraging Human Computation},
author = {Edison Thomaz and Aman Parnami and Irfan Essa and Gregory Abowd},
doi = {10.1145/2526667.2526672},
year = {2013},
date = {2013-01-01},
urldate = {2013-01-01},
booktitle = {Proceedings of ACM International SenseCam and Pervasive Imaging (SenseCam '13)},
keywords = {activity recognition, behavioral imaging, computational health, ubiquitous computing, wearable computing},
pubstate = {published},
tppubtype = {inproceedings}
}
8.
Edison Thomaz, Thoma Pleotz, Irfan Essa, Gregory Abowd
Interactive Techniques for Labeling Activities Of Daily Living to Assist Machine Learning Proceedings Article
In: Proceedings of Workshop on Interactive Systems in Healthcare, 2011.
Abstract | Links | BibTeX | Tags: activity recognition, behavioral imaging, computational health, wearable computing
@inproceedings{2011-Thomaz-ITLADLAML,
title = {Interactive Techniques for Labeling Activities Of Daily Living to Assist Machine Learning},
author = {Edison Thomaz and Thoma Pleotz and Irfan Essa and Gregory Abowd},
url = {https://wish2011.wordpress.com/accepted-papers/
https://users.ece.utexas.edu/~ethomaz/papers/w1.pdf},
year = {2011},
date = {2011-11-01},
urldate = {2011-11-01},
booktitle = {Proceedings of Workshop on Interactive Systems in Healthcare},
abstract = {Over the next decade, as healthcare continues its march away from the hospital and towards the home, logging and making sense of activities of daily living will play a key role in health modeling and life-long home care. Machine learning research has explored ways to automate the detection and quantification of these activities in sensor-rich environments. While we continue to make progress in developing practical and cost-effective activity sensing techniques, one large hurdle remains, obtaining labeled activity data to train activity recognition systems. In this paper, we discuss the process of gathering ground truth data with human participation for health modeling applications. In particular, we propose a criterion and design space containing five dimensions that we have identified as central to the characterization and evaluation of interactive labeling methods.
},
keywords = {activity recognition, behavioral imaging, computational health, wearable computing},
pubstate = {published},
tppubtype = {inproceedings}
}
Over the next decade, as healthcare continues its march away from the hospital and towards the home, logging and making sense of activities of daily living will play a key role in health modeling and life-long home care. Machine learning research has explored ways to automate the detection and quantification of these activities in sensor-rich environments. While we continue to make progress in developing practical and cost-effective activity sensing techniques, one large hurdle remains, obtaining labeled activity data to train activity recognition systems. In this paper, we discuss the process of gathering ground truth data with human participation for health modeling applications. In particular, we propose a criterion and design space containing five dimensions that we have identified as central to the characterization and evaluation of interactive labeling methods.
9.
Kihwan Kim, Jay Summet, Thad Starner, Dan Ashbrook, M. Kapade, Irfan Essa
Localization and 3D Reconstruction of Urban Scenes Using GPS Proceedings Article
In: Proceedings of IEEE International Symposium on Wearable Computers (ISWC), pp. 11–14, IEEE Computer Society, 2008.
BibTeX | Tags: IMWUT, navigation, wearable computing
@inproceedings{2008-Kim-LRUSU,
title = {Localization and 3D Reconstruction of Urban Scenes Using GPS},
author = {Kihwan Kim and Jay Summet and Thad Starner and Dan Ashbrook and M. Kapade and Irfan Essa},
year = {2008},
date = {2008-09-01},
urldate = {2008-09-01},
booktitle = {Proceedings of IEEE International Symposium on Wearable Computers (ISWC)},
pages = {11--14},
publisher = {IEEE Computer Society},
keywords = {IMWUT, navigation, wearable computing},
pubstate = {published},
tppubtype = {inproceedings}
}
Other Publication Sites
A few more sites that aggregate research publications: Academic.edu, Bibsonomy, CiteULike, Mendeley.
Copyright/About
[Please see the Copyright Statement that may apply to the content listed here.]
This list of publications is produced by using the teachPress plugin for WordPress.