Physiological Parameter Estimation With Glass

BioGlass is a research project by MIT and Georgia Tech aiming to estimate physiological measures from a head-mounted wearable device. Yes, that would be Google Glass. I’ve just started experimenting with Glass and decided to see if I could visualize my heart beat rate from inertial sensor data collected with the device.

Sure enough, it was quite easy (but I had to be completely still).


BioGlass’s goal is to go beyond heart and breathing data, and estimate stress levels, for example. It looks like they are on the right track.

Recognizing Smoking Gestures with Inertial Sensors

Here is another paper for Mobisys 2014 from researchers at UMass Amherst that I found quite good. Instead of eating detection, it focuses on smoking gestures.

Authors: Parate, Chiu, Chadowitz, Ganesan, Kalogerakis (UMass Amherst)

Idea: A wristband with inertial sensors (IMU) that can detect smoking gestures and sessions in real-time.

Approach: An arm trajectory-based (quaternion) method segments the signal, extracting candidate hand-to-mouth gestures. Trajectory-based features are used for distinguishing smoking features from non-smoking features (confounding gestures). A probabilistic model is employed to find out which hand-mouth gestures belong to individual smoking sessions. For evaluation, multiple IMUs and 3D animation were used for obtaining ground-truth.

Study: (Dataset) 28 hours, 15 subjects, 17 smoking sessions, 10 eating sessions, resulting in 369 smoking puffs, 252 food bites.

Results: 91% precision, 81% recall.

Study II: Study in the wild with 4 participants, 4 hours per day for 3 days. Thirty smoking sessions, missed 3 sessions; Not too many false positives low, less than 2 per day.

Note: Wristband with IMU (9-axis: accel, gyro, compass) provides 3D orientation of the wrist. Interesting approach for obtaining ground truth (3D animation of arm). Good description of challenges (concurrent activity while smoking). Good overview of pipeline, segmentation, features. Comparison to BiteCounter, mPuff (respiration chest band)

Published: Mobisys 2014

You can download the paper here.

Ubicomp 2014

Three weeks ago I headed to Seattle for the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, also simply know as Ubicomp. It is my favorite conference of the year, and this one did not disappoint; the work presented was very good. I co-organized a workshop on Personal Informatics (PI) on Sunday before the conference program began and then attended three days of paper presentations. Here is a small sample of the papers/presentations I enjoyed:

StudentLife: Assessing Mental Health, Academic Performance and Behavioral Trends of College Students using Smartphones
Rui Wang et al. (Presentation)

BiliCam: Using Mobile Phones to Monitor Newborn Jaundice
Lilian de Greef et al. (Presentation)

MobileMiner: Mining Your Frequent Behavior Patterns on Your Phone
Vijay Srinivasan et al. (Presentation)

Your activity tracker knows when you quit smoking
Ken Kawamoto et al. (Presentation)

Connecting Personal-scale Sensing and Networked Community Behavior to Infer Human Activities
Nicholas D Lane et al. (Presentation)

Chroma: A Wearable Augmented-Reality Solution for Color Blindness
Enrico Tanuwidjaja et al.

InterruptMe: Designing Intelligent Prompting Mechanisms for Pervasive Applications
Veljko Pejovic and Mirco Musolesi

Aside from the papers, three highlights of the conference for me were the closing keynote by Gaetano Borriello (historical perspective on Open Data Kit), the evening reception at the Experience Music Project (fantastic venue, especially with the ISWC design exhibit), and of course, the telepresence robots (for remote attendees).

Looking forward to Ubicomp 2015 in Osaka!

Summer Visits and Activities

Summer is coming to an end quickly, as it always happens. In fact, at Georgia Tech, Fall classes started this week. Most of my time in July and August was dedicated to examining on-body sensor streams and evaluating ways to process and make sense of them. I am getting ready to run some studies in the Fall, and it was immensely useful to concentrate on improving the methods and techniques I will be relying on for analysis and activity inference.

At the end of July, I had the chance to visit Prof. Bonnie Spring’s lab in the School of Preventive Medicine at Northwestern University. We are exploring a collaboration where some of the eating recognition systems that I am building as part of my dissertation work will be employed in diet/behavior studies. I will save all the details of this possible collaboration to another post, but I am thrilled about the opportunity to work with Prof. Spring and her team. In my view, working closely with experts in the medical sciences beyond computer scientists interested in addressing health problems is critical to making meaningful impact. Hopefully I will have more to share soon.

Finally, just last week I attended the third annual Intel Science and Technology Center for Pervasive Computing (ISTC-PC) retreat, which was held close to Seattle, WA. Intel has been a major support of my graduate research work. As expected, I met many colleagues and had numerous productive conversations. Joe Paradiso gave the keynote speech after dinner:


Intel executives and researchers often come to this event. Bruce Horn, one of the members of the original Macintosh team, and none other than the man who created the Finder, is now the head of an Intel research unit. His signature is inside the case of the original Mac 128K, alongside other legends like Jobs, Raskin, Espinosa and Hertzfeld:


Here is a great post where he describes joining the Mac team. Naturally, I had to ask him to sign my MacBook Air:


Fun times. I hope the Georgia Tech folks tolerate my appreciation for the history of computing.

My Smartphone Knows I am Hungry

Really? Well, at least that’s what the authors of this paper published at the Workshop on Physical Analytics (Mobisys 2014) claim. Yes, I did say I was focusing on CHI2014 papers over the next few weeks, but I ran into this one and had to read it. After all, my dissertation topic is in large part about inferring eating activities.

Authors: Chen, Wang, Zhou, Campbell (Dartmouth College)

Idea: Use of behavior data and location history from smartphones to predict if an individual is going to eat or not in the future. The goal is to provide just-in-time feedback to people about healthy food and drink choices.

Approach: Built a CART (classification and regression trees) model to predict food purchases of a group of undergraduate students. Top six features were (1) the current building, (2) arrival time of current building, (3) departure time of previous building, (4) activity ratio of last building, (5) arrival time of previous building, and (6) conversation duration at previous building. A prediction baseline was set with random guessing and compared with a generic model (10 fold cross-validation on all the data), personalized model with and without adaptation. With adaption, the prediction model is trained for each individual and updated over time to reflect most recent behavior.

Study: 10-weeks, 25 undergraduate students, used dining card purchase records as ground truth.

Results: 74.2% accuracy, 52.7% precision, 55.1% recall (using three weeks of training, personalized model with adaptation).

Published: Workshop on Physical Analytics (WPA) 2014 – Mobisys 2014

Personal Tracking as Lived Informatics

As part of my CHI 2014 paper reading series, today I read “Personal Tracking as Lived Informatics” by Rooksby et al. from the University of Glasgow in the UK. They offer a compelling take on activity tracking. It is a good read, even if a bit contradictory at times, in my opinion.

Idea: Understand people’s practices and relationships with activity tracking devices and personal tracking in everyday life. This paper claims that personal tracking has been studied very narrowly so far, where devices serve as nothing more than activity detectors or as part of interventions.

Approach: Unstructured and follow up interviews with 22 participants.

Findings: Tracking is social (but not in the ’share on Twitter/Facebook’ social, there are different styles of tracking (directive, documentary, diagnostic, reward-based, gadget-based), multiple trackers are used, tracking information used for meeting daily or short-term goals.

Notes: People chose different metrics to examine out of the devices they used, data collected and analyzed for short-term use (people not concerned about saving the data), tracking punctuated memorable life experiences (e.g., finishing a marathon), self-esteem. Idea that tracking is actually almost always prospective, since it’s about where one’s heading (e.g., losing weight, training for a marathon.

Feedback for Improving Medication Taking

Over the next couple of weeks I will be writing several posts about CHI 2014 papers that caught my eye. I will format my summaries in 4 short paragraphs: Idea, Approach, Study, Notes. This structure will fit most papers I think. I will start with a paper by Matthew Lee and Anind Dey titled “Real-Time Feedback for Improving Medication Taking“. As I understand it, this is Matt’s PhD work at CMU under Anind. Matt is now at Phillips Research.

Idea: Study the effect of an instrumented pillbox and ambient display providing real-time feedback on medication taking. Research questions: (1) does real-time feedback improve consistency of medication taking?, (2) does real-time feedback improve self-efficacy for medication taking?, and (3) do medication taking behaviors change when removing real-time feedback after long-tern use?

Approach: Added various sensors and communication capabilities to a pill-box. Activity uploaded to server, projected back onto a visualization on a tablet that participants kept around the house.

Study: Longitudinal study with 12 people over 10 months. Participants divided in control, feedback groups. Two months for baseline, no feedback for both groups. After 6 months, real-time feedback was removed from feedback group.

Notes: The real-time feedback improved medication-taking behaviors and self-efficacy, but when it was removed after long-term use, performance was not sustained.

Disasters in Personal Informatics Ubicomp Workshop

Together with Jon Froehlich, Jakob Eg Larsen and Matthew Kay, I am organizing a Ubicomp workshop later this year. The theme: disasters in personal informatics. Here is a synopsis of what we are trying to accomplish:

In this workshop, our goal is to uncover, analyze, discuss, and learn from the failures of PI and QS research—failures that are most often not captured or surfaced in traditional publications because of embarrassment, perceived irrelevance, or simply lack of space. We want to provide an explicit forum to share stories of failure, perhaps even entire lines of research that did not succeed, in order to synthesize lessons learned and help progress the PI research community forward.

More details can be found on the official workshop web site. Please considering submitting – I believe this will be a rewarding experience for all who are working in the Personal Informatics space.

Variation of Serum Cholesterol in Man

Unlike many of my colleagues, I am not going to CHI this year. Instead, I am staying put in Atlanta and laying the groundwork for some of my upcoming studies. With my many April deadlines and to-dos behind me, I am looking forward to resuming my paper reading routine. This morning I set aside 20 minutes for a paper published all the way back in 1939, “Long Term Study of the Variation of Serum Cholesterol in Man“.

In this work, Turned and Steiner examined how cholesterol levels change in individuals over time. They had the opportunity to study 10 hospitalized patients over a period of 12-14 months at approximate weekly intervals. These patients were tested under different conditions (e.g., administration of thyroid medication, on a high-fat diet, on a low-fast diet, etc.). Thyroid admin resulted in drop in cholesterol and rise in basal metabolic rate, whereas a high fat diet caused a small increase in cholesterol. A low fat diet failed to influence cholesterol level.

As it becomes possible to monitor our physiology non-invasively with wrist-worn devices such as this and this, I am curious about which biomarkers might be fruitful to track, and how often.

QS Public Health Symposium



Last week I had the opportunity to attend the Quantified-Self Public Health Symposium organized at UCSD by Bryan Sivak (U.S. Department of Health and Human Services), Larry Smarr (Calit2 and ), and Gary Wolf (Quantified Self Labs). The intent of the one-day event was to provide a conducive environment for researchers, policy makers and developers of QS-type tools to discuss issues around data and its potential for use in public health. Many attendees such as Susannah Fox have written about the highlights of the day from their perspective. Here are my notes:

At the beginning of the day, Larry Smarr said he believes this is the “year of scale” when it comes to personal health informatics. Stephen Downs (Robert Wood Foundation) reiterated this point with a remark about our “rapidly expanding oceans of personal health data” and expanded on how RWJ Foundation has identified that a cultural change is needed around the value of health in society. Gary Wolf pointed out the uniqueness of the researchers involved in QS, ranging from public health epidemiologists and CS/HCI types, and also claimed that we are at “the beginning of a new program in the human sciences”, which I very much agree. Susannah Fox from the Pew Research Center made a number of excellent points based on the data she has been collecting and studying. When it comes to self-quantification, do “people want the unforgiving bright light of numbers in front of them”?, she asked. Fox also talked about how it is important to respect cultural rituals.

Brian Sivak was one of the major forces in the event. It was a pleasure to see someone from a government agency be so forward-thinking about the opportunities in this field. He discussed a number of topics, from areas he considers critical (e.g., data ownership, privacy and standards) to the possibility of a broad public health campaign around activity tracking. He also introduced the theme of sharing documents such as terms of service and privacy policy agreements amongst tool developers to facilitate the exchange of personal health data.

Later in the day, in a session reserved for “toolmakers”, Anne Wright from Fluxtream discussed how personal health data should be as portable as personal financial data, where one can download documents from banks and upload them to tools such as Mint and Quicken. In terms of visualizing and making sense of data, she mentioned the need for different types of interfaces at different levels (e.g., personal dashboards versus deep history introspection). Margareth McKenna from Runkeeper brought with her a number of research questions that are difficult to answer even in light of large amounts of data, such as what happens when people are not logging their exercise and activities (“sensors can only capture so much”). She also emphasized the need to disseminate tools to everyone, echoing others during the day who talked about the “digital health divide”. Andy Hickl from A.R.O. presented the “11-week problem”, at which point self-tracking devices go to the drawer. In other words, “what should we care?”, or how can we make self-tracking data meaningful? At the end of the session, Gary Wolf came back and proposed a research toolkit that could be shared within the community.

In one of the sessions focused on study designs, a highlight for me was Eric Hekler, a behavior psychologist and deep thinker around behavior change. He discussed how he lost faith in randomized clinical trials and how we should be asking people what they truly want from data and devices. He also made the point that we should support people to come up with their own interventions and find out ways to “get more functional life years”.

To me, the high-point of the day was a panel with Larry Smarr and Lee Hood. Hood, whose work has revolutionized biology and genetics, is leading a longitudinal, Framingham-like study with the goal of better understanding how digital technologies and ongoing self-tracking can be leveraged to quantify what it means to be healthy. Another goal of the study is to look at the progression of disease, as observed by these devices. Smarr, one of the most recognized self-trackers in the Quantified-Self movement today, started his tracking journey to understand his own personal health issues, in a way doctors could not, or were simply unwilling to listen.

In addition to the plenary talks and discussions, the day was filled with coffee breaks, where I had the chance to chat and exchange ideas with many researchers and developers, such as Kevin Patrick from UCSD and Mike Lee, who leads MyFitnessPal. Overall, a great day in the campus of UCSD.