Feedback for Improving Medication Taking

Over the next couple of weeks I will be writing several posts about CHI 2014 papers that caught my eye. I will format my summaries in 4 short paragraphs: Idea, Approach, Study, Notes. This structure will fit most papers I think. I will start with a paper by Matthew Lee and Anind Dey titled “Real-Time Feedback for Improving Medication Taking“. As I understand it, this is Matt’s PhD work at CMU under Anind. Matt is now at Phillips Research.

Idea: Study the effect of an instrumented pillbox and ambient display providing real-time feedback on medication taking. Research questions: (1) does real-time feedback improve consistency of medication taking?, (2) does real-time feedback improve self-efficacy for medication taking?, and (3) do medication taking behaviors change when removing real-time feedback after long-tern use?

Approach: Added various sensors and communication capabilities to a pill-box. Activity uploaded to server, projected back onto a visualization on a tablet that participants kept around the house.

Study: Longitudinal study with 12 people over 10 months. Participants divided in control, feedback groups. Two months for baseline, no feedback for both groups. After 6 months, real-time feedback was removed from feedback group.

Notes: The real-time feedback improved medication-taking behaviors and self-efficacy, but when it was removed after long-term use, performance was not sustained.

Variation of Serum Cholesterol in Man

Unlike many of my colleagues, I am not going to CHI this year. Instead, I am staying put in Atlanta and laying the groundwork for some of my upcoming studies. With my many April deadlines and to-dos behind me, I am looking forward to resuming my paper reading routine. This morning I set aside 20 minutes for a paper published all the way back in 1939, “Long Term Study of the Variation of Serum Cholesterol in Man“.

In this work, Turned and Steiner examined how cholesterol levels change in individuals over time. They had the opportunity to study 10 hospitalized patients over a period of 12-14 months at approximate weekly intervals. These patients were tested under different conditions (e.g., administration of thyroid medication, on a high-fat diet, on a low-fast diet, etc.). Thyroid admin resulted in drop in cholesterol and rise in basal metabolic rate, whereas a high fat diet caused a small increase in cholesterol. A low fat diet failed to influence cholesterol level.

As it becomes possible to monitor our physiology non-invasively with wrist-worn devices such as this and this, I am curious about which biomarkers might be fruitful to track, and how often.

ISTC-PC Summer 2013 Retreat

One of the sponsors of my research at Georgia Tech is Intel. My group, the Ubicomp group, is a member of the Intel Science and Technology Center for Pervasive Computing (ISTC-PC), and together with other academic institutions and researchers, we explore how sensing and computing technologies are changing the landscape of a variety of domains, such as health and wellbeing.

Every year, Intel organizes a retreat event with all ISTC-PC members. This year we all traveled to Suncadia, about one hour east of Seattle for 2 days of face-to-face conversation. Some of the topics covered included the future of sensing, privacy & security, activity recognition and behavior journaling.


It was a very productive event with good discussions and the opportunity to meet many friends in the research community. And the location was truly spectacular.

On-Body Sensing for Dietary Monitoring

As my research progresses, it is becoming clearer that my dissertation topic will focus on systems that help people keep track of what they eat. This is an interesting problem because it is not possible to track diet as easily and objectively as physical activity, which one can capture using an activity tracker like the Fitbit. Yet, diet is critical as a health determinant.

There is a large body of work around automatic dietary monitoring (ADM), and Oliver Amft (TU Eindhoven) and Gerhard Troster (ETH Zurich) have explored this domain fairly extensively from the perspective of on-body sensing. Today I decided to take a closer look at one of their papers, an article for the IEEE Pervasive Computing magazine titled “On-body Sensing Solutions for automatic dietary monitoring“. It was published in 2009.


The article discusses the motivation for ADM and the weaknesses of self-report techniques. One of the highlights is a discussion of ADM on-body sensing options, ranging from gastric activity to the thermic effect of food intake (TEF):

“TEF starts imme­diately after food reaches the stomach and peaks after roughly 60 minutes. For unrestrained eating in people of normal weight, skin temperature above the liver increased between 0.8 and 1.5K”

The approach they chose to investigate in more detail involves modeling the intake cycle, covering intake gestures, chewing, and swallowing using a probabilistic context-free grammar. They frame it as a hierarchical recognition task that is constituted of detecting a sequence of sub-events.

It is a good read.

The Real Win: Data + Collaborations

At TEDMED this year, Deborah Estrin made a strong case for leveraging our “digital breadcrumbs” to “help us get a clearer picture of our personal health”. She referred to these “breadcrumbs” as our “small data”, and told the story of her father’s decaying health; while invisible to his caregivers and doctors, his changing daily behaviors could have been picked up by his phone and computer. She finished her talk with a call-to-action: we need to get web services and mobile providers to give us our data back so that we can use it for our own purposes, such as for personal health inferences.

Needless to say, I agree with Estrin. Collecting digital traces of our lives is something that I have been doing for almost 10 years now, first with Slife Labs and now in my research. In fact, being plugged-in to this concept has forced me to pay attention, and as a result, I’ve seen many efforts with similar messaging come and sometimes go (hello Attention Trust, YouData, the Locker Project). In my opinion, however, we don’t necessarily need access to what is centrally stored about us by AT&T, Verizon and Google to make an impact. By working at the node level, by installing software on our phones, computers and soon TVs, we can collect all the data we need to build predictive models of human behavior. In fact, this has been done, and Estrin knows this too – she mentions the company in her talk. Additionally, many of these companies already provide APIs that give us access to a lot of the data we would want. From my experience, the biggest barrier to realizing the small data vision that Estrin proposes is getting the green light from the medical community, to not only to validate and refine these approaches but also incorporate them into traditional medical practices. That’s where the challenge lies; I do not see access to data as a barrier, at least not at the personal level.

Another significant health initiative related to personal activity data tracking was announced yesterday. The California Institute for Telecommunications and Information Technology (Calit2) in San Diego and the Robert Wood Johnson foundation unveiled a project called Health Data Exploration, with the goal of bringing everyday activity data to researchers. In the spirit of the Quantified-Self movement, it is framed as a conversation, an exploration, and not as a research program per se. But if I am correct, the ideal outcome would be to have companies like FitBit, Jawbone and Nike open their activity databases for “the public good”.

There is a clear overlap between these two initiatives. I see the value of the enormous amount of data that companies like Fitbit have compiled. Analyzing that data would certainly lead to new insights about health and human behavior, so I am curious to see how the project unfolds. But again, as a researcher, today I feel that I am more restricted due to the skepticism of the medical community towards adopting new strategies and technologies than by lack of data. Even starting collaborations with medical professionals has been a challenge.

Here is what would be very exciting to me as a computer scientist working in the health domain: an avenue to work directly with doctors and patients. I am not sure how this could be structured, but in an age when health accountability matters, it might be possible to incentivize healthcare institutions to work alongside technologists more closely. Yes, I am aware of the difficulties associated with the regulatory process, getting FDA approval, dealing with HIPAA, etc, but my perspective is that we will have to deal with these hurdles eventually no matter which road we take. In practical terms, I would love to see a resource, perhaps a web site, acting as a meeting place where electrical engineers, biomedical engineers, computer scientists and even designers could propose studies to healthcare organizations around new and innovative technologies and approaches. One of these studies could be Estrin’s small data, perhaps applied in the context of congestive heart failure or Alzheimer’s Disease. Doctors and caregivers would visit this resource and possibly sign up to collaborate, depending on their needs and areas of specialization. Realistic? I am not sure, especially considering that physical presence matters in these kinds of collaborations. But to me the value of a project like this is crystal clear. Especially if doctors and scientists really do show up to the party and decide to dance together.

Lively Revisiting Home Monitoring

One of the reasons why activity recognition in the home is interesting to me is because it has the potential to enable so many important health applications, such as remote monitoring. There are many people, especially older adults, who would much rather stay at home while battling chronic diseases than move into an assisted living facility or hospital. But without a range of supportive services, which are often prohibitively expensive, it becomes virtually impossible to properly care for someone in their own home.

There is a wide range of commercial products centered on supporting independent living, from fall detection to medication compliance systems. One piece of the puzzle that is missing is a communication channel that offers caregivers a holistic view of an individual’s patterns of daily living at home. This would enable caregivers to observe everyday behaviors on a regular basis and hopefully anticipate problems.

Today I read about the Lively system, a sensor network and base station designed for eldercare remote monitoring. It combines a number of wireless sensors that could be attached to objects:

One type of sensor goes on pill boxes, while another measures whether people are eating and drinking on a regular schedule by indicating when refrigerator or pantry doors are opened, both using accelerometers. A third variety is a key fob with a Bluetooth Low Energy transmitter than lets the server know when the user is out of range, typically 125 meters (about 410 feet). This measure acts as a proxy for indicating when the person has left home.

Researchers have attempted to use sensor networks this way, with moderate success. For example, Tapia took the idea of sensors in the environment and showed how one could learn more about an individual’s high-level activities from low-level sensors. Rantz and Skubic demonstrated how a sensor network could be used as an early-warning system for conditions such as urinary tract infection in older adults. The only limitation of these system has been the large number of sensors required, sometimes in the order of 50-80 sensors per home. That is too many.

It’s clear that there is room for sensor networks in health monitoring at home, and Lively is betting that a few strategically located wireless sensors can tell us most of what we need to know about someone’s well-being. I do agree with this direction, and I am now curious to see how the company does in the future, including whether it raises significant funds through Kickstarter, which is a good indicator of how demand exists for a product like this.

The Privacy of Smartphone Health Sensing

It seems like every other day a new story in the media reports on the rise of the smartphone as the uber-sensor for medical applications. Yesterday I had the chance to read the New York Times story “Apps Alert the Doctor When Trouble Looms“, which highlights apps that monitor patterns in device usage (e.g. phone calls made, text messages received) and link pattern deviations to a variety of possible medical conditions.

The notion of extracting behavior features from smartphone usage is an idea I am fond of. I am following, the company that brought this idea to market, very closely. So, naturally, I read the story with interest. Surprisingly, the most remarkable aspect of the story was not the story itself, but the comments from readers.

As of now, 8 out of 10 comments note that the notion of using a smartphone as a diagnostic tool or to alert a doctor is preposterous. The majority of these negative comments were “upvoted” using the “Recommend” link that the New York Times publishing platform provides. Many comments touched on the privacy issue:

This is insane Big Brotherism

No way around it, such is a Violation of Privacy!

An interesting perspective came from someone battling chronic conditions:

As someone dealing with several chronic conditions I feel qualified to state that this is the worst idea I have ever read. The possibilities for the app to go wrong are endless.

A doctor also chimed in:

You want to know the first thing i thought about as a doctor? Picturing me getting sued by a patient or their family and the lawyer saying: this app shows that you received this information, yet there is no record that you acted on it. im literally supposed to act on it, pull the chart, and then note what the app said and what i did, and why i did it, and why i didnt do something else. tort reform before i use this app.

Two observations.

First of all, the variety of viewpoints on an important topic from many of the represented parties constitute, in my opinion, one of the best practical examples of Social Construction of Technology theory (SCOT) at play I have ever witnessed. In a few words, human action shapes technology and technology is interpreted differently by different social groups.

Secondly, as we move forward developing technologies that assess an individual’s health condition from everyday behavior, do we need to pay more attention to these loud voices asking us to stop? Or should we remember that just 10 years ago, the idea of sharing what we share today on social networks was equally ludicrous and ended up going mainstream anyway? As a technologist, I am not surprised to find myself leaning towards the latter. But I do believe it’s time for more effort to be put into thinking deeply about the privacy implications of this work.

What do you think?

ISTC-PC 2012 Retreat

Due to a very hectic schedule in the last few months, which included a paper deadline, the Ubicomp conference, a qualifying exam presentation and the beginning of classes, I haven’t been able to blog here nearly as often as I should. With a little bit more breathing room now, I will try to catch up in the next post or two.

Let me start with the ISTC retreat. Back in August, I had the opportunity to attend the first ISTC-PC retreat in the beautiful Alderbrook Resort & Spa, located about 2 hours from Seattle by car and ferry. It was a great event, and an opportunity to meet up with other researchers that comprise the relatively new Intel Science and Technology Center for Pervasive Computing.

Many professors and researchers from the University of Washington were in attendance and it was wonderful to see old friends and meet new ones. Intel was well represented and sent a large entourage of engineers and executives from many relevant units, from embedded systems to mobile devices and personal computers. I found it particularly rewarding to meet and have conversations with Deborah Estrin (Cornell-NY), Tanzeem Choudhury (Cornell) and their students, since we all share a deep interest in health modeling and applications.

It was encouraging to hear that Intel is very eager to enable our vision of the future as well. In particular, one of the issues they are addressing is how to make mobile device sensors less dependent on the CPU and associated chip infrastructure in the interest of conserving battery life. Designing so-called “sensor hubs” might be one direction to pursue.

Overall it was a great experience and I am looking forward to more interactions with ISTC collaborators and Intel in the future.

Recognizing Home Activities with IMS

Today I am in Pittsburgh for Ubicomp 2012, presenting our work in recognizing activities in the home using infrastructure-mediated sensing:

Recognizing Water-Based Activities in the Home Through Infrastructure-Mediated Sensing

Activity recognition in the home has been long recognized as the foundation for many desirable applications in fields such as home automation, sustainability, and healthcare. How- ever, building a practical home activity monitoring system remains a challenge. Striking a balance between cost, pri- vacy, ease of installation and scalability continues to be an elusive goal. In this paper, we explore infrastructure-mediated sensing combined with a vector space model learning ap- proach as the basis of an activity recognition system for the home. We examine the performance of our single-sensor water-based system in recognizing eleven high-level activi- ties in the kitchen and bathroom, such as cooking and shav- ing. Results from two studies show that our system can es- timate activities with overall accuracy of 82.69% for one in- dividual and 70.11% for a group of 23 participants. As far as we know, our work is the first to employ infrastructure- mediated sensing for inferring high-level human activities in a home setting.

Edison Thomaz, Vinay Bettadapura, Gabriel Reyes, Megha Sandesh, Grant Schindler, Thomas Plo ̈tz, Gregory D. Abowd, Irfan Essa

Download the full paper and leave your questions, comments and suggestions here.

Health Sensors Approaching Escape Velocity

Rock Health, a technology/startup incubator focused on health companies and applications, just released a very clear report on medical and wellness sensors. It’s a graphical, somewhat summarized version of a post I wrote a while back about health sensing technologies. Definitely worth a look:

The key message here is that health-focused sensors will be big. They project hundreds of millions of devices by 2014.