For the past eight years, Google Health has been shorthand for tech’s hubris in healthcare. A cautionary tale. The search giant’s failure to drive adoption of Google Health, the name of its unsuccessful and untethered personal health record (PHR), was a topic the industry mulled over for years.
One common conclusion was that Big Tech just didn’t get healthcare.
In the years since Google Health’s demise, Google launched a biotech company called Calico. It spun up a life sciences company named Verily, which, in turn, helped create various new entities, including a joint venture with Sanofi named Onduo. It has grown its Google Cloud for healthcare business substantially. And yet, it wasn’t until recently that the company began to reclaim the old Google Health brand.
While it has had a couple of cringe-worthy, hyped projects (like glucose-sensing contact lenses and noninvasive, optical sensor-based cancer diagnostic devices) for the most part, Google’s health initiatives have been quiet R&D projects. Instead of splashy consumer health launches, Google has worked closely with academics, medical researchers, and clinicians. They’ve focused on the cloud.
You might argue that — at least in some parts of the company — Google found a little humility since announcing the shuttering of their PHR in 2011.
This recent resurrection of Google Health as a business unit is a sign that the company is ready to go-to-market again with some ambitious products. This report will focus on one that has yet to be announced by the company, but it has been years in the making.
I predict Google’s next big health product is being built around a previously unrevealed acquisition that Google made a year ago — well before it had its eyes on Fitbit. While the transaction was a few orders of magnitude smaller than the $2.1 billion Fitbit deal, this one will open up a new front in its competition against Apple and Amazon for digital health dominance.
Google Health is back. This report will give you a better sense of what happens next.
Executive summary: Google Health is building a health-sensing, camera-equipped monitor under its Nest brand.
In September 2018, Google Health acquired a San Francisco-based startup called Knit Health, which had developed a smart baby monitor, called the Knit Monitor, and recently pivoted to focus on physician-referred, pediatric assessments of sleep and respiratory issues in kids of any age. Most of the Knit team joined Google Health, which also acquired its IP.
Google was already working on a similar product at the time of the acquisition with contributions from Nest’s CTO and the founder of Senosis, a health startup the company acquired in 2017.
Google Health’s forthcoming product is likely to be positioned as the first health-focused device and service from Nest. I predict that while it will offer a number of features for monitoring infants and young children, the device will offer health-monitoring for people of all ages, including seniors.
Project lead: Yoky Matsuoka, VP of Google Health, resigned last month.
As with any pre-market product at a Big Tech company, there is a good chance the one described above won’t see the fluorescent overhead lighting of CES. An exciting patent from a Google engineer is not enough to build a case that a product launch is imminent. (If it was, we’d have seen that Google’s heart sensing toilet seat by now.)
While I think the case for this project is strong, one red flag popped up while I was pulling this research together: Google Health’s Vice President Yoky Matsuoka, who happened to be leading this project, resigned in October to take an executive role at Panasonic. Few media outlets besides Reuters and The Japan Times covered her recent move, but her past job changes have been well-documented:
- Matsuoka was a co-founder of Google X.
- She was the VP of Technology at Nest for years and through its acquisition by Google.
- She was a CEO at (now out-of-business) Quanttus, the health-sensing wearable startup that fed a lot of talent to Apple.
- Like many of her coworkers, she then joined Apple’s health team.
- She then boomeranged back to Google as Nest’s CTO before changing her title to VP of Google Health when Google folded Nest into its new health unit last fall.
- After less than two years, she left to join Panasonic in October 2019.
We’ll get back to Matsuoka, but first:
A brief history of Knit Health, the startup Google secretly acquired.
David Janssens, Paul Silberschatz, and Evan Shapiro — three alums from well-known design consultancy, IDEO — founded Knit Health in February 2015. About 18 months later, Janssens described the company’s product as the “world’s first human health sensor using a camera, deep learning, computer vision, and human-centered design.” Here’s how Janssens saw Knit’s opportunity, which went well beyond simple sleep tracking or basic baby monitoring:
Human sleep is an uncharted frontier of health information, and a camera is the perfect sensor for studying health during sleep. Sleep is a natural barometer of our bodies. When we’re stressed, we don’t sleep well. Days before the obvious symptoms arrive, our sleep patterns can reveal the onset of illness. With the right camera, breathing and heart rate can be measured, even through a down comforter. Those measurements can help determine short-term issues like a cough, or long-term chronic issues like asthma. Sleep itself has a major impact on our short-term and long-term health. When we sleep well we have more energy to exercise and eat better. We are also less likely to be obese, have hypertension, diabetes and mental health issues like depression. Sleep is the perfect opportunity to capture continuous data about our health every day.
Knit Health’s device eschewed the cloud computing trend for edge computing.
Most connected baby monitors or wearables stream video feeds and data to a parent’s smartphone locally over their home WiFi network. If the monitor includes any kind of service for analyzing the video feed or crunching the data coming off an ankle-worn baby monitor, that data is likely sent to the cloud for remote analysis.
Knit Health’s hardware strategy bucked the cloud computing trend in favor of a smarter device that analyzed the data stream in real-time on the device itself. Pricing and processing power of chips have come down enough to enable the Knit monitor to work this way, but it also wouldn’t have worked otherwise.
Knit’s co-founder and CTO Evan Shapiro explained during a podcast interview in 2016 that if the device compressed the video feed and transmitted it via the average home WiFi network, the compression artifacts that would have been added to the data would make the company’s sensitive computer vision algorithms useless. Knit needed to analyze the raw, uncompressed video right on the device itself, or it couldn’t detect the signals it needed to for its analysis.
Keeping this sensitive data and video stream out of the cloud also makes security and privacy issues less of a concern. Longterm, the local neural network embedded in the user’s personal Knit device might become increasingly tuned into their child’s individual habits and possessions — the company hoped.
During that same podcast interview in November 2016, Shapiro explained how embedding the intelligence in the camera enabled Knit’s computer vision-powered analysis to be incredibly sensitive to tiny motions:
We can see respiration through swaddles, blankets, sleep suits. We have had kids bury themselves in piles of stuffed animals and we have been able to detect their respiration signal. So, we are pulling out signals that are in the video that are really too small to see with the unaided eye. With a bit of signal processing and the context of knowing exactly where that child is and which features in that video are which parts of the child, versus a stuffed animal, part of the crib, or surroundings, we can actually extract that information.
Crowdfunding the Knit Learning Health Monitor to $100,000
Knit launched a Kickstarter for their original baby monitor offering in November 2016, and that resulted in a successful campaign that cleared $100,000, which was the stated goal. The device was named the Knit Learning Health Monitor in the campaign and it was bundled with one-year of insights via the Knit Assist subscription service.
The crowdfunding campaign’s video opened up with a pitch to parents: If your baby gets a good night’s sleep, so do you. And that makes you a better parent. The original mission of the company, according to the video, was to solve human needs around parenting and sleep. Janssens, beta customers, and other early Knit team members explain the offering like so:
It’s a camera that goes on the wall above your child’s crib or their bed. It connects to an app on your phone. The camera has a brain. It can actually think and process what it sees. Like whether they are asleep or awake. We can detect crying or coughing. It can track tiny motions associated with them breathing in their chest. And what we provide to you when you wake up is a sleep graph of their night. We provide sort of the story behind it. Tied to that, I think more importantly, are video clips that let you know what happened when they woke up. The thing that you want to know from your monitor at night is whether your baby is breathing. You can actually see the movement because of a color processing change, and you’re like ‘ah, question answered. I can go back to sleep now.’
Knit’s co-founder and COO Paul Silberschatz says in the video that the service also provides parents with suggestions for how to help their children get better sleep. That might include changing the temperature, the light level, and maybe even whether they sleep better with a particular stuffed animal. Early testers of Knit’s service say that when they check their child’s sleep graph from the night before and find that they didn’t sleep well, it might change what they had planned for the day given their kid might be more emotional and tired during the day. They called Knit’s data “the single best predictor for how [our kid’s] day is going to be, which is huge for us.”
Anna Shaw, an early Knit employee focused on user experience, explains in the video that Knit wants to grow with parents and help explain their child’s sleep habits as they get older. Janssens goes on to hint at Knit’s ambitions beyond babies and says that what Knit is developing “could be used in a variety of ways to help a lot of people”.
Evan Shapiro, Knit’s CTO and co-founder, concludes the video pitch by explaining why they wanted to do a crowdfunding campaign. It boiled down to a need for a larger dataset to fine-tune Knit’s algorithms and user experience. Shapiro said the device is complete from an engineering perspective and has already been tested in “users’ homes for months on end” as of October 2016. “This product is ready to go,” he said.
While the Knit Monitor’s crowdfunding campaign was successful in reaching its $100,000 goal, repeated delays in shipping the products over the course of the following year led to many refunds. Ultimately, Knit announced that it was changing its focus from monitoring babies.
Knit’s “difficult decision” and a pivot to 14-day pediatric sleep assessment service.
In May 2017, Knit sent an email to its backers announcing that it had made the “hard decision” to broaden its focus for the Knit Learning Health Monitor, which meant it would no longer be focused on infants exclusively. The company offered refunds because its Kickstarter promised a one-year subscription to Knit Assist, the analytics service portion of its offering.
By September 2017, Knit articulated its new focus as “helping assess sleep issues in children to help families get the help they need.” Its marketing materials shifted from testimonials from parents of infants or toddlers to parents of kids aged 7-, 11-, and 13-years-old. Knit began to position its offering as a way to help parents ensure their kids got enough sleep to enable them to grow and develop into healthy adults. Knit’s blog began to focus on how it can help with issues related to ADHD, screen time, jet lag, circadian rhythms, poor diet, acting out in school, tonsil removal, fear of the dark, dentist-recommended oral appliances, bedtime routines, and more.
Knit Health began to position its device and service as an alternative to a professional sleep lab. One blog post boasted:
The Knit Health technology is completely unique in the field of sleep health assessment. Understanding the science behind Knit reveals why it is the only in-home pediatric sleep assessment solution that can signal the presence of potential sleep issues and provide daily feedback necessary for improving long-term sleep health.
Knit Health no longer sold its product as a longterm monitor, but rather as a 14-day guided assessment:
Our intelligent camera detects motion, breathing and sleep stage, generating the caliber of data only available in a sleep lab — even when your child is under their blanket. With computer vision technology Knit can capture the moments that matter, making discovery of sleep issues possible, anytime during the night.
In order to fully analyze your child’s sleep, we need to capture 14 nights of sleep data. This enables our technology to evaluate routines, breathing, quality of sleep and more. During that time you’ll have access to the Knit Sleep Assistant app where you can track nightly sleep metrics, and a dedicated Knit Sleep Guide in the event you have questions.
At the end of your assessment, you’ll receive a personalized sleep report that includes in-depth, actionable insights and easy to implement recommendations from our sleep experts. The comprehensive report is yours to keep and share with pediatricians or other providers.
Knit’s website asked visitors to type in the name of the physician who referred them before the company would send them a device. For visitors without a referral, Knit invited them to join a waitlist.
The pivot was first announced in May 2017, and a little more than a year later, Google acquired Knit in September 2018.
Knit’s monitor always worked for people of any age, but just one at a time.
Before we transition to Knit’s move into Google Health, a few areas are worth digging into a little. The first is that Knit’s technology worked with any aged human from the beginning. Even as the startup was in the midst of its crowdfunding campaign for its baby monitor, CTO Evan Shapiro acknowledged that it worked on people of all ages:
It works on humans in general. It works on adults. The constraints right now are that we don’t handle more than one person or person and pet per bed. If you put it on an adult, which we’ve done — teenagers, 10-year-olds, young toddlers, young infants — it is smart enough and has a generalized sense of what a person is to work. It is really designed for babies — maybe newborns until about three-years-old. That’s kind of the starting target market. But we definitely have users who have older kids who use it for various reasons and it will work on anyone really.
Knit recognized that pointing a camera at an infant was already an accepted behavior for parents. Paying a company to record and analyze your health as you — an adult — slept might be harder to accept from a privacy standpoint.
Knit Health’s anti-wearable ethos.
Considering Google’s ongoing $2.1 billion acquisition of Fitbit, it’s worth pointing out here that a key marketing message for Knit was that its offering required no wearables. The most obvious sign of the importance of this messaging was in Knit’s crowdfunding campaign’s title: “Knit Monitor —Tracks Sleep & Breathing w/o Wearables.”
One way to think about Knit’s anti-wearable ethos is to consider its competition. Knit could easily dismiss competition from basic baby monitor cameras, but some of its baby-focused, health-sensing competitors — like Owlet — used wearables to track infants while they slept. Knit CEO David Janssens fleshed out the company’s anti-wearable strategy in a Medium post that coincided with the company’s Kickstarter campaign. In it he also admitted he had stopped using Fitbit devices (oops):
The data sounds exciting, but this stuff is so frustrating. The day my Fitbit lost its first charge was the day I stopped tracking my steps. A wearable asks too much of us: remember to charge it, remember to wear it, remember to turn it on. And don’t get me started with wearables for kids (imagine cleaning a wearable after a diaper blowout). What if a device existed that helped you understand your overall health, but you didn’t have to wear it, charge it, or turn it on? It could provide more context, more accurate personal data AND remember everything for you. Enter cameras.
Knit Health’s two patents.
In addition to the majority of Knit Health’s team joining Google Health as new employees in September 2018, Knit Health also assigned its patents to Google in October 2018.
At the time of the acquisition, the Knit Health team was working on two related patents, which named the inventors as the three Knit co-founders mentioned above as well as Knit Health’s lead engineer Adam Carlucci. Knit’s first patent was entitled, “Remote biometric monitoring system“. Its second patent was titled, “Subject detection for remote biometric monitoring,” and included the same four inventors. Knit’s second patent seems to have been in the works prior to the acquisition but filed with the USPTO following the deal.
Meanwhile, Google’s Nest team develops its own health-sensing baby monitor.
A few weeks after Knit Health concluded its successful crowdfunding campaign, Yoky Matsuoka rejoined Google as the new CTO of Nest, which was an Alphabet company at the time and not yet rolled into Google Health. By the time Knit Health was processing its last refunds to its crowdfunding backers, Matsuoka and her team were busy filing patent applications for similar technology that monitored infants and seniors in the home.
The first of these patents was entitled “Thoughtful Elderly Monitoring in a Smart Home Environment”, which Matsuoka filed in October 2017 with one of Nest’s lead engineers, Mark Stefanski. Matsuoka next filed a patent solo in December 2017 for a way to launch a one-way visual communication session from a tactile user interface, which sounds like a way to start a video call to a senior. In February 2018, Matsuoka is named on three patents related to health-sensing and infants: Enhanced visualization of breathing or heartbeat of an infant or other monitored subject; infant monitoring system with video-based temperature baselining and elevated temperature detection; and infant monitoring system with observation-based system control and feedback loops. The patents could just have easily been describing much of what Knit Health was building at the time.
Matsuoka’s team for the infant-related patents included Nest engineers Mike Dixon (all three patents) and William Greene (just one). The other name on all three of the patents might be a familiar one to diligent digital health readers: Shwetak Patel.
How Nest’s 2017 acquisition of Senosis factors in.
Shwetak Patel is a research at the University of Washington who founded a health algorithms startup called Senosis. Rumor had it that Google had quietly acquired Senosis from Patel and his employer, the University of Washington, in mid-2017. Because of the school’s government funding, journalists at Washington state-based GeekWire were able to use a Freedom of Information Act request to learn more about the acquisition when they received the documents in September 2018. GeekWire could then reveal that Nest had actually acquired Senosis, but didn’t want the school, Patel, or anyone else to discuss the deal until after a Nest health product went to market. According to GeekWire, here’s what a lawyer from Google told the school right before the acquisition was finalized:
“If a Nest health product doesn’t use the licensed technology, an unrelated press release regarding Nest’s acquisition of (Senosis) could again negatively affect our product rollout.”
As GeekWire explained, “Senosis aimed to turn smartphones into monitoring devices that collect health metrics to diagnose pulmonary function, hemoglobin counts and other critical health information.” Its apps included SprioSmart, SpiroCall, HemaApp, and OsteoApp. Senosis also had an algorithm that detected jaundice in an infant.
What’s ironic about the timing of that GeekWire report is that September 2018 was the same month that Google acquired Knit Health. Since Senosis was acquired by Nest in mid-2017, that also means both Knit Health and Senosis were acquired under Yoky Matsuoka.
Whether or not technology from Senosis makes its way into Google Health’s device is an open question, but it is clear that Senosis founder Shwetak Patel was working on the product with Yoky Matsuoka and Nest’s engineers.
Google Health and Knit Health both saw the senior market as an opportunity.
In June 2017, just before Nest acquired Senosis, Yoky Matsuoka said during an interview that Nest’s mission statement indicates that the company should move into home health. Nest’s mission statement at the time was “to create a home that cares for the people inside it and the world around it.” Matsuoka said the reason she rejoined Nest and Google was to pursue this idea of creating Nest products that care for people in their homes. Given that we spend so much time in our homes, and a lot happens in our homes, Matsuoka said it made sense to focus there, “whether that’s for the elderly, or younger people, or people with disability.”
Matsuoka goes on to hint that Nest would not create a product that specifically targets the elderly population:
Elderly specific technology doesn’t sell because as we get older we will never admit that we’re getting older. We would never buy something that’s tuned for the elderly because we don’t believe it true. So I think that’s another reason that I bet the way we enter the elderly market is to not build something very tailored for the elderly but [that works] for everybody. [It] just turns out to be really really helpful for the elderly.
A year later in the summer of 2018, CNBC reported that Nest was in discussions with senior care facilities about how smart home products may be helpful in caring for the elderly and enabling independent living.
Knit Health’s CTO Evan Shapiro also said that the senior market was one vertical that Knit had pegged as a potential future market.
How this might fit into Dr. Feinberg’s vision for Google Health.
In November 2018, Geisinger Health’s CEO Dr. David Feinberg joined Google as Head of Google Health, a new business unit that would bring together various health-related businesses under Google, including Nest, Google Fit, and Google Brain. At the big HLTH event in Las Vegas a few weeks ago, Feinberg gave a short presentation titled, “Meet Google Health” that promised to reveal what Google was working on today and its vision for the future. The presentation was short on details for anything patient or consumer-facing, but — while it’s a stretch — one general framing from Feinberg caught my ear:
We think of hospitalization by and large as a failing of outpatient care, and outpatient care is a failure of home care. And, fundamentally, home care is a failure of community care. So, you’ll see, and we look forward to sharing with you, ways we can move patients in that direction.
What Knit Health built is the beginnings of a bedside monitor. Google Health may take a similar approach to Knit’s initial tack and begin selling these devices as a health-sendings baby monitor under the Nest brand, but they could be useful for anyone in a bed.
As Feinberg’s presentation made clear, Google Health has a ton of seemingly disparate health initiatives underway. He spent a good amount of time on the various conditions and medical issues Google Health has been able to detect from eye scans. I mentioned the algorithms from Senosis above, but there are many other projects like these in Google Health and Verily’s R&D teams. You have likely heard about many already. Just to make the point, here’s one example you might not have come across yet:
Jeffrey Rogers’ secretive Project Prime work for Google (2014-2015)
Project Prime was a two-year project lead by Jeffrey Rogers, a director of engineering at Google from 2014 to 2015, who is now a research lead at IBM Research. The project ended up finishing early in about 20 months, and it produced a dozen patents that named Rogers as an inventor. A few of the patents name Verily’s Brian Debusschere. One names a different ex-Verily engineer, Paul Reid, and another names Google self-driving car engineer Paul Karplus.
Here’s how Rogers described Project Prime on LinkedIn:
Executive 2-year position to define and demonstrate a new business opportunity at convincing scale. Created Project Prime; an internal start-up focused on healthcare/medical technologies. Defined vision (including target disease), technical approach, business opportunity, and secured funding. Formed and lead diverse skilled team balanced to meet 2-year timeline; hiring direct reports and external contractors. Key Accomplishments—Defined criteria to predict acute cardiac events from validated clinical data early enough that they could be avoided (up to 30 days in advance). Successfully met 24-month goals in 19 months. Developed 6 sensors to measure needed physiological signals, validated sensors met requirements, developed software and analysis backend, and deployed system to 4 hospitals. Defined commercialization roadmap and established needed partnerships. Attained FDA clearance for clinical use of 1 of the developed sensors.
Conclusion: Google Health’s Island of Misfit Toys problem
Ten years ago if you said the words Google Health, most people working in health IT knew you were talking about the tech company’s PHR. Google had an almost singular focus at the time for its health initiatives. As we move into 2020, Google has the opposite problem. Its decade of medical and health-related R&D has created an Island of Misfit Toys problem within the company. A big part of Feinberg’s job over the past year had to be focused on deciding which to kick off the island.
Considering the acquisitions of Knit Health and Senosis, Yoky Matsuoka’s two years working on the project, and the Knit Health team mostly sticking around to continue working at Google Health, I think it’s a fair prediction to say that Google will roll out a health-sensing home monitor under its Google Nest brand in 2020. While Matsuoka’s departure last month might indicate that Feinberg canceled the project, I think it’s more likely she left because much of the R&D work had been completed and her track record isn’t to stick around in one gig for too long.
Will we see a Google Nest Health Monitor at CES in January? I’ll update this section then.