Beware of the high promises of the Covid-19 “Tracker” applications

From the world started their battle against the Covid-19 outbreak, mobile apps have promised to do it all: identify infections, predict who may be most at risk, know how long the virus survives on surfaces, estimate the fraction of asymptomatic carriers, select medical resources To prevent people from exposing themselves, the list goes on. And while some mobile apps may be helpful as we adjust to life with this virus, there is also evidence that by skewing our understanding of this disease, certain apps are more harmful than helpful.

Currently, there are no fewer than seven major Covid-19 related applications in the US. If we count only those backed by reputable governments or healthcare organizations. Most of course will attract few users and disappear, but there is a mobile app that has already attracted attention for its surprising discoveries. The COVID Symptom Tracker was first released in the UK by a research team at King’s College, and has been promoted in the United States by Harvard and Stanford Medical Schools. The Symptom Tracker had over 1.6 million downloads in its first week of release in late March. The response was so fast and remarkable that the researchers needed just five days of data to trigger the first preprint of the scientific findings. But if the initial scans coming out of the COVID Symptom Tracker are a sign of what’s to come, then app developers have a lot of work ahead of them as they struggle with a large volume of low-quality data.

Every day, Covid Symptom Tracker users must submit a report on their health. You can also see an estimate of cases in your area. The app offers a self-diagnosis of Covid-19, which is not necessarily accurate but undoubtedly useful while governments evaluate and ration the tests. (Given the paucity of diagnostic tests, both the UK and US governments have restricted testing to people with severe symptoms.) However, an undesirable side effect of the targeted tests is contaminating the data that fuels subsequent analyzes, such as estimating the prevalence of the Covid-19 population and identifying the most relevant symptoms. This damage is exposed in the preprint of the scientific findings published by the Tracker application team. As this app and others like it become more popular, it’s critical that we understand what data coming out of self-reported symptom trackers can and cannot tell us.

Initially, the researchers’ goal was to use self-reported symptoms to predict test results. They started by gathering an analytical sample (also known as training data) containing symptoms and self-reported diagnostic test results. Given the rationing of the test, we can assume that all users included in the sample experienced severe symptoms.

The Tracker app study provided the first scientific evidence to support loss of smell and taste, or anosmia, as a symptom of Covid-19, which some doctors and patients had suspected since the early days of the pandemic. However, the researchers went further, stating that anosmia is the best predictor of infection, even better than the usual suspects, such as persistent cough. But the predictive ability of anosmia is nothing more than a mirage of triage tests.

In particular, approximately one third of the analytical sample tested positive for coronavirus, while two thirds tested negative, meaning that a perfect predictive model would mark one in three users of the application as positive for the virus. Obvious symptoms, such as persistent cough, affected half of the analytical sample, so using cough as a single predictor would have meant marking half of the users as likely to be infected. Therefore, out of every 100 users, 50 are predicted to be positive, but since 67 have reported negative results, we immediately know that 17 out of 50 positives are false positives. To have the best chance of correctly predicting all true positives while making the fewest possible false negative errors, the symptom should affect approximately one third of the sample. What proportion of users report loss of smell and taste? You guessed it, one in three. It is no wonder that anosmia is the diagnosis of Nate Silver from Covid-19.

Anosmia seems to better predict infection than cough just because it is less common in the analytical sample. And it is less common due to triage testing. Unlike cough, loss of smell and taste was not yet recognized as a Covid-19 symptom in March. So to get a test, you had cough, but not necessarily loss of smell and taste. And since the analytical sample included only the people who had been analyzed, it over-represented the qualifying symptoms. But this finding is unlikely to apply to the largest population.