Inspiration
Melina has been displaying Covid-19 symptoms for a few days. She has been advised home care by doctors. Her symptoms went from bad to worse when she finally got to the hospital.
Empirical evidence has shown that Covid-19 cases can turn critical abruptly, even after symptoms have improved, often within a couple of hours. With limited capacity, it is challenging for hospitals to proactively keep regular track of symptoms of ambulant patients across rotational shifts.
We propose YourHomeNurse - a system for automated regular calls to patients, reporting their symptoms, tracking their conditions on an online dashboard and triggering alarms in critical cases.
What it does
We propose a service of automated regular phone calls to create individual track records of Covid-19 specific symptoms.
The system can be operated from a hospital or health ministry where medical personnel oversees the incoming anamnesis results and track records for an effective triage - and can take immediate action.
For patients, phone calls present a familiar and convenient way to stay in contact with their healthcare providers. No tedious installations, no forgotten passwords, no fiddly logins. After a simple signup giving contact information and preferences for the monitoring (e.g. regarding the frequency) patients receive regular phone calls (e.g., twice per day) to collect in a conversation-like dialogue their symptoms which are reported to a dashboard. This dialogue can as well offer the option for connecting directly to a doctor.
How we built it
We use state-of-the-art speech and language technologies for building the corona symptom diary tracker. This system is based on previous work we have done developing call bots in financial industries and for voicebots in the context of typical phone service use cases in clinics. It applies algorithms from the realm of Artificial Intelligence and, more specifically, from the automated processing of natural language (NLP) as well as text-to-speech language generation and speech-to-text language recognition techniques.
In more detail, the call bot system has three main components: (1) the speech to text system (ASR) that transcribes what the patients are speaking into text, (2) the chatbot system that understands the meaning of the given text, records to the patient’s database and gives a response in text format, (3) the text to speech system (TTS) that produces natural pronunciation for the output text of the chatbot and replies to the patients in the phone line. This call bot system uses the SIP communications protocol to serve contents generated by server sessions to patients over public telephone networks in real time.
The main component of the system is illustrated in the follow diagram:
The "brain" of the system is the chatbot logic with pre-defined conversational flow that considers the context of the conversation (e.g., the previous symptoms, current conditions, additional questions from the patient). The policy tracker takes into account symptoms detected from the system and detected intents from responses of patients through the phone line. It connects to the skill manager that defines the current flow of the conversation, checking the current state of the dialogue, recording the information into the database and that finally generates next questions or answers returned to the patients. Finally, the symptoms recorded are being evaluated and trigger alarms when necessary. Technically, the intent classifier and symptom - named entity detector employs neural networks with word embeddings and Bert embeddings for different cases with accuracy over 90% for general domains.
The solution’s impact to the crisis
Our solution presents an efficient way for the ongoing remote assessment of ambulant Covid-19 patients. Facilitating regular oversight is particularly relevant for the Covid-19 specific disease cycle which can abruptly turn critical.
As an effective tool for triage, it supports medical personnel on the ground in directing the available capacity to the critical cases for a timely treatment. This will hopefully help to lower the mortality rate and contain the infection. We imagine that with a growing database, the system could be developed further to recognise patterns in the disease cycle and to classify and flag critical cases or even preventively trigger alarms to search for doctoral support.
The prototype
We have conducted interviews with a healthcare management consultant, with ICU nurses and with doctors in four countries: Spain, Germany, India and the UK. Through the interviews, we have collected ideas, suggestions and developed a better understanding of the situation on the ground.
Special thanks to all of the nurses and doctors that were willing to help us during the hackathon!!
We have developed a working prototype for automatically collecting and tracking symptoms over the phone through conversation. For doctors, the data collected is presented on an online dashboard for easy assessment.
You can listen to our prototypes through the following phone lines:
- Corona-Anamnesis (Germany): +49 711 26898013
The voicebot asks you for your Corona-related symptoms. It notes down and reports your symptoms to a system using auto transcription (speech-to-text) and voice recognition.
- Corona information and statistics updates:
Infoline UK: +44 20 3389 5018
Infoline Germany: +49 711 26898119
The phone line with a voice bot that provides you updated information and statistics of Corona virus infection in your region. The data is collected through APIs and webscraping for UK and Germany.
For more information, we have set up a website during our hackathon here: https://praxisconcierge.de/yourhomenurse/.
What's next
Remote self care, supported by hospitals, is perhaps our new reality. We are fully aware that this transition is not only a technical challenge but requires a shift in existing healthcare processes, for which we will need the support of the administration.
With the project ready to be rolled out in a few weeks, we look for partnerships with hospitals and health ministries to provide our system as an extension to their medical care or helplines. For an efficient monitoring and timely treatment of patients.
The team
Our team on this project combines scientific expertise in Machine Learning and, in particular, Natural Language Processing (NLP), with a strong track record of successful business initiatives and project management in the spaces of microfinance, climate and financial modelling, eCommerce, healthcare and education. Combined, we have more than two decades of experience running operations and delivering software solutions in highly dynamic startup environments, and have worked with diverse teams, both remotely and on the ground, in the UK, USA, Vietnam, India, China, Singapore, Poland, France and Germany. This background gives us confidence that even under the currently constrained circumstances, we will be able to deliver on the promise.
Simon Kuttruf (Germany): Diploma in Mathematics, MBA from College des Ingenieurs in Paris, has worked in Software Development and PM, for firms in financial consulting, healthcare technology, engineering and energy systems, has held positions as Chief Data Scientist and Chief Technology Officer and has founded two companies.
Dieu Thu Le (Germany): PhD in machine learning, more than 10 years of experience working in NLP area, strong research background in AI dialogue systems, IBM best paper award for building a debater chatbot in Brussel 2018, co-founder of an AI call bot company.
Mansi Sharma (Belgium): a former CNN journalist and education professional, co-founder of a career guidance firm based in New Delhi from 2009 to 2018, recipient of training in leadership for Education for Sustainable Development, interested area including communications, sustainability and social justice.
Partners
Special thanks
Special thanks to all of those who have given us advices, shared ideas and supported us during the Hackathon!
Irene Meulenkamp, Product Management, Netherlands
Alexandra Toader, Healthcare Management Consultant, UK
Nuria San Pablo, Nurse ICU, Spain
Dr. DK Gupta, Medical Doctor, India
Dr. Mallika Goswami, Medical Doctor, UK
Dr. Alfred Kuttruf, Medical Doctor, Germany
James Cameron, NHS Primary Care Lead, UK
Elisabetta Aloisi, Medical Advisor
Susana Paco, Computational Biology
Built With
- amazon-web-services
- natural-language-processing
- python
- pytorch
- speech-to-text
- text-to-speech
- twilio