I was approached in the summer of 2014 with an intriguing and potentially valuable project. I was asked to explore whether it was possible to predict a mental health crisis for a user of mental health services. With an increasing demand in the UK for such services, yet no corresponding increase in funding, I felt that there was value in contributing to potential solutions to address this.
My task was to employ data from the patient record system of a mental health trust to design an algorithm identifying users most at risk of an upcoming mental health crisis. This could ultimately be used by clinicians and managers, alongside other technologies, to direct vital resources to those most in need of support. It could also aid the trust in making more strategic decisions about resourcing services.
Challenges
I found that I encountered several challenges in this project that often came from the data itself. It was a complex data set with no obvious documentation, which meant that I was reliant on the business intelligence team at the trust who worked with the structures every day. I had to learn about how the underlying concepts of treatment episodes translated into data elements, and gauge whether the users of the system (normally clinicians, doctors and nurses) had entered the data accurately and fully.
Clearly, the data system had not been designed for such an algorithm, but on closer inspection it appeared to be sufficiently robust. The initial reasons for collecting the data included individual service user treatment (the notes used by healthcare professionals to track users), risk management (to collect evidence in case of any issues or queries), and statutory reporting purposes (as the trust was subject to reporting obligations). Therefore, although the data was not specifically designed to monitor crises, it had the potential to do so.
The task also had practical technical challenges, as access to the data was via a VPN into the mental health trust’s business services network. Analysis was conducted in SQL Server and the results were exported to Excel. Other tools would have perhaps been more suitable, but sometimes you must work with what is available.
My approach
I first tried to understand the precise definition of the term ‘crisis’. Once this was understood, I could set up queries about when exactly individual service users had historically experienced a crisis. I was also interested in the length of the crisis period, as well as where this was managed (the community or the hospital). I then considered a range of different factors, and evaluated the extent to which these correlated with the observed mental health crises.
The next step was to construct the algorithm that could predict the likelihood of a future crisis for a specific service user. The best way forward seemed to be to use the estimated likelihood of a factor contributing to a crisis taken from a historical data set. This could then be applied in estimating a forward-looking likelihood of crisis for the service users of today’s cohort.
Another term that was important to grasp was ‘prediction’. Given the relatively rare nature of incidents of crisis within the data set, it became clear that the data was not strong enough for a very specific prediction, such as: “Mrs Jones will be having a crisis next Thursday.” Instead, a probability measure seemed to be more appropriate, calculating the probability that any given service user on any given date would suffer a mental health crisis in the subsequent four weeks, such as: “The likelihood that Mrs Jones will have a crisis in the next four weeks is 8%.”
Results
By tracking actual crisis events that occurred after these predictions were made, this approach could be easily monitored and it could be determined whether the original estimated likelihood was realistic. The results are as follows:
[Graph]
Practical application
Although on an individual basis, clinicians would be more able to understand a service user and predict a crisis than the algorithm would, the practicalities of the everyday running of the mental health services indicate a need for it. The algorithm can be applied to a service user every day, which would be infeasible for a clinician for a number of reasons (time, sickness, holidays, etc.).
However, although the algorithm may be beneficial, a likelihood score would be practically difficult to apply to service users – i.e. what does a probability of 8% actually suggest? Users were consequently sorted into high, medium and low risk groups based on the relative likelihood of a crisis. This gave clinicians a better sense of how to deal with service users, and suggested how support could be distributed.
This system could then assess the balance of service users from high to low risk, and match them with the necessary resources and skill mix currently available within the services. This provided robust evidence that the services were greatly in need of further funding and resources.
The algorithm and the new system of risk levels also had use as a regular management tool. It could be used to oversee caseloads, understand users whose risk level had changed, and help to balance need. This new perspective on management was transformative to teams in the services who had only ever seen lists of individuals on caseloads.
Reflections
During this project, I was commissioned to work for a large commercial company developing a digital health business. I therefore worked alongside employees of that business as well as other freelancers and external software development companies within a core team. As a team we were commissioned within the NHS. This provided an interesting perspective on how commercial enterprises can work successfully with the NHS, and was a brilliant model for bringing together technical and managerial skills that are lacking within the NHS.