Why risk assessment in mental health care is quite hopeless at predicting the events we most want to prevent

 Gormley

 

Risk has assumed a huge role in mental health care in general and is at the centre of much thinking about the use of compulsory treatment.

We like to believe that we can predict, in a useful way, many of the bad things that may happen. In health and social care much effort has been devoted to developing the process of ‘risk assessment’. A suicide, a homicide committed by a person with a mental illness, the death of a child as a result of abuse, are appalling events. Every reasonable effort should be made to anticipate them and act to prevent them. The key here is every reasonable effort.

 

The sad reality about risk and prediction

The sad truth is that we are quite hopeless at predicting the very events that are the most tragic and that we most want to prevent. The main reason for this is their rarity: the more infrequent an event is, the more difficult it is to predict it with a worthwhile degree of accuracy. This is brute statistical fact.

Unfortunately statistics are not very intuitive, at least not for non-statisticians. Very intelligent people, including psychiatrists, psychologists, lawyers, judges, journalists, senior civil servants and politicians find statistics very hard to grasp. Probabilities are what statistics is largely about; that is, the precise probability (or chance) of a particular event occurring. It seems to me that our brains are not ‘hard-wired’ so as to make it easy for us. We are dominated by a need to create narratives to explain events. Creating narratives – usually with the benefit of hindsight – outweighs the probabilities that are the stuff of statistics and that are in this case, at least, a truer reflection of the real world.

So how does it work? How are our conjectures undermined by statistical realities? Let’s take as an example, suicide. There was recently a tragic case that was heard in the Supreme Court of the UK of a young woman was allowed home on leave from a psychiatric hospital where she had been admitted as a voluntary patient following a suicide attempt in the context of a severe depressive illness.

The case

Melanie Rabone had suffered from depression since 2000. On 4 March 2005 she had been admitted to hospital after a suicide attempt where she tied a pillow-case around her neck. She was discharged on 18 March after an earlier overnight leave on the 14th. She then went on a family holiday to Egypt, but on 31 March, after her return, she cut her wrists with broken glass. She was not readmitted, apparently as no bed was available. Follow-up with the trainee psychiatrist on 6 April revealed that she had occasional thoughts of suicide and frequent thoughts of deliberate self-harm. On 11 April she tied lamp flex around her neck. Her face became swollen and there were ligature marks around her neck. Her parents also found a hosepipe and tape hidden in her room. The medical notes stated she had a “severe depressive episode ….. possible psychosis. High risk self-harm and suicide”. She was admitted informally, and prescribed medication and placed under 15 minute observations. A risk assessment using Trust documents was commenced by one of the nurses on the ward, but further information was required for it to be completed. On 13 April Melanie’s father “expressed grave concern about Melanie’s current condition and her being sent out on leave or discharged too soon”. There were further conversations involving her parents along these lines during the week. Nursing reports in the notes from 16 April stated that her mood was lifting, but Mr Rabone phoned the ward on 18 April saying that she was not improving and that she had expressed fleeting suicidal thoughts. On 19 April, her consultant who had been on leave when Melanie was admitted, returned from leave. The following was the record in the Nursing notes of a meeting with Melanie and her mother:

Present: consultant, Trainee psychiatrist, staff nurse. Melanie seen with her mother. She states she self harmed at home due to feeling angry at herself because of the thoughts she has. Realises that does not achieve anything. Feels trapped at home ‘slightly’. Would like to be more independent. Stated enjoyed recent trip to Egypt. Does not regret leaving employment. Wishes to look for something else. Does not want to stay in destructive cycle. Struggling to recognise how she can stop same. Feels she is lacking in confidence and has low self-esteem. Identified ways of addressing issues herself. Would like to leave for up to a week. Would start looking for job and see friends. Leave agreed as long as Melanie when seeing her friends does not talk about herself and become centre of attention. Reasons for this also discussed. Mother concerned about same as unable to keep eye on her. Consultant advised Melanie has to take responsibility for own actions and when has previously harmed herself it has been when parents keeping an eye on her. Melanie in agreement that will not self-harm

Plan – for 2 days/nights leave

Melanie left the ward with her mother that evening, and some time after 5pm on the next day she hung herself from a tree in a park. The hospital conceded the decision to allow home leave was negligent.

Both the High Court and the Court of Appeal judged that an ‘operational duty’ under Article 2 of the Human Rights Act 1998 – the state’s duty ‘to protect life’ – did not apply. The requirement for a breach of the State’s obligations under Article 2 is for the presence of a ‘real and immediate risk’ not to have been reasonably ascertained, or, if it has been, for reasonable action then not to have been taken to avoid that risk eventuating in a death. Further appeal was allowed to the Supreme Court.

The hospital had admitted negligence, but the decision of the Supreme Court went further, concluding that the State (through its agent, the psychiatric hospital) did have an operational duty under Article 2 to protect the life of Melanie and that it had failed to take reasonable measures to do so. The patient should not have been allowed leave, it decided. An important reason for the judgement was that the risk of suicide was not adequately assessed by the hospital. Had it been, it was judged, the patient would not have been given leave, and if she had refused to stay, she should have been detained on a compulsory order.

Please be clear that I am not making any comment on whether the treatment decisions were right or not, or whether Melanie should have been detained on an order.  My concern is only with the issue of risk assessment.

 

 What goes into creating a sound risk assessment?

Let’s look at this in some detail. Most people shudder when numbers and formulae are introduced, but what I’m about to discuss is, I hope, not that complicated. Try to bear with me, but I will understand if you skip the statistics.

There are three key ingredients in a predictive test – any kind of predictive test – from a statistical point of view. The first two are characteristics of the risk assessment test itself, the third is the rate of the terrible event to be predicted – how commonly it occurs:

  1. Of the eventual suicides, what proportion of those is accurately predicted by the test? We want this to be as high as possible.
  2. Of those who do not suicide, what proportion of those non-suicides is accurately predicted by the test? This ingredient is often ignored when people think about risk prediction. It is absolutely critical though in telling us how many ‘false positives’ there will be – that is, how many people will eventually be classified as being a high suicide risk, but who will not commit suicide. (This characteristic has to be considered in conjunction with the first. For example, I could correctly predict 100% of those who will commit suicide in the first test; I could do this by predicting that everyone will commit suicide. I’ll get them all, but it’s is not very sensible. It will be right for everyone who eventually commits suicide, but wrong for all those who do not, and they will hugely outnumber those who do. If, for example, 1 in 100 will eventually commit suicide, I will be wrong 99 times out of a 100. So the test must tell us whom to exclude as posing a high risk of suicide). We want this proportion to be as high as possible.
  3. The rate at which the event to be avoided – and thus predicted – occurs in the people who might be at risk. In this case, this is the rate of suicide among psychiatric inpatients.

Let’s give these three characteristics a name: the first is technically the sensitivity of the test – how sensitive it is at picking up those who eventually suicide. I’ll call it the ‘inclusion proportion’; the second is the specificity – how specific the test is, that is, how well it excludes those who will not suicide. I’ll call it the ‘exclusion proportion’. The frequency of the event to be prevented is called the base rate – that is how often it occurs in the population of people that we are taking about (for example, how many suicides occur in a year in patients admitted to psychiatric hospitals)

 

An example: inpatient suicide

 So let’s look at some basic figures concerning suicide among patients admitted to a psychiatric hospital. I looked at this in 2009 when there were 120,000 admissions to psychiatric hospitals in England involving 108,000 patients (some of whom were obviously admitted more than once).   In that year there were 84 suicides during the period of the admission. There are no statistics concerning how many of these occurred during a leave period, but the 84 suicides includes all those that did occur during such leave periods. But let’s stick with the 84.

Now let’s try to estimate how many of the 120,000 admissions involved patients like the one in the Supreme Court case, that is, those who were admitted because they were regarded as posing a suicide risk rather than for other reasons, such as disturbed behaviour based on delusions or hallucinations. The best estimate comes from a study in London that looked in detail at reasons for patients being admitted to a number of psychiatric hospitals. This study found that the prevention of suicide was a reason in 36% of the admissions. In 21% it was the ‘major’ reason for the admission; in the remaining 15% it was a ‘contributory’ reason. It’s important to bear in mind that suicides among inpatients are not restricted to those who are admitted because of a risk of suicide. Suicides may occur in patients with a psychosis or with substance abuse where there might have been little or no warning beforehand. However, let’s assume that all 84 suicides in 2009 occurred in the 21% of admissions, that is, 25,200, who were admitted primarily because of the risk of suicide. The rate of suicide then was approximately 1 in 300 admissions – and, given the allowances we have made above, this has to be an overestimate of the rate.

 

The power of a narrative versus statistical reality

It is at this point that a very typical human trait needs to be introduced – the need to construct narratives. In a very important book, the Nobel Prize winning psychologist of economics, Daniel Kahneman presents many examples in his book Thinking Fast, and Slow (2012) of how our need to build narratives to explain the behaviour of others may, on occasions, lead us badly astray. In most everyday situations, such an approach works well enough. We can predict pretty well, although not perfectly, whether our partner or a friend will like a particular film, or a new dish, or how they will react to a particular setback at work. We construct a narrative based on our knowledge of the person’s history and previous reactions in similar circumstances. But sometimes our automatic narrative building becomes a serious handicap.

Kahneman gives a wonderful example, one that deserves wide recognition. He calls it the Linda Problem. This is the description of

Linda:

Linda is thirty-one years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in antinuclear demonstrations (The experiments were done in the 1980s, hence the reference to the antinuclear demonstrations). Now to the nub of the issue. He posed the following question:

Which alternative is more probable?

  1. Linda is a bank teller.
  2. Linda is a bank teller and is active in the feminist movement.

 

I am sure that many readers will pick the second alternative as the more probable. I know this because Kahneman found that 85% to 90% of undergraduates at several major universities in the US endorsed this alternative. Even if one gets the right answer, in some way it goes against the grain. It is logically more likely – when you think more carefully about it – that Linda is a bank teller than that she is both a bank teller and active in a feminist movement: there are more female bank tellers than females who are bank tellers and feminists.

Here is another version of exactly the same problem:

Linda is a woman.

Which alternative is more probable?

  1. Linda has brown hair.
  2. Linda has brown hair and brown eyes.

It is more likely that someone has brown hair, than that they have both brown hair and brown eyes. It’s obvious in an example where we do not feel somehow driven to construct a narrative.

 

Where the narrative can trip us up

So, what are the implications of our insistent need to construct explanatory narratives? For any one of the 25,200 patients admitted to a psychiatric hospital in 2009 primarily because of the risk of suicide, if they had indeed committed suicide, a very plausible explanatory narrative could have been constructed, with hindsight, which would have indicated the event was very likely to happen. They were admitted because the psychiatric history seemed to offer precisely such a narrative. Common in such narratives would have been suicidal ideas, a history of self-harm, of depressive symptoms (such as pervasive sadness, inability to find pleasure in anything, insomnia, poor concentration, feelings of hopelessness or guilt), a family history of suicide, of recent troubling life events or losses, of social problems, and so on. It would have appeared more or less obvious. And very likely staff would be seen as having failed the patient, or as being culpable. Yet, at most, only 1 of the 300 for whom the plausible narrative could have been constructed actually committed suicide.

Nevertheless, can we pick the 1 in 300 who did?

 

How the statistics on inpatient suicide unfold in a risk assessment

How well then, using the best available tools, could the 1 in 300 suicide be predicted. Unfortunately, we need to use some mathematics rather than the somewhat easier method of narrative building. Matthew Large and colleagues in Australia recently examined the accuracy of risk prediction methods for inpatient suicide (including all periods of leave). The best sensitivity (inclusion proportion) and specificity (exclusion proportion) that could be achieved taking into account all potential predictors, known as risk factors – such as history of suicide attempts, suicidal ideas, depressive symptoms , alcohol or substance abuse, family history of suicide, recent social difficulties, and so on – was 0.64 and 0.85, respectively.

The best way of showing the statistics is by using a ‘probability tree’. I hope you will find this kind of illustration as illuminating as I do. It shows the way in which the inclusion proportion (sensitivity), the exclusion proportion (specificity) and base rate come together to determine how many persons predicted to commit suicide by the risk assessment test, in fact, turn out on follow-up to do so.

This is how the tree looks for predicting inpatient suicide:

Consider 1000 patients admitted to hospital primarily for suicide risk, 3 of whom will eventually commit suicide as an inpatient (i.e. roughly one in 300)

Screen Shot 2015-09-13 at 10.55.43

(Those marked in red score positively on the test: that is as being at ‘high risk’ of suicide).

The result comes as a shock to most of us. Using the best predictive tool available – one that under normal clinical circumstances, due for example to time constraints, is unlikely to be feasible in practice – only 1.3%, 1 in 77, of those considered to be at ‘high risk’ will commit suicide. To make matters worse, about 1 out of 3 of those who will commit suicide will not be classed as ‘high risk’ at all (the inclusion proportion being only 0.64, around two-thirds). The conclusion must be that predicting who will commit suicide during their inpatient stay is in practice simply not feasible. Placing restrictions, including involuntary detention, on the freedom of all 152 patients who are ‘high risk’, in the example above, would not be acceptable, as it will unnecessarily restrict the freedom of 77 patients who would not have committed suicide for every one that will. Furthermore, concentrating on the ‘high risk’ group may lead to less treatment attention to those who are considered at ‘low risk’ who will nevertheless include one third of the suicides. Indeed, less attention to that group may result in more suicides in that group. Please recall that all of the patients were already in a sense ‘high risk’ – they were all admitted because they were suicidal. In fact we are here considering the ‘high risk’ group within an already ‘high risk’ group.

You may think of another explanation for the rarity of inpatient suicides. It could be due to the effectiveness of psychiatric treatment for people who are at high risk of suicide. If this were the case, and up to 99% of suicides were successfully prevented, it would put psychiatry near the top of the medical specialty league for lives saved. This is very unlikely to be the explanation for the rarity of suicide among those at ‘high risk’. As far as I know, suicides are rare in countries that have much poorer psychiatric services than ours.

 

Homicides

When it comes to predicting homicides by persons with a mental disorder, the figures are even more deflating. There are at any one time about 250,000 persons in England who have a psychotic illness such as schizophrenia. In any one year, about 25 homicides will be carried out by a patient with a psychotic illness – that is, 1 in 10,000. There are many violence risk assessment instruments, but rarely do they attain an ‘inclusion proportion’ (sensitivity) and ‘exclusion proportion’ (specificity) as good as 0.7 and 0.7, respectively. If you care to construct a probability tree using these figures, you will find that only 1 in 5000 patients classified as ‘high risk’ will commit a homicide in a year. The tests are basically useless for predicting homicide.

What about the prediction of serious violence? If serious violence, short of homicide, occurs in say 1% of patients per annum, using this unusually good risk assessment tool – under research conditions and probably a far-cry from the situation in the clinic – only 2 out of 100 patients who fall into the ‘high risk’ category will commit such an act in a year. (You can show this for yourself by constructing a probability tree with the three numbers – ‘inclusion proportion’ (sensitivity), ‘exclusion proportion’ (specificity), and base rate). However, the narrative constructed for the two who do will be entirely plausible. It will feature the risk factors that are associated with violence – a past history of violence, substance abuse, paranoid delusions, poor compliance with treatment, a history of having experienced violence, living in a violent subculture, and so on. But so would the narratives of the other 98 persons rated as ‘high risk’.

 

Some ‘costs’ of a risk emphasis or ‘risk thinking’

For reasons that are somewhat difficult to understand, mental health care in England has placed an extremely strong emphasis on risk. Other countries are much less preoccupied with, for example, homicides perpetrated by persons with mental illness. In England there have been fears that ‘community care has failed’, despite there being no evidence this has been the case. Fears have nonetheless been generated that many dangerous mentally ill persons are now free in the community who in the past would have been safely locked up. Sensational stories in the media when a homicide has occurred have fuelled such fears. Politicians have reacted by insisting that public protection be a key aim, if not the key aim, of mental health services. What has been overlooked is that despite the move to community care, and despite its considerable problems of implementation, the number of homicides by persons with mental illness has not increased; indeed, if anything it has fallen, most certainly as a much smaller proportion of all homicides perpetrated in England. Suicide rates have remained about the same over the past 10 years..

The emphasis on ‘risk thinking’ brings with it certain costs. It is an important principle that reducing uncertainty always carries a cost – even when the attempts to do so are unsuccessful. There are a number of costs can be incurred in mental health care: I will only mention two here: moral costs and discrimination against persons with a mental illness.

 Moral costs

Our sadly limited ability to assess risk with a degree of practical utility and its bedevilment by a high rate of ‘false positives’ – that is, persons predicted to commit suicide or to be seriously violent but who would never have done so – means that many people will be treated restrictively or coercively, but unnecessarily so. This obviously incurs a high moral cost.

Discrimination

Finally, there is a problem of discrimination. A good example is the state’s insistence, through the Department of Health in England, on a routine risk assessment for all persons referred to mental health services. There are many persons in society, hugely exceeding in number those with a mental disorder, who present an equal, indeed much greater risk of violence to others. But they’re not selected for any form of routine, systematic risk assessment. Fairness demands that all people presenting an equal potential risk be treated equally. I suggest that fairness requires risk assessment, if we’re to have it, to be only in response to some kind of reasonable ‘trigger’ event, applicable to all. Simply being referred to mental health services is not an acceptable trigger. So those who could be subjected to a risk assessment might include all who’ve been involved in violence of some kind, those with traumatic injuries, known substance misusers, those who’ve been threatening to neighbours or in the workplace, and so on. That would be fair – but obviously daft. However, our society seems well able to tolerate unfairness when it affects only those with mental illness. Their rights are easily discounted.

 

So what should we do about risk?

Obviously clinicians must do everything they reasonably can to prevent serious negative outcomes involving their patients – but the emphasis must be on ‘reasonably’. And the management of risk, at least in the sense in which it has come to so heavily influence mental health care, I suggest, should not be a primary aim motivating involuntary treatment.

Risk data have been largely drawn from population (or epidemiological) studies – for example, a group of people who commit a serious offence is compared with a group who don’t, and the differences between the groups in respect of a number of factors are noted – for example, whether there was a past history of violence, substance abuse, childhood behavioural problems, and so on. These differences are then combined in a risk assessment instrument according to what gives the best separation between the two groups. A question appropriate to the nature of this kind of population-based information, then, is whether it points to population-level interventions that may reduce risk. Examples might be the provision of effective drug and alcohol services, parenting interventions for families of children with conduct disorders, and so on.

At a practice level, thoughts of suicide or violence may be of particular value as indicators of the severity of an underlying illness or of the patient’s distress that serve to alert the clinician to the need for urgent and effective treatment. However, in many or most cases, such signs will not be present, or will be no more evident or severe in the few episodes that do lead to serious consequences than in the many that do not. Clinicians are influenced by present harms and harms that are clearly foreseeable – for example, when a person is severely depressed and is failing at work, and where they are planning to resign from a job which they have up to now enjoyed and which pays extremely well. Clinician’s will obviously take seriously a clear and direct threat of suicide or violence. However, the majority of cases where a serious outcome of this kind will ensue will be missed – and that simply cannot be helped where the base rate of such outcomes is low. Essentially, clinicians will operate with high specificity (exclusion rate) but very low sensitivity (inclusion rate).

A suggestion that Nikolas Rose, a sociologist, and I have made is that there would be value in a public engagement exercise concerning risk in mental health care. In other areas it has been shown that people may think about risk in ways that are not well understood by scientists or public institutions, and that they may be prepared to accept a higher level of risk than was anticipated, especially when various benefits are clearly evident. If people were to understand the severe limitations of risk assessment, coupled with the fact that the greatest risk of violence from a mentally ill person, for example, is to members of their family or friends, it would not surprise if they were to opt for mental health services that were accessible and responsive in emergencies as offering a better approach than decisions made on the basis of risk. Yet public engagement has rarely been used in the context of mental health.

Importantly, clinicians working in general psychiatry – forensic psychiatry is different, I think – must think about the degree to which they are prepared to allow professional practice to be reshaped by the risk approach. They owe it to their patients (and to the idea of a just society) that they do not support practices that discriminate unfairly against those with mental illness.

 

Some other publications

 

1  Szmukler, G (2003) Risk assessment: ‘numbers’ and ‘values’ Psychiatric Bulletin 27, 205-207

This paper shows the astonishing effect of the ‘base rate’ (the frequency in the population of the outcome that is to be prevented) on the rate of ‘false positives’ using the risk assessment test. For example, using the best ever (experimental) risk assessment test for violence, the test will be wrong 75 times out of a 100 when the ‘base rate’ is 10%, but increases to 97 times out of 100 when the ‘base rate’ is 1%.

2  Szmukler G, Rose N. (2013) Risk assessment in mental health care: costs and values. Behavioral Sciences and the Law 31, 125-40. doi: 10.1002/bsl.2046.

This paper looks in some detail at the costs of risk assessment in mental health care with a focus on violence by people with a mental illness.  These include the moral and discrimination costs I have discussed here as well as costs arising from changes in clinical practice and in the potential for a loss of trust.  Unfortunately this paper is only accessible to those who have Athens or Shibboleth rights.

 3  Large MM, Ryan CJ, Singh SP, Paton MB, Nielssen OB. (2011) The predictive value of risk categorization in schizophrenia. Harvard Review of Psychiatry. 19, 25-33.

Offers a similar analysis to the one I have presented.

 

 Homicides by persons with a mental illness

 

The three papers below show that homicides by people with a mental illness have not increased with the introduction of community care – indeed, the study of Large and his colleagues  suggests there has been a decrease.

The fourth paper is one I wrote following an Independent Homicide Inquiry into a homicide involving a patient in my Mental Health Trust when I was the Medical Director. It points to some of the weaknesses of this type of post-incident inquiry when the event is a rare one. It led to the early retirement of one of our country’s most eminent forensic psychiatrists.

1  Taylor, P. J., Gunn, J. (1999) Homicides by people with mental illness: myth and reality. British Journal of Psychiatry 174, 9–14.

2  Szmukler G, Thornicroft G, Holloway F, Bowden P (1999) Homicides and community care: The evidence. British Journal of Psychiatry, 174: 564 -565.

3  Large M, Smith G, Swinson N, Shaw J, Nielssen O. (2008) Homicide due to mental disorder in England and Wales over 50 years. British Journal of Psychiatry. 193, 130-133.

4  Szmukler G (2000) Homicide Inquiries: What sense do they make? Psychiatric Bulletin 24: 6-10