Reliability and Validity in Diagnosis and Classification of Schizophrenia

How can we Define Reliability in Diagnosis?

Reliability in research refers to the consistency of achieving the same results if different researchers were to perform the same experiment over and over again. A study is considered reliable if it can be repeated and give the same results.

Reliability in diagnosis refers to the level of agreement that different psychiatrists can achieve on a single diagnosis for an individual, both over time and across cultures, provided that the disorder’s symptoms have not changed.

• It must be repeatable and consistent.

• The diagnosis must be stable over time if the definition of the disorder has not changed.

To ensure a diagnosis and its classification are reliable, test-retest reliability is usually measured by the intraclass correlation (ICC) and the inter-rater reliability test, also measured as the kappa score. Both are statistical results that examiners can use to confirm the reliability of their results.

An Example of Test-Retest Reliability

Let's explore an example of test-retest reliability in schizophrenia tests.

An Example of Inter-Rater Reliability

Let's explore an example of inter-rater reliability measures in schizophrenia.

Rosenhan (1973) Schizophrenia Study

Rosenhan’s (1973) study, one of the most influential studies affecting the diagnosis and treatment of psychiatric disorders today, highlighted the problems with reliability in the diagnosis of psychiatric disorders. Rosenhan emphasised that psychiatric hospital staff were somewhat desensitised and unaware of how drastic the experience of being admitted as a patient can be.In this study, Rosenhan himself and seven healthy individuals went to 12 different psychiatric hospitals.

• These were three women and five men from various backgrounds, from psychologists to a painter. They used pseudonyms.

The pseudopatients asked the hospitals for an appointment and then complained upon arrival that they heard unfamiliar but same-sex voices, specifically:

• Empty
• Hollow
• Thud

These words were explicitly chosen to show the pseudopatient has symptoms such as hallucinations, and to suggest that the pseudopatient believes their life is ‘empty and hollow’.

• The pseudopatients falsified their name, occupation, and profession to protect their identity. However, they were free to talk about their history and circumstances and were asked to present events from their life story as if they had happened.

Once admitted, the pseudopatients immediately ceased all symptomatic behaviours. They behaved as they normally would and talked to other patients and staff as they would before they were admitted as pseudopatients. They were not told when they could leave the clinic. Instead, they were told they would have to convince the staff that they were sane to leave the clinic.

Pseudopatients who were asked how they were doing told staff they were fine and no longer had symptoms. They behaved accordingly and followed the rules. They also noted observations about the hospitals, secretly at first, but when it was obvious that no one cared or took note, they wrote them down without secrecy.

Fig. 2 - In Rosenhan's study, patients behaved normally after entering the hospital.

All but one of the patients were discharged with a diagnosis of schizophrenia in remission. Their length of stay ranged from 7 to 52 days (an average of 19 days). They were not classified as ‘sane’; the label of schizophrenia remained. The pseudopatients were not detected during their stay, although they remained sane throughout. There was a lack of adequate observation, but Rosenhan attributes this to a failure of the hospital rather than the patient and the lack of opportunity.

Interestingly, actual patients at the hospital recognised the pseudopatients as sane. About 35 of 118 patients recognised the sanity of the pseudopatients and even suggested their real occupation:

• ‘You are not crazy. You are a journalist or a professor. You are checking out the hospital.’

Patients were more likely than staff to recognise normality, even though the pseudopatient insisted he was sick.

Rosenhah suspects this failure to recognise sanity is due to type 2 errors, also known as false-positive diagnoses. Rosenhan explained that diagnoses in this area of health care are neither useful nor reliable and represent a depressing and frightening reality that we must combat. Hospitals of this type cannot identify and treat these behaviours, and they also convey a sense of powerlessness.

Rosenhan stated:

The study impacted the way we treat patients with mental disorders today. It led to a condemnation of the use of institutionalised hospitals and a focus on care in the community.

Validity in Classification and Diagnosis of Schizophrenia

Validity is the legitimacy of a test, i.e., whether what the psychiatrist uses to diagnose a person measures what it purports to measure.

If it is valid, the diagnosis represents something real and is different from other disorders.Cronbach and Meehl (1955) established several types of validity.

What are the Issues with Validity and Reliability when Diagnosing and Classifying Schizophrenia?

The following issues cause problems with validity and reliability:

1. Comorbidity
2. Symptom overlap
3. Gender bias
4. Culture bias

What is Comorbidity?

Comorbidity can be a problem in any type of disorder, but what exactly is comorbidity?

Fig. 3 - Comorbidity in schizophrenia can be a common problem in diagnosis.

If two disorders can occur in one person, the validity of the diagnosis begs the question, who is to say that it is only one disease and not several?Unfortunately, in the case of schizophrenia, there are many comorbidities with disorders such as OCD (obsessive-compulsive disorder) and depression. As a result, there are significant problems with the disorder’s validity.Buckley et al. (2009) found that comorbid depression occurs in 50% of patients with schizophrenia.

This finding raises some concerns:

• Is it because psychiatrists or diagnostic manuals are poor at recognising the difference between these two disorders?

• If they are so similar that there are problems with comorbidity, why can not they be considered a single disorder?

Since there is no truly distinct disorder, it can be difficult to diagnose and treat schizophrenia with confidence.

Symptom Overlap

As there are an abundance of symptoms in schizophrenia as well as other mental health disorders, diagnosis and classification can be tricky.

Schizophrenia and bipolar disorder are good examples.Bambole et al. (2013) noted that patients diagnosed with schizophrenia and bipolar disorder have both positive and negative symptoms, making it difficult to distinguish between them.If these two disorders have too many symptoms in common, it questions the validity of the classification system and thus the validity of the diagnosis.For example, if a patient has these symptoms:

• Avolition

• Delusions of grandeur

The ICD may diagnose schizophrenia, while the DSM suggests bipolar disorder, indicating an apparent problem in the way these manuals claim to measure a disorder, and, therefore, a weakness in their validity.

Gender Bias in Diagnosing and Classifying Schizophrenia

Gender bias is problematic in many areas of research, and it is not exempt from psychological research.

There may be gender-specific criteria, or there may be a bias based on stereotypes that the clinician believes in.

Cultural Bias in Diagnosing and Classifying Schizophrenia

Cultures differ across the world, and research has long had issues with the generalisability of data because of this.

Fig. 4 - Culture across the world differ in their approaches to diagnosing schizophrenia, flaticon.com/authors/flat-icons

Why is that so?

• There may be a cultural problem with understanding what is ‘normal’ behaviour because in some cultures. In Africa, hearing voices and doing things that are normally considered positive symptoms of schizophrenia are more accepted than in other cultures where this is considered bizarre and irrational.

• Racial biases could also play a role. When certain ethnic minorities are overdiagnosed compared to White people, especially by White psychiatrists, it indicates a high degree of miscommunication, bias, and lack of understanding of the patient and their symptoms.

• Since countries of origin generally do not support these high diagnosis rates, there is no genetic susceptibility. Why are African Americans in another country more likely to be diagnosed with schizophrenia than people in Africa and the West Indies?

Reliability and Validity of Classification and Diagnosis of Schizophrenia

In light of the above, it is also important to acknowledge that:

• No objective test for schizophrenia exists, as there is no biological basis to assess it. It can only be diagnosed by clinical interview, leading to the abovementioned problems. If bias and lack of reliability and validity exist, schizophrenia cannot be diagnosed with certainty.
• When people are misdiagnosed, several problems arise, especially because schizophrenia has a stigma attached to it. Angermeyer and Matschinger (2003) surveyed adults of German nationality and found that labelling people with schizophrenia has a strong negative impact on public opinion.
• In the case of misdiagnosis, treatment is likely to be ineffective because it does not address the root cause and does not properly treat symptoms.