I have been wanting to write this part for a while now; let's get started!

So, lostlovefairy told me about how people have started relying on AI more and more to the extent that patients would come to the doctor and tell them to do a certain procedure because ChatGPT or another similar AI model told them that based on the entered symptoms.

This isn't just the medical field, but in every field—even science—AI is taking over and seemingly doing better than humans in the same scenario, and even when it might seem like it's not exactly an issue, it certainly is.

If you've ever known COMPAS (Correctional Offender Management Profiling for Alternative Sanctions), which was developed and owned by Northpointe (now Equivant), used to assess the likelihood of becoming a recidivist.

sources to look for more in-depth research: Sam Corbett-Davies, Emma Pierson, Avi Feller, and Sharad Goel (October 17, 2016) "A computer program used for bail and sentencing decisions was labeled biased against blacks. It's actually not that clear." The Washington Post. Retrieved January 1, 2018.

Aaron M. Bornstein (December 21, 2017). "Are Algorithms Building the New Infrastructure of Racism?". Nautilus, No. 55. Retrieved January 2, 2018.

The term recidivist means that the person does the same crime again.

In other terms, COMPAS is software that uses an algorithm to assess potential recidivism risk. Northpointe created risk scales for general and violent recidivism and for pretrial misconduct. According to the COMPAS Practitioner's Guide, the scales were designed using behavioral and psychological constructs "of very high relevance to recidivism and criminal careers."

Wikipedia source link here →

I started on clinical diagnosis... why did we end up discussing COMPAS?

There's a reason; let me explain. So, as you can see, COMPAS was a diagnostic tool used in order to assess criminal offenses. And it seemed effective enough, until the court, jury, and police started to use it everywhere. You see where I am going with this? A person could be scaled by numbers to decide if they could reoffend or not.

And while it's not based on numbers everywhere... everything is becoming less and less narrow with terms of how things are measured and scaled up, and that's problematic because if you're trying to simplify everything with the basis of something that's a black box, what you miss out on is transparency and accuracy—two things that matter the most.

Here's some key points as summarized by ChatGPT for COMPAS (since there's too much context that i couldn't summarize right away and hence the help):

COMPAS was praised for offering an objective, data-driven way to predict recidivism. It was designed to assess the risk of offenders re-offending, aiding judges, parole boards, and law enforcement in making more informed decisions about sentencing, parole, and rehabilitation. In theory, this promised a more consistent and unbiased approach than relying solely on human judgment.

The tool provided a quick, standardized assessment across various cases, potentially reducing judicial workload and saving time in overburdened court systems. It allowed for streamlined decision-making in complex criminal cases, offering quantitative risk scores based on numerous factors.

COMPAS was initially heralded for its perceived objectivity—the idea being that algorithms, unlike humans, would not be swayed by emotions, personal biases, or inconsistent reasoning. It was marketed as a way to remove subjective biases from decision-making and promote fairness.

Major consequence as a result of it:

Racial bias and injusticeLack of Transparency ("Black Box" Nature)

→ Like many machine learning algorithms, COMPAS functioned as a "black box," with its proprietary algorithm not fully disclosed to judges or defendants. This meant that neither legal professionals nor those affected could fully understand how the tool was generating risk scores. This lack of explainability and transparency raised serious concerns about due process, fairness, and the ability to challenge the system's decisions.

Over-Simplification of Human Behavior

→ Predicting human behavior, especially something as complex as criminal recidivism, is inherently difficult. COMPAS reduced human actions to a set of data points, which could lead to oversimplified conclusions. It failed to account for personal rehabilitation efforts, changes in life circumstances, or nuanced factors that could only be interpreted through human judgment.

Reinforcement of Systemic BiasesEthical and Legal Accountability

→ Similar to AI in healthcare, there were questions about who should be held accountable when COMPAS's risk scores led to unjust or disproportionate punishments. The tool's decisions had real-world consequences, but because it was a machine-driven process, it complicated the ability to assign responsibility for flawed outcomes.

Over-reliance on automated decision-making

→ The judicial system began to place too much faith in the numerical scores generated by COMPAS, sometimes overlooking the importance of holistic human judgment. Judges and parole boards may have treated the algorithm's outputs as infallible rather than as one tool among many in a broader decision-making process. This over-reliance on automation could have led to harsher sentences or denials of parole based solely on risk scores rather than a thorough review of individual cases.

COMPAS was quite reliable until it eventually had a downfall. And the reasons are as above. Now, let's come to the medical diagnostics aspect of why relying on AI, or really, just any machine learning, is a bad idea.

To understand clinical diagnostics vs. statistical diagnostics better and truly understand why machine learning, and particularly AI, started to catch the spark of people with regards to clinical diagnostics, I'll suggest referring to the topics of:

CLINICAL VERSUS STATISTICAL PREDICTION [The Alignment Problem, Brian Christian]
IMPROPER MODELS: KNOWING WHAT TO LOOK AT [The Alignment Problem, Brian Christian]
OPTIMAL SIMPLICITY [The Alignment Problem, Brian Christian]

A detailed summary of individual topics felt slightly unnecessary, so I'll be summarizing them as a whole, drawing conclusions from the three topics and how they relate to the discussion as provided by ChatGPT:

1. Simple models can be surprising effective: Across all three topics, there is a recurring theme: simple, interpretable models can often perform as well or better than complex, opaque models. Dawes' work on improper linear models, Rudin's efforts in recidivism prediction, and medical diagnostics highlight the effectiveness of models that use only key, well-selected features. These models are not only competitive but also more transparent and interpretable for human decision-makers.

2. Simplicity and Interpretability Matter: Both Dawes and Rudin emphasize the importance of understanding which variables to look at (i.e., feature selection) rather than relying solely on complex algorithms to combine vast amounts of data. Rudin, in particular, argues that the current clinical models are often based on expert intuition (handcrafted), which leaves room for optimization through data-driven approaches. She pushes for a future where we don't just rely on expert-based heuristics but instead use computational power to build better, simpler models directly from data.

3. Challenges of Complex Models: While complex models like neural networks (used in some medical tools and self-driving cars) can handle vast amounts of data, they suffer from opacity—often referred to as "black boxes." This makes it difficult to interpret or trust the outputs without knowing exactly why the model made a particular prediction. When human lives are at stake, as in clinical diagnostics, the lack of transparency in these models becomes a significant barrier to widespread adoption.

ML algorithms, particularly data-driven ones like the ones Rudin develops, can significantly improve the accuracy of clinical diagnostics by analyzing vast datasets, identifying patterns, and making predictions that might elude human experts. They have the potential to automate and optimize many tasks in healthcare, from disease prediction to personalized treatment recommendations.

But then you'd wonder, Why would you not use them if they're effective? We already use tools in clinical diagnostics and healthcare, but bringing ML entirely into the system and letting it do the job is not going to help us in the long run.

Here are a few reasons why healthcare would be at risk, given the nature of AI:

1. Lack of Interpretability

2. Regulatory Hurdles

→ Medical diagnostics are heavily regulated, and getting approval for new algorithms requires proving not only their accuracy but also their reliability and safety. If models can't be fully understood or explained, it becomes difficult to meet these regulatory requirements.

3. Clinician Resistance

→ Doctors are used to relying on their own expertise and judgment, and there may be resistance to relying on algorithms—especially those that seem to work in a "black box" fashion. Trust in AI tools remains a major obstacle, as does the reluctance to change well-established clinical practices.

4. Data Quality: In clinical settings, data quality and availability can be a limiting factor. Models depend on large amounts of data to function properly, and poor-quality or incomplete data could result in inaccurate predictions. Simple models, by contrast, often rely on clearly defined, well-known variables, reducing the risk of misinterpretation from flawed data.

While machine learning algorithms hold enormous potential to revolutionize clinical diagnostics through efficiency and accuracy, their adoption is slowed by concerns over interpretability, trust, and regulatory challenges. Simple, interpretable models, as championed by Dawes and Rudin, offer a middle ground—balancing accuracy with transparency, which is critical in healthcare settings where human decision-makers must fully understand and trust the tools they use. The future of clinical diagnostics may lie in optimizing these simpler, more transparent models rather than pushing for increasingly complex, black-box algorithms.

---

AI models are typically trained on large datasets with common patterns. They may not perform well on unseen, rare, or novel conditions, which human doctors are better equipped to handle through experience, intuition, and deep knowledge. Unpredictable scenarios could lead AI to fail, especially in edge cases that lie outside its training data.

Developing interpretable AI models (e.g., simple models like the ones Cynthia Rudin advocates for) that explain their decisions and reasoning clearly will help clinicians trust AI predictions. By making AI systems more transparent, human experts can scrutinize the AI's recommendations and understand where they came from, allowing for more informed final decisions.

AI should complement human expertise rather than replace it. AI can handle routine, repetitive tasks, such as analyzing large datasets or providing initial diagnostic suggestions, while humans focus on more complex, ambiguous, or rare cases that require deeper insight. AI can be a "second opinion" or a tool to augment clinicians' decision-making, improving overall accuracy.

And last but not least: Establish clear regulatory guidelines and protocols for how AI should be used in diagnostics. This includes setting limits on where AI can be applied independently and where human intervention is required. It can also involve building ethical frameworks that dictate AI's role, ensuring that patients are protected and human oversight is maintained at critical junctures.

In conclusion, AI holds significant promise in enhancing medical diagnostics, particularly by handling large datasets and identifying patterns that humans may overlook. However, full reliance on AI is filled with risks due to concerns around trust, interpretability, bias, and lack of human connection. The best path forward is human-AI collaboration—where AI serves as a powerful tool to augment, not replace, the expertise and judgment of clinicians. By combining the strengths of both, healthcare outcomes [or criminal justice outcomes] can be improved while ensuring patient safety and ethical standards.

Sara.