While IBM seeking buyers for Dr.Watson this year, Its a case study to review why Watson failed which decade back promised Cancer Diagnostics AI , ahead of time  ? 


AI Diagnosis and Treatment advisory Platforms have challanges

This is important industry case study will help those who are reviewing potential AI solutions targeted for Healthcare Diagnosis Intervension and Treatment Recommendations. I am summarizing few important points of this case study and visible reasons of failure.

A. Good point with Watson is its massive patient data repository ….

  • In past 10 years, IBM collected significant data from cancer patients (namely focussed for USA Healthcare systems) and developed a advance analatyics model to recommend treatment from list of a fixed list generated by panel of US Doctors (MSKCC’s world-renowned oncologists)
  • The treatment list based on Patient Characteristics Parameters namely Age/Gendor/Total Billirubin/Prior Therepies/MStage/Primary Tumer Size/Tumer Grade/Neuropathy Grade/Ejection Fraction/Nyha Cardiac Disease Grade/CT Stage/Estrogen Recepter Status/Performance Status/Menopausal Status/Cancer Stage/Primary Tumer etc.
  • Doctors see these recommendations along with published articles backing up these decisions.

B. the key reason IBM Watson failed to deliver promise as under –

  • Gaps in knowledge about complex diseases whose outcomes often depend on many factors that may not be fully captured in clinical databases.
  • Lack deep expertise in how healthcare works, Challenge of implementing AI in hospital patient settings.
  • A overpromised premature launch without proper test and trail runs, technically invited interoperability challenges and over-reliance on human (panel of doctors) input to generate results
  • Lack of data-collection standards, ( ** This is still a issue with all healthcare AI which makes taking an algorithm that was developed in one setting and applying it in others difficult. The customization problem is still severe in healthcare.)
  • The treatment recommendations by Watson are not based on its own insights from these data. Instead, they are based exclusively on training by human (Doctors) overseers, who laboriously feed Watson information about how patients with specific characteristics should be treated.
  • System is not intelligent ( means watson doesn’t create new knowledge and is artificially intelligent only in the most rudimentary sense of the term. (Win at“Jeopardy! -live television show trials shared in above link”- its programming is akin to a different game-playing machine )
  • There is no way to test the safety of a digital treatment advised, and no way to ensure its compliance with regulations, and how to incorporate it into the daily work of doctors and nurses.

C. What we could learn for not repeating such mistakes in future with Healthcare AI ?

  • Proper Identification of Problem and Areas to Address ( The most successful applications of AI in healthcare to date have been when the technology aims to solve discrete and narrow problems, Such applications include alert systems that warn doctors which of their patients might be at risk for readmissions or severe outcomes and chatbots that help answer basic questions. )
  • Planning and Execution shall be supported by Process Definition, Testing and Trials. They should have conducted a clinical trial before commercializing Watson, noting that many practices in medicine are widely accepted even though they aren’t supported by a randomized controlled trial. Bringing the best information to bear on medical decision making is a no-brainer.
  • Data Standards and connected network of institutions feeding data about specific cohorts of patient
  • Then the gaps in knowledge about complex diseases whose outcomes often depend on many factors that may not be fully captured in clinical databases shall be addressed in first place.
  • Having Data samples and Panel Experts from all global regions . Having only US Medical Expert is not sufficient to understand treatment protocols in Asia or Europe.