Assessment, Criticism and Improvement of Imprecise Subjective Probabilities for a Medical Expert System

Three paediatric cardiologists assessed nearly 1000 imprecise subjective conditional probabilities for a simple belief network representing congenital heart disease, and the quality of the assessments has been measured using prospective data on 200 babies. Quality has been assessed by a Brier scoring rule, which decomposes into terms measuring lack of discrimination and reliability. The results are displayed for each of 27 diseases and 24 questions, and generally the assessments are reliable although there was a tendency for the probabilities to be too extreme. The imprecision allows the judgements to be converted to implicit samples, and by combining with the observed data the probabilities naturally adapt with experience. This appears to be a practical procedure even for reasonably large expert systems.