Evaluating Knowledge Discovery and Data Mining
Foster Provost and David Jensen. "Evaluating Knowledge Discovery
and Data Mining." A tutorial to be given at the Fourth International
Conference on Knowledge Discovery and Data Mining. August.
- Abstract
- Both the science and practice of KDD stand to benefit from a common
understanding of the strengths and limitations of the many frameworks
for evaluating results. We will explain and criticize a wide variety
of evaluation techniques, illustrating the similarities, but focusing
on the important small differences. We first discuss the difference
between evaluating models and evaluating model-building algorithms,
which leads into a description of the traditional scientific frameworks
for comparing KDD results. We then show where these frameworks
are weak statistically and recommend techniques for strengthening
them. Next, we discuss weaknesses of these frameworks when it
comes to the practical application of data mining results. We
show how to make evaluations more robust for a wide variety of
real-world data mining scenarios, comparing and contrasting metrics
such as sensitivity, specificity, positive predictive value, precision,
and recall, and frameworks such as lift and ROC curves. Finally,
expanding our view, we consider the general problem of searching
for interesting patterns. We describe a diverse collection of
techniques, including Bayesian and Bonferroni adjustments, blindfold
trials, interestingness criteria, and the use of prior domain
knowledge.
- Links
- The Fourth International Conference on Knowledge Discovery and
Data Mining (KDD98)
Fineprint