Computer Science · Computational Theory and Mathematics
Real External Predictivity\nof QSAR Models. Part 2.\nNew Intercomparable Thresholds for Different Validation Criteria and\nthe Need for Scatter Plot Inspection
Nicola Chirico (2050864), Paola Gramatica (2050867)
See on the mapAbstract
The evaluation of regression QSAR model performance,\nin fitting,\nrobustness, and external prediction, is of pivotal importance. Over\nthe past decade, different external validation parameters have been\nproposed: <i>Q</i><sub>F1</sub><sup>2</sup>, <i>Q</i><sub>F2</sub><sup>2</sup>, <i>Q</i><sub>F3</sub><sup>2</sup>, rm2̅, and the Golbraikh–Tropsha method.\nRecently, the concordance correlation coefficient (CCC, Lin), which\nsimply verifies how small the differences are between experimental\ndata and external data set predictions, independently of their range,\nwas proposed by our group as an external validation parameter for\nuse in QSAR studies. In our preliminary work, we demonstrated with\nthousands of simulated models that CCC is in good agreement with the\ncompared validation criteria (except <i>r</i><sub><i>m</i></sub><sup>2</sup>) using the cutoff values normally applied for the acceptance of\nQSAR models as externally predictive. In this new work, we have studied\nand compared the general trends of the various criteria relative to\ndifferent possible biases (scale and location shifts) in external\ndata distributions, using a wide range of different simulated scenarios.\nThis study, further supported by visual inspection of experimental\nvs predicted data scatter plots, has highlighted problems related\nto some criteria. Indeed, if based on the cutoff suggested by the\nproponent, rm2̅ could also accept not predictive models\nin two of the possible biases (location, location plus scale), while\nin the case of scale shift bias, it appears to be the most restrictive.\nMoreover, <i>Q</i><sub>F1</sub><sup>2</sup> and <i>Q</i><sub>F2</sub><sup>2</sup> showed some problems in one of\nthe possible biases (scale shift). This analysis allowed us to also\npropose recalibrated, and intercomparable for the same data scatter,\nnew thresholds for each criterion in defining a QSAR model as really\nexternally predictive in a more precautionary approach. An analysis\nof the results revealed that the scatter plot of experimental vs predicted\nexternal data must always be evaluated to support the statistical\ncriteria values: in some cases high statistical parameter values could\nhide models with unacceptable predictions.