English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Machine learning (ML) has been widely used in the literature to automatesoftware engineering tasks. However, ML outcomes may be sensitive torandomization in data sampling mechanisms and learning procedures. Tounderstand whether and how researchers in SE address these threats, we surveyed45 recent papers related to three predictive tasks: defect prediction (DP),predictive mutation testing (PMT), and code smell detection (CSD). We foundthat less than 50% of the surveyed papers address the threats related torandomized data sampling (via multiple repetitions); only 8% of the papersaddress the random nature of ML; and parameter values are rarely reported (only18% of the papers). To assess the severity of these threats, we conducted anempirical study using 26 real-world datasets commonly considered for the threepredictive tasks of interest, considering eight common supervised MLclassifiers. We show that different data resamplings for 10-foldcross-validation lead to extreme variability in observed performance results.Furthermore, randomized ML methods also show non-negligible variability fordifferent choices of random seeds. More worryingly, performance and variabilityare inconsistent for different implementations of the conceptually same MLmethod in different libraries, as also shown through multi-dataset pairwisecomparison. To cope with these critical threats, we provide practicalguidelines on how to validate, assess, and report the results of predictivemethods.

Run, Forest, Run? On Randomization and Reproducibility in Predictive Software Engineering