An articulated procedure for ex-ante evaluation of Design Science Research artifacts

Abstract

Evaluation is a core activity of the Design Science Research (DSR) paradigm and one of the least supported by operational guidance in the literature, particularly when the evaluated artifact is a guideline, a framework or a process model that does not allow controlled experimentation. The FEDS framework (Framework for Evaluation in Design Science) provides a consolidated evaluative conceptualization, but does not prescribe how to select evaluators under objective criteria, how to structure the consultation instrument, or which statistical tools to apply depending on the structure of the data. In parallel, there exists a consolidated repertoire of quantitative methods for expert-judgment validation that combines contributions from the Ibero-American tradition (competence coefficient K, Kendall's W, Torgerson cut-points) with internationally circulating content-validity developments (Hernández-Nieto's CVC, Aiken's V), though it rarely appears articulated with reference DSR frameworks. In response, this article proposes an articulated seven-phase procedure for the ex-ante evaluation of non-experimentable DSR artifacts, positioned within the artificial ex-ante quadrant of FEDS, whose analytical phase enables the selection of the statistical instrument according to the type of judgment sought and the ordinal structure of the data. The work rests on a documentary review, a conceptual articulation between the traditions, and an internal verification of the procedure against criteria of operability, replicability, traceability and theoretical integration declared ex-ante. It is concluded that the procedure is proposed as an operational methodological contribution applicable by researchers evaluating non-experimentable DSR artifacts.