ABSTRACT The absence of a systematic and flexible evaluation methodology, that can characterise the effect of methods and tools on the development process, has made any process improvement program a difficult task. The DESMET project aims to develop and validate an evaluation methodology which will quantify the effect of the use of development methods and tools. DESMET (Determining an Evaluation Methodology for Software Methods and Tools) is a project partly funded by the DTI (IEDS ATP/1 150) and SERC (GR/F 38556). The collaborators in the project are the National Computing Centre, GEC-Marconi Software Systems, Racal Research Ltd and The University of North London. The first phase of the project has critically surveyed current evaluation approaches which were classified as 'quantitative' or 'qualitative'. It was concluded that DESMET must support both types and provide guidelines to identify which method is appropriate to the particular circumstances of the user. To achieve this, the design of DESMET was based on a multi-module architecture. The paper outlines the architecture of DESMET and the rationale behind it. Formal experimentation is one of the quantitative techniques that DESMET will support. The paper presents a new framework for conducting formal experiments in a software engineering environment. The paper will also discuss how managerial and sociological issues can affect formal experimentation in software engineering. |