It can be stated that there are two phases in the scale development process. The first phase is designing the scale; and the other phase is pre-application. There are different validity concepts specific to each phase. Particularly, while content and face validities have leading role in designing scale, the construct and criterion validities have come to the forefront in pre-application phase. In fact, content validity is prerequisite for construct validity. Content validity is often established through qualitative expert reviews. Content validity information often derives from reviews to be undertaken by subject matter experts as qualitative data. However, there are some studies about transforming these qualitative data into quantitative form; because it is difficult to report qualitative reviews of experts. Content Validity Index (CVI) and kappa statistics have come to the forefront in the studies. The purpose of the present study is to examine the consistency of Content Validity Index (CVI) and kappa statistics. For the purpose, in designing phase for a developing scale, data obtained from ten expert judges were analyzed with two approaches. In pre-application, the consistency of Content Validity Index (CVI) and kappa statistics with average variance extracted (AVE) and factor loadings obtained from factor analysis were examined.