Validity evidence based on response processes

  1. Padilla García, José Luis
  2. Benítez, María Isabel
Revista:
Psicothema

ISSN: 0214-9915

Año de publicación: 2014

Volumen: 26

Número: 1

Páginas: 136-144

Tipo: Artículo

DIALNET GOOGLE SCHOLAR lock_openAcceso abierto editor

Otras publicaciones en: Psicothema

Resumen

Antecedentes: la evidencia de validez basadas en los procesos de respuestas fue incluida explícitamente por primera vez como fuente de evidencias de validez en la última edición de los Standards for Educational and Psychological Testing. En este artículo, presentamos la teoría, la relación con otras fuentes de evidencias de validez, y los métodos disponibles para realizar estudios de validación cuyo objetivo sea obtener evidencias de validez sobre los procesos de respuesta. Método: una extensa revisión de la literatura junto con propuestas teóricas y prácticas. Resultados: el artículo aporta argumentos para determinar cuando la evidencia de validez basada en los procesos de respuesta es crítica para apoyar el uso del test para un objetivo particular, y ejemplos de cómo realizar un estudio de validación para obtener tales evidencias de validez. Conclusiones: hay métodos para obtener evidencias de validez basadas en los procesos de respuesta. Debe prestarse especial atención a los estudios de validación mediante el método de entrevista cognitiva por sus características y posibilidades. Futuros problemas de investigación plantean como combinar datos de métodos diferentes �cualitativos y cuantitativos�, para elaborar argumentos de validez que apoyen el uso del test para un objetivo particular.

Referencias bibliográficas

  • American Psychological Association, American Educational Research Association, & National Council on Measurement in Education (1985). Standards for educational and psychological testing. Washington, D.C.: American Psychological Association.
  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, D.C.: American Educational Research Association.
  • Althof, S.E., Perelman, M.A., & Rosen, R.C. (2011). The Subjective Sexual Arousal Scale for Men (SSASM): Preliminary development and psychometric validation of a multidimensional measure of subjective male sexual arousal. The Journal of Sexual Medicine, 8, 2255-2268.
  • Beatty, P., & Willis, G.B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71, 287-311.
  • Benitez, I., & Padilla, J.L. (2013). Analysis of nonequivalent assessments across different linguistic groups using a mixed methods approach: Understanding the causes of differential item functioning by cognitive interviewing. Journal of Mixed Methods Research, Online first. doi: 10.1177/1558689813488245.
  • Borsboom, D., Mellenbergh, G.J., & Van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 1061-1071.
  • Brod, M., Hammer, M., Christensen, T., Lessard, S., & Bushnell, D.M. (2009). Understanding and assessing the impact of treatment in diabetes: The Treatment-Related Impact Measures for Diabetes and Devices (TRIM-Diabetes and TRIM-Diabetes Device). Health and Quality of Life Outcomes, 7, 83.
  • Castillo, M., Padilla, J.L., Gomez, J., & Andres, A. (2010). A productivity map of cognitive pretest methods for improving survey questions. Psicothema, 22, 475-481.
  • Castillo, M., & Padilla, J.L. (2012). How cognitive interviewing can provide validity evidence of the response processes to scale items. Social Indicators Research. Online first. doi: 10.1007/s11205-012-0184-8.
  • Cepeda, N.J., Blackwell, K.A., & Munakata, Y. (2013). Speed isn't everything: Complex processing speed measures mask individual differences and developmental changes in executive control. Developmental Science, 16, 269-286.
  • Cizek, G.J., Rosenberg, S.L., & Koons, H.H. (2007). Sources of validity evidence for educational and psychological tests. Educational and Psychological Measurement, 68, 397-412.
  • Cizek, G.J. (2012). Defining and distinguishing validity: Interpretations of score meaning and justifications of test use. Psychological Methods, 17, 31-43.
  • Collins, D. (2003). Pretesting survey instruments: An overview of cognitive methods. Quality of Life Research, 12, 229-238.
  • Conrad, F.G.; Blair, J. (2009). Sources of error in cognitive interviews. Public Opinion Quarterly, 73, 32-55.
  • Day, R.F. (2010). Examining the validity of the Needleman-Wunsch algorithm in identifying decision strategy with eye-movement data. Decision Support Systems, 49, 396-403.
  • Deal, L.S., DiBenedetti, D.B., Williams, V.S., & Fehnel, S.E. (2010). The development and validation of the daily electronic Endometriosis Pain and Bleeding Diary. Health and Quality of Life Outcomes, 8, 64.
  • Elling, S., Lentz, L., & de Jong, M. (2012). Combining concurrent think-aloud protocols and eye-tracking observations: An analysis of verbalizations and silences. IEEE Transactions on Professional Communication, 55, 206-220.
  • Embretson, S. (1983). Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93, 179-197.
  • Ercikan, K., Arim, R., & Law, D. (2010). Application on think aloud protocols for examining and confirming sources of differential item functioning identified by experts review. Educational Measurement: Issues and Practices, 29, 24-35.
  • Ferdous, A.A., & Plake, B.S. (2005). Understanding factors that influence decisions of panelists in a standard setting study. Applied Measurement in Education, 18, 257-267.
  • Gadermann, A.M., Guhn, M., & Zumbo, B.D. (2011). Investigating the substantive aspect of construct validity for the Satisfaction with Life Scale adapted for children: A focus on cognitive processes. Social Indicator Research, 100, 37-60.
  • Garrett (1937). Statistics in psychology and education. New York: Longmans, Green.
  • Gehlbach, H., & Brinkworth, M.E. (2011). Measure twice, cut down error: A process for enhancing the validity of survey scales. Review of General Psychology, 15, 380-387.
  • Hawthorne, G., Davidson, N., Quinn, K. McCrate, F., Winkler, I., Lucas, R., Kilian, R., & Molzahn, A. (2006). Issues in conducting crosscultural research: implementation of an agreed international protocol designed by the WHOQOL Group for the conduct of focus groups eliciting the quality of life of older adults. Quality of Life Research 15, 1257-1270.
  • Ivie, J.L., & Embretson, S.E. (2010). Cognitive process modeling of spatial ability: The assembling objects task. Intelligence, 38, 324-335.
  • Jabine, T.B., Straf, M.L., Tanur, J.M., & Tourangeau, R. (1984). Cognitive aspects of survey methodology: Building a bridge between disciplines. Washington, DC: National Academy Press.
  • Kane, M.T. (1992). An argument-based approach to validity. Psychological Bulletin, 112, 527-535.
  • Kane, M.T. (2006). Validation. In B.L. Robert (Ed.), Educational Measurement (4th ed., pp. 17-64). Wesport, CT: Praeger.
  • Kane, M.T. (2013). Validation as a pragmatic, scientific activity. Journal of Educational Measurement, 50, 115-122.
  • Krall, J.S., & Lohse, B. (2010). Cognitive testing with female nutrition and education assistance program participants informs validity of the Satter eating competence inventory. Journal of Nutrition Education and Behavior, 42, 277-283.
  • Krosnick, J.A. (1999). Survey research. Annual Review of Psychology, 50, 537-567.
  • Martin, E. (2004). Vignettes and respondent debriefing for questionnaire design and evaluation. En Presser, S., Rothgeb, J.M., Couper, M.P., Lessler, J.T., Martin, E., Martin, J., & Singer, E. (2004). Methods for Testing and Evaluating Survey Questionnaires. (pp. 149-173). New York: Wiley-Interscience.
  • Messick, S. (1989). Validity. In R.L. Linn (Ed.), Educational Measurement 3rd ed. (pp. 13-103). New York: McMillan.
  • Messick, S. (1990). Validity of test interpretation and use. Research Report 90-11. Education Testing Service.
  • Miller, K., Chepp, V., Willson, S., & Padilla, J.L. (2014 in press). Cognitive Interviewing Methodology: A Sociological Approach for Survey Question Evaluation. New York, NJ: John Wiley and Sons.
  • Mislevy, R.J., Steinberg, L.S., & Almond, R.G. (2002). On the role of task model variables in assessment design. In S.H. Irvine & P.C. Kyllonen (Eds.), Item generation for test development (pp. 97-128). Mahwah, NJ: Lawrence Erlbaum.
  • Olt, H., Jirwe, M., Gustavsson, P., & Emami, A. (2010). Psychometric evaluation of the Swedish adaptation of the Inventory for Assessing the Process of Cultural Competence Among Healthcare Professionals-Revised (IAPCC-R). Journal of transcultural nursing: Official journal of the Transcultural Nursing Society / Transcultural Nursing Society, 21, 55-64.
  • Sireci, S.G. (2009). Packing and unpacking sources of validity evidence: History repeats itself again. In R.W. Lissitz (Ed.), The Concept of Validity (pp. 19-39). Charlotte, NC: Information Age Publishing, Inc.
  • Sireci, S.G. (2012). "De-constructing" Test Validation. Paper presented at the annual conference of the National Council on Measurement in Education as part of the symposium "Beyond Consensus: The Changing Face of Validity" (P. Newton, Chair). April 14, 2012, Vancouver, Canada.
  • Sireci, S., & Faulkner-Bond, M. (2014). Validity evidence based on test content. Psicothema, 26, xx-xx???.
  • Sireci, S.G., & Parker, P. (2006). Validity on trial: Psychometric and legal conceptualizations of validity. Educational Measurement, 25, 27-34.
  • Sireci, S.G., Baldwin, P., Martone, A., Zenisky, A.L., Kaira, L., Lam, W., Shea, C.L., Han, K., Deng, N., Delton, J., & Hambleton, R.K. (2008). Massachusetts adult proficiency tests technical manual: Version 2: Amherst, MA: Center for Educational Assessment. Available at http:// www.umass.edu/remp/CEA_TechMan.html.
  • Skorupski, W.P., & Hambleton, R.K. (2005). What are panelists thinking when they participate in standard setting studies? Applied Measurement in Education, 18, 233-256.
  • Tourangeau, R. (1984). Cognitive science and survey methods: A cognitive perspective. En T. Jabine, M. Straf, J. Tanur & R. Tourangeau (Eds.), Cognitive Aspects of Survey Methodology: Building a Bridge Between the Disciplines (pp. 73-100). Washington, DC: National Academy Press.
  • Tourangeau, R., Rips, L.J., & Rasinski, K. (2004). The Psychology of Survey Response. Cambridge: Cambridge University Press.
  • Wang, X., & Sireci, S.G. (2013). Evaluating the cognitive levels measured by test items using item response time. Paper presented at the annual conference of the American Educational Research Association. April, 2013, San Francisco, USA.
  • Webber, M.P., & Huxley, P.J. (2007). Measuring access to social capital: The validity and reliability of the Resource Generator-UK and its association with common mental disorder. Social Science & Medicine (1982), 65, 481-492.
  • Willis, G.B. (2005). Cognitive interviewing. Thousand Oaks: Sage Publications.
  • Willson, S., & Miller, K. (2014 in press). Data collection. In K. Miller, V. Chepp, S. Willson & J.L. Padilla (Eds.), Cognitive Interviewing Methodology: A Sociological Approach for Survey Question Evaluation. New York, NJ: John Wiley and Sons.
  • Zumbo, B.D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R.W. Lissitz (Ed.), The Concept of Validity (pp. 65-83). Charlotte, NC: Information Age Publishing, Inc.
  • Zumbo, B.D., & Shear, B.R. (2011). The concept of validity and some novel validation methods. In Northeastern Educational Research Association (p. 56). Rocky Hill, CT.
Fuente de los datos: Dialnet