Worse than expectedA z-curve reanalysis of motor cortex stimulation studies of embodied language comprehension

  1. Pablo Solana 1
  2. Julio Santiago 1
  1. 1 Universidad de Granada
    info

    Universidad de Granada

    Granada, España

    ROR https://ror.org/04njjy449

Revista:
Psicológica: Revista de metodología y psicología experimental

ISSN: 1576-8597

Año de publicación: 2023

Volumen: 44

Número: 2

Tipo: Artículo

DOI: 10.20350/DIGITALCSIC/15661 DIALNET GOOGLE SCHOLAR lock_openAcceso abierto editor

Otras publicaciones en: Psicológica: Revista de metodología y psicología experimental

Resumen

Dozens of TMS and tDCS studies suggest a functional involvement of motor cortex in action language comprehension, supporting the embodied cognition view. In a recent study (Solana & Santiago, 2022, Neurosci. Biobehav. Rev., 141, 104834), we evaluated the soundness of this literature by means of p-curve analyses and tests for excess significance. The analysis estimated a low average power (≈ 30%) and showed signs of publication bias, which led us to conclude that this body of findings does not stand on solid ground. Yet, the employed techniques seem to perform poorly when high heterogeneity is present (as is the case) and cannot quantify the amount of publication bias. For these reasons, in the present paper, we reanalyzed the same set of studies with a method that does not have these limitations: z-curve analysis. Z-curve not only replicated our prior conclusions but showed an even more heartbreaking situation, with a lower power estimation (≈ 20%) and clear signs of a strong publication bias (around 6-10 unpublished contrasts expected to exist for each published one). Researchers on this topic should consider these shortcomings before drawing conclusions from this body of findings, as well as start implementing robust and transparent research practices to improve their reliability.

Referencias bibliográficas

  • Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22(4), 577-660. https://doi.org/10.1017/S0140525X99002149
  • Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59, 617-645. https://doi.org/10.1146/annurev.psych.59.103006.093639
  • Bartoš, F., & Schimmack, U. (2020). zcurve: An R Package for Fitting Z-curves. R package version 2.3.0. https://CRAN.R-project.org/package=zcurve
  • Bartoš, F., & Schimmack, U. (2022). Z-curve 2.0: Estimating replication rates and discovery rates. MetaPsychology, 6, MP.2021.2720. https://doi.org/10.15626/MP.2021.2720
  • Bem, D. J. (2003) Writing the empirical journal article. In J. M. Darley, M. P. Zanna & H. L. Roediger (Eds.), The compleat academic: A career guide (2nd Edition). American Psychological Association.
  • Birba, A., Vitale, F., Padrón, I., Dottori, M., de Vega, M., Zimerman, M., ... & García, A. M. (2020). Electrifying discourse: Anodal tDCS of the primary motor cortex selectively reduces action appraisal in naturalistic narratives. Cortex, 132, 460-472. https://doi.org/10.1016/j.cortex.2020.08.005
  • Boes, A. D., Kelly, M. S., Trapp, N. T., Stern, A. P., Press, D. Z., & Pascual-Leone, A. (2018). Noninvasive brain stimulation: challenges and opportunities for a new clinical specialty. The Journal of Neuropsychiatry and Clinical Neurosciences, 30(3), 173-179. https://doi.org/10.1176/appi.neuropsych.17110262
  • Boulenger, V., Mechtouff, L., Thobois, S., Broussolle, E., Jeannerod, M., & Nazir, T. A. (2008). Word processing in Parkinson's disease is impaired for action verbs but not for concrete nouns. Neuropsychologia, 46(2), 743-756. https://doi.org/10.1016/j.neuropsychologia.2007.10.007 Brunner, J., & Schimmack, U. (2020). Estimating population mean power under conditions of heterogeneity and selection for significance. Meta-Psychology, 4, MP.2018.874. https://doi.org/10.15626/MP.2018.874
  • Buccino, G., Riggio, L., Melli, G., Binkofski, F., Gallese, V., & Rizzolatti, G. (2005). Listening to actionrelated sentences modulates the activity of the motor system: A combined TMS and behavioral study. Cognitive Brain Research, 24(3), 355-363. https://doi.org/10.1016/j.cogbrainres.2005.02.020
  • Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365-376. https://doi.org/10.1038/nrn3475
  • Glenberg, A. M., Sato, M., & Cattaneo, L. (2008). Use-induced motor plasticity affects the processing of abstract and concrete language. Current Biology, 18(7), R290-R291. https://doi.org/10.1016/j.cub.2008.02.036
  • Hardwicke, T. E., & Wagenmakers, E. J. (2023). Reducing bias, increasing transparency and calibrating confidence with preregistration. Nature Human Behaviour, 7(1), 15-26. https://doi.org/10.1038/s41562-022-01497-2
  • Ioannidis, J. P., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials, 4(3), 245-253. https://doi.org/10.1177/1740774507079441
  • John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524-532. https://doi.org/10.1177/0956797611430953
  • Lakens, D. (2019). The value of preregistration for psychological science: a conceptual analysis. Japanese Psychological Review, 62(3), 221–230. https://doi.org/10.24602/sjpr.62.3_221
  • Lakens, D. (2022a). Bias detection. In D. Lakens (Ed.), Improving your statistical inferences. https://lakens.github.io/statistical_inferences/
  • Lakens, D. (2022b). Sample size justification. Collabra: Psychology, 8(1), 33267. https://doi.org/10.1525/collabra.33267
  • Mahon, B. Z., & Caramazza, A. (2008). A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content. Journal of Physiology-Paris, 102(1-3), 59-70. https://doi.org/10.1016/j.jphysparis.2008.03.004
  • McShane, B. B., Böckenholt, U., & Hansen, K. T. (2020). Average power: A cautionary note. Advances in Methods and Practices in Psychological Science, 3(2), 185-199. https://doi.org/10.1177/2515245920902370
  • Montero-Melis, G., Van Paridon, J., Ostarek, M., & Bylund, E. (2022). No evidence for embodiment: The motor system is not needed to keep action verbs in working memory. Cortex, 150, 108-125. https://doi.org/10.1016/j.cortex.2022.02.006
  • Morey, R. D., Kaschak, M. P., Díez-Álamo, A. M., Glenberg, A. M., Zwaan, R. A., Lakens, D., … & ZivCrispel, N. (2022). A pre-registered, multi-lab non-replication of the action-sentence compatibility effect (ACE). Psychonomic Bulletin & Review, 29, 613-626. https://doi.org/10.3758/s13423-021- 01927-8
  • Papesh, M. H. (2015). Just out of reach: On the reliability of the action-sentence compatibility effect. Journal of Experimental Psychology: General, 144(6), e116. https://doi.org/10.1037/xge0000125
  • Pulvermüller, F. (2005). Brain mechanisms linking language and action. Nature Reviews Neuroscience, 6(7), 576-582. https://doi.org/10.1038/nrn1706
  • R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/
  • Renkewitz, F., & Keiner, M. (2019). How to detect publication bias in psychological research. Zeitschrift für Psychologie / Journal of Psychology, 227(4), 261–279. https://doi.org/10.1027/2151- 2604/a000386
  • Repetto, C., Colombo, B., Cipresso, P., & Riva, G. (2013). The effects of rTMS over the primary motor cortex: The link between action and language. Neuropsychologia, 51(1), 8-13. https://doi.org/10.1016/j.neuropsychologia.2012.11.001
  • Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638
  • Simmons, J., Nelson, L., & Simonsohn, U. (2018, January 8). P-curve handles heterogeneity just fine. DataColada. http://datacolada.org/67
  • Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: a key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534-547. https://doi.org/10.1037/a0033242
  • Solana, P. (2023). The embodiment and disembodiment of language. Nature Reviews Psychology, 2, 391. https://doi.org/10.1038/s44159-023-00203-x
  • Solana, P., & Santiago, J. (2022). Does the involvement of motor cortex in embodied language comprehension stand on solid ground? A p-curve analysis and test for excess significance of the TMS and tDCS evidence. Neuroscience and Biobehavioral Reviews, 141, 104834. https://doi.org/10.1016/j.neubiorev.2022.104834
  • Van Aert, R. C., Wicherts, J. M., & van Assen, M. A. (2016). Conducting meta-analyses based on p values: Reservations and recommendations for applying p-uniform and p-curve. Perspectives on Psychological Science, 11(5), 713-729. https://doi.org/10.1177/1745691616650874
  • Vitale, F., Padrón, I., Avenanti, A., & de Vega, M. (2021). Enhancing motor brain activity improves memory for action language: A tDCS study. Cerebral Cortex, 31(3), 1569-1581. https://doi.org/10.1093/cercor/bhaa309
  • Vukovic, N., Feurra, M., Shpektor, A., Myachykov, A., & Shtyrov, Y. (2017). Primary motor cortex functionally contributes to language comprehension: An online rTMS study. Neuropsychologia, 96, 222-229. https://doi.org/10.1016/j.neuropsychologia.2017.01.025
  • Winter, A., Dudschig, C., Miller, J., Ulrich, R., & Kaup, B. (2022). The action-sentence compatibility effect (ACE): Meta-analysis of a benchmark finding for embodiment. Acta Psychologica, 230, 103712. https://doi.org/10.1016/j.actpsy.2022.103712
  • Witt, J. K., Kemmerer, D., Linkenauger, S. A., & Culham, J. C. (2020). Reanalysis Suggests Evidence for Motor Simulation in Naming Tools Is Limited: A Commentary on Witt, Kemmerer, Linkenauger, and Culham (2010). Psychological Science, 31(8), 1036-1039. https://doi.org/10.1177/0956797620940555
  • Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018). Making replication mainstream. Behavioral and Brain Sciences, 41, E120. https://doi:10.1017/S0140525X17001972
Fuente de los datos: Dialnet