The bibliometric journey towards technological and social changeA review of current challenges and issues

  1. Daniel Torres-Salinas 1
  2. Nicolás Robinson-García 1
  3. Evaristo Jiménez-Contreras 1
  1. 1 Universidad de Granada
    info

    Universidad de Granada

    Granada, España

    ROR https://ror.org/04njjy449

Revista:
El profesional de la información

ISSN: 1386-6710 1699-2407

Año de publicación: 2023

Título del ejemplar: Digital native media ecosystem

Volumen: 32

Número: 2

Tipo: Artículo

DOI: 10.3145/EPI.2023.MAR.28 DIALNET GOOGLE SCHOLAR lock_openAcceso abierto editor

Otras publicaciones en: El profesional de la información

Resumen

Repasamos las tendencias y desafíos actuales en el campo de la bibliometría. Para ello, llevamos al lector por una ruta bibliométrica con seis estaciones: la explosión de bases de datos, la inflación de métricas, su relación con la Ciencia de Datos, la búsqueda de sentido, la bibliometría evaluativa, y la diversidad y profesión. Esta ruta abarca tres dimensiones, una tecnológica, una teórica y una social, del campo de la bibliometría en relación con la evaluación de la investigación. Finalmente, abogamos por los principios de una bibliometría evaluativa, equilibrando el poder de la métrica con el juicio de los expertos y la política científica

Referencias bibliográficas

  • Alonso, Sergio; Cabrerizo, Francisco-Javier; Herrera-Viedma, Enrique; Herrera, Francisco (2009). “h-Index: A review focused in its variants, computation and standardization for different scientific fields”. Journal of informetrics, v. 3, n. 4, pp. 273-289. https://doi.org/10.1016/j.joi.2009.04.001
  • Anderson, Chris (2008). “The end of theory: The data deluge makes the scientific method obsolete”. Wired, 23 June. https://www.wired.com/2008/06/pb-theory
  • Aria, Massimo; Cuccurullo, Corrado (2017). “bibliometrix: An R-tool for comprehensive science mapping analysis”. Journal of informetrics, v. 11, n. 4, pp. 959-975. https://doi.org/10.1016/j.joi.2017.08.007
  • Arroyo-Machado, Wenceslao; Torres-Salinas, Daniel; Costas, Rodrigo (2022). Wikipedia knowledge graph dataset [Data set]. Zenodo. https://doi.org/10.5281/zenodo.6346900
  • Arroyo-Machado, Wenceslao; Torres-Salinas, Daniel; Herrera-Viedma, Enrique; Romero-Frías, Esteban (2020). “Science through Wikipedia: A novel representation of open knowledge through co-citation networks”. Plos one, v. 15, n. 2, e0228713. https://doi.org/10.1371/journal.pone.0228713
  • Athar, Awais (2014). Sentiment analysis of scientific citations (UCAM-CL-TR-856). University of Cambridge, Computer Laboratory. https://doi.org/10.48456/tr-856
  • Banda, Juan M.; Tekumalla, Ramya; Wang, Guanyu; Yu, Jingyuan; Liu, Tuo; Ding, Yuning; Artemova, Katya; Tutubalina, Elena; Chowell, Gerardo (2022). A large-scale COVID-19 Twitter chatter dataset for open scientific research - An international collaboration [Data set]. Zenodo. https://doi.org/10.5281/zenodo.7029821
  • Barré, Rémi (2019). “Les indicateurs sont morts, vive les indicateurs! Towards a political economy of S&T indicators: A critical overview of the past 35 years”. Research evaluation, v. 28, n. 1, pp. 2-6. https://doi.org/10.1093/reseval/rvy029
  • Bastedo, Michael N.; Bowman, Nicholas A. (2010). “U.S. news & world report college rankings: Modeling institutional effects on organizational reputation”. American journal of education, v. 116, n. 2, pp. 163-183. https://doi.org/10.1086/649437
  • Bell, Gordon; Hey, Tony; Szalay, Alex (2009). “Beyond the data deluge”. Science, v. 323, n. 5919, pp. 1297-1298. https://doi.org/10.1126/science.1170411
  • Benedictus, Rinze; Miedema, Frank; Ferguson, Mark W. J. (2016). “Fewer numbers, better science”. Nature, v. 538, n. 7626, pp. 453-455. https://doi.org/10.1038/538453a
  • Bérubé, Nicolas; Ghiasi, Gita; Sainte-Marie, Maxime; Larivière, Vincent (2020). Wiki-Gendersort: Automatic gender detection using first names in Wikipedia. SocArXiv. https://doi.org/10.31235/osf.io/ezw7p
  • Björneborn, Lennart; Ingwersen, Peter (2001). “Perspective of webometrics”. Scientometrics, v. 50, n. 1, pp. 65-82. https://doi.org/10.1023/A:1005642218907
  • Boekhout, Hanjo; Van-der-Weijden, Inge; Waltman, Ludo (2021). Gender differences in scientific careers: A large-scale bibliometric analysis. Arxiv. http://arxiv.org/abs/2106.12624
  • Bollen, Johan; Van-de-Sompel, Herbert; Smith, Joan A.; Luce, Rick (2005). “Toward alternative metrics of journal impact: A comparison of download and citation data”. Information processing & management, v. 41, n. 6, pp. 1419-1440. https://doi.org/10.1016/j.ipm.2005.03.024
  • Bu, Yi; Waltman, Ludo; Huang, Yong (2021). “A multidimensional framework for characterizing the citation impact of scientific publications”. Quantitative science studies, v. 2, n. 1, pp. 155-183. https://doi.org/10.1162/qss_a_00109
  • Cabezas-Clavijo, Álvaro; Torres-Salinas, Daniel (2021). “Bibliometric reports for institutions: Best practices in a responsible metrics scenario”. Frontiers in research metrics and analytics, v. 6. https://doi.org/10.3389/frma.2021.696470
  • Clarivate Analytics (2018). InCites: Indicators Handbook. Clarivate. https://help.prod-incites.com/inCites2Live/8980-TRS/version/default/part/AttachmentData/data/InCites-Indicators-Handbook-6%2019.pdf
  • Costas, Rodrigo; Corona, Carmen; Robinson-García, Nicolás (2022). Could ORCID play a key role in meta-research? Discussing new analytical possibilities to study the dynamics of science and scientists. SocArXiv. https://doi.org/10.31235/osf.io/sjck6
  • Cronin, Blaise (1984). The citation process: The role and significance of citations in scientific communication. London: Taylor Graham. ISBN: 0 947568 0 1 8
  • Cronin, Blaise (2013). “From signtometrics to scientometrics: A cautionary tale of our times”. Journal of information science theory and practice, v. 1, n. 4, pp. 6-11. https://doi.org/10.1633/JISTaP.2013.1.4.1
  • Daraio, Cinzia; Lenzerini, Maurizio; Leporelli, Claudio; Moed, Henk F.; Naggar, Paolo; Bonaccorsi, Andrea; Bartolucci, Alessandro (2016). “Data integration for research and innovation policy: An ontology-based data management approach”. Scientometrics, v. 106, n. 2, pp. 857-871. https://doi.org/10.1007/s11192-015-1814-0
  • De-Bellis, Nicola (2009). Bibliometrics and citation analysis: From the Science Citation Index to cybermetrics. Scarecrow Press. ISBN: 978 0 810867130
  • Delgado-López-Cózar, Emilio; Ràfols, Ismael; Abadal, Ernest (2021). “Letter: A call for a radical change in research evaluation in Spain”. Profesional de la información, v. 30, n. 3, e300309. https://doi.org/10.3145/epi.2021.may.09
  • Egghe, Leo; Rousseau, Ronald (1990). Introduction to informetrics. Quantitative methods in library, documentation and information science. Elsevier Science Publishers. ISBN: 0 444 884939 http://eprints.rclis.org/6011
  • Elsevier (2019). Research metrics guidebook. Elsevier. https://www.elsevier.com/__data/assets/pdf_file/0020/53327/ELSV-13013-Elsevier-Research-Metrics-Book-r12-WEB.pdf
  • European Commission (2022). Reforming research assessment: The Agreement is now final. European Commission. Directorate-General for Research and Innovation. https://onx.la/e1aef
  • Garfield, Eugene (2004). “Historiographic mapping of knowledge domains literature”. Journal of information science, v. 30, n. 2, pp. 119-145. https://doi.org/10.1177/0165551504042802
  • Gorraiz, Juan; Wieland, Martin; Ulrych, Ursula; Gumpenberger, Christian (2020). “De profundis: A decade of bibliometric services under scrutiny”. In: Daraio, Cinzia; Glänzel, Wolfgang (eds.). Evaluative informetrics: The art of metrics-based research assessment. Festschrift in Honour of Henk F. Moed. Cham: Springer, pp. 233-260. ISBN: 978 3 030 47665 6 https://doi.org/10.1007/978-3-030-47665-6_11
  • Gregorio-Chaviano, Orlando; Repiso, Rafael; Calderón-Rehecho, Antonio; León-Marín, Joaquín; Jiménez-Contreras, Evaristo (2021). “Dialnet Métricas como herramienta de evaluación bibliométrica: Aportes al análisis de la actividad científica en Ciencias Sociales y Humanidades”. Profesional de la información, v. 30, n. 3, e300318. https://doi.org/10.3145/epi.2021.may.18
  • Grodzinski, Noah; Grodzinski, Ben; Davies, Benjamin M. (2021). “Can co-authorship networks be used to predict author research impact? A machine-learning based analysis within the field of degenerative cervical myelopathy research”. Plos one, v. 16, n. 9, e0256997. https://doi.org/10.1371/journal.pone.0256997
  • Gumpenberger, Christian; Glänzel, Wolfgang; Gorraiz, Juan (2016). “The ecstasy and the agony of the altmetric score”. Scientometrics, v. 108, n. 2, pp. 977-982. https://doi.org/10.1007/s11192-016-1991-5
  • Haustein, Stefanie; Bowman, Timoty D.; Costas, Rodrigo (2016). “Interpreting ‘altmetrics’: viewing acts on social media through the lens of citation and social theories”. In: Sugimoto, Cassidy R. (ed). Theories of informetrics and scholarly communication. De Gruyter, pp. 372-406. https://doi.org/10.1515/9783110308464
  • Herzog, Christian; Hook, Daniel; Konkiel, Stacy (2020). “Dimensions: Bringing down barriers between scientometricians and data”. Quantitative science studies, v. 1, n. 1, pp. 387-395. https://doi.org/10.1162/qss_a_00020
  • Hicks, Diana; Wouters, Paul; Waltman, Ludo; De-Rijcke, Sarah; Ràfols, Ismael (2015). “The Leiden Manifesto for research metrics”. Nature, v. 520, n. 7548, pp. 429-431. https://doi.org/10.1038/520429a
  • Jappe, Arlette (2020). “Professional standards in bibliometric research evaluation? A meta-evaluation of European assessment practice 2005-2019”. Plos one, v. 15, n. 4, e0231735. https://doi.org/10.1371/journal.pone.0231735
  • Jefferson, Osmat-Azzam; Koellhofer, Deniz; Warren, Ben; Jefferson, Richard (2019). The Lens MetaRecord and LensID: An open identifier system for aggregated metadata and versioning of knowledge artefacts [Preprint]. LIS Scholarship Archive. https://doi.org/10.31229/osf.io/t56yh
  • Khan, Samiya; Liu, Xiufeng; Shakil, Kashish A.; Alam, Mansaf (2017). “A survey on scholarly data: From big data perspective”. Information processing & management, v. 53, n. 4, pp. 923-944. https://doi.org/10.1016/j.ipm.2017.03.006
  • Kozlowski, Diego; Larivière, Vincent; Sugimoto, Cassidy R.; Monroe-White, Thema (2022). “Intersectional inequalities in science”. Proceedings of the National Academy of Sciences, v. 119, n. 2, e2113067119. https://doi.org/10.1073/pnas.2113067119
  • Lammey, Rachael (2020). “Solutions for identification problems: A look at the Research Organization Registry”. Science editing, v. 7, n. 1, pp. 65-69. https://doi.org/10.6087/kcse.192
  • Leydesdorff, Loet (1998). “Theories of citation?”. Scientometrics, v. 43, n. 1, pp. 5-25. https://doi.org/10.1007/BF02458391
  • Leydesdorff, Loet (2009). “How are new citation-based journal indicators adding to the bibliometric toolbox?”. Journal of the American Society for Information Science and Technology, v. 60, n. 7, pp. 1327-1336. https://doi.org/10.1002/asi.21024
  • Leydesdorff, Loet; Etzkowitz, Henry (1998). “Triple Helix of innovation: Introduction”. Science and public policy, v. 25, n. 6, pp. 358-364. https://www.leydesdorff.net/spp98
  • Ma, Anqi; Liu, Yu; Xu, Xiujuan; Dong, Tao (2021). “A deep-learning based citation count prediction model with paper metadata semantic features”. Scientometrics, v. 126, n. 8, pp. 6803-6823. https://doi.org/10.1007/s11192-021-04033-7
  • Mateo, Ferrán (2015). “Producción científica en español en Humanidades y Ciencias Sociales. Algunas propuestas desde Dialnet”. El profesional de la información, v. 24, n. 5, pp. 509-516. https://doi.org/10.3145/epi.2015.sep.01
  • Moed, Henk F. (2017). Applied evaluative informetrics. Springer International Publishing. ISBN: 978 3 319 60522 7 https://doi.org/10.1007/978-3-319-60522-7
  • Moed, Henk F.; Burger, W. J. M.; Frankfort, J. G.; Van-Raan, Anthony F. J. (1985). “The use of bibliometric data for the measurement of university research performance”. Research policy, v. 14, n. 3, pp. 131-149. https://doi.org/10.1016/0048-7333(85)90012-5
  • Moher, David; Bouter, Lex; Kleinert, Sabine; Glasziou, Paul; Sham, Mai Har; Barbour, Virginia; Coriat, Anne-Marie; Foeger, Nicole; Dirnagl, Ulrich (2020). “The Hong Kong Principles for assessing researchers: Fostering research integrity”. Plos biology, v. 18, n. 7, e3000737. https://doi.org/10.1371/journal.pbio.3000737
  • Orduña-Malea, Enrique; Martín-Martín, Alberto; Ayllón, Juan M.; Delgado-López-Cózar, Emilio (2016). La revolución Google Scholar: Destapando la caja de Pandora académica. Universidad de Granada. ISBN: 978 84 338 5985 3
  • Ortega y Gasset, José (2005). Misión del bibliotecario. Edición conmemorativa del 50 aniversario luctuoso del autor y de la celabración de la Día Nacional del Bibliotecario. Consejo Nacional para la Cultura y las Artes.
  • Pardo-Guerra, Juan-Pablo (2022). The quantified scholar: How research evaluations transformed the British social sciences. Columbia University Press. ISBN: 978 0 231197809
  • Priem, Jason; Piwowar, Heather; Orr, Richard (2022). OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts (arXiv:2205.01833). arXiv. http://arxiv.org/abs/2205.01833
  • Priem, Jason; Taraborelli, Dario; Groth, Paul; Neylon, Cameron (2010). Altmetrics: A manifesto – altmetrics.org. http://altmetrics.org/manifesto
  • Ràfols, Ismael; Molas-Gallart, Jordi; Chavarro, Diego-Andrés; Robinson-García, Nicolás (2016). On the dominance of quantitative evaluation in ‘peripheral’ countries: Auditing research with technologies of distance (SSRN Scholarly paper No 2818335). https://doi.org/10.2139/ssrn.2818335
  • Ràfols, Ismael; Stirling, Andy (2021). “Designing indicators for opening up evaluation: insights form research assessment”. In: Dahler-Larsen, Peter. A research agenda for evaluation. Edward Elgar, pp. 165-193. https://doi.org/10.4337/9781839101083.00015
  • Robinson-García, Nicolás; Amat, Carlos-Benito (2018). “¿Tiene sentido limitar la coautoría científica? No existe inflación de autores en Ciencias Sociales y Educación en España”. Revista española de documentación científica, v. 41, n. 2, e201. https://doi.org/10.3989/redc.2018.2.1499
  • Robinson-García, Nicolás; Costas, Rodrigo; Sugimoto, Cassidy R.; Larivière, Vincent; Nane, Gabriela F. (2020). “Task specialization across research careers”. eLife, v. 9, e60586. https://doi.org/10.7554/eLife.60586
  • Robinson-García, Nicolás; Jiménez-Contreras, Evaristo; Torres-Salinas, Daniel (2016). “Analyzing data citation practices using the data citation index”. Journal of the Association for Information Science and Technology, v. 67, n. 12, pp. 2964-2975. https://doi.org/10.1002/asi.23529
  • Robinson-García, Nicolás; Ràfols, Ismael (2020). “The differing meanings of indicators under different policy contexts. The case of internationalisation”. In: Daraio, Cinzia; Glänzel, Wolfgang (eds.). Evaluative informetrics: The art of metrics-based research assessment: Festschrift in honour of Henk F. Moed. Springer, pp. 213-232. https://doi.org/10.1007/978-3-030-47665-6_10
  • Robinson-García, Nicolás; Repiso, Rafael; Torres-Salinas, Daniel (2018). “Perspectiva y retos de los profesionales de la evaluación científica y la bibliometría”. El profesional de la información, v. 27, n. 3, pp. 461-466. https://doi.org/10.3145/epi.2018.may.01
  • Robinson-García, Nicolás; Sugimoto, Cassidy R.; Murray, Dakota; Yegros-Yegros, Alfredo; Larivière, Vincent; Costas, Rodrigo (2019). “The many faces of mobility: Using bibliometric data to measure the movement of scientists”. Journal of informetrics, v. 13, n. 1, pp. 50-63. https://doi.org/10.1016/j.joi.2018.11.002
  • Robinson-García, Nicolás; Torres-Salinas, Daniel; Zahedi, Zohreh; Costas, Rodrigo (2014). “New data, new possibilities: Exploring the insides of Altmetric.com”. El profesional de la información, v. 23, n. 4. http://doi.org/10.3145/epi.2014.jul.03
  • Rose, Michael E.; Kitchin, John, R. (2019). “Pybliometrics: Scriptable bibliometrics using a Python interface to Scopus”. SoftwareX, v. 10, 100163. https://doi.org/10.1016/j.softx.2019.100263
  • Saier, Tarek; Färber, Michael; Tsereteli, Tornike (2022). “Cross-lingual citations in English papers: A large-scale analysis of prevalence, usage, and impact”. International journal on digital libraries, v. 23, n. 2, pp. 179-195. https://doi.org/10.1007/s00799-021-00312-z
  • Saltelli, Andrea; Bammer, Gabriele; Bruno, Isabelle; Charters, Erica; Di-Fiore, Monica; Didier, Emmanuel; Espeland, Wendy-Nelson; Kay, John; Lo-Piano, Samuele; Mayo, Deborah; Pielke Jr., Roger; Portaluri, Tommaso; Porter, Theodore M.; Puy, Arnald; Ràfols, Ismael; Ravetz, Jerome R.; Reinert, Erik; Sarewitz, Daniel; Stark, Philip B.; Stirling, Andrew; Van-der-Sluijs, Jeroen; Vineis, Paolo (2020). “Five ways to ensure that models serve society: a manifesto”. Nature, v. 582, pp. 482-484. https://doi.org/10.1038/d41586-020-01812-9
  • Small, Henry (2006). “Tracking and predicting growth areas in science”. Scientometrics, v. 68, pp. 595-610. https://doi.org/10.1007/s11192-006-0132-y
  • Small, Henry; Sweeney, Ed; Greenlee, Edward (1985). “Clustering the science citation index using co-citations. II. Mapping science”. Scientometrics, v. 8, n. 5, pp. 321-340. https://doi.org/10.1007/BF02018057
  • Sugimoto, Cassidy R.; Larivière, Vincent (2018). Measuring research: What everyone needs to know. Oxford University Press. ISBN: 978 0 190640125
  • Szomszor, Martin; Adie, Euan (2022). “Overton: A bibliometric database of policy document citations”. Quantitative science studies, v. 3, n. 3, pp. 624-650. https://doi.org/10.1162/qss_a_00204
  • Tahamtan, Iman; Bornmann, Lutz (2022). “The social systems citation theory (SSCT): A proposal to use the social systems theory for conceptualizing publications and their citations links”. Profesional de la información, v. 31, n. 4, e310411. https://doi.org/10.3145/epi.2022.jul.11
  • Tekles, Alexander; Bornmann, Lutz (2020). “Author name disambiguation of bibliometric data: A comparison of several unsupervised approaches”. Quantitative science studies, v. 1, n. 4, pp. 1510-1528. https://doi.org/10.1162/qss_a_00081
  • Torres-Salinas, Daniel (2022a). “InfluScience2: Disponible la nueva versión de nuestra plataforma altmétrica”. InfluScience. https://influscience.eu/novedades/influscience2-disponible-la-nueva-version-de-nuestra-plataforma-altmetrica
  • Torres-Salinas, Daniel (2022b). “1985: Cinco lecciones inmortales de Moed para bibliómetras profesionales”. Anuario ThinkEPI, v. 16, e16a14. https://doi.org/10.3145/thinkepi.2022.e16a14
  • Torres-Salinas, Daniel; Arroyo-Machado, Wenceslao (2022a). Material complementario de ‘Nuevas formas de trabajar con la información científica: Introducción al uso de las APIs en contextos bibliométricos’. https://doi.org/10.5281/zenodo.6323589
  • Torres-Salinas, Daniel; Arroyo-Machado, Wenceslao (2022b). “APIs en contextos bibliométricos: Introducción básica y corpus exhaustivo”. Anuario ThinkEPI, v. 16. https://doi.org/10.3145/thinkepi.2022.e16a09
  • Torres-Salinas, Daniel; González-Molina, Antonio (2019). A modo de acta: “II Reunión de los Servicios de Evaluación Científica” en la Universidad de Córdoba. https://doi.org/10.5281/zenodo.2557725
  • Torres-Salinas, Daniel; Jiménez-Contreras, Evaristo (2012). “Hacia las unidades de bibliometría en las universidades: modelo y funciones”. Revista española de documentación científica, v. 35, n. 3, pp. 469-480. https://doi.org/10.3989/redc.2012.3.959
  • Torres-Salinas, Daniel; Robinson-García, Nicolás; Jiménez-Contreras, Evaristo; Delgado-López-Cózar, Emilio (2012). “Towards a “Book Publishers Citation Reports”. First approach using the ‘Book Citation Index’”. Revista española de documentación científica, v. 35, n. 4, pp. 615-624. https://doi.org/10.3989/redc.2012.4.1010
  • Torres-Salinas, Daniel; Robinson-García, Nicolás; Herrera-Viedma, Enrique; Jiménez-Contreras, Evaristo (2018). “Consideraciones metodológicas sobre uso del impacto normalizado en convocatorias Severo Ochoa y María de Maetzu”. El profesional de la información, v. 27, n. 2, pp. 367-374. https://doi.org/10.3145/epi.2018.mar.15
  • Torres-Salinas, Daniel; Valderrama-Baca, Pilar; Arroyo-Machado, Wenceslao (2022). “Is there a need for a new journal metric? Correlations between JCR Impact Factor metrics and the Journal Citation Indicator - JCI”. Journal of informetrics, v. 16, n. 3, 101315. https://doi.org/10.1016/j.joi.2022.101315
  • Van-Eck, Nees-Jan; Waltman, Ludo (2017). “Citation-based clustering of publications using CitNetExplorer and VOSviewer”. Scientometrics, v. 111, pp. 1053-1070. https://doi.org/10.1007/s11192-017-2300-7
  • Visser, Martijn; Van-Eck, Nees-Jan; Waltman, Ludo (2021). “Large-scale comparison of bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic”. Quantitative science studies, v. 2, n. 1, pp. 20-41. https://doi.org/10.1162/qss_a_00112
  • Wang, Yuzhuo; Zhang, Chengzhi; Li, Kai (2022). “A review on method entities in the academic literature: Extraction, evaluation, and application”. Scientometrics, v. 127, n. 5, 2479-2520. https://doi.org/10.1007/s11192-022-04332-7
  • Wildgaard, Lorna; Schneider, Jesper W.; Larsen, Birger (2014). “A review of the characteristics of 108 author-level bibliometric indicators”. Scientometrics, v. 101, n. 1, pp. 125-158. https://doi.org/10.1007/s11192-014-1423-3
  • Wilsdon, James (2018). “Has the tide turned towards responsible metrics in research?”. The Guardian, July 10. https://www.theguardian.com/science/political-science/2018/jul/10/has-the-tide-turned-towards-responsible-metrics-in-research
  • Wilsdon, James; Allen, Liz; Belfiore, Eleonora; Campbell, Philip; Curry, Stephen; Hill, Steven; Jones, Richard; Kain, Roger; Kerridge, Simon; Thelwall, Mike; Tinkler, Jane; Viney, Ian; Wouters, Paul; Hill, Jude; Johnson, Ben (2015). The metric tide: Report of the independent review of the role of metrics in research assessment and management. HEFCE. http://doi.org/10.13140/RG.2.1.4929.1363
  • Wouters, Paul; Zahedi, Zohreh; Costas, Rodrigo (2019). “Social media metrics for new research evaluation”. In: Glänzel, Wolfgang; Moed, Henk F.; Schmoch, Ulrich; Thelwall, Mike (eds.). Springer handbook of science and technology indicators. Springer, pp. 687-713. https://doi.org/10.1007/978-3-030-02511-3_26
  • Xia, Feng; Wang, Wei; Bekele, Teshome Megersa; Liu, Huan (2017). “Big scholarly data: A survey”. IEEE transactions on big data, v. 3, n. 1, pp. 18-35. https://doi.org/10.1109/TBDATA.2016.2641460
  • Yang, Zhan; Xu, Wei; Chen, Runyu (2021). “A deep learning-based multi-turn conversation modeling for diagnostic Q&A document recommendation”. Information processing & management, v. 58, n. 3, 102485. https://doi.org/10.1016/j.ipm.2020.102485
  • Zacharewicz, Thomas; Lepori, Benedetto; Reale, Emanuela; Jonkers, Koen (2019). “Performance-based research funding in EU Member States - a comparative assessment”. Science and public policy, v. 46, n. 1, pp. 105-115. https://doi.org/10.1093/scipol/scy041
  • Zahedi, Zohreh; Costas, Rodrigo (2018). “General discussion of data quality challenges in social media metrics: Extensive comparison of four major altmetric data aggregators”. Plos one, v. 13, n. 5, e0197326. https://doi.org/10.1371/journal.pone.0197326