Bibliometric indicators have been widely used to assess the value of published scientific output in Poland in recent years. The article presents an overview of academic staff assessment criteria in Poland, selected bibliographic databases developed by libraries, and commercial and open-access citation indexes commonly used in Poland.
Currently, among the activities of a contemporary Polish academic library, those most frequently performed are tasks such as registering staff publications, dealing with bibliometric queries, compiling citation reports and retrieving bibliometric indicators from Polish and international citation indexes, as well as the organisation of training sessions for academic staff in this area. On the basis of a survey, the authors compare the number of bibliometric queries dealt with at various Polish academic libraries, focus on the methods used, and identify the bibliographic indicators most commonly included in the bibliographic databases developed by these libraries.
2. Literature review
The topic of bibliometric analyses carried out by university libraries has been studied previously. In one such study, Hunter and Hambelton (2011) presented the potential advantages of the bibliometric approach to the evaluation of national library roles, while Ball and Tunger (2006) focused on bibliometric analysis as an example of a new business area for libraries. Corrall, Kennan and Afzal (2013) discussed the results of a survey of Australian, New Zealand, Irish, and British libraries and their bibliometric activity, including difficulties, barriers, and the need for staff training and support, while MacColl (2010) presented results of OCLC research reports documenting the role research libraries play in university research assessment in the Netherlands, Denmark, the UK, Ireland and Australia.
Similarly, in Poland the topic of bibliometric assessment has been discussed, among others, by Przyłuska & Maczuga (2011), who presented the most commonly used measures of publishing activity evaluation, such as the number of publications, the number of citations, the Impact Factor, and the Hirsch Index, and Garczyska (2009) focused on Polish university staff collective bibliographies, providing an introduction to the construction of collective bibliographies, analysing types of recorded documents, and evaluation indicators, and discussing the usefulness of such resources.
3. Academic staff assessment in Poland
For many years the scientific output of academic staff at Polish universities has been subject to various forms of evaluation for the purposes of academic promotion, the financial support of research, or simply for the periodic assessment of academic staff and units. Both individual members of staff and whole research teams are subject to assessment, and this process also includes larger academic units such as institutes, faculties or universities.
Academic staff assessment in Poland is conducted at least every two years, and in some cases at minimum every four years (for example when assessing academic teachers with the scientific title of a professor, employed on the basis of a nomination). Assessment criteria include: teaching performance, professional development, improvement in terms of qualifications achieved, accomplishments, and the number and quality of scientific publications.
In Poland, this type of assessment is based both on traditional methods (expert opinion), and more recently on quantitative methods. The latter include various numerical indicators, such as the number of citations, a journal Impact Factor coming from the Journal Citation Reports (JCR) published by Thomson Reuters, and the h-index. ‘The annual JCR impact factor is a ratio of citations and recent citable items published, calculated by dividing the number of current year citations to the source items published in that journal during the previous two years’.1 The h-index, proposed by J. E. Hirsch (2005), is defined as the number of papers with citation number h. These indicators, although not without criticism from academic communities, can provide objective supplementary information on a researcher’s scientific output.
As regards the evaluation of academic units in Poland, a new assessment system was accepted in 1998 by the Scientific Research Committee. This system is applied to those units seeking financial support for their statutory activity and is based on a parametric approach in which the results of a unit’s scientific activity have been subject to a numerical evaluation. As the rating has important consequences (for example, setting the level of financial support granted to fund the unit’s research) ongoing discussion and evaluation of the system have existed since its inception. In the following years, the system has been modified several times by the Ministry of Science and Higher Education. As a result, research units have been obliged to submit an annual unit assessment form, including, among other data, information about the unit staff’s publications and citations. On 30th December 2010, the Committee for the Evaluation of Scientific Units was appointed as a consultative and advisory body to the Minister. The main task of the Committee is to stipulate the parameters and criteria for the comprehensive evaluation of scientific units, and to perform such evaluations no less regularly than every four years. Assessment criteria include accepted scientific achievements, such as publication in scientific journals as specified by the Ministry’s list of scientific journals2, a list published by the Ministry of Science and Higher Education annually. The list consists of three parts:
- Part A: includes scientific journals covered in Thomson Reuters Journal Citation Reports (JCR), with the number of scores for publication set by the Ministry
- Part B: includes local scientific journals not covered in Thomson Reuters Journal Citation Reports (JCR), with the number of scores for publication set by the Ministry
- Part C: includes scientific journals covered in the European Reference Index for the Humanities (ERIH), and not covered in Thomson Reuters JCR, with the number of scores for publication set by the Ministry
4. Bibliographies of Staff Publications
One of the most important activities of academic libraries is the registration of scientific publications of the university staff. In the 1990s, academic units, mainly libraries, began registering faculty academic output in databases of staff publications, while ceasing to prepare bibliographies in print. In the late 1990s, in accordance with the directives of the Ministry of Science and Higher Education and guidelines set by respective university authorities, the transformation of databases into both bibliographic and bibliometric tools was initiated. At the present, some bibliographies also include links to the full contents of registered documents, thus functioning to some extent as full-text databases.
Each library has established its own model of co-operation with their university authorities regarding the documentation and dissemination of research results3. Statistical reports and bibliometric analyses currently compiled by libraries are used both for organizational purposes inside the university (staff, faculty and unit assessment) and for the purpose of the external evaluation of faculties by the Ministry of Science and Higher Education.
An example of a database involved in registering staff publications, including various indicators, and, at the same time, containing links to full texts of publications, is the Bibliographic List of AGH University Staff Publications. The database has been created and developed by the staff at the Main Library of the AGH University of Science and Technology located in Cracow, Poland, with the technical support of the AGH University Computer Centre. In many other libraries, in-house or commercial systems are used (e.g. Expertus).
4.1. AGH University Main Library and the ‘Bibliographic List of Staff Publications’
The AGH University Main Library is the largest technical library in Cracow, and one of the largest in Poland. Its role is to support education and scientific research conducted at the university. The library offers services mostly to the parent university its and students; however, it caters to the needs of other users as well. The collection subject matter reflects the academic courses taught at the university, and most of the collection is available in electronic form.
In 2007, the university Rector issued a directive obliging university staff to register their publications in the Main Library bibliographic database, with registration being performed by the Main Library Reference Department. Scientific output is registered in the ‘Bibliographic List of Staff Publications’ which has been available online since 19994. The database covers the scientific and educational output of academic staff and doctoral students, published within their employment period, or in the case of doctoral students, during their doctoral studies. Original copies of any publication are required for the purposes of registration.
Reference Department staff formally analyse university staff publications, categorising them according to the type of publications, assigning scores in accordance with the guidelines from the Ministry of Science and Higher Education, and from the university authorities. The rules for allotting scores are set in the Ministry directives and depend on the type of publication and the journal rank.
The number of bibliographic descriptions entered into the database has been increasing year on year, as shown in Table 1.
|2. book excerpts (chapters)||640||705||724||644||596||576||650||638|
|3. conference papers||2,267||1,980||1,976||2,021||2,437||2,496||2,725||2,988|
|5. patent descriptions||92||116||145||146||208||268||238||277|
|7. journal editions||35||20||26||17||9||16||21||10|
|8. other (e.g. blogs, broadcasts, online presentations)||33||51||51||36||53||89||67||72|
In total, from 2008 to 2015, 51,102 bibliographic descriptions were registered in the database. Publications registered before 2008 are incorporated in the previous version of the database, and they are systematically exported to the newer version.
Unregistered publications are not taken into consideration for the purposes of staff assessment. The bibliographic database allows for the analysis of publications using tools for generating statistical data which are in turn necessary for carrying out bibliometric analyses and compiling ranking lists. It supports assessment of both individual authors and individual chairs, particular faculties, or the university as a whole, and it provides data for tracking the university, and its staff scientific development. The database is a demonstration of the university’s track record, and provides a clear view of its scientific potential. The content of the database is also of interest for students writing doctoral or graduate theses, as they frequently begin their literature searches using the university staff’s scientific output as a starting point.
5. Bibliometric databases used for academic staff assessment
Among the contemporary evaluation methods of scientific achievement in Poland, bibliometrics plays the leading role. The term ‘bibliometrics’ was proposed by A. Pritchard in 1969: “It is suggested that a better name for this subject [...] is bibliometrics i.e. the application of mathematics and statistical methods to books and other media of communication” (Pritchard, 1969).
Most bibliometric analyses performed for staff assessment and for unit parametric evaluation are based on the following indicators: the number of citations, h-index, the Impact Factor of the journal, including the assessed author’s or unit publication, and number of scores allotted for a given publication according to the Ministry of Science and Higher Education lists. The necessary data is retrieved from bibliographic and bibliometric databases, such as: international commercial indexes (Web of Science, Scopus), and Polish open-access databases developed by academic libraries, such as BazTech (technical sciences) or BazEkon (economics).
5.1. International citation databases
The Ministry of Science and Higher Education recommends that commercial citation indexes (Web of Science and Scopus) are used for the purpose of staff and unit assessment. These indexes are available to Polish universities and scientific research units under a national licence. The indexes register articles from selected journals, selected conference materials, and books. Bibliometric queries answered with the use of these indexes most frequently include searches for citations of publications (often self-citations are listed separately). Citation reports are compiled for each year, and they include both cited and citing articles, bibliographic data, and cited references.
Web of Science (WoS) offers two options for citation searches:
- via ‘Search’: searching for citations of publications covered in WoS, and
- via ‘Cited Reference Search’: searching for citations of publications covered in WoS, their citation variants (e.g. with incomplete or incorrect bibliographic data), and citations of publications not included in WoS, but cited by sources covered by WoS.
Scopus offers similar functionalities, named respectively ‘Search’ and ‘View Secondary Documents’; however, the ‘View Secondary Documents’ option only shows citation variants and citations of publications not covered by Scopus, but cited by Scopus sources.
Citations of publications found via ‘Cited Reference Search’/’View Secondary Documents’, in contrast to ‘Search’, frequently do not include full data (e.g. inaccurate or missing publication or source title, publication year, volume, issue or page number), therefore identification of the cited work is arduous, and in some cases can prove unfeasible. Other problems include: the cited authors’ identical surnames and initials, errors that have occurred in the database while scanning authors’ names due to Polish diacritical marks, or citations quoting the full forename and the first letter of the author’s surname instead of the author’s surname and first name initial (Chadaj & Turecka, 2008). When finding an inaccuracy of this sort, library staff suggest corrections to the database technical support team. For these reasons, the preparation of a comprehensive citation report requires a sufficient knowledge of citation database functionalities, awareness of the most common errors, and a sizeable workload. Owing to all of the possible issues mentioned above, academic staff frequently seek library assistance.
5.1.1. Web of Science
Web of Science provides access to multidisciplinary databases and Thomson Reuters citation indexes. The Web of Science Core Collection, offered to Polish scientific institutions under a national licence, includes access to the following citation indexes:
- Science Citation Index Expanded (1945 – present)
- Social Sciences Citation Index (1956 – present)
- Arts & Humanities Citation Index (1975 – present)
- Conference Proceedings Citation Index- Science (1990 – present)
- Conference Proceedings Citation Index- Social Science & Humanities (1990 – present)
- Book Citation Index- Science (2005 – present)
- Book Citation Index- Social Sciences & Humanities (2005 – present)
- Emerging Sources Citation Index (2015 – present)
and chemical indexes:
- Current Chemical Reactions (2010 – present)
- Index Chemicus (2010 – present)
All databases can be simultaneously searched. Web of Science platform also hosts Journal Citation Reports (JCR), which include journal bibliometric indicators, e.g. the previously mentioned Impact Factor.
Scopus is a multidisciplinary abstract and citation database developed by Elsevier, available to Polish scientific institutions under a national licence since 2012. Scopus covers more than 21,000 peer-reviewed journals, 83,000 conference proceedings, as well as books and trade publications. As with Web of Science, Scopus gives a h-index, which includes documents published since 1970.
For citation searching, open-access bibliographic and bibliometric databases are also used, e.g. Google Scholar, or Polish databases, namely, BazTech and BazEkon.
5.2. Polish databases
Increasingly frequently, Polish bibliographic databases are used for citation searches, especially when a publication has been tailored mainly to the local community and published in a Polish journal. Examples of such databases created and developed by Polish academic libraries, are BazTech and BazEkon.
The BazTech database was jointly developed and is continuously updated by 23 scientific libraries, and is administered by the Main Library of the Cracow University of Technology. It includes abstracts of articles published in Polish scientific and technical journals since 1999, and since 2006, apart from bibliographic information and abstracts, cited references have also been included. This latter feature enables users to retrieve information about citations from the database. BazTech also offers open access to full-texts of selected articles (with the publisher’s consent).
The BazEkon database was developed and is continuously updated by 6 libraries of Polish universities of economics, and offers access to abstract and bibliographic information for articles published in Polish economic journals. The Main Library of the Cracow University of Economy oversees this project.
6. Bibliometric queries in Polish academic library – a survey
6.1. The method
In 2016, the authors conducted a survey among Polish academic libraries regarding the registration of university staff publications, the number of bibliometric queries dealt with, and the methods used for citation searching. The purpose of the survey was to establish to what extent academic libraries support staff assessment and unit evaluation processes by compiling citation reports and registering the university staff scientific output. We also wished to investigate other questions: How detailed are the bibliometric analyses? Which citation indexes are used? Do databases registering scientific publications include bibliometric indicators? (An English translation of the questionnaire is available from the LIBER Quarterly Dataverse, at http://doi.org/10.7910/DVN/XWBK9D.)
The questionnaire was sent to 74 academic libraries in Poland, affiliated with the Conference of Directors of Polish Academic Libraries5 (with the exclusion of the libraries of schools of arts). In total, the questionnaire was directed at 32 university libraries, 16 technical university libraries, 10 medical university libraries, and 16 other academic libraries.
6.2. The questionnaire
The questionnaire included 11 closed questions (with yes or no answers, and the option of adding a commentary). Questions 1 to 5 concerned bibliometric queries dealt with by the library, the databases used, the method of citation searching, and the h-index. In question 6, respondents needed to specify the number of citation reports for individual staff members and for university units compiled during the six years preceding the survey (2010–2015). Questions 7 and 8 focussed on the number of people involved in the queries, and their position within the library structure (the reference department, another department, an individual position). In question 9, respondents were asked whether they provide training support to university staff in relation to citation searches. Questions 10 and 11 centred on the registration of university staff scientific publications in library databases, and the bibliometric indicators included in these databases.
31 libraries returned completed questionnaires (12 technical university libraries, 11 university libraries, 5 medical university libraries, 3 other academic libraries; see Figure 1).
All the libraries register their parent university staff’s publications in up-to-date bibliographic databases. Records in these databases, apart from bibliographic descriptions, include various indicators useful for staff and unit assessment. Most frequently (25 answers – 81%) this is the number of scores for a given publication in accordance with the Ministry of Science and Higher Education list which is valid for the year of publication. 22 libraries (71%) include the Impact Factor value, and 5 libraries (16%) add information concerning citations. 4 libraries (13%) of all the libraries participating in the survey (which constitutes 50% of university libraries whose profile is connected with medicine and physical education) include the Index Copernicus Value (ICV)6 in their bibliographic database. See Figure 2.
All the libraries (100%) which took part in this study stated that they conduct citation analyses for individual members of staff upon request, and 26 libraries (84% respondents) carry out collective analyses (for university units) on demand. 9 libraries (29%) prepare mandatory citation reports for all members of the parent university research staff.
All respondents (100%) compile citation reports on the basis of the Web of Science, and 25 libraries (81%) also use Scopus; 13 libraries (42%) search for citations in Google Scholar. As regards Polish databases, BazTech is used for citation searches by 5 libraries (16%) while 3 respondents (10%) use BazEkon for this purpose.
Citations in Web of Science are found mostly via ‘Cited Reference Search’ (24 replies – 77%). 7 respondents (22%) find citations exclusively via ‘Search’.
Out of 25 libraries which use Scopus for citation search, 13 libraries find citations in Scopus using both ‘Search’ and ‘View Secondary Document’s options, and 12 libraries only use ‘Search’.
The h-index value is usually retrieved from WoS and Scopus. However, upon special request, 15 respondents (48%) calculate a researcher’s h-index based on citations found via ‘Cited Reference Search’ or ‘View Secondary Documents’. The library of the Academy of Physical Education in Wroclaw have implemented a citation module for automatic publication citation searches in Web of Science, as part of the functionality of their bibliographic database in the Expertus library system.
Bibliometric queries are usually answered by the staff of Reference Departments (24 replies – 77% answers). At the Wroclaw University of Technology Library, a Scientometric Analysis Laboratory has been separated within the Scientific and Technical Information Section. Other library units that deal with the queries are: the Electronic Resources Department (Lodz University of Technology), the E-library Department (Poznan University of Technology Library), the Department of Bibliometrics (Szczecin University Library), the Independent Section of Bibliography, the Bibliometrics and Information Resource (Lodz University), the Bibliographic Section (Gdansk Medical University), the Bibliography and Documentation Department (Silesian Medical University), the Bibliometrics Section (Lodz Medical University) or the work is supported by the staff of faculty libraries (Bialystok University).
28 respondents (90% of surveyed libraries) run one-to-one training on citation searching for university staff, and 19 libraries (61%) offer group training. At the Silesian Medical University, citation search training is included in the syllabus on doctoral studies. In addition, libraries organise training sessions run by Thomson Reuters or Elsevier representatives.
6.4. Bibliometric queries – quantitative analysis
In order to estimate the demand from the university community and thus the library staff workload, the authors attempted to carry out a quantitative analysis of individual citation queries7 dealt with from 2011–2015. Unfortunately, few libraries keep these types of statistics. Only 12 respondents (39%) provided the authors with full data regarding the number of bibliometric queries addressed in the 5-year period under study here.
Among these libraries, the greatest number of bibliometric queries was answered at the Silesian Medical University Library, the Wroclaw University of Technology Library, and the Jagiellonian University Collegium Medicum Library in Cracow.
Table 2 shows the number of individual bibliometric queries performed by these libraries over the period in question.
|Year||Silesian Medical University Library||Wroclaw University of Technology Library||Jagiellonian University Collegium Medicum Library|
The Silesian Medical University Library compiles detailed citation reports for each parent university academic staff member annually, and in recent years, twice a year. In total in 2011–2015 the three-person team at the Bibliography and Documentation Department dealt with 9,650 queries. Up to 2013 (inclusively), citations were searched for via ‘Cited Reference Search’, and since 2014, because of the considerable amount of work required to analyse and compile search results, only via Search, but twice a year. The bibliography of University Staff Publications, updated by the staff of the department, includes, apart from bibliographic descriptions of registered publications, additional bibliometric indicators such as: the Ministry of Science and Higher Education scores, Impact Factor, the number of citations together with information about who the publication was cited by and where.
In the Wroclaw University of Technology Library (currently the Centre for Scientific and Technical Information) bibliometric queries are dealt with by an 8-person team working in the Scientific and Technical Information Section. Citation reports are compiled annually for every university staff member on the basis of Web of Science via ‘Cited Reference Search’. For example, in 2015, the team searched for citations for 1,920 university staff; 1,167 of them had citations in WoS in 2015; 5,269 publications were cited, and 12,407 citations were found in total. Their database for the registration of staff scientific output includes bibliometric indicators such as the Ministry of Science and Higher Education scores, the journal Impact Factor, information about whether the journal is included in the Thomson Reuters Master Journal List, and citations of publications. The database provides links to full texts of publications, if the university has access to them.
The library of the Jagiellonian University Collegium Medicum compiles citation reports on request (not for every university employee); therefore the number of queries is lower than in the above-mentioned libraries, though still relatively high. Reports are prepared on the basis of Web of Science and Scopus; for staff, detailed reports via ‘Cited Reference Search’ are drawn up; for university units via the function ‘Search’ (in Scopus respectively: Search + View Secondary Documents in the case of staff, Search for the university units). The queries are dealt with by the Reference Department 3-person team. The publication database includes bibliometric indicators, namely: the Ministry of Science and Higher Education scores, and the journal Impact Factor.
The authors have used the term ‘collective bibliometric queries’ in this text to refer to queries concerning university units, i.e. chairs, institutes, faculties or the entire university. The number of staff working for each university unit is not uniform; therefore the number of publications for which citation reports are compiled varies significantly. For this reason, collective queries have not been subject to quantitative analysis. However, it is worth mentioning that apart from reports for individual staff, collective queries are also performed at Polish academic libraries. The Silesian Medical University library answered 1,403 collective queries over the 5-year period surveyed, which was the highest figure among all the libraries which responded to the questionnaire.
Practical measures undertaken by libraries depend on the needs of the community that the library provides services for.
All of the surveyed libraries support the evaluation process through the registration of scientific publications from the parent university staff in bibliographic databases. The bibliographies of university staff publications, currently compiled by libraries, are useful for statistical purposes, for the assessment of the scientific output of the university staff, for the parametric evaluation of institutions, and for addressing various kinds of queries. The larger the range of options for retrieving data from a bibliographic database, the wider its utility for staff and unit assessment. This diversity of means and the necessity to meet various requirements results in the particular selection of indicators included in the database. Currently, the most frequently included bibliometric indicators are the Ministry of Science and Higher Education score, and the Impact Factor value8 of the journal in which the article was published. Libraries of medical universities and academies of physical education often include the Index Copernicus Value as well. Some of the databases provide information on citations of the publications registered by university staff. As a result, the bibliographies are developing into bibliometric databases.
University units and staff members order citation reports, an important element in any assessment, to be compiled at libraries (all surveyed libraries compile citation reports for individual university staff members upon request, and 26 libraries (84%) for the university units upon request). Citations are searched for primarily via extended functions (‘Cited Reference Search’ and ‘View Secondary Documents’). The h-index is retrieved from Web of Science and Scopus, but on request it is additionally calculated by the library staff on the basis of citations found via ‘Cited Reference Search’ (WoS) or ‘View Secondary Documents’ (Scopus). This demonstrates the high level of accuracy of the bibliometric queries answered by Polish academic libraries. Some libraries have even appointed teams to deal with bibliometric enquiries, a development which shows the increase in specialization in library task profiles in certain areas.
Bibliometric queries in Poland are answered predominantly on the basis of international commercial databases, of which Web of Science is utilised by all respondents (Scopus – 77%). The open-access Google Scholar database is used to a much lesser extent. Polish databases provide information on publications cited locally, the most often drawn on for this purpose being BazTech.
Running one-to-one or group training on citation searching also frequently falls under the auspices of the academic library. Due to the demand for bibliometric information required for the purposes of assessment, university authorities now value their libraries even more as information centres, and their documentation functions are perceived as crucial for university development overall.
Research Data: The full dataset of the survey (in Polish) and an English translation of the Questionnaire can be downloaded from the LIBER Quarterly Dataverse, at http://doi.org/10.7910/DVN/XWBK9D.