In Search of the Perceived Quality and Impact of Accredited South African Law Journals: Exploring the Possibility of a Ranking System. A Baseline Study: 2009 - 2014

  • Marita Carnelley Faculty of Law at the North-West University
Keywords: ranking, journals, law, quality, impact, subsidy, Department of Higher Education and Training, public universities

Abstract

The DHET Research Output Policy (2015) indicates that there has been a change in the government’s approach to research funding. Previously all research published in any accredited journal was rewarded equally. A decision has been taken, however, that a shift will be made towards rewarding better quality and higher impact peer-review research. Additional mechanisms such as biometric/bibliometric data, including citations, assessments by discipline-specific panels of experts and/or post-publication reviews may be used to determine the quality and impact of publications. The policy notes that the DHET may distinguish between "high" and "low" impact journals after proper consultation.

This article highlights the need for consultation by the legal fraternity with the DHET about the implementation of these possible mechanisms in the light of the special considerations applicable to the evaluation of law journals: most journals publish mainly local legal content, there is a limited number of active legal academics, the nature of legal research is not empirical, and a premium is placed on the writing of books.

The research evaluates the available data between 2009 and 2014 in an attempt to assess if it would be appropriate to introduce a legal journal ranking system in South Africa. The article discusses direct and indirect forms of quality evaluation to inform possible ranking systems. This includes the data from the ASSAf expert panel evaluation of law journals in 2014 and other bibliometric data based on whether the journal is featured in international accredited lists, the size of its print-run, author prominence, rejection-rate, usage studies, and evaluations based on citations. An additional ranking system is considered, based on the five best outputs submitted to the National Research Foundation by applicants applying for rating.

The article concludes that a law journal ranking system would be inappropriate for South Africa. None of the systems meet the minimum requirements for a trustworthy ranking of South African law journals, as the data available are insufficient, non-verifiable and not based on objective quality-sensitive criteria. Consultation with the DHET is essential and urgent to avoid the implementation of inappropriate measures of quality and impact assessmen

Author Biography

Marita Carnelley, Faculty of Law at the North-West University

Professor

Faculty of Law

References

Bibliography

Literature

Altbach 2001 Higher Education
Altbach PG "Academic Freedom: International Realities and Challenges" 2001 Higher Education 205-219

ASSAf Report on Strategic Approach to Research Publishing
ASSAf Report on a Strategic Approach to Research Publishing in South Africa (ASSAf Pretoria 2006)

ASSAf Report on Grouped Peer Review of Scholarly Journals in Law
ASSAf Report on Grouped Peer Review of Scholarly Journals in Law and Related Legal Fields (ASSAf Pretoria 2014)

Brophy 2006 Connecticut Law Review
Brophy AL "Law [Review]'s Empire: The Assessment of Law Reviews and Trends in Legal Scholarship" 2006
Connecticut Law Review 101-107

Budden et al 2008 Trends Ecol Evol
Budden AE et al "Double-Blind Review Favours Increased Representation of Female Authors" 2008 Trends Ecol Evol 4-6

Campbell, Goodacre and Little 2006 J L & Soc'y
Campbell K, Goodacre A and Little G "Ranking of UK Law Journals. An Analysis of the Research Assessment Exercise (RAE) 2001 Submissions and Results" 2006 J L & Soc'y 335-363

Carnelley 2015 Obiter
Carnelley M "Identifying Predatory Open-Access Academic Journal Publishers, in Light of the South African Department of Higher Education and Training's Decision to Retrospectively De-accredit Certain Journals" 2015 Obiter 519-538

Crespi 1997 Int'l Law
Crespi GS "Ranking International and Comparative Law Journals: A Survey of Expert Opinion" 1997 Int'l Law 869-886
Crespi 1998 Wm & Mary Envtl L & Pol'y Rev

Crespi GS "Ranking the Environmental Law, Natural Resources Law and Law Use Planning Journals: A Survey of Expert Opinion" 1998 Wm & Mary Envtl L & Pol'y Rev 273-297

Crespi 1999 Fla St U L Rev
Crespi GS "Ranking Specialised Law Reviews: A Methodological Critique" 1999 Fla St U L Rev 837-849

Currie and Pandler 2011 J Bank Finance
Currie RR and Pandler GS "Survey. Finance Journal Rankings and Tiers: An Active Scholarly Assessment Methodology" 2011 J Bank Finance 7-20

Curtis 2008 Globalisation, Societies and Education
Curtis B "The Performance‐Based Research Fund: Research Assessment and Funding in New Zealand" 2008 Globalisation, Societies and Education 179-194

Datt, Tran and Tran-Nam 2009 ATF
Datt K, Tran A and Tran-Nam B "Ranking Tax Journals – The Way Forward" 2009 ATF 341-369

DHET Research Output Policy
Department of Higher Education and Training Research Output Policy (DHET Pretoria 2015)

DHET White Paper
Department of Higher Education and Training White Paper for Post-School Education and Training: Building an Expanded, Effective and Integrated Post-School System (DHET Pretoria 2013)

DHET Directorate Communiqué
Department of Higher Education and Training Directorate: University Policy and Development Support Communiqué 2 of 2015 (DHET Pretoria August 2015)

Doyle 2004 Leg Ref Serv Q
Doyle J "Ranking Legal Periodicals and some other Numeric Uses of the Westlaw and Lexis Periodical Databases" 2004 Leg Ref Serv Q 1-396

Editorial 2012 EJIL
Editorial 2012 "Impact Factor – The Food is Bad and What's More There is not Enough of It: EJIL – The Beginning of the Existential Debate; Masthead Changes; In this Issue" 2012 EJIL 607-612

Editorial 2015 Nature
Editorial 2015 "Nature Journals Offer Double-Blind Review" 2015 Nature 274

Eisenberg and Wells 2014 Economic Inquiry
Eisenberg T and Wells MT "Ranking Law Journals and the Limits of Journal Citation Reports" 2014 Economic Inquiry 1301-1314

European Union Expert Group on Assessment of University-based Research Assessing University-based Research
European Union Expert Group on Assessment of University-based Research Assessing University-based Research (Publications Office of the EU Luxembourg 2010)

Fuld 1953 NYU L Rev
Fuld SH "A Judge Looks at the Law Review" 1953 NYU L Rev 915-921

George and Guthrie 1999a Fla St U L Rev
George TE and Guthrie C "An Empirical Evaluation of Specialised Law Reviews" 1999 Fla St U L Rev 813-836

George and Guthrie 1999b Fla St U L Rev
George TE and Guthrie C "In Defence of Author Prominence: A Reply to Crespi and Korobkin" 1999 Fla St U L Rev 877-896

Given, Kelly and Wilson "Bracing for Impact"
Given LM, Kelly W and Wilson R "Bracing for Impact: The Role of Information Science in Supporting Societal Research Impact" in Proceedings of the 78th ASIS&T Annual Meeting: Information Science (6-10 November 2015 St Louis, MO) 1-10

Grossman 2003 Colum J Gender & L
Grossman JL "Feminist Law Journals and the Ranking Conundrum" 2003 Colum J Gender & L 522-531

HEFCE Research Excellence Framework
Higher Education Funding Council for England Research Excellence Framework 2014: Overview Report by Main Panel C and Sub-panels 16 to 26 (HEFCE Bristol 2015)

HEFCE Research Assessment Exercise 2001
Higher Education Funding Council for England Research Assessment Exercise 2001 Law Panel General Overview (HEFCE Bristol 2001)

Jarvis and Coleman 1997 Arizona L Rev
Jarvis RM and Coleman PG "Ranking Law Reviews: An Empirical Analysis Based on Author Prominence" 1997 Arizona L Rev 15-24

Jarvis and Coleman 2007 L Libr J
Jarvis RM and Coleman PG "Ranking Law Reviews by Author Prominence – Ten Years Later" 2007 L Libr J 573-588

Kahn 2011 SAJS
Kahn M "A Bibliometric Analysis of South Africa's Scientific Outputs – Some Trends and Implications" 2011 107(1/2) SAJS 1-6

Korobkin 1999 Fla St U L Rev
Korobkin R "Ranking Journals: Some Thoughts on Theory and Methodology" 1999 Fla St U L Rev 851-876

Liefner 2003 Higher Education
Liefner I "Funding, Resource Allocation and Performance in Higher Education System" 2003 Higher Education 469-489

Mallard, Lamont and Guetzkow 2009 Sci Technol Hum Values
Mallard G, Lamont M and Guetzkow J "Fairness as Appropriateness. Negotiating Epistemological Differences in Peer Review" 2009 Sci Technol Hum Values 573-606

Marashi et al 2015 SAJS
Marashi S-A et al "Can Scientific Journals be Classified Based on their 'Citation Profiles'?" 2015 SAJS 1-3

Maru 1976 Am B Found Res J
Maru O "Measuring the Impact of Legal Periodicals" 1976 Am B Found Res J 227-249

McWhirter Legal 100
McWhirter DA The Legal 100: A Ranking of Individuals Who Have Most Influenced the Law (Birch Lane Press Secaucus 1998)

Ministry of Education Policy and Procedures
Ministry of Education Policy and Procedures for Measurement of Research Output of Public Higher Education Institutions (Department of Education Pretoria 2003)

Moed, Van Leeuwen and Reedijk 1998 J Doc
Moed HF, Van Leeuwen Th N and Reedijk J "A New Classification System to Describe the Ageing of Scientific Journals and Their Impact Factors" 1998 J Doc 387-419

Mouton Bibliometric Analysis
Mouton J A Bibliometric Analysis of the State of Research at UKZN. Final Report (Centre for Research on Evaluation, Science and Technology Stellenbosch 2015)

Mouton and Valentine 2017 SAJS
Mouton J and Valentine A "The Extent of the South African Authored Articles in Predatory Journals" 2017 SAJS 1-9

NRF/RISA Evaluation and Rating
NRF/RISA Evaluation and Rating. Guidelines and Instructions for Completing the Application for Evaluation and Rating via the NRF Online Submission System (NRF/RISA Pretoria 2015)

OCLC Research Assessment Regimes
Online Computer Library Centre A Comparative Review of Research Assessment Regimes in Five Countries and the Role of Libraries in the Research Assessment Process. A Pilot Study (OCLC Truro 2009)

Osterloh and Frey Research Governance in Academia
Osterloh M and Frey BS Research Governance in Academia: Are There Alternatives to Academic Rankings? (Institute for Empirical Research in Economics Zurich 2009)

Perry 2006 Va J Law Technol
Perry R "The Relative Value of American Law Reviews: A Critical Appraisal of Ranking Methods" 2006 Va J Law Technol 1-40

Pouris and Pouris 2015 SAJS
Pouris AEM and Pouris A "An Assessment of South Africa's Research Journals: Impact Factors, Eigenfactors and Structure of Editorial Boards" 2015 111(3/4) SAJS 1-8

Pritchard 1998 Minerva
Pritchard RMO "Academic Freedom and Autonomy in the United Kingdom and Germany" 1998 Minerva 101-124

Ramsay and Stapledon 1997 MULR
Ramsay I and Stapledon GP "A Citation Analysis of Australian Law Journals" 1997 MULR 676-692

Smyth 1998 U Tas LR
Smyth R "Academic Writing and the Courts: A Qualitative Study of the Influence of Legal and Non-Legal Periodicals in the High Court" 1998 U Tas LR 164-182

Smyth 2012 UNSWLJ
Smyth R "Who Publishes in Australia's Top Law Journals?" 2012 UNSWLJ 201-245

Steinbuch 2009 Loy LA L Rev
Steinbuch R "On the Leiter Side: Developing a Universal Assessment Tool for Measuring Scholarly Output by Law Professors and Ranking Law Schools" 2009 Loy LA L Rev 87-123

Svantesson 2009 Legal Studies
Svantesson DJB "International Ranking of Law Journals – Can it be done and at What Costs?" 2009 Legal Studies 678-691

Svantesson and White 2009 Bond LR
Svantesson DJB and White P "Entering an Era of Research Ranking – Will Innovation and Diversity Survive?" 2009 Bond LR 173-191

Van Gestel 2015 Legal Studies
Van Gestel R "Sense and Non-Sense of a European Ranking of Law Schools and Law Journals" 2015 Legal Studies 165-185

Van Gestel and Vrancken 2011 GLJ
Van Gestel R and Vrancken J "Assessing Legal Research: Sense and Nonsense of Peer Review Versus Bibliometrics and the Need for a European Approach" 2011 GLJ 901-929

WITS Research Publication Incentive (RINC) Policy
University of the Witwatersrand Faculty of Commerce, Law and Management Faculty Research Committee Research Publication Incentive (RINC) Policy (WITS Johannesburg circa 2015)

Yoon 2013 JLA
Yoon AH "Editorial Bias in Legal Academia" 2013 JLA 309-339

Internet sources

Akademie van Wetenschappen 2014 http://www.knaw.nl/nl/actueel/nieuws/
wetenschapsorganisaties-presenteren-nieuw-evaluatieprotocol-voor-onderzoek
Akademie van Wetenschappen 2014 Standard Evaluation Protocol 2015 – 2021 http://www.knaw.nl/nl/actueel/nieuws/wetenschapsorganisaties-presenteren-nieuw-evaluatieprotocol-voor-onderzoek accessed 10 October 2017

Elsevier date unknown https://www.elsevier.com/solutions/scopus
Elsevier date unknown About Scopus https://
www.elsevier.com/solutions/scopus accessed 10 October 2017

IBSS 2013 http://media2.proquest.com/documents/IBSS+
Editorial+Policies+and+ Principles.pdf
International Bibliography of the Social Sciences 2013 IBSS Editorial Policies and Principles http://media2.proquest.com/ documents/
IBSS+Editorial+Policies+and+Principles.pdf accessed 10 October 2017
London School of Economics 2010 http://blogs.lse.ac.uk/impactofsocial
sciences/files/ 2010/11/chart3.jpg

London School of Economics 2010 Maximising the Impact of Academic Research http://blogs.lse.ac.uk/impactofsocialsciences/files/2010/11/chart3.jpg accessed 10 October 2017
NSD date unknown https://dbh.nsd.uib.no/publiseringskanaler/OmKriterier.
action?requestlocale=en
Norwegian Register for Scientific Journals, Series and Publishers date unknown Criteria for Inclusion of New Scientific Publication Channels https://dbh.nsd.uib.no/publiseringskanaler/OmKriterier.action?request_locale=en accessed 10 October 2017

PBRF date unknown http://www.tec.govt.nz/funding/funding-and-performance/
New Zealand Performance-Based Research Fund date unknown Funding and Performance http://www.tec.govt.nz/Funding/funding-and-performance/ accessed 10 October 2017
SciELO date unknown http://www.scielo.org.za/avaliacao/avaliacao_en.
htm

Scientific Electronic Library Online date unknown 'SciELO SA Criteria': Criteria, Policy and Procedures for Admission and Permanence of Scientific Journals in the South African Collection of SciELO http://www.scielo.org.za/avaliacao/avaliacao_en.htm accessed 10 October 2017

Testa 2016 http://wokinfo.com/essays/journal-selection-process/
Testa J 2016 Journal Selection Process http://wokinfo.com/essays/journal-selection-process/ accessed 10 October 2017

Tome and Lipu 2004 http://citeseerx.ist.psu.edu/viewdoc/download?doi=
10.1.1.897.6809&rep=rep1&type=pdf
Tome L and Lipu S 2004 Research and Development Discussion Paper #6: Indicators of Journal Quality http://citeseerx.ist.psu.edu/viewdoc/download
?doi=10.1.1.897.6809&rep=rep1&type=pdf accessed 19 December 2017

UKZN Research Office date unknown http://research.ukzn.ac.za/DoHETAccredited Journals.aspx
UKZN Research Office date unknown DHET Accredited Journals http://research. ukzn.ac.za/DoHETAccreditedJournals.aspx accessed 10 October 2017

Washington and Lee University 2009-2014 http://lawlib.wlu.edu/LJ/
Washington and Lee University 2009-2014 Law Journals: Submissions and Ranking, 2009 – 2014 http://lawlib.wlu.edu/LJ/ accessed 10 October 2017

WITS date unknown https://www.wits.ac.za/sajhr/#sthash.OYn340c9.dpuf
University of the Witwatersrand date unknown South African Journal on Human Rights https://www.wits.ac.za/sajhr/#sthash.OYn340c9.dpuf accessed 10 October 2017
Views
  • Abstract 9
  • pdf 6
  • XML 0
Views and downloads are with effect from 11 January 2018
Published
2018-01-22
Section
Articles