In Search of the Perceived Quality and Impact of Accredited South African Law Journals: Exploring the Possibility of a Ranking System. A Baseline Study: 2009 - 2014

Authors

  • Marita Carnelley Faculty of Law at the North-West University

DOI:

https://doi.org/10.17159/1727-3781/2018/v21i0a3459

Keywords:

ranking, journals, law, quality, impact, subsidy, Department of Higher Education and Training, public universities

Abstract

The DHET Research Output Policy (2015) indicates that there has been a change in the government’s approach to research funding. Previously all research published in any accredited journal was rewarded equally. A decision has been taken, however, that a shift will be made towards rewarding better quality and higher impact peer-review research. Additional mechanisms such as biometric/bibliometric data, including citations, assessments by discipline-specific panels of experts and/or post-publication reviews may be used to determine the quality and impact of publications. The policy notes that the DHET may distinguish between "high" and "low" impact journals after proper consultation.

This article highlights the need for consultation by the legal fraternity with the DHET about the implementation of these possible mechanisms in the light of the special considerations applicable to the evaluation of law journals: most journals publish mainly local legal content, there is a limited number of active legal academics, the nature of legal research is not empirical, and a premium is placed on the writing of books.

The research evaluates the available data between 2009 and 2014 in an attempt to assess if it would be appropriate to introduce a legal journal ranking system in South Africa. The article discusses direct and indirect forms of quality evaluation to inform possible ranking systems. This includes the data from the ASSAf expert panel evaluation of law journals in 2014 and other bibliometric data based on whether the journal is featured in international accredited lists, the size of its print-run, author prominence, rejection-rate, usage studies, and evaluations based on citations. An additional ranking system is considered, based on the five best outputs submitted to the National Research Foundation by applicants applying for rating.

The article concludes that a law journal ranking system would be inappropriate for South Africa. None of the systems meet the minimum requirements for a trustworthy ranking of South African law journals, as the data available are insufficient, non-verifiable and not based on objective quality-sensitive criteria. Consultation with the DHET is essential and urgent to avoid the implementation of inappropriate measures of quality and impact assessmen

Google_Scholar4.png  ScienceOpen_Log0343104.png

Downloads

Download data is not yet available.

Metrics

Metrics Loading ...

Author Biography

Marita Carnelley, Faculty of Law at the North-West University

Professor

Faculty of Law

References

Bibliography

Literature

Altbach 2001 Higher Education

Altbach PG "Academic Freedom: International Realities and Challenges" 2001 Higher Education 205-219

ASSAf Report on Strategic Approach to Research Publishing

ASSAf Report on a Strategic Approach to Research Publishing in South Africa (ASSAf Pretoria 2006)

ASSAf Report on Grouped Peer Review of Scholarly Journals in Law

ASSAf Report on Grouped Peer Review of Scholarly Journals in Law and Related Legal Fields (ASSAf Pretoria 2014)

Brophy 2006 Connecticut Law Review

Brophy AL "Law [Review]'s Empire: The Assessment of Law Reviews and Trends in Legal Scholarship" 2006

Connecticut Law Review 101-107

Budden et al 2008 Trends Ecol Evol

Budden AE et al "Double-Blind Review Favours Increased Representation of Female Authors" 2008 Trends Ecol Evol 4-6

Campbell, Goodacre and Little 2006 J L & Soc'y

Campbell K, Goodacre A and Little G "Ranking of UK Law Journals. An Analysis of the Research Assessment Exercise (RAE) 2001 Submissions and Results" 2006 J L & Soc'y 335-363

Carnelley 2015 Obiter

Carnelley M "Identifying Predatory Open-Access Academic Journal Publishers, in Light of the South African Department of Higher Education and Training's Decision to Retrospectively De-accredit Certain Journals" 2015 Obiter 519-538

Crespi 1997 Int'l Law

Crespi GS "Ranking International and Comparative Law Journals: A Survey of Expert Opinion" 1997 Int'l Law 869-886

Crespi 1998 Wm & Mary Envtl L & Pol'y Rev

Crespi GS "Ranking the Environmental Law, Natural Resources Law and Law Use Planning Journals: A Survey of Expert Opinion" 1998 Wm & Mary Envtl L & Pol'y Rev 273-297

Crespi 1999 Fla St U L Rev

Crespi GS "Ranking Specialised Law Reviews: A Methodological Critique" 1999 Fla St U L Rev 837-849

Currie and Pandler 2011 J Bank Finance

Currie RR and Pandler GS "Survey. Finance Journal Rankings and Tiers: An Active Scholarly Assessment Methodology" 2011 J Bank Finance 7-20

Curtis 2008 Globalisation, Societies and Education

Curtis B "The Performance‐Based Research Fund: Research Assessment and Funding in New Zealand" 2008 Globalisation, Societies and Education 179-194

Datt, Tran and Tran-Nam 2009 ATF

Datt K, Tran A and Tran-Nam B "Ranking Tax Journals – The Way Forward" 2009 ATF 341-369

DHET Research Output Policy

Department of Higher Education and Training Research Output Policy (DHET Pretoria 2015)

DHET White Paper

Department of Higher Education and Training White Paper for Post-School Education and Training: Building an Expanded, Effective and Integrated Post-School System (DHET Pretoria 2013)

DHET Directorate Communiqué

Department of Higher Education and Training Directorate: University Policy and Development Support Communiqué 2 of 2015 (DHET Pretoria August 2015)

Doyle 2004 Leg Ref Serv Q

Doyle J "Ranking Legal Periodicals and some other Numeric Uses of the Westlaw and Lexis Periodical Databases" 2004 Leg Ref Serv Q 1-396

Editorial 2012 EJIL

Editorial 2012 "Impact Factor – The Food is Bad and What's More There is not Enough of It: EJIL – The Beginning of the Existential Debate; Masthead Changes; In this Issue" 2012 EJIL 607-612

Editorial 2015 Nature

Editorial 2015 "Nature Journals Offer Double-Blind Review" 2015 Nature 274

Eisenberg and Wells 2014 Economic Inquiry

Eisenberg T and Wells MT "Ranking Law Journals and the Limits of Journal Citation Reports" 2014 Economic Inquiry 1301-1314

European Union Expert Group on Assessment of University-based Research Assessing University-based Research

European Union Expert Group on Assessment of University-based Research Assessing University-based Research (Publications Office of the EU Luxembourg 2010)

Fuld 1953 NYU L Rev

Fuld SH "A Judge Looks at the Law Review" 1953 NYU L Rev 915-921

George and Guthrie 1999a Fla St U L Rev

George TE and Guthrie C "An Empirical Evaluation of Specialised Law Reviews" 1999 Fla St U L Rev 813-836

George and Guthrie 1999b Fla St U L Rev

George TE and Guthrie C "In Defence of Author Prominence: A Reply to Crespi and Korobkin" 1999 Fla St U L Rev 877-896

Given, Kelly and Wilson "Bracing for Impact"

Given LM, Kelly W and Wilson R "Bracing for Impact: The Role of Information Science in Supporting Societal Research Impact" in Proceedings of the 78th ASIS&T Annual Meeting: Information Science (6-10 November 2015 St Louis, MO) 1-10

Grossman 2003 Colum J Gender & L

Grossman JL "Feminist Law Journals and the Ranking Conundrum" 2003 Colum J Gender & L 522-531

HEFCE Research Excellence Framework

Higher Education Funding Council for England Research Excellence Framework 2014: Overview Report by Main Panel C and Sub-panels 16 to 26 (HEFCE Bristol 2015)

HEFCE Research Assessment Exercise 2001

Higher Education Funding Council for England Research Assessment Exercise 2001 Law Panel General Overview (HEFCE Bristol 2001)

Jarvis and Coleman 1997 Arizona L Rev

Jarvis RM and Coleman PG "Ranking Law Reviews: An Empirical Analysis Based on Author Prominence" 1997 Arizona L Rev 15-24

Jarvis and Coleman 2007 L Libr J

Jarvis RM and Coleman PG "Ranking Law Reviews by Author Prominence – Ten Years Later" 2007 L Libr J 573-588

Kahn 2011 SAJS

Kahn M "A Bibliometric Analysis of South Africa's Scientific Outputs – Some Trends and Implications" 2011 107(1/2) SAJS 1-6

Korobkin 1999 Fla St U L Rev

Korobkin R "Ranking Journals: Some Thoughts on Theory and Methodology" 1999 Fla St U L Rev 851-876

Liefner 2003 Higher Education

Liefner I "Funding, Resource Allocation and Performance in Higher Education System" 2003 Higher Education 469-489

Mallard, Lamont and Guetzkow 2009 Sci Technol Hum Values

Mallard G, Lamont M and Guetzkow J "Fairness as Appropriateness. Negotiating Epistemological Differences in Peer Review" 2009 Sci Technol Hum Values 573-606

Marashi et al 2015 SAJS

Marashi S-A et al "Can Scientific Journals be Classified Based on their 'Citation Profiles'?" 2015 SAJS 1-3

Maru 1976 Am B Found Res J

Maru O "Measuring the Impact of Legal Periodicals" 1976 Am B Found Res J 227-249

McWhirter Legal 100

McWhirter DA The Legal 100: A Ranking of Individuals Who Have Most Influenced the Law (Birch Lane Press Secaucus 1998)

Ministry of Education Policy and Procedures

Ministry of Education Policy and Procedures for Measurement of Research Output of Public Higher Education Institutions (Department of Education Pretoria 2003)

Moed, Van Leeuwen and Reedijk 1998 J Doc

Moed HF, Van Leeuwen Th N and Reedijk J "A New Classification System to Describe the Ageing of Scientific Journals and Their Impact Factors" 1998 J Doc 387-419

Mouton Bibliometric Analysis

Mouton J A Bibliometric Analysis of the State of Research at UKZN. Final Report (Centre for Research on Evaluation, Science and Technology Stellenbosch 2015)

Mouton and Valentine 2017 SAJS

Mouton J and Valentine A "The Extent of the South African Authored Articles in Predatory Journals" 2017 SAJS 1-9

NRF/RISA Evaluation and Rating

NRF/RISA Evaluation and Rating. Guidelines and Instructions for Completing the Application for Evaluation and Rating via the NRF Online Submission System (NRF/RISA Pretoria 2015)

OCLC Research Assessment Regimes

Online Computer Library Centre A Comparative Review of Research Assessment Regimes in Five Countries and the Role of Libraries in the Research Assessment Process. A Pilot Study (OCLC Truro 2009)

Osterloh and Frey Research Governance in Academia

Osterloh M and Frey BS Research Governance in Academia: Are There Alternatives to Academic Rankings? (Institute for Empirical Research in Economics Zurich 2009)

Perry 2006 Va J Law Technol

Perry R "The Relative Value of American Law Reviews: A Critical Appraisal of Ranking Methods" 2006 Va J Law Technol 1-40

Pouris and Pouris 2015 SAJS

Pouris AEM and Pouris A "An Assessment of South Africa's Research Journals: Impact Factors, Eigenfactors and Structure of Editorial Boards" 2015 111(3/4) SAJS 1-8

Pritchard 1998 Minerva

Pritchard RMO "Academic Freedom and Autonomy in the United Kingdom and Germany" 1998 Minerva 101-124

Ramsay and Stapledon 1997 MULR

Ramsay I and Stapledon GP "A Citation Analysis of Australian Law Journals" 1997 MULR 676-692

Smyth 1998 U Tas LR

Smyth R "Academic Writing and the Courts: A Qualitative Study of the Influence of Legal and Non-Legal Periodicals in the High Court" 1998 U Tas LR 164-182

Smyth 2012 UNSWLJ

Smyth R "Who Publishes in Australia's Top Law Journals?" 2012 UNSWLJ 201-245

Steinbuch 2009 Loy LA L Rev

Steinbuch R "On the Leiter Side: Developing a Universal Assessment Tool for Measuring Scholarly Output by Law Professors and Ranking Law Schools" 2009 Loy LA L Rev 87-123

Svantesson 2009 Legal Studies

Svantesson DJB "International Ranking of Law Journals – Can it be done and at What Costs?" 2009 Legal Studies 678-691

Svantesson and White 2009 Bond LR

Svantesson DJB and White P "Entering an Era of Research Ranking – Will Innovation and Diversity Survive?" 2009 Bond LR 173-191

Van Gestel 2015 Legal Studies

Van Gestel R "Sense and Non-Sense of a European Ranking of Law Schools and Law Journals" 2015 Legal Studies 165-185

Van Gestel and Vrancken 2011 GLJ

Van Gestel R and Vrancken J "Assessing Legal Research: Sense and Nonsense of Peer Review Versus Bibliometrics and the Need for a European Approach" 2011 GLJ 901-929

WITS Research Publication Incentive (RINC) Policy

University of the Witwatersrand Faculty of Commerce, Law and Management Faculty Research Committee Research Publication Incentive (RINC) Policy (WITS Johannesburg circa 2015)

Yoon 2013 JLA

Yoon AH "Editorial Bias in Legal Academia" 2013 JLA 309-339

Internet sources

Akademie van Wetenschappen 2014 http://www.knaw.nl/nl/actueel/nieuws/

wetenschapsorganisaties-presenteren-nieuw-evaluatieprotocol-voor-onderzoek

Akademie van Wetenschappen 2014 Standard Evaluation Protocol 2015 – 2021 http://www.knaw.nl/nl/actueel/nieuws/wetenschapsorganisaties-presenteren-nieuw-evaluatieprotocol-voor-onderzoek accessed 10 October 2017

Elsevier date unknown https://www.elsevier.com/solutions/scopus

Elsevier date unknown About Scopus https://

www.elsevier.com/solutions/scopus accessed 10 October 2017

IBSS 2013 http://media2.proquest.com/documents/IBSS+

Editorial+Policies+and+ Principles.pdf

International Bibliography of the Social Sciences 2013 IBSS Editorial Policies and Principles http://media2.proquest.com/ documents/

IBSS+Editorial+Policies+and+Principles.pdf accessed 10 October 2017

London School of Economics 2010 http://blogs.lse.ac.uk/impactofsocial

sciences/files/ 2010/11/chart3.jpg

London School of Economics 2010 Maximising the Impact of Academic Research http://blogs.lse.ac.uk/impactofsocialsciences/files/2010/11/chart3.jpg accessed 10 October 2017

NSD date unknown https://dbh.nsd.uib.no/publiseringskanaler/OmKriterier.

action?requestlocale=en

Norwegian Register for Scientific Journals, Series and Publishers date unknown Criteria for Inclusion of New Scientific Publication Channels https://dbh.nsd.uib.no/publiseringskanaler/OmKriterier.action?request_locale=en accessed 10 October 2017

PBRF date unknown http://www.tec.govt.nz/funding/funding-and-performance/

New Zealand Performance-Based Research Fund date unknown Funding and Performance http://www.tec.govt.nz/Funding/funding-and-performance/ accessed 10 October 2017

SciELO date unknown http://www.scielo.org.za/avaliacao/avaliacao_en.

htm

Scientific Electronic Library Online date unknown 'SciELO SA Criteria': Criteria, Policy and Procedures for Admission and Permanence of Scientific Journals in the South African Collection of SciELO http://www.scielo.org.za/avaliacao/avaliacao_en.htm accessed 10 October 2017

Testa 2016 http://wokinfo.com/essays/journal-selection-process/

Testa J 2016 Journal Selection Process http://wokinfo.com/essays/journal-selection-process/ accessed 10 October 2017

Tome and Lipu 2004 http://citeseerx.ist.psu.edu/viewdoc/download?doi=

1.1.897.6809&rep=rep1&type=pdf

Tome L and Lipu S 2004 Research and Development Discussion Paper #6: Indicators of Journal Quality http://citeseerx.ist.psu.edu/viewdoc/download

?doi=10.1.1.897.6809&rep=rep1&type=pdf accessed 19 December 2017

UKZN Research Office date unknown http://research.ukzn.ac.za/DoHETAccredited Journals.aspx

UKZN Research Office date unknown DHET Accredited Journals http://research. ukzn.ac.za/DoHETAccreditedJournals.aspx accessed 10 October 2017

Washington and Lee University 2009-2014 http://lawlib.wlu.edu/LJ/

Washington and Lee University 2009-2014 Law Journals: Submissions and Ranking, 2009 – 2014 http://lawlib.wlu.edu/LJ/ accessed 10 October 2017

WITS date unknown https://www.wits.ac.za/sajhr/#sthash.OYn340c9.dpuf

University of the Witwatersrand date unknown South African Journal on Human Rights https://www.wits.ac.za/sajhr/#sthash.OYn340c9.dpuf accessed 10 October 2017

LIST OF ABBREVIATIONS

AHRLJ African Human Rights Law Journal

Am B Found Res J American Bar Foundation Research Journal

Am Sociol Rev American Sociological Review

Annual Survey Annual Survey of the South African Law

Arizona L Rev Arizona Law Review

ASSAf Academy of Science of South Africa

ATF Australian Tax Forum

Bond LR Bond Law Review

CARSA Child Abuse Research in South Africa

CCMA Commission for Conciliation, Mediation and Arbitration

CESifo Centre for Economic Studies Publishing Group

Chi-Kent L Rev Chicago-Kent Law Review

CILSA Comparative and International Law Journal of Southern Africa

Colum J Gender & L Columbia Journal of Gender and the Law

Conn L Rev Connecticut Law Review

DHET Department of Higher Education and Training

EJIL European Journal of International Law

Fla St U L Rev Florida State University Law Review

GLJ German Law Journal

Haifa L Rev Haifa Law Review

HEFCE Higher Education Funding Council for England

IBSS International Bibliography of the Social Sciences

ILJ Industrial Law Journal

Ind LJ Indiana Law Journal

Int'l J Legal Info International Journal of Legal Information

Int'l Law The International Lawyer

ISI Thomson Reuters Web of Science (formerly

ISI Web of Knowledge)

IT Information Technology

J Bank Finance Journal of Banking and Finance

J Doc Journal of Documentation

J L & Soc'y Journal of Law and Society

JJS Journal of Juridical Science

JLA Journal of Legal Analysis

L Libr J Law Library Journal

LDD Law, Democracy and Development

Leg Ref Serv Q Legal Reference Services Quarterly

Loy LA L Rev Loyola of Los Angeles Law Review

Merc LJ South African Mercantile Law Journal

MULR Melbourne University Law Review

NMMU Nelson Mandela Metropolitan University

NRF National Research Foundation

NSD Norwegian Register for Scientific Journals,

Series and Publishers

NYU L Rev New York University Law Review

NZ New Zealand

OCLC Online Computer Library Centre

PBRF New Zealand Performance-Based Research Fund

PER Potchefstroom Elektroniese Regstydskrif

RISA Research and Innovation Support and Advancement

SACJ South African Journal of Criminal Justice

SAJELP South African Journal of Environment al Law and Policy

SAJHR South African Journal of Human Rights

SAJS South African Journal of Science

SALJ South African Law Journal

SAYIL South African Yearbook of International Law

Sci Technol Hum Values Science, Technology and Human Values

SciELO Scientific Electronic Library Online

Stell LR Stellenbosch Law Review

THRHR Tydskrif vir Hedendaags Romeins-Hollandse Reg

Trends Ecol Evol Trends in Ecology and Evolution

TSAR Tydskrif vir die Suid-Afrikaanse Reg

U Tas LR University of Tasmania Law Review

UK United Kingdom

UKZN University of KwaZulu-Natal

UNSWLJ University of New South Wales Law Journal

US / USA United States of America

Va J Law Technol Virginia Journal of Law and Technology

WITS University of the Witwatersrand

Wm & Mary Envtl L & Pol'y Rev William and Mary Environmental Law and

Policy Review

Published

22-01-2018

How to Cite

Carnelley, M. (2018). In Search of the Perceived Quality and Impact of Accredited South African Law Journals: Exploring the Possibility of a Ranking System. A Baseline Study: 2009 - 2014. Potchefstroom Electronic Law Journal, 21, 1–50. https://doi.org/10.17159/1727-3781/2018/v21i0a3459

Issue

Section

Articles