In Search of the Perceived Quality and Impact of Accredited South African Law Journals: Exploring the Possibility of a Ranking System. A Baseline Study: 2009 – 2014

The DHET Research Output Policy (2015) indicates that there has been a change in the government’s approach to research funding. Previously all research published in any accredited journal was rewarded equally. A decision has been taken, however, that a shift will be made towards rewarding better quality and higher impact peer-review research. Additional mechanisms such as biometric/bibliometric data, including citations, assessments by discipline-specific panels of experts and/or post-publication reviews may be used to determine the quality and impact of publications. The policy notes that the DHET may distinguish between "high" and "low" impact journals after proper consultation. This article highlights the need for consultation by the legal fraternity with the DHET about the implementation of these possible mechanisms in the light of the special considerations applicable to the evaluation of law journals: most journals publish mainly local legal content, there is a limited number of active legal academics, the nature of legal research is not empirical, and a premium is placed on the writing of books. The research evaluates the available data between 2009 and 2014 in an attempt to assess if it would be appropriate to introduce a legal journal ranking system in South Africa. The article discusses direct and indirect forms of quality evaluation to inform possible ranking systems. This includes the data from the ASSAf expert panel evaluation of law journals in 2014 and other bibliometric data based on whether the journal is featured in international accredited lists, the size of its print-run, author prominence, rejection-rate, usage studies, and evaluations based on citations. An additional ranking system is considered, based on the five best outputs submitted to the National Research Foundation by applicants applying for rating. The article concludes that a law journal ranking system would be inappropriate for South Africa. None of the systems meet the minimum requirements for a trustworthy ranking of South African law journals, as the data available are insufficient, non-verifiable and not based on objective quality-sensitive criteria. Consultation with the DHET is essential and urgent to avoid the implementation of inappropriate measures of quality and impact assessment.


Introduction: A changing research landscape
There is something fundamentally absurd about the idea of ranking research. At the same time, no one can seriously argue that all research is equal in importance and quality. Either way, we are doubtless witnessing a dramatic change in the management and organisation of research. One aspect of the change is a move towards the ranking of research. 1 The assessment of the quality of research in higher education is a relatively new international phenomenon. 2 Academics are required to publish more, to publish faster, 3 and to demonstrate the quality and significance of their research to their employers and state funders. 4 This trend is prevalent in South Africa as well, and is part of a larger corporatisation movement to introduce managerial mechanisms. Initially developed to measure performance in profit-making enterprises, quality assessment was extended into academia to "improve efficiency and economy" 5 at universities and to advance capacity, quality and innovation. 6 The merits of this trend and its potential impact on academic freedom are excluded from this discussion. 7 In developing any national research policy: It … is worth considering three fundamental issues: first, whether there is a need to assess the quality of research outputs and, if so, whether it is better to control the assessment process centrally or devolve the process to individual institutions; second, if research assessment is deemed useful, then The need for the quality assessment of research output is recognised and accepted in South Africa as well as internationally, and a form of such assessment is already in use. 9 Research assessment is seen as useful and necessary, and is currently practised in the form of rewarding all contributions to accredited peer-reviewed journals in the form of the government "publication output subsidy". 10 Over the past decade or so public universities have successfully encouraged their academics to publish peer-reviewed 11 research in accredited journals in line with this policy, even though the actual number of academics has remained fairly stable. 12 Official opinion will have it that although the policy has resulted in increased output, it has not necessarily led to an improvement in quality. 13 Academic performance measurement generally leads to a growth in measurable output but seldom results in a higher quality of the research or direct reallocation of funds to the best performers. 14 Both the 2013 White Paper for Post-School Education and Training 15 and particularly the 2015 DHET Research Output Policy 16 indicate that a different government approach is to be adopted in an attempt to reward better quality and higher impact peer-reviewed research. 17 Leaving aside the problematic definitions of "quality" and "impact", 18 the question may well be asked why the specific focus on quality and impact is 8 OCLC Research Assessment Regimes 8. 9 Ministry of Education Policy and Procedures. 10 Ministry of Education Policy and Procedures 4: "While the policy recognises different types of research output for purposes of subsidy, it does not support differentiation within types of output." South African public universities are rewarded by the DHET for the number of accredited publications their academic staff publishes (Mouton Bibliometric Analysis 9). 11 Peer-review is not necessarily double-blind peer-review (Budden et al 2008 Trends Ecol Evol 4;Editorial 2015 Nature 274). in the number of predatory publishers and journals, some of which also appear on accredited lists. These seem to focus on profit rather than on an adherence to strict peer-review quality standards, making it easier for authors to get published. 24 In addition, there is a negative perception of the quality of South African journals. 25 Because of the small pool of academics in any specific discipline, it is inevitable that the number of journal submissions as well as of peer reviewers will be limitedcasting doubt over the independence of the peerreview system. 26 The negative perceptions about quality prompted the DHET to amend its research assessment policy as from 2016 to ensure that it received value for its subsidy investment in academic research. 27 In terms of this policy, the DHET will continue to determine the quality of research output by proxy. 28 However, from 2016 the quality assessment of research output may include additional mechanisms such as biometric/bibliometric data (including citations), 29 discipline-specific panels of experts and post-publication reviews by the DHET. 30 The DHET "may consider introducing such measures as the categorising of journals as 'high' or 'low' impact; citation indexes or other relevant and appropriate quality measurements as prerequisites, after due and extensive consultation process with this sector." 31 No formal consultation process with legal academia on this issue has yet begun at the time of the writing of this article, but certain universities 24 See in general Mouton and Valentine 2017 SAJS 2;Carnelley 2015 Obiter 519-538. 25 ASSAf Report on Strategic Approach to Research Publishing 29. 26 The increase of specialised journals may also have exacerbated this problem of a lack of an experienced pool of academics per discipline which is in line with research in Canada and the USA that noted that law reviews generally publish articles from their own faculty rather than outsiderseven if they are cited less frequently. This is indicative of what is referred to as "editorial bias in legal academia" (Yoon 2013 JLA 336). In South Africa, DHET has attempted to ameliorate this problem by requiring from 2016 that, to qualify for subsidy, in-house journals should not publish a volume where at least 75% of the submissions emanate from multiple external institutions (DHET Research Output Policy Clause 5.10(c)). 27 The same is true for other jurisdictions ( Van Gestel and Vrancken 2011 GLJ 905). The OCLC Report confirms that in the absence of evidence to government that their research funding results "in good value of quality and impact", it is difficult to objectively defend research budgets (OCLC Research Assessment Regimes 8). 28 The assessment was and is done through ASSAf in terms of the Ministry of Education Policy and Procedures (until 2015), and from 2016 in terms of the DHET Research Output Policy 22.2. 29 ASSAf Report on Strategic Approach to Research Publishing 7. are already differentiating between legal journals on national and international accredited lists. 32 It must be stated that whatever form any additional research assessment takes, it could never be truly objective. 33 Then, the additional burdens of the cost of the implementation of such a system and the inconvenience to individuals and universities by diverting attention to non-core business issues must also be reckoned with. 34 The task facing the South African legal fraternity is to agree on additional quality improvement measures that would generally be regarded as suitable and that could be implemented successfully as a means of determining the impact or quality (or perceptions of the impact or quality) of the South African legal research output, or as a means of distinguishing between the exceptional and the average. The consequences of non-engagement may result in the DHET determining measure(s) that may or may not be appropriate for legal academics.
When one considers the terms "high" and "low" quality and "impact", it stands to reason that a ranking system, identifying the better quality journals could be introduced. 35 However, such a system has not been universally acceptable for all legal jurisdictions.
In the USA, for instance, quality is assessed through a direct journal ranking system as opposed to an assessment of individual articles. 36 This system is well developed with an extensive published discourse about the types of journal ranking systems employed. More about these will form the basis of the discussion in the latter part of this article. was so controversial that it was never implemented and in Australia the 2010 formal journal rankings were abandoned by 2011. 38 The Netherlands and the UK have not even attempted to implement a ranking system. 39 The Netherlands uses "qualitative reviews by panels of international experts for its external reviews" to assess research outputs. 40 The UK implemented an external post-publication evaluation process where selected individual articles submitted by universities are assessed on merit. 41 The 2014 UK Law sub-committee confirmed the view "that peer review remains the most reliable method of assessing research quality in law". 42 In the UK the external evaluation process of selected research output submissions, rather than the ranking of journals, serves as quality-control. 43 In 2001  It is submitted that should the DHET in South Africa implement a system of differentiating among journals based on their "low" and "high" impact, whether by a panel of experts and/or the use of bibliometric data and/or through other methods, it is conceivable that a journal ranking system would follow, either officially or unofficially, unless a suitable alternative could be found.

Journal ranking systems
It has been argued that a journal ranking system could serve to increase the quality and impact of scholarship and create incentives for journal editors to select and publish only quality submissions; which, in turn, would motivate academics to strive to produce work of higher quality. 45 However, this is not always the case as "ground-breaking 'must read' articles are as likely to be published in less prestigious journals as those held in particular high regard". 46 Research has shown that a ranking system may "stifle diversity and innovation", as journal editors may prefer to publish mainstream articles to increase their rankings rather than new and experimental research. 47 An unintended consequence (also of incentivising editors to publish only high quality outputs) may be a chilling effect on young academics, wary of rejection, or, where those young academics do submit outputs, editors may reject them. In the light of the above observation regarding the relatively small pool of persons working in legal fields in South Africa, this could be catastrophic going forward. And this would be even more devastating in respect of the project of transforming academia in respect of the development of black academics.
Academics could benefit from a ranking system as it could act as a guide to their choice of journal. 48 Publication in higher-ranking journals would afford prestige, as it would signal potential superior quality, 49 which could lead to 44 Svantesson favourable outcomes regarding promotions and career paths. 50 The reputations of law schools would benefit 51 if their journals obtained highranking status. 52 Higher-ranked journals in turn could benefit other stakeholders: they would be more widely purchased, 53 read and cited, with accompanying benefits, as serious scholars are likely to prefer making use of more prestigious journals. 54 From a journal editor's perspective, it could also translate into potential reviewers approached for review being more likely or inclined to review submissions.
For funders and managers, exceptional quality could be rewarded and promoted. 55 For the journals, a negative change in ranking may give rise to self-evaluation and reflection. 56 If journal rankings become established and respected in the legal and academic community, they can have a significant effect on the content of legal scholarship produced nationwide. This conclusion suggests that attempts to rank journals are extremely significant to the scholarly enterprise … 57 According to Perry 58 a ranking system should adhere to certain minimum requirements. 59 First, it should be based on quality-sensitive criteria. 60 This could be problematic because, as mentioned above, one journal may contain both excellent and mediocre articles. 61 Secondly, the ranking methodology must be sensitive to changes in quality and must make allowance for regular revision and updating. 62 Thirdly, the ranking must be based on objective criteria, free from bias, 63  Ranking systems generally distinguish between general journals and specialised journals. In South Africa the number of specialised journals is limited and this distinction is not made in this article.
available data to fulfil the goals of the ranking. 64 The data should be readily verifiable and not susceptible to manipulation. 65 For a ranking system to be successful, it should thus be "carefully designed. If the ranking method is not defensible, then the resultant ranking will not fulfil its goals". 66 These requirements will be used hereunder to evaluate the ranking systems discussed.
It has been acknowledged that a multi-factor combination ranking rather than a single factor system is preferred, 67 although using a combination of factors may be simply "too burdensome"especially where a single factor data system has not yet been collected and coded. 68 The use of a multifactor method is, however, not beyond criticism, as the person determining the ranking has: … to determine how the different factors should be combined to generate the ultimate ranking. The weight that is assigned to each factor is crucial, and since this determination is purely subjective (and most likely controversial), a complex ranking method can [also] never be objective. 69 The question arises whether it is possible to achieve a successful ranking system in South Africa, given the limited number of law journals published in the country, including those of a highly specialised nature, and taking into account the limited number of legal academics in the country.
The ASSAf Report on Grouped Peer Review of Scholarly Journals in Law notes the following special considerations for the evaluation of law journals. 70 One: legal content is more locally-orientated than other disciplines, as the legal principles under scrutiny are mostly jurisdictionspecific. 71 Two: the limited number of active academics in a specific sub- area impacts on the number of possible submissions 72 and qualified peerreviewers. Three: the type of research is generally not primary research or empirical in nature 73 and can be carried out with minimal resources, unlike other types of research that require extensive funding. 74 Four: a premium is placed on the writing of books and chapters in books, resulting in the fact that not all legal scholarship can be found in journal articles. 75 And five: research can be focused on applied legal practice, where research academics are agents for legal changes or developments. 76 Law is always a "discipline in transition" where quality and impact should not rely solely on historical accuracy and data. 77 Legal commentary is aimed at the legal profession and a case note suggesting an alternative approach may have a profound effect on the law that more "lengthy and academic papers published in prestigious journals do not have". 78 Similarly, textbooks clarifying a complex legal issue in an accessible manner may "appear trivial to researchers from other disciplines [but] are in fact highly valuable and more sophisticated than they may seem at first glance". 79 Such work, [law review articles] … has earned the real respect of the bench. We admire the law review for its scholarship, its accuracy, and, above all, for its excruciating fairness. We are well aware that the review takes very seriously its role as judge of judgesand to that, we say, more power to you. By your criticisms, your views, your appraising cases, your tracing the trends, you render the making of 'new' law a little easier. In a real sense, you thus help to keep our system of law an open one, ever ready to keep pace with the changing patterns. 80 Taking the above into consideration, the aim of this article is to focus on the consequences of a possible law journal ranking system for South Africa in determining perceived quality and impact. The author will assess how the existing journal information for the period from 2009 to 2014 would have been evaluated, had the various US-type journal ranking systems been 72 The Report notes that in certain instances there may be criticism that a specific journal accepts too many submissions from a particular university, such as UNISA. However, that university may have many academics working in that area (ASSAf imposed. The article includes direct and indirect quality evaluations. 81 The direct evaluation consists of the ASSAf expert panel evaluation. Indirect evaluations through biometric/bibliometric data are based on whether the journal is featured in internationally accredited lists, the size of its print run, author prominence, the rejection rate, usage studies based on the perusal of library and electronic databases, and evaluations based on citationsin other journals as well as by the courts. An additional and uniquely South African ranking system is also considered, based on the five best outputs submitted to the National Research Foundation by applicants applying for an NRF rating. In this article the principles, advantages and disadvantages, and the outcome of the principles applied (with some nuances) to the South African law journals will be discussed. Finally, the spread of the rankings per journal and an average of all the rankings are shown, flawed as they may be. The article concludes with an assessment of whether the application of the various systems resulted in a consistent ranking outcome or whether the results showed a marked difference in ranking depending on the ranking system used. This information could form the basis of a more informed decision about the viability of ranking systems for law journals in South Africa or whether an alternate evaluation system by the DHET is called for.
Of the twenty-three peer-review law journals that met the ASSAf minimum criteria for accreditation, twenty-one will be considered for this article. 82 It should be noted from the outset that the immediate problem was the "lack of reliable, comparable and comprehensive data", 83 and this article should therefore be treated as exploratoryas a starting point for a debate about the quality and impact evaluation of South African law journals. It is not intended to be comprehensive and neither the parts nor the whole is without fault or beyond criticism. 84 The various aspects could and should be improved upon by additional research and debate. 85 That said, the author 81 The Perry framework is adopted for this article  Va J Law Technol 7-37). Also see Currie and Pandler Journal of Banking and Finance 7.

82
The ASSAf Report on Grouped Peer Review of Scholarly Journals in Law evaluated 24 law journals and recommended that 23 remain accredited. The scope of the article is limited to journals publishing predominantly legal articles. The two multidisciplinary journals, Acta Criminologica and CARSA, were excluded from this discussion. SAJELP was also excluded as it was found not to meet the ASSAf criteria by being out of date at that time, although this has subsequently been rectified. It would be difficult to have rankings of speciality law journals in South Africa as there is only one (or maybe two at most) journal in each specific speciality area. For expects to make a useful contribution by exposing the dangers of a ranking system and the need for engagement with the DHET.

Direct quality and impact assessment through expert panels
In an ideal world, direct quality evaluation for all law journals would be performed by a panel of experts who regularly evaluate journal contributions objectively, according to prescribed criteria. 86 However, it is unrealistic to expect academics who are over-burdened as it is and not expert in all areas of the law to find the time to devote to additional and continuous quality peer reviewing. 87 Even where sub-specialisation panels are utilised and the number of specialised journals is limited, the process would remain timeconsuming, subjective and therefore problematic. 88 The logical conclusion would be that academics would evaluate their own articles as well as those of their peers, making the system inherently subjective, biased 89 and potentially "self-perpetuating". 90 [Peer-review, w]hile this is an important step in the right direction … can, nevertheless give unreliable assessments. Studies have shown peer reviews can produce inconsistent results … 91 The process of ranking journals is complex and even experts might find it challenging to evaluate the varying quality of different journals. 92 The South African ASSAf panels mentioned above have been active for some time and periodically assess sample journals through the ASSAf Committee on Scholarly Publishing in South Africa to determine whether, post-publication, journals still meet the minimum set criteria for inclusion in the accreditation lists. 93 This process is not a journal ranking system and it does not designate "low" or "high" impact status. For now, it provides the best available data for the purposes of this paper.
South African law journals were assessed by the ASSAf in 2014 and the results were published in their Report on Grouped Peer Review of Scholarly Journals in Law. The evaluation of the journals was based on the best practices set out in the ASSAf Editorial Process-related Criteria, 94 which are aimed at eventually promoting available quality open-access online research. 95 As mentioned earlier, the criteria used by the panels included editorial process-related criteria set out in the Code of Best Practices, 96 as 93 DHET Research Output Policy 3.12.

94
The Forum of Editors of Law Journals of Southern Africa also subscribe to these best practices. 95 This quality assurance process is seen as a precursor to the identification of journal titles to be loaded onto the open access platform Scientific Electronic Library Online (SciELO)-South Africa. Journals of a sufficiently high quality will be included in this fully indexed, free online, multi-national platform featured on the Thomson-Reuters Web of Knowledge portal (ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 7). Journals not already on SciELO, were invited to SciELO as "SciELO will become an important tool for the DHET to consider articles for subsidy purposes". (ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 18). In the legal publishing scenario this is problematic, as some of the journals and/or their publishers indicated upfront that they are not interested in making use of the SciELO platform (ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 18). The invitation to join the platform was nonetheless made, but fewer than a quarter of the law journals are listed on the SciELO platform (ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 18). 96 ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 12. The criteria include: the longevity of the journal; the number of original peer-reviewed papers published per year during the last five years; the number of manuscripts submitted and rejected out of hand or after peer review; the average length of published papers; the "author demography" of the papers submitted and published; the number and nature of the peer reviewers used per manuscript and per year, including the institutional and national/international spread; the quality and average length of the peer review reports; the average delay before publication; the frequency of publication; the professional stature and experience of the editor, his selection and length of service; the success in addressing the major issues in the field; the number and professional stature and experience of the editorial board members, the selection process, turnover and involvement; the mix from developed or developing countries; the editorial policy and guidelines; the conflict of interest policy; the annual errata published; value-adding features; the number of pages per issues; the peer review process and professional associations (ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 12-13).
well as business-related criteria 97 and bibliometric assessments. 98 Although the process does not rank journals, the panels do comment on the quality of the contributions in their individual assessments. Descriptions of the quality of the journals are used in this article to determine if and/or how the journals could/should be ranked. They were, however, not standardised and objective, and as such neither very accurate nor very helpful. 99 For instance, would a journal described as "high" quality be the same as "generally high"? Is there a difference in quality between being "good overall" and "generally good"? If the first is inconsistent and of varied quality, and the other included "very good contributions", does that mean that the latter should be rated higher? Where the evaluating panel described a journal as "generally good" would that imply that some contributions were poorer and others better?
If the ranking of law journals should become inevitable, these panels may be in the best position to carry out such a task, although a truly accurate ranking may remain elusive and subjective.
For the ranking of journals based on the available information, flawed and subjective as it may be, five different groups are identifieda sort of scale of perceived quality as expressed by the ASSAf Report: The SALJ referees were unanimous that the SALJ publishes articles of a high quality. 100 In fact, it noted that the SALJ is "South Africa's premier law journal". 101 97 Business-related criteria include the frequency, regularity and punctuality of publication; the print-run, distribution patterns and the redundant stock; the production model and service providers; advertising and sponsorship; the subscription base, marketing and costs; e-subscriptions, accessibility and searchability; the format and the use of multimedia. In addition, the annual income and expenditure; the distribution to international destinations; and indexing in Thomson ISI and/or IBSS, or any other international database are considered (ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 13). 98 These refer to the citation practice and the number of authors listed; ISI-type impact factors; whether reviews are a regular feature and if the articles are not in English, whether an English abstract is mandatory (ASSAf Report on Grouped Peer Review of Scholarly Journals in Law 13). 99 Perry 2006  The second group included the Annual Survey and the ILJ, which were regarded as examples of "the best work done in South Africa". 102 The SAJHR, 103 the SAYIL 104 and Fundamina 105 fell into the third group, being described as of a "high" quality without reservations.
The fourth group, including Stell LR, LDD, TSAR, JJS, SACJ, Obiter, THRHR and CILSA, was described as publishing good material, subject to a qualification. Articles in Stell LR 106 received special mention and the quality was labelled "generally high". LDD 107 articles were noted to be "generally very good". The quality of the articles in TSAR 108 was described as "good overall" and those in JJS, 109 SACJ 110 and Obiter 111 as "generally good". The contributions in the THRHR 112 were also described as "high", although there was concern that some articles seemed to be primarily descriptive and to have very little theoretical content. The CILSA publications were described as of a "high quality", but concerns were raised about the lack of variety and about not keeping pace with changes in the area. 113 The fifth group consisted of journals that were regarded as publishing articles of varying quality, but as being nonetheless worthy of accreditation. This group includes AHRLJ, De Jure, PER, SA Public Law, Acta Juridica, Merc LJ and Speculum Juris. The Report noted that in AHRLJ 114 the quality of the contributions varied between and within issues, with some being excellent whilst others are average, but most were judged as being "good". good. SA Public Law occasionally had "very good contributions", 119 and the Merc LJ was "generally good with some exceptions". 120 Speculum Juris 121 contributions were described as "a mixture of more academic and more practical articles [that] compares well with general national law journals in Europe, America and the UK".
In conclusion, it is submitted that the information generated through the ASSAf Report does not meet the Perry minimum standards for a trustworthy and acceptable ranking system, mainly because it is neither objective nor free from bias, but also as it is impractical and not easily verifiable. A ranking based on the five groups was nonetheless included in the summary of the data under 5 hereunder.
As an aside, this article would not be complete without mentioning that an alternative method of determining the perceived quality of journals exists in the form of perception-based questionnaires or surveys similar to those of the Crespi 122 and Campbell, Goodacre and Little 123 studies. However, this system is also controversial because of its subjectivity and the fact that discretionary viewpoints cannot be standardised. 124 In addition, the respondents may not be equally familiar with all of the journals 125 and research has shown that "geographical origin, research orientation and affiliation with a journal" play a significant role in the assessment made by of the respondent. 126 As far as the outcome of these studies is concerned, there may be some consensus about who should make the list, but the ranking of the journals remains unclear. 127

Indirect quality evaluation by bibliometric data
Not everything that counts can be counted, and not everything that can be counted counts. 129

Introduction
Although there has been a rise in the use of bibliometric indicators in legal scholarship, it is not yet regarded as being on a par with or as effective as the expert peer review process. 130 This type of data is also biased in favour of English publications and older legal sub-disciplines. 131 Van Gestel and Vranken 132 ask the following question: What problems are bibliometric research indicators really meant to solve? … The purpose seems to have shifted [from furthering the scholarly quality of individual publications] towards creating an instrument for oversight, management and policy, which is just as ineffective in guaranteeing a lasting high quality of scholarly publications as substantive assessment by peers. Implementing both systems cumulatively would only add to the burden on the time and efforts of researchers to justify their work, leaving less time for research and education.
Bibliometric data serve the purpose of gauging a journal from another perspective to get an indication of the productivity and depth of impact amongst discipline peers. It is regarded as being more objective, as it circumvents the "old boy's network", is cheaper and more transparent. 133 The systematic use of bibliometric data is, by its very nature, rooted in history. It assesses the past as a possible indication of future performance, but excludes new discoveries, new researchers and new universities. 134 Obtaining reliable data is problematic 135 because law journals themselves do not always present a full picture of academic endeavour. earlier and discussed later, books and chapters in books are also important sources of information in the legal field. 136 With the available information, the following eight bibliometric systems were considered for this article: inclusion in international accredited lists; the printrun; author prominence; the rejection rate; library usage; the citation index; court citations; and NRF rating choices.

International accredited lists
Although this article is limited to the ranking of law journals on the DHET list of accredited journals, some of these journals also appear on international accreditation lists recognised by the DHET. Could it be argued that journals appearing on international lists should be ranked higher? It may well be that inclusion on numerous accreditation lists could have a positive influence on their impact because it makes the journals more accessible. But are these journals necessarily of a higher quality?
On the one hand it could be argued that they are not necessarily so, as the criteria used for inclusion in any of the lists are similar to those used by the DHET. No information is available about the reasons why all the journals are not on international lists. For instance, did they apply and were they rejected, or did they not apply at all?
Certain South African public institutions award greater financial incentives to academics who publish in journals accredited in international lists, indicating perceptions of their better quality or greater impact. 137 Most importantly, to be accepted for and remain on these international lists the journal must undergo an additional systematic and continuous evaluation by experts, using set criteria of scholarly expertise, 138 including peer- reviewed 139 high quality analytical research under an international editorial board of academics. Most of these lists have their own areas of expertise that journals must adapt to for inclusion. NSD and ISI focus on the diversity of authorship. ISI specifically considers the citation index of the journal within the context of the discipline. SciELO is focused on the DHET criteria as confirmed by the ASSAf evaluation panel reports; and Scopus on indexing and citations.
For the purposes of this article and because these journals have been subjected to additional external scrutiny and evaluation and have been found to meet their specific criteria, the journals that appear on international lists, and the number of times they appear are ranked higher for the purposes of this article.
Only one South African law journal appeared in four additional lists in 2014, 140 namely the SAJHR, which is ranked first. AHRLJ, CILSA and PER were each listed in two of these lists, 141 and are jointly ranked second. The titles of ten journals appeared in only one international list: SALHR, SALJ, JJS, THRHR and AHRJ in IBSS; and SAJHR, SAYIL and the SACJ in the NSD list. The SciELO list also included De Jure, Fundamina and LDD.
In conclusion, it is reiterated that this system is not a true reflection of the quality of a journal's research output and as such does not meet the Perry minimum requirement for quality-sensitive criteria. It is nevertheless included in the summary under 5 hereunder.

Ranking based on the print run
The print run of a journal used to be an indication of its popularity, coverage and visibility, including in South Africa, as market forces generally differentiate between journals based on their relevance and impact. 142 The ASSAf Report noted the print-run of all hard-copy law journals, but the information could not be verified independently. Print run as a measuring tool has limited usefulness, as the number of copies of the journals printed 139 IBSS requires submissions to be "ideally peer-reviewed" (IBSS 2013 http://media2.proquest.com/documents/IBSS+Editorial+Policies+and+Principles.pd f), although NSD's requirements are more stringent: "a system of quality assurance, generally a double peer-review system" (NSD date unknown https://dbh.nsd.uib.no/publiseringskanaler/OmKriterier.action?Requestlocale=en). does not necessarily indicate the quality of the research per volume. Serial journal subscription should also be taken into account. Moreover, in later years some journals have decreased their print runs as they became available electronically. Others, like PER, have never been available in hard copy, while TSAR was unwilling to release its print run information to ASSAf, 143 this making the inclusion of these two journals in the ranking impossible.
The stated number of hard copies printed is as follows: SALJ (1150) Although seemingly straightforward, print run is not an effective measuring tool and it is evident that it does not meet the Perry minimum requirements for an acceptable ranking system, in that the criteria are not qualitysensitive. In addition, in the light of the move towards open online access, the print run will become less relevant. It is still included in the ranking data hereunder at 5, however.

Ranking based on author prominence
We do not believe that we need to provide a detailed justification. Right or wrong, good or bad, justified or unjustified, prestige speaks volumes in the legaland legal academicworld … Accordingly, we think our decision to 143 ASSAf  (21)  21 attempt a prestige-based ranking of specialised reviews will strike most readers as intuitive. 159 In 1997 Jarvis and Coleman 160 ranked law reviews in the US by author prominence and in 1999 George and Guthrie 161 did the same for speciality law journals. 162 The rationale for this ranking methodology is that it "reflects the common-sense intuition that the prestige of a journal depends largely upon the prestige of the authors whose articles it publishes". 163 Law journals were ranked over a five-year period, using a 1 000-point contributor scale according to the prestige of the authors of lead articles at the time of publication. 164 The scale ranged from 1 000 points for an article by the US President to 750 for a US Circuit Court judge to 625 for a law professor at a first-tier law school. 165 Although the creators acknowledged that the scale itself was subjective, they argued that the exact points were not as important as the consistency in comparing journals. 166 The main flaw in this method is its subjectivity, 167 which is highlighted by Crespi, 168 Perry 169 and Korobkin. 170 It is the status or prominence of the author which determines the rankings rather than the scholarliness or quality of the article published. 171 Author prominence does "not necessarily correlate with creativity, innovation, profundity, style, usefulness, or impact on legal thought or practice". 172 It incentivises editors to select articles based on the prestige the author might lend the journal, rather than the quality of the submission. Specialist law reviews in the US are generally peer-reviewed and faculty edited rather than the generalist law reviews that are edited by graduate students ( The average contributor score for each volume per year was used for the calculations. Although this was considered, no difference was made between law faculties in South Africa as no official faculty ranking system exists. The seniority of judges, magistrates and attorneys was not considered as this information is not readily available. A and B-NRF ratings were considered to be of greater prominence, although C-rated academics were placed at their appointment level.   (524) with SACJ (521) next.
In conclusion, even though this ranking system does not meet the Perry minimum criteria for its lack of quality-sensitivity, it is nonetheless included under 5 hereunder. Furthermore, in the South African historical context focussing on author prominence as the basis of a ranking system would not be an appropriate measure. It might dis-incentivise editors from publishing articles by younger (black) academics as they may not yet register high on the prominence scale, resulting in the possibility of constraining the academic transformation process.

Ranking based on the rejection rate
Another ranking system for journals might be based on the rejection or acceptance rates of submitted articles. The argument is that the more selective the journal, the higher its rejection rate, which could indicate a higher quality of scholarly publication. Perry 176 argues that this method is unreliable for the following reasons: cooperation from the editors is required in collecting and producing the data, and the information may be unreliable and unverifiable; and differences in rejection policies may distort the 176 Perry 2006  rejection rates and result in the artificial alteration of the rates. He asks, for instance, if the response "revise and re-submit" is a rejection or not? 177 Is the non-acceptance of a quality article because of systematic preferences a rejection? The rejection rate focuses only on the selection made by the journal among the articles submitted to it, and not the selection made by the authors of the journals to which they will submit their articles; and the solicitation of articles may decrease the rejection rate, which would therefore not represent a true reflection of the quality of the material. 178 These arguments are also relevant vis-à-vis South African law journals. In addition, one should not lose sight of the fact that a journal with a greater output of volumes per annum may receive more submissions 179 and that the subjective nature of peer review may influence the rejection rate. 180 I refer yet again to the ASSAf Report on Grouped Peer Review of Scholarly Journals in Law as the only available information on this topic. The Report reflects the number of rejections by most of the law journals, although the data could not be independently verified. The ranking according to the rejection rate result in Merc LJ being ranked first, AHRLJ second and JJS third. SALJ and LDD would be ranked fourth and fifth respectively.
In conclusion, apart from the data's being unverifiable and subjective, it does not meet the Perry minimum requirement of quality-sensitivity. The data are nonetheless included under 5 hereunder.

Ranking based on library and electronic database usage
This type of ranking is based on the library usage of journalseither through actual physical viewing of the journal in the library or online by way of electronic downloads. 200 This method cannot be regarded as very effective in South Africa, as there are no national or other verifiable statistics available for the physical access to journals. Research in the US has shown that the data would be university-, region-or database-specific, depending on the interest and focus area of academics at a specific university. 201 The results would be demand-driven and dependent on what the "hot" topics 202 are, and could ultimately be manipulated by an in-house academic who requires his/her students to use specific materials. 203 Online databases are not very reliable sources of statistics, as companies are competitive and secretive. 204 When accessing information from outside the system, there is always the possibility of a manipulated result because, without individual logon identification, multiple views or author views may not be highlighted. 205 The frequency of downloading is also not a true reflection of the frequency of actual use 206 and may not be indicative of the quality or impact of the article. 207 As it was not possible to obtain any data of electronic usage per journal for the period, no rating could be made on this basis.

Ranking based on journal citations
One of the standard ways of determining the quality of a journal for ranking purposes in various disciplines is to look at the number of citations it receives: 208 The overall impact of a given journal is equivalent to the aggregate impact of all articles that were published in it during the relevant time period. Consequently, the citation frequency of all articles published in a certain journal in a given set of subsequent texts constitutes a rough measure of that journal's impact on the professional discourse within a specific circle, although … it needs to be adjusted to serve as an approximate measure of the academic value of this journal. 209 The argument is that the more often the journal is cited, the higher the quality and the more influential the journal. 210 The citation frequency of law journals can be seen as an indication of important scholarship, although it may not always be the case. 211 Although citation is regarded as a more objective measure of quality, 212 it is not without its problems, such as technical difficulties in the "citing-cited matching process" 213 underlying the bias 214 and limitations that underpin the system. 215 The quality of each journal may vary and an article published in a frequently cited journal may not necessarily be of a high quality itself. 216 In addition, this ranking system is "overly sensitive to the presence of one or two 206 Perry 2006  remarkable articles" getting numerous citations. 217 In addition, editors could be tempted to publish articles that they know would be controversial and thus more likely to be cited 218 or to publish articles by more famous authors likely to be cited and not necessarily because of the quality of the article itself. 219 A journal may also be cited for the "wrong" reason. 220 Korobkin inter alia names a few instances where the citing of a journal "provides little or no evidence of scholarly value." For instance, the article may contain an unoriginal argument that is easy to access; the source may be a friend or ally of the author; the author may wish to impress the source; 221 the source may be incorrect or present an opposite view from that of the author. 222 Other negatives of the citation system have been noted by Tome and Lipu: 223 Scholars working in the same field tend to cite one another, resulting in "strong geographical and regional tendencies" 224 and favouring English as the medium. 225 Other problems are that the citations may include negative citations, selfcitation, collegial citation or selective citation without necessarily reflecting the quality of the journal. 226 Citations are inherently predisposed towards older journals, 227 although the number of citations may decline over the years. 228 The extent of the size of 217 Perry 2006  the publication may impact on the number of citations because of an increased word-count, even though the material published may be more mediocre. 229 Articles available online in their entirety are more likely to be cited than those available in hard copy only. 230 Impact factors also differ for each subject field, document type and journal size. 231 The citation conventions in the legal field vary: articles are comprehensively referenced and every citation may not be of equal scholarly value. 232 In addition, there may be inconsistencies in the citations of general law journals vis-à-vis specialist law journals. 233 Certain areas of the law could be regarded as more interesting or popular at any given time and this could impact positively on the number of citations. 234 Low impact indices are not unusual in law. The London School of Economics Maximising the impact of academic research noted that the average h-index score in law is 1.25 as opposed to 4.83 in Economics and 5.04 in Geography. 235 Research in the US has highlighted that various citation indices could provide inconsistent results. 236 Those concerned with citations 237 in different disciplines have attempted to nuance or adapt their citation systems accordingly to mitigate the negatives. 238 Eisenberg and Wells argued that the use of ISI is inappropriate 229 Perry 2006 Va J Law Technol 33. To alleviate the problem of the varied number of articles per journal, it has been suggested that the solution would be to divide the number of submissions by the number of publication outlets (Ramsay and Stapledon 1997 MULR 683). This solution was adopted in this article. 230 Doyle 2004  in the legal field as it is under-represented and slanted towards the sciences, and may be detrimental to scholarship. 239 The most popular social sciences index in South Africa is the Web of Science SSCI Social Science Citation Index, but a certain preference for country and discipline is present. 240 As only the SAJHR was included in this list during the relevant period, this index is disregarded in this article.
The Washington and Lee citation databases count only citations by US Law journal articles in the Westlaw Database. 241 As a result, the citation counts of smaller legal jurisdictions such as South Africa, which tend to be jurisdiction-specific, are not well-represented. 242 In this article the existing Washington and Lee Law Journal Ranking Project 243 data will be used as an example. It has been argued that its socalled Combined Factor balances the raw citations with the impact factor to give a more realistic measure of influence. 244 This system considers the wider readership of legal materials such as practitioners, judges, academics and policy makers. 245 It should be noted that compared to ISI it has been described as "surprisingly inconsistent, with no statistically significant correlation". 246 structure of the article; specifically the protocol, the methodology, the descriptive section or the theoretical framework (Marashi et al 2015 SAJS 2). 239 Eisenberg and Wells 2014 Economic Inquiry 1313. 240 Datt, Tran and Tran-Nam 2009 ATF 344. It is for this reason that the ISI citation analysis is not used for the Humanities cluster that includes law as a discipline. 241 Datt, Tran and Tran-Nam 2009 ATF 350; Van Gestel 2015 Legal Studies 168. 242 Datt, Tran and Tran-Nam 2009 ATF 350. 243 For a discussion of the system, see Doyle 2004 Leg Ref Serv Q 25-27;Datt, Tran and Tran-Nam 2009 ATF 345;Steinbuch 2009 Loy LA L Rev 107-112. Van Gestel describes their ranking as the "most influential and comprehensive" ( Van Gestel 2015 Legal Studies 168). Steinbuch argues that this system is "highly right tailed", meaning that very few journals score highly. The rankings described in this article are based on citations found in the fulltext Westlaw journals database of journals published in the preceding eight years. 247 For our purpose, it is important to note that fourteen of the relevant journals appear on the list. By 2014 the South African journals were ranked as follows:  247 Doyle 2004 Leg Ref Serv Q 25. It also includes the number of citations in full-text US state and federal case databases (Doyle 2004 Leg Ref Serv Q 25-26). Although Google Scholar could be another possibility, this is excluded from this article as the 2014 data are no longer available. 248 The Washington and Lee website (Washington and Lee University 2009-2014 http://lawlib.wlu.edu/LJ/) defines the combined-score as follows: "The combinedscore is a composite of each journal's impact-factor and total cites count. The combined-score is, by default, weighted with approximately a third of the weight given to impact-factor and two-thirds given to total cites. The resulting score is then normalized." 249 The Washington and Lee website (Washington and Lee University 2009-2014 http://lawlib.wlu.edu/LJ/) defines impact factor as: "Impact-factor shows the average number of annual citations to articles in each journal (rounded to two decimal places). The method by which impact-factor is calculated is to conduct each of the Westlaw searches for citing articles in 8 separate yearly slices." In conclusion, the citation system does not meet the Perry minimum requirements: comprehensive South African data is unavailable and it is not quality-sensitive. In addition, it may not be productive to rank South African law journals and pit the editors against each other in such a small pool of legal academia: The notion of competition among editors has its disturbing side, in that behaviour modification to manipulate a journal's citation ranking may not be a positive feature. Without improving the intellectual quality of a general law review it is possible to increase a journal's citation ranking by excluding less popular topics … 251

Court citations
The first legal citation index dates back to the Shepard's Citations of 1873, wherein the index listed court cases and publications referred to. 252 Because of the nature of legal research, it was argued that it may be possible to rank the journals according to the number of citations of the journal by the courts. 253 As such, it may be a measure of the value of legal academic scholarship, 254 especially if the aim of legal research is seen to be legal change and the development of the law.
The question, however, is which courts should be used for the purposes of this research. In the US "there are at least two paradigmatic versions of the method": 255 one, the citations in the appellate courts, which would reflect the influence the journal exercises on the understanding and development of the law and especially the resolving of practical legal problemswhich shows creativity and innovation; 256 and two, the frequency of citations in the trial courts because of the nature and variety of work done in these courts - Although the data obtained are flawed and incomplete, if this system is used the ILJ would be ranked first, with SALJ second, THRHR third and TSAR and SAJHR fourth and fifth respectively. The high ranking of the ILJ is not unexpected as it includes labour judgments, and possibly because the databases searched include a high number of CCMA awards and where labour law sources are cited a slanting towards more citations from the ILJ is inevitable.
Because of the unreliability of the information, this cannot be a proper basis for an official ranking system. It does not meet the Perry minimum requirements of reliability and quality-sensitivity. The data are nonetheless included under 5 hereunder.

Ranking based on the outputs chosen by successful NRF-rating applicants as their best 261
In terms of the NRF rating application requirements, applicants who apply for rating must submit what they regard as their five best publications of the past eight years for peer-evaluation. 262 These submissions may include locally or internationally published books, chapters in books, or journal articles.
When one looks at the NRF-rating data over a period of six years, 2009-2014, the raw data reveal the frequencies with which the journals relevant 261 Although this application is based on the South African system, the idea originated from a study done by Campbell (12); Obiter (9); SACJ (8); AHRLJ (8); Acta Juridica (7); Fundamina (6); LDD (6); ILJ (5); JJS (5); SAYIL (5) and Speculum Juris (5). It is not surprising to note that specialised journals such as Fundamina, SACJ and the ILJ were selected less frequently than the generalist law journals, keeping in mind that there are fewer specialist academics in South Africa.
When one breaks down the data further into rating categories, the information becomes more nuanced. The A-rated legal academics, who are in the minority, displayed a preference for submitting internationally published books and international journal articles for the NRF review process. 263 B-rated academics, in contrast, submitted a variety of materials. Apart from a modicum of conference proceedings and reports, there was an almost equal distribution on average of South African journal articles, international articles and chapters in books. 264 There was also a noteworthy number of book submissions in this class (15%). 265 The C-rated academics concentrated on submissions published in South African journalsat all levels. 266 Although chapters in books and international journals also played a role, these were substantially fewer than the South African journals. 267 The submission of chapters and books increases as the rating becomes higher. 268 P-and Y1-rated scientists generally included more international journals in their submission lists. 269 From the above it may be inferred that there seems to be a trend that the higher an academic's NRF-rating, the less significant the role South African law journals play in the assessment of career-defining publications. This 263 This is similar to the research done by Campbell, Goodacre and Little 2006 J L & Soc'y 346 in the UK, which found that the percentage of journal submissions in Law is generally less than in other disciplines. 264 South African journals (29%), international articles (25%) and chapters in books (26%). 265 There was no significant difference between the sub-categories of the B-rated scientists, although B2 academics seem to publish less in South African journals and more in international journals. 266 C1 (53%), C2 (56%) and C3 (65%).

267
With C-rated academics, international journals are 10% (C1), 36% (B2) and 8% (B3), whilst chapters in books are 18%, 15% and 8% respectively. Books represent 13%, 12% and 4% of the submissions. 268 The submission of chapters and books increased from 7% and 6% respectively for C3 rated academics to 20% and 14% respectively for C1 rated academics. 269 For other Y-rated academics, just under half the submissions (46%) consisted of South African law journals, with about 10% books and 10% chapters in books. The remainder were mostly international journals. does not detract, however, from the submission above, namely that the esteem in which a journal is held is reflected in the number of highly-rated academics quoting it.
An additional caveat should be added to this analysis, namely that fewer than twenty per cent of law academics in South Africa were rated in 2014. 270 A reason for this could be inter alia a pre-conception that applicants will be given a C-rating, irrespective of their real standing in the field nationally, as many fields in law do not lend themselves to publication in international journals. Whatever the merit of the argument, rating is not compulsory at most universities, and many legal academics are not interested in applying for a rating. And, going forward, recent developments at the NRF to reduce incentive funding may suggest that very few academics will apply for rating in future. If this system were to be used as a guide, the ranking of the top five journals would be as follows: SALJ, TSAR, Merc LJ, THRHR and Stell LR. However, apart from the reasons given above, the data do not meet the Perry minimum requirements for an overall quality-sensitive measurement. The data are nevertheless included under 5 hereunder.

Consolidated data
The evidence above that none of the systems discussed is flawless, trustworthy and quality-sensitive, but it also highlights the problems related to ranking systems in general as well as specific ranking systems in particular.
The question can rightly be asked whether the various systems should not be weighted, as some of the measures could be regarded as inherently more important than others. The choice of systems and weightings could be in the interests of credibility and trustworthiness. 271 However, any exclusions or weightings would remain purely subjective. 272 For the purposes of illustration, all the available data are included.
To consider the consistency in the outcome of the various systems of ranking South African law journals, the spread of the outcomes per journal and the average of all the imperfect results are shown.  So what does all of this mean? The spread per journal of the outcome of the various ranking systems used shows that there is very little consistency in the outcome of the data. 273 The ranking of a journal will thus depend largely on the ranking system(s) chosen, making the use of ranking as a quality and impact measure random, unfair, irrational and unreliable.

Conclusion
The aim of this article is to consider whether a journal ranking system would be useful to assist the DHET in determining quality and impact in the legal field. None of the journal ranking systems discussed meets the Perry minimum requirements for a reliable and trustworthy system. None can claim to be objective, with quality-sensitive criteria free from bias. In addition the data are difficult to retrieve, unverifiable and possibly open to manipulation. For South Africa as a small jurisdiction the negatives clearly outweigh any possible benefits of the use of a data ranking system. 274 The short response to the proposal of a ranking system for South African law journals would be negative, making consultation with the DHET urgent, 273 One may also be tempted to look at the table and conclude that TSAR is consistently ranked high, but this does not take into consideration its refusal to divulge to ASSAf the print run information that may have impacted the outcome. 274 Svantesson and White 2009 Bond LR 191. especially since the decision is directly linked to government funding. Not acting could result in an inappropriate and prejudicial system's being imposed on the legal fraternity, especially in the light of the tendency to use journal citations as a fall-back position.
More research and debate are required. These may indicate that expert panels should be appointed to determine quality and impact, which may or may not result in the giving of an expanded brief to ASSAf. New developments on the electronic front, particularly around databases, and a usable, verifiable and trustworthy citation system may in future be developed for law to create indicators of quality and impact.
The quest for a solution should be part of a serious national debate on substantive quality and impact and the development of a matrix to give effect to the complexity of the legal discipline, as suggested by Van Gestel. 275 The South African Law Deans' Association may have a role to play in this regard.
Whatever the final decision, there are deep concerns currently about a law journal ranking system for South Africa and I can only agree with the sentiment eloquently expressed by Svantesson and White: [N]one of the ranking methodologies ... has the sort of scientific rigour that can be expected from an exercise with such profound implications. … [T]he results… are bound to be influenced by intentional and/or unintentional biases. Consequently, there may be a lack of 'procedural fairness' as to who will be the winners and who will be the losers under any particular ranking scheme. Finally, we are concerned about the consequences that inevitably will flow from research ranking. Those consequences may be particularly detrimental for a small jurisdiction … 276 [In addition …] we have little faith that existing research ranking schemes are capable of producing accurate and scientifically valid ranking of research. At the same time, we are convinced that attempts to rank research carry with them serious detrimental effects on research diversity, and potentially, research quality. This is particularly so where research funding is based on the outcome of the ranking exercise. 277 In addition, any ranking system could probably be open to challenge should it have a detrimental effect for any given journal. Administrative justice requires, at least, rationality, and it would not appear that the implementation of any of the systems discussed here and found to be wanting.