1 Introduction

Ranking law journals is a recent phenomenon that is gaining influence not only in individual jurisdictions, such as the US,1 the UK,2 Australia,3 and Israel,4 but also international legal academia.5 As an increasingly popular means used to assess legal publications, journal rankings in different regions are being made in a variety of shapes and sizes.6 The existence of contradictory journal rankings increases the need for transnational comparison as well as scholarly debate with respect to the possible purposes and functions of journal rankings. One major question is to what extent is it likely that the best quality legal scholarship is represented by journal-ranking schemes and elite law journals which are embedded in these schemes? This article is an attempt to answer this question by analyzing journal rankings in China where academic law journal publishing is alleged to be of poor quality. Law journals are sometimes claimed to be useless to practitioners,7 merely repeating existing knowledge8 and producing too many pieces that will never be cited.9 However, the elite law journals are thought to be the exception to this rule. They are deemed to be the place where the very best articles are published. Important faculty decisions concerning tenure, promotion, and well-being of legal scholars are therefore decided on the basis of publications in these journals to a great extent.10 All this makes these elite journals the pinnacle of Chinese legal scholarship. Elite law journals refer to those which are labelled as “core”, “top” and “A-list” by at least four major journal-ranking schemes. Every few years they rank academic journals according to quantitative metrics, such as citation scores. Today the most common method to tell the quality of articles has become ‘[…] judging which journals they are published in.’11

The validity of this claim will be challenged in this paper for two reasons. The first reason relates to the methodology of journal-ranking schemes that are based on bibliometrics. The use of bibliometric indicators has serious flaws for the field of law, not only in China but also in other parts of the world. What will be shown hereafter is that in a variety of situations citations and similar metrics do not express the value and quality of legal research. Citation scores might say something about publication impact but they are only a proxy for quality. Apart from the limits of bibliometrics, rankings are incomplete since they only represent a small part of the law journals and have a preference for general interest journals over more specialized law journals. The second reason relates to the qualitative evaluation methods used by elite law journals. As shown hereafter, I selected fifteen elite law journals as a sample to explore how they define and select the best quality according to their “author guidelines” and procedures for article selection. First, with respect to the author guidelines, journals typically define certain quality criteria for submissions. However, there seems to be a lack of consensus on what “quality indicators” actually mean. Secondly, a majority of the editorial boards claim that they have adopted peer review, but peer review may be influenced by the phenomenon of guanxi, meaning that authors and editors affiliated with the same institution probably take advantage of their personal relationships to influence publishing decisions. Another reason for concern is the rate of in-house publications. This refers to an editorial preference for articles written by authors from the law schools. The in-house publishing rate of Chinese elite law journals also gives rise to suspicion with respect to the independency of editorial decision-making.

After critically examining the available quantitative and qualitative evaluation methods in Chinese legal academia in Section 2, it will be questioned whether serious reform is necessary in order to either improve current journal-ranking schemes and peer review mechanisms to strike a better balance between these two evaluation methods, or find alternative evaluation methods in Section 3. Section 4 concludes with a discussion of the results.

2 Methodological Flaws of Bibliometrics-Based Journal-Ranking Schemes

Academic journal-ranking schemes, based on bibliometric indicators, are used to select and rank journals from different disciplines, including law. There are four major journal-ranking schemes in China, namely: A) The Guide of the Core Journal of China (GCJC) published by Beijing University Library, which examines and ranks journals based on nine types of bibliometric indicators combined with expert review, covering 28 “core” law journals in its latest edition from 2011.12 B) The Chinese Social Sciences Citation Index (CSSCI) run by the Chinese Social Sciences Research Evaluation Center in Nanjing University, which examines journals by multiple bibliometric indicators and qualitative review13 and covers 21 “resource” law journals in the 2014–2015 version.14 C) The Chinese Humanities and Social Sciences Citation Database (CHSSCD) run by the Chinese Academy of Social Sciences, which examines journals according to the “AMI bibliometric indicator system”.15 It covers 22 elite law journals including one top, two authoritative, and 19 core journals in the 2014 version. Top journals in the CHSSCD are considered superior to the authoritative journals and core journals according to the AMI system, but there are no substantive differences between them.16 D) The Research Center for China Science Evaluation System (RCCSE) run by Wuhan University, which examines journals by six types of bibliometric indicators combined with expert review, covers 16 A-list law journals including two “A+” and 14 “A” journals in 2015–2016 (A+ journals in the RCCSE are superior to A journals according to bibliometric scores).17 The bibliometric indicators and ranking methods of these major schemes are shown in Table 1 below.

Table 1

The bibliometric indicators and ranking methods of major schemes.

Ranking Method/Journal Ranking Quantitative Method(s) Qualitative Method

GCJC [2011] search sum,
total cites,
cites by other journals,
rate of reprint,
impact factor,
ratio of prize or indexed in other system,
ratio of funded paper,
web-download numbers
expert review
CSSCI [2014–2015] citation frequency,
impact factor,
immediacy index,
citation extent,
areal distribution,
cited half-life,
qualitative review
CHSSCD [2014] Multiple-layer indicator system, the first layer includes:
attraction power, management Power,
and impact power.
\
RCCSE [2015–2016] Total cites,
2-year impact factor,
immediacy index,
proportion of funded paper,
web-downloads,
proportion of indexed in other system
expert review

There are many ways to rank journals, but the most prevalent method in China is the use of quantitative bibilometric indicators.18 As shown above, ranking schemes provide a list of journals based on several metrics, such as various aggregated scores for citations (e.g. impact factor, cited half-life citations, and immediacy),19 online search sums, web-download numbers, reprint ratio (republication),20 ratio of funded papers (prized papers).21 In doing so, the schemes seemingly strive to achieve a high level of ranking validity. The question then is, does the mixture of metrics allow an accurate prediction of the (substantive) quality of the papers published in the elite journals? As I will argue hereafter, one may have serious doubts.

2.1 Metrics are Only Proxies for Quality and can be Misleading

At first sight, there seems little wrong with current metrics used to rank journals: Citation scores may indicate research quality. Scholars will frequently cite pieces that are innovative and original.22 Impact factor scores for journals are average scores of articles they publish. Articles published in a high impact journal then generally receive more citations than those published in low impact journals. Immediacy and half-life scores are used to measure how fast an article or journal attracts citations during a fixed period of time. Online search sums and number of downloads indicate popularity and academic value. The reprint ratio (articles getting republished in other journals) and ratio of funded papers (articles funded or prized by institutions or research funding) indicate the external recognition a publication has received.

However, metrics are usually applied to measure the “impact” of scholarly publications on other scholars. Therefore, there can only be an indirect relationship between scoring on an indicator and the quality of the content in publications.23 First, there can be negative citations, where scholars disagree with other authors and cite them to show their disagreement. Elsewhere, a citation is merely used to flatter the cited author or editor, not because the cited article is of high value in terms of ideas.24 Second, not all articles published in high impact journals are necessarily good quality and not all articles published in lower impact journals are poor quality. Hence one should not confuse impact for content. Judging the quality of a paper by looking at where it appears is a biased and ineffectual method of measuring what is supposed to be measured.25 Third, an immediacy index is unfair to those articles published later during a certain year, since the time for being cited is much shorter than for publications published earlier. Likewise, the half-life citation scores are also unfair to research with slower communication patterns, which should not imply that the articles are of lower quality.26 Fourth, high numbers of abstract views or downloads may suggest originality, innovation, thoroughness, insightfulness and so on, but again this is not necessarily the case. There are frequently downloaded papers with relatively small citation numbers. It could mean that the reader is disappointed after having downloaded and read the paper. Or, the title or abstract looks promising, but the content is irrelevant to the reader. In practice, it is nearly impossible to trace who searched and downloaded. Therefore, we do not know what these unreliable numbers actually mean.27 Moreover, there is the risk of manipulating numbers. For instance, take the Social Science Research Network (SSRN), where high download numbers are sometimes confused for high quality and authors attempt to inflate download numbers by self-downloading or using a network of friends.28 In light of this, one must realize that it is merely impossible to measure quality through such metrics. Last, being funded and being republished are also factored when ranking journals. Funded papers imply that a given publication has won a certain type of competition based on its academic value, leading the reader to believe that the content is excellent – but this depends on the rules of the competition. Republished papers are also presumed to be of high quality because they become widely disseminated widely, but this runs the risk of becoming a self-fulfilling prophecy. It is beyond the scope of this paper to discuss all the specific limitations of using these external awards, but the bottom line is that ranking schemes should not replace reading the content of publications with counting numbers. In summary, metric indicators are invented to explore the pattern of knowledge dissemination but they are an imperfect tool to evaluate quality of journals since they have inherent flaws. Adding more metric indicators does not solve the problem. Aggregated scores still cannot measure the quality of a journal’s content.

2.2 Incompleteness of Chinese Law Journal-Ranking Schemes

Another main issue of badly designed journal-ranking schemes is the incompleteness in terms of coverage of all available journals. Journal rankings are also incomplete if there is no proper division between different types of journals. Chinese ranking schemes usually put specialized journals at a disadvantage. Non-elite journals are not included in the major rankings, resulting in confusion about how and why elite law journals are superior to non-elite journals.

To be more specific, law journals of general interest publish articles on a variety of subjects, making them attractive to a potentially large number of readers, whereas specialized journals almost by definition have a smaller readership. Journal-ranking schemes that simply put generalist and specialized journals together according to citation numbers are comparing apples and oranges. Hence the competition stimulated by rankings disadvantage specialized journals in less popular fields of law. As non-elite journals are not included in the ranking schemes, they become condemned in the eyes of the scholarly legal community, especially as soon as scholars find out that publishing in high-ranked journals delivers considerable credits for promotion, tenure and prestige.29 In addition, the legitimacy of shortened elite law journal lists begins to be undermined due to the absence of non-elite journals in ranking lists. How can it prove that elite journals are superior to non-elite ones if there is no actual comparison between them? The issues caused by the incompleteness of ranking schemes are easy to see in China, as shown in Table 2 below.

Table 2

The elite journal ranking lists of major schemes in China.

GCJC [2011]
28 journals included
CSSCI [2014–2015]
31 journals included
CHSSCD [2014]
32 journals included
RCCSE [2015–2016]
24 journals included

1. 法学研究
Chinese Journal of Law
1. 比较法研究
Journal of Comparative Law
1. 法学研究
Chinese Journal of Law
1. 中国法学 (A+)
China Legal Science
2. 中国法学
China Legal Science
2. 当代法学
Contemporary Law Review
2. 中国法学
China Legal Science
2. 法商研究 (A+)
Studies in Law and Business
3. 法商研究
Studies in Law and Business
3. 法律科学
Science of Law
3. 法学
Law Science
3. 法学 (A+)
Law Science
4. 法学
Law Science
4. 法商研究
Studies in Law and Business
4. 中外法学
Peking University Law Journal
4. 法学家 (A+)
Jurists Review
5. 政法论坛
Tribune of Political Science and Law
5. 法学
Law Science
5. 法制与社会发展
Law and Social Development
5. 现代法学 (A)
Modern Law Science
6. 现代法学
Modern Law Science
6. 法学家
Jurists Review
6. 政法论坛
Tribune of Political Science and Law
6. 政法论坛 (A)
Tribune of Political Science and Law
7. 法律科学
Science of Law
7. 法学论坛
Legal Forum
7. 法学家
Jurists Review
7. 法律科学 (A)
Science of Law
8. 中外法学
Peking University Law Journal
8. 法学评论
Law Review
8. 现代法学
Modern Law Science
8. 中外法学 (A)
Peking University Law Journal
9. 法学评论
Law Review
9. 法学研究
Chinese Journal of Law
9. 法商研究
Studies in Law and Business
9. 法学研究 (A)
Chinese Journal of Law
10. 法制与社会发展
Law and Social Development
10. 法学杂志
Law Science Magazine
10. 法律科学
Science of Law
10. 政治与法律 (A)
Journal of Politics and Law
11. 比较法研究
Journal of Comparative Law
11. 法制与社会发展
Law and Social Development
11. 环球法律评论
Global Law Review
11. 法制与社会发展 (A)
Law and Social Development
12. 法学家
Jurists Review
12. 华东政法大学学报
Journal of East China University of Political
12. 当代法学
Contemporary Law Review
12. 法学评论 (A)
Law Review
13. 环球法律评论
Global Law Review
13. 环球法律评论
Global Law Review
13. 知识产权
Intellectual property
13. 当代法学 (A)
Contemporary Law Review
14. 法学杂志
Law Science Magazine
14. 清华法学
Tsinghua Law Review
14. 清华法学
Tsinghua Law Review
14. 法学论坛 (A)
Legal Forum
15. 法学论坛
Legal Forum
15. 现代法学
Modern Law Science
15. 比较法研究
Journal of Comparative Law
15. 河北法学 (A)
Hebei Law Science
16. 当代法学
Contemporary Law Review
16. 政法论丛
ZhengFa LunCong30
16. 法学评论
Law Review
16. 法学杂志 (A)
Law Science Magazine
17. 政治与法律
Journal of Politics and Law
17. 政法论坛
Tribune of Political Science and Law
17. 政治与法律
Journal of Politics and Law
18. 行政法学研究
Administrative Law Review
18. 政治与法律
Journal of Politics and Law
18. 法学论坛
Legal Forum
And:
8 second-tier law journals (A–)
19. 中国刑事法杂志
Criminal Science
19. 知识产权
Intellectual property
19. 华东政法大学学报
Journal of East China University of Political Science and Law
20. 河南省政法管理干部学院学报
Journal of Henan Administrative Institute of Politics and Law
20. 中国法学
China Legal Science
20. 法学杂志
Law Science Magazine
21. 华东政法大学学报
Journal of East China University of Political Science and Law
21. 中外法学
Peking University Law Journal
21. 河北法学
Hebei Law Science
22. 河北法学
Hebei Law Science
And: 22. 行政法学研究
Administrative Law Review
23. 法律适用
Journal of Law Application
10 “extended” resource law journals And:
24. 甘肃政法学院学报
Journal of Gansu Political Science and Law Institute
10 “Extended” core law journals
25. 人民检察
People’s Prosecutorial Semimonthly
26. 知识产权
Intellectual property
27. 国家检察官学院学报
Journal of National Prosecutors College
28. 清华法学
Tsinghua Law Review

The first type of incompleteness is that the above mentioned ranking lists do not categorize journals. They put different types of journals together, such as national and regional journals (e.g. Chinese Journal of Law and Hebei Law Science in the GCJC list),31 academic-oriented journals and professional-oriented journals (e.g. the China Legal Science vs. Journal of National Prosecutors College in the GCJC list), as well as generalist journals and specialized journals (e.g. Contemporary Law Review vs. Intellectual property in the CSSCI list). Consequently, national journals, in particular scholarly-oriented journals and generalist journals with broader audience, are on the “top 10” list. Regional journals, less scholarly-oriented journals, and specialized journals with a narrower audience are generally ranked much lower. For example, in Hebei Law Science (regional law journal), People’s Prosecutorial Semimonthly (professional-oriented law journal) and Intellectual Property (specialized law journal) are shown at the bottom of Table 2.32 Similar cases are numerous; the message delivered in these ranking schemes is that the quality of journals with a broader audience or general topics are necessarily superior to that of journals of a narrower scope. This is not true. For example, one often cannot be sure that publications in Chinese Journal of Law are better than articles published in Intellectual Property. All in all, without proper grouping standards, journals will automatically form a hierarchy not decided by quality, but by the size of the audience of readers.

Lists of elite journals are also too short to show the bigger picture of academic legal publishing in China. Loosely speaking, China has nearly one thousand different law journals,33 but the longest ranking list (CHSSCD) covers only three percent of all available law journals. There is nothing wrong with this percentage as such, but the problem is that we know nothing about the remaining 97 percent of law journals. We do not know what the quality of the articles published in these non-elite law journals is. More importantly, we do not know how and to what extent elite law journals are superior to non-elite law journals. The best counter-example is the Washington and Lee law journal ranking, which covers nearly all law journals in the US and classifies different types of journals in detail.34 It can clearly demonstrate, for example, the extent to which the impact of a top law journal is greater than that of lower rank journals. Lastly, another downside of the Chinese journal-ranking schemes is their neglect towards the publishing practice of the scholarly legal community. A simple fact is that legal scholars do not only publish articles in law journals but also seek placement in other academic journals that focus on social sciences. A number of social science journals are also included in ranking schemes but are ranked independently from ranking of law journals. The separation between these two types of rankings makes the comparison between law journals and social science journals impossible.35

After examining the current Chinese journal rankings, it is doubtful whether they show a representative image of Chinese academic legal publishing, select the best journals, and enhance competition for the sake of better research quality. General interest journals have benefitted from this to the detriment of specialized journals in smaller fields with fewer (potential) readers.

3 Are Elite Law Journals Able to Define and Select Best Quality?

If journal rankings are unable to represent the highest quality legal scholarship available in China, what does this mean? Do we need to abolish rankings and turn to alternative research evaluation methods that rely less on metrics? To answer these questions, let us first look at the qualitative evaluation methods, including how journals try to incorporate quality criteria in their author guidelines and how these journals adhere to the quality criteria via (peer) review process. As a sort of “litmus test”, a small sample of elite law journals were chosen. As shown in Table 3, there are 15 law journals included in all four major ranking schemes, which could be seen as the “elite of the elite”.

Table 3

The Inclusions of Elite Law Journals across the major Chinese journal-ranking schemes.


Elite Law Journals Included in Ranking Schemes
Chinese Journal of Law GCJC, CSSCI, CHSSCD, RCCSE
China Legal Science GCJC, CSSCI, CHSSCD, RCCSE
Studies in Law and Business GCJC, CSSCI, CHSSCD, RCCSE
Law Science GCJC, CSSCI, CHSSCD, RCCSE
Tribune of Political Science and Law GCJC, CSSCI, CHSSCD, RCCSE
Modern Law Science GCJC, CSSCI, CHSSCD, RCCSE
Science of Law GCJC, CSSCI, CHSSCD, RCCSE
Peking University Law Journal GCJC, CSSCI, CHSSCD, RCCSE
Law Review GCJC, CSSCI, CHSSCD, RCCSE
Law and Social Development GCJC, CSSCI, CHSSCD, RCCSE
Jurists Review GCJC, CSSCI, CHSSCD, RCCSE
Law Science Magazine GCJC, CSSCI, CHSSCD, RCCSE
Legal Forum GCJC, CSSCI, CHSSCD, RCCSE
Contemporary Law Review GCJC, CSSCI, CHSSCD, RCCSE
Journal of Politics and Law GCJC, CSSCI, CHSSCD, RCCSE
Journal of Comparative law GCJC, CSSCI, CHSSCD
Journal of East China University of Political Science and Law GCJC, CSSCI, CHSSCD
Hebei Law Science GCJC, CHSSCD, RCCSE
Tsinghua Law Review GCJC, CSSCI, CHSSCD
Global Law Review GCJC, CSSCI
Administrative Law Review GCJC, CHSSCD
Intellectual property GCJC, CHSSCD
Criminal Science GCJC
Journal of Henan Administrative Institute of Politics and Law GCJC
Journal of Law Application GCJC
Journal of Gansu Political Science and Law Institute GCJC
People’s Prosecutorial Semimonthly GCJC
Journal of National Prosecutors College GCJC

3.1 Author Guideline as Criteria of Qualitative Evaluation

At present, elite law journals usually announce author guidelines on official websites to make requests for submissions. Author guidelines are actually a set of requirements that typically consist of 1) the citation format, 2) length of submissions, 3) substantive requirements and 4) ethical guidelines and academic codes of conduct. The information of author guidelines of these 15 elite law journals is presented in Table 4 below.

Table 4

The author guideline of 15 “super” elite law journals.

Super Elite Law Journals Format of Citation Length of Submissions Substantive Requirements Dishonorable Behaviors

Chinese Journal of Law36 Principle of necessity of citations, principles of citing authoritative resources, not citing non-academic resources, and adding access date to citations of online resources 20000 Chinese characters of text, not more than 25000 characters Basic theory building, innovative methodology, empirical research, interdisciplinary research, and critical studies Plagiarism, self-plagiarism, data tempering and falsification, citation falsification, imposter of authorship, etc.
China Legal Science37 General rules of making footnotes, specific rules of citing different forms of resources, including books, thesis, anthology, translated works, law codes, newspapers, dictionaries, etc. Abstract with less than 300 Chinese characters, and three to six key words, no specific requirement for text length Explicit research orientation in its field, high value of topics, innovation,
reasoning method,
depth of argument, acceptable writing norms,
and potential of future citations
Repetitive submissions or duplicate submissions
Studies in Law and Business38 \ \ problem-based research, innovative topics, insightful views,
sufficient arguments,
logically rigorous,
acceptable writing norms,
high theoretical values,
societal relevance,
empirical research,
interdisciplinary research
\
Law Science39 Specific rules of citing different forms of resources, including books, thesis, anthology, translated works, law codes, newspapers, dictionaries, etc. “Relatively short” Innovative notions,
thoughtful,
societal relevance,
promoting development of law,
slight writing style,
cutting-edge research
\
Tribune of Political Science and Law40 \ \ Originality \
Modern Law Science41 Specific rules of bibliography and citations \ \ \
Science of Law42 General and specific rules of citations Abstract with less than 200 Chinese characters Innovational topics,
independent views,
high academic values,
theoretical,
societal relevance,
sufficient argument,
logically rigorous,
formal writing norms
Duplicate submissions,
plagiarism
Peking University Law Journal43 General rules of citations,
specific examples of citing different forms of resources
\ Explicit topics,
innovation,
academic rigorousness,
acceptable academic norms
\
Law Review44 General rules of making footnotes, specific rules of citing different forms of resources, including books, thesis, anthology, translated works, law codes, newspapers, dictionaries, etc. Abstract with less than 200 Chinese characters and three to five keywords Originality Fabrication,
fraud,
plagiarism,
involving state-secrets,
infringement of intellectual property rights
Law and Social Development45 General rules of making footnotes, specific rules of citing different forms of resources, including books, thesis, anthology, translated works, law codes, newspapers, dictionaries, etc. Abstract with less than 300 Chinese characters and three to eight keywords Theoretical
(This journal does not publish introductory articles, book reviews, translation of papers published abroad, conference summary, literature review.)
Plagiarism,
infringement of intellectual property rights,
duplicate submissions
Jurists Review46 General rules of making footnotes, specific rules of citing different forms of resources, including books, thesis, anthology, translated works, law codes, newspapers, dictionaries, etc. Abstract with less than 300 Chinese characters and less than five keywords Originality,
societal relevance,
impacting research,
awareness of research question,
solid literature review,
sufficient argument,
proper citations,
accurate writing style
Repetitive submissions, duplicate submissions,
transferring text types and submitting
Law Science Magazine47 Specific rules of citing different forms of resources, including books, thesis, anthology, translated works, law codes, newspapers, dictionaries, etc. Approximately 8000 Chinese characters for text (good quality text can be relatively long),
abstract with less than 300 Chinese characters and three to five keywords
Innovative topics,
rigorous arguments,
original views,
societal relevant,
\
Legal Forum48 Footnotes with no specific rules Text with 9000 to 12000 Chinese characters,
Abstract with less than 200 Chinese characters and three to five keywords
Innovation or new development of law \
Contemporary Law Review49 General rules of making footnotes, specific rules of citing different forms of resources, including books, thesis, anthology, translated works, law codes, newspapers, dictionaries, etc. Abstract with 200 to 300 Chinese characters and three to eight keywords \ \
Journal of Politics and Law50 Specific rules of citing different forms of resources, including books, thesis, anthology, translated works, law codes, newspapers, dictionaries, etc. Text with not less than 9000 Chinese characters Original topics,
sufficient argument,
problem-based research and explicit solutions, theoretical-oriented or application-oriented research
\

Not all the categories in Table 4 are directly relevant to the concept of quality. Format of citation is to remind potential contributors to pay attention to the layout of citing prior works. Dishonorable behaviors mainly refer to plagiarism or fraud. It is to warn potential contributors to obey the very basic academic norms.51 The length of submissions shows a significant diversity between elite law journals as to the size of their articles, which has little to do with quality. All the above-mentioned requirements seem to be the common threshold of academic publishing but they are distant from the very content of submissions.

One of the categories in Table 4 that touches upon quality criteria is “substantive requirements for content”. Except for three journals, all sampled journals have clarified their standards in relation to quality indicators (terms). As far as these indicators are concerned, “innovation” is mentioned eight times (a similar term, “originality”, is mentioned four times), “argumentative” is mentioned six times, “theory” is mentioned five times, and “societal relevance” is mentioned five times.

Originality or innovation is valued most in Chinese elite law journals. None of these elite law journals offer further interpretation on what originality means. In recent years, Chinese legal publications have faced criticism that they lack scholarly originality.52 In response to this, Liu and Xie suggest that originality or innovation refers to something new, which is supposed to offer added value to the existing body of legal knowledge, however, they do not further expatiate its potential meanings and implications.53 The question then arises: what does originality possibly entail?

Greater insights into this question might be found in the United States of America (USA) and Europe. In the USA, originality is also an important criterion exercised in the selection process of law reviews. Originality (uniqueness), then, refers to a fresh idea that has not been developed in previous literature. As complete uniqueness is not always possible, the author can alternatively summarize and rehash prior debate useful to others, or, present old theories in a new light to foster further developments in legal scholarship.54 In other scholarly contributions, originality is defined as ‘[…] something new that has not been said before (i.e. academic preemption)’,55 or ‘[…] newly discovered knowledge and original ways to disseminate or report existing knowledge’.56 In Europe, van Gestel and Vranken suggest there are multiple ways of being original, such as presenting research results that contribute something new to the scholarly debate on a subject, raising a new question (e.g. innovative angle or method, new source material, different insights into a familiar problem), building a (partly) new legal theory (e.g. critiquing a doctrinal opinion, or revealing inconsistencies in the literature), or studying a new research subject.57 Hence, they argue the term “originality” is not a matter of strict objectivity and should be broadly interpreted.58 As a result, there is no consensus on how to understand originality. Other quality indicators in these Chinese elite law journals are also likely vague and multi-interpretative to different individual evaluators. This will cause a series of potential problems. For example, there is confusing logic between these indicators, so it is uncertain whether a good quality paper must fulfill all these indicators, or merely a few of them. The terms “innovation” and “originality” refer to the same sort of thing, but one cannot exclude that reviewers interpret them differently. Furthermore, the terms “theory” and “societal relevance” appear to indicate different quality features, but one may wonder to what extent they always go together in the view of evaluators. In a case note the author may, for example, pay more attention to the practical relevance, but less to theory. Does this mean case notes should not be published in high ranked journals anymore? The indeterminacy and variety of the quality indicators used by the elite law journals lead to further questions in relation to the submission review process. In other words, peer review might offer little certainty if there is no consensus on the interpretation of the criteria.

3.2 The Peer Review Process as a Mechanism to Realize Quality Criteria

As shown in Table 5 below, three types of peer review procedures can be found, including editor review (six of the 15 core journals are using editorial review), external peer review (seven out of 15 journals are using external referees) and more complicated “triple-reading review”, which usually goes through editor review, peer review, and chief-editor review successively (two of the elite law journals are using it). Generally speaking, the main difference between these review procedures is who actually reads and evaluates the content of submissions. Editor review means that the publishing decision is made by editors without seeking help from external referees who shares the same expertise as the author. External peer review means that the editorial board approaches independent referees whose opinion tends to be highly valued. Triple-reading review is a combination of the two former types of review process, in which editors decide whether a submission is sent to external referees and whether this submission is publishable after peer review. Table 5 shows the popularity of external peer review (nine journals adapt such review). Peer review is considered superior to editorial review since external referees are thought to be more competent and objective than editors in scrutinizing the quality of submissions, at least in theory.59 Still, it seems that editorial boards have the final say in making publishing decisions.

Table 5

Review process of 15 elite law journals.

Elite Law Journals Review Process

Chinese Journal of Law Blind peer review. It applies to foreign authors, prestigious scholars, “in-house” authors, authors have a (personal) relationship with editors, other situations in which editorial board considers necessary
China Legal Science Blind peer review. Peers are asked to grade articles on the aspects of topic value, innovation, method, thoroughness, writing norms, potential citations
Studies in Law and Business Blind peer review
Law Science “Triple-reviewing system” applying to all submissions. It consists of editor review, blind peer review, and chief-editor review
Tribune of Political Science and Law (blind) Peer review
Modern Law Science (blind) Peer review
Science of Law Blind editor review
Peking University Law Journal “Multiple-reviewing system” consists of editor review, blind peer review, committee review, and chief-editor review
Law Review Blind review
Law and Social Development Blind editor review
Jurists Review Blind editor review
Law Science Magazine Blind editor review
Legal Forum Blind editor review
Contemporary Law Review Peer review
Journal of Politics and Law Blind peer review

The review process for most elite law journals is concealed or blind.60 Concealed review process masks the identity of the author from editors or external referees, guaranteeing that their review is made fully on academic merits of content, without prejudice to prestige of authors, biases and relationships between reviewers and authors. Sometimes, concealed review is more strictly conducted as “double-blind” peer review, meaning both the author and referees do not know the identity of the other. In the case of single blind review, the referees know who the author is, not vice versa.61 Hence, the various review processes adapted by Chinese elite law journals, if conducted correctly, seem to be functional and fair to realize their high-quality standards. Nevertheless, this is not always true in practice, as discussed below.

3.2.1 The Ambiguous Review Process

Little is known about how different types of peer review are conducted in detail by editorial boards. Few elite journals reveal how reviewers decide whether one submission is more innovative than another, for example. Certain journals, such as China Legal Science, strive to solve this issue by way of making peer review procedures more “objective”. External referees are asked to score submissions on their topical value, innovation, research method, thoroughness, writing, and so on. Afterwards, submissions with the highest scores are selected.62 Still, the issue behind this evaluation method remains the same. There is no shared understanding of these indicators; the review remains intuitive and subjective. One may argue that good quality legal writing is easy to recognize but hard to describe. The question is, however, whether referees recognize the same things as “quality”.

3.2.2 “Guanxi” as an Editorial Bias in Review Process

Chinese academic journals are usually hosted by higher education institutions and particularly prestigious law schools. Editors have broad discretion over the review process. Often, they are colleagues or friends of authors who work in the same faculty. Such personal relationships can easily lead to favors, meaning mediocre submissions have a chance to bypass the gatekeeping function of a review process in order to be published. This so called “guanxi effect” in academic journal publishing has now extended beyond the simple relationship directly linking editors and authors.63 In recent years, guanxi in academia at large has been under scholarly scrutiny in China.64 Undoubtedly, this is phenomenon also applies to law faculties. The fact that certain journals have introduced blind peer review in reaction to guanxi proves that this phenomenon should be taken seriously. Most elite law journals claim a concealed review process in order to exclude the influence of external factors (reputation or personal relationships). For example, Chinese Journal of Law makes it clear that the review process is especially designed to prevent the guanxi effect.

3.2.3 In-House Publications as Possible Counter-evidence Against Success of Review Process

One consequence of guanxi is the relatively high number of in-house publications in certain elite law journals which has become an issue for Chinese law journals in recent years. The organization Law Innovation collected data pertaining to this problem, claiming high in-house publishing rates do not always mean corruption, but journals will damage fairness and development of legal academia if they publish too many articles from their own faculty members.65

Chinese elite law journals have different editorial policies concerning in-house publishing. The journal nearly immune to in-house publishing is China Legal Science, which hardly published works by its own faculty members between 2010 and 2014 (0 percent from 2010–2013, 2.02 percent in 2014).66 However, this journal appears to be an anomaly. The remarkable counter-example is the Tribune of Political Science and Law which always keeps one of the highest rates of in-house publishing at approximately 20 percent (17.07 percent in 2010, 23.81 percent in 2011, 20.72 percent in 2012, 21.62 percent in 2013, 24 percent in 2014).67 As for other journals, rates vary over time from under 10 percent to over 20 percent. In general, in-house publishing is prevalent between elite law journals. Hence, Law Innovation has drawn two warning lines: the “yellow line” (20 percent) and “red line” (25 percent),68 suggesting the rate should be kept between 15 and 20 percent.69

Why do elite journals publish so many in-house articles if they have a rigorous review process that ought to select submissions purely on the basis of their content? Perhaps their own faculty members write the best articles. Therefore, mediocre submissions in most cases are produced by authors from second-tier law schools who do not meet high publication standards. However, this interpretation sounds less likely than another interpretation: Chinese editorial boards are biased. This is not surprising, since a similar case can be found in the USA. It has been proven that American law reviews are systematically biased in favor of publishing their own faculty authors reserve their weaker articles for their own law reviews.70 Moreover, in a small community, editors are able to tell who the authors are just by topic and writing style, giving way to guanxi. At present, the extent to which guanxi raises the number of in-house publications is unknown, but it is certain that editorial boards are discontent with the pressure from established professors who are friends or former supervisors of editorial board members.71

Overall, qualitative evaluation methods used by Chinese elite law journals appear problematic; they are based on vague quality criteria, dubious review mechanisms, and vulnerable to certain social influence (guanxi). Therefore, they are not the perfect alternative to quantitative evaluation methods.

4 Discussion and Conclusion

The concepts of “journal ranking” and “elite law journal” have been widely acknowledged in many countries. In the Netherlands, for example, although a majority of Dutch legal scholars prefer a substantive assessment of scholarly legal publications to counting citation, it is being debated whether a metrics-based journal ranking should be introduced.72 The case in China teaches us that journal ranking and peer review can be problematic, hence it is meaningless to believe high ranking law journals represent the highest quality of legal scholarship. Moreover, the issues in Chinese legal publishing suggest the quality of legal publications is in the end irrelevant to evaluation methods. Rather, it has to do with articulated substantive criteria and the extent to which they are realized in the assessments of legal publications. It is still, however, necessary to promote pragmatic improvements for the current Chinese evaluative system. This can be a shared experience among foreign legal communities that are facing an inevitable rise of metrics-based journal rankings or seeking for better alternative evaluation methods.

A first improvement can be a change of ranking methodology. It is not meaningful to add up more metric indicators because the current hodgepodge of metric indicators is already too complex and incomprehensible to serve as a solid yardstick for research quality. Apart from that, there is still added value in making a better arrangement to reduce the disadvantage of specialized and sound academic journals not included in the current system. For example, why not rank all academic law journals in China, if the idea is to organize a status competition? In that case less-well known journals at least have the chance to move up the ladder. A ranking scheme that is tailored to legal research, excluding other sciences, would probably be better capable of considering the typical features of academic legal research and its unique style of knowledge dissemination (i.e. citation culture) and diverse subject matter. A second improvement relates to peer review. A debate on how to operationalize abstract indicators such as originality is needed. Otherwise, referees will interpret these indicators differently. A discussion needs to be had beyond these superficial semantic standards, penetrating the indispensable components of any form of academic legal writing – namely, the presence of a clear research question, a balanced use of sources that actually support the arguments and interpretations in the text and a more careful explanation of the limits of the conclusions that follow from the research question.73

As for peer review, guanxi is critiqued as an expense of high quality writing. A double-blind peer review process sounds like an ideal antidote to guanxi but the key question is to what extent this will actually work in practice. If one looks at the current rate of in-house publications of many journals one may have doubts as to the possibility of completely ruling out guanxi as a means to bypass peer review. By borrowing from the innovative exercise of “Publons” where reviewers make public their review history to gain credits, a form of “open access” peer review, might help to suppress guanxi.74 The logic of Publons is simple if placed in the context of China. Rational reviewers will think carefully whether they can continue to help their friends when their reviews become transparent and traceable for the public. Otherwise, their prestige will be undermined. No journal will approach them for being famous by bowing to guanxi. If guanxi cannot be prevented by trusting journal editors, then what is left is transparency of review processes.

No matter which of the above-mentioned improvements would be applied, the Chinese scholarly legal community should take action. The long-term negative consequences have already begun to emerge; legal scholars are “locked” into a system in which they are forced to concentrate on publishing in a narrow range of elite journals, which, in turn, do not have sufficient room to publish a vast amount of high quality legal articles that are produced every year. Chinese law journals need to realize that the current ranking system leads to a distorted competition in which the status of journals becomes more important than the quality of the work that is published in it. Apart from this, Chinese legal scholars can also do something themselves about the disadvantages of the system, for example, by no longer seeing Chinese journals as the single outlet to disseminate academic legal knowledge. In a globalizing legal world, there is room for recognition outside the realm of a narrowly defined core of elite journals that dictate the market of scholarly legal research.