Contemporary legal scholarship is scientifically stagnant. While the use of empirical methods has significantly increased in other disciplines, such as in political sciences, law is falling short of unfolding its explanatory potential. This paper argues that Computer Assisted Qualitative Document Analysis Software (CAQDAS) can be a useful tool for conducting empirical legal research. We present a generic research process consisting of (1) matching methods and research questions, (2) compiling source documents, (3) coding and (4) analysis and discussion on how these stages can be implemented in legal research (Section 2). This generic approach has two limitations. First, it must be translated into effective coding. Second, it is further constrained by the technical functionalities of the CAQDAS program itself. By using this generic research process, we seek to show the different possibilities for conducting software supported content analysis through one of the most prevalent CAQDAS programs: ATLAS.ti. The technical discussion of the programme is illustrated with a proof of concept that we conducted on decisions rendered on a specific World Trade Organization (WTO) law provision (Section 3). The article concludes with an evaluation of ATLAS.ti and its advantages and limitations (Section 4). Although not a panacea to empirical legal scholarship, ATLAS.ti is a low-investment high-return research tool with great potential utility to transform legal research.1
1.1 Problem description
Mainstream legal scholarship seems to be caught in a deep sleep when it comes to experimenting with new methods. More often than not, a “tendency of the court” or “a development in jurisprudence” is postulated without specifying the number of judgments examined or the time period covered by the statement. It is disciplinary standard practice in doctrinal legal research that (empirical) observations are virtually unaccounted for.
While we certainly witness a budding rejuvenation of empirical legal research, the empirical element therein often refers to “fancy” empiricism such as statistical methods, experiments and interviews. These offer undeniable opportunities for generating innovative research, providing insights about law in action or simply what the content of legal norms should look like. However, there are also empirical methods that can be deployed within “traditional interpretative legal scholarship”2 and the latter should no longer be allowed to rest on the untouchable pedestal of a prestigious discipline. Schuck observed in 1989 that:
[T]he neglect of empirical work is a bad, increasingly worrisome thing for our scholarship and teaching, and the reasons for its persistence are so deeply embedded in the incentive structure and professional norms of the law schools that they are exceedingly resistant to change.3
Arguably, this observation still holds almost 30 years later. Today, we need to pay more attention to the unavoidable changes and realities that affect our discipline. For instance, the unprecedented availability of legal documents allows us to identify new possibilities for research. In the 80s, a regular law professor would use a typewriter, painstakingly collecting hard copies of judgments for analysis that had to be obtained individually from court registrars. Until well into the 21st century, legal casebooks relied on hardcopies of judgments that were then digitized.4 It is only now that a majority, but still not all, legal documents are available online, ready for exhaustive and systematic analysis. Here, quantity becomes quality in two ways. First, descriptive observations about jurisprudence should increase in accuracy with the number of observations approaching the actual number of cases. Second, a greater number of possible observations mean that there may literally be thousands of cases requiring analysis, confronting the “traditional” legal scholar with ever more case work and cognitive overload.5,6
This means that researchers should try more to engage with the new tools to support this process, especially in “classical” legal endeavors such as legal interpretation and case law research. Improving our empirical approaches in the traditional interpretation of law is therefore not an opportunity, but a necessity.
When putting forward the topic of software supported systematic content analysis in a legal forum7 that discussed the “opportunities” of empirical approaches in legal research, there was some adamant resistance; and one that reflects the most basic, heart-felt opposition of many lawyers: “do you think that a recognized law professor would not be capable of reading all the Supreme Court judgments on a given topic and come up with a convincing legal analysis?” In reply, consider the following: of course s/he can, but if that law professor then writes an article without specifying which judgments s/he identified, on the basis of which keywords, and does not document the process of analysis,8 the results will (unnecessarily) fail to meet important tenets of scientific inquiries, for instance documentation, replicability and verifiability. Without these features, credibility is less about convincing evidence and research quality, but about the reputation of who makes a statement. Note that the example works well with a recognized law professor and much less so with a student. An overpowering reliance on authority enshrines hierarchy, and ultimately moves the legal discipline closer to bible studies than a scientific inquiry. Incidentally, this modus operandus also prevents researchers in the discipline from ever truly standing “on the shoulders of giants”, that is relying on previous discoveries.9
1.2 Solution description
This article argues that software supported content analysis can help address both the problems of lacking scientific and methodological robustness and cognitive overload. More specifically, it discusses the use of Computer Assisted Qualitative Data Analysis Software (CAQDAS or QDAS), such as ATLAS.ti, QSR NVivo and MAXQDA as a solution to enhance the scientific validity of legal research.
In one of the most extensive meta-studies conducted on content analysis, Hall and Wright10 analyzed empirical legal studies conducted in the United States between 1956 and 2006, documenting an exponential increase in the spread of empirical methodologies. They trace the origins of content analysis back to the period between 1950–1970.11 Studies using systematic content analysis mainly pursued one of two objectives, namely prediction and understanding behavior.12
Although the study of Hall and Wright was published almost ten years ago, the main findings remain valid to this day: even though empirical legal research is growing significantly, a structured methodological consensus, even dialogue, is largely missing. Certainly, the use of systematic content analysis in mainstream legal research publications is erratic. Moreover, it remains absent from the general parts of the curriculum at law faculties.
In light of this, this paper argues that software supported content analysis has the capacity to enhance the systematicity of research while continuing to rely on and benefit from the interpretative capacities of the researcher. To be clear, this approach does not suggest replacing human analysis, but rather enhancing it through software support. CAQDAS software packages have different possibilities of how they can support data collection, management, search and query, coding, annotation, data visualization, import/export interfaces and collaborative possibilities. The purpose of this article is to explore the value of ATLAS.ti in conducting content analysis in law. While the use of ATLAS.ti is widespread in the social sciences, legal studies conducted with it remain rare.13 The program is relatively easy to use, and does not require the kind of investment that other more sophisticated empirical approaches such as statistical computing or behavioral experiments do. ATLAS.ti comes from a grounded theory tradition (building hypotheses inductively from the data14) and was specifically developed to deal with concepts. Therefore, it is potentially very well-equipped to support classical legal interpretative scholarship.
2 Description of a generic research process
Software supported content analysis is an empirical method that combines quantitative and qualitative research possibilities.15 According to leading scholars in the field, systematic content analysis can contribute to the following stages of the empirical research process: (1) ensuring an epistemological fit between research question and methodology, (2) selecting source documents; (3) coding source documents; and (4) analyzing research results.16 Each research stage will be discussed below. After the high-level description of research stages a range of technicalities, that only appear at the level of the implementing solution and the specific software deployed, will be discussed in the subsequent section.
2.1 Epistemological fit
Our preliminary question is what type of research inquiries can be addressed through content analysis; in which cases is there a methodological fit between the research question and the given methodology? Hall and Wright identify an “epistemological fit” between content analysis and research questions when investigating: ‘(1) the bare outcomes of legal disputes, (2) the legal principles one can extrapolate from those outcomes, and (3) the facts and reasons that contribute to those outcomes and principles.’17 They further argue that:
[C]ontent analysis is perfectly suited for examining aspects of judicial method. Questions researchers have pursued include the types of authorities judges cite in their opinions; the argumentative, interpretive, or expressive techniques judges use in different circumstances.18
The limitations of this type of research questions lie in the descriptive nature of the methodology and the resulting weakness of such research trying to answer the respective “why” questions. The usefulness of content analysis in answering the “why” questions in legal analysis may be overstated. While noting the possible discrepancy between the text of judgments on one hand and “real” institutional court processes on the other hand,19 the argumentative and discursive nature of law relies on courts providing reasons.20 Therefore, content analysis in the legal discipline may be used in order to inventorize, for instance, the reasons why courts have come to a certain outcome in different cases.21 Many legal research questions are conveniently addressable with content analysis. However, as presented in Section 2.3, its utility depends greatly on how far research questions are translated into effective coding.
2.2 The identification of primary sources
During the data collection stage, the researcher has to decide what the scope of the study will be, and how this will influence the population of the study (for instance, the source documents). Will the entire population be covered (census) or not, and if not what are other sampling methods?
Ad-hoc discovery of primary sources is a prevalent practice in legal scholarship. While the advent of digital search tools and databases makes systematic searches necessary, it is not a common practice in legal research to establish how the underlying dataset has been collected and created. Particularly, in comparative law and EU law, one often finds studies in which it is unclear which cases, or how many, were identified and through which parameters. For the sake of transparency, accountability and replicability, legal researchers must systematically start to account for the identification and selection of primary sources used in a given legal research.22
This includes questions such as whether all possible observations (e.g. existing cases) were considered, which databases were used in order to identify cases, and whether those databases provide an exhaustive compilation of existing judgments, or only a selection. For instance, many national databases are limited to certain higher jurisdictions and thus are not fully exhaustive, while most international law databases (European Court of Justice (ECJ), World Trade Organization (WTO) and International Court of Justice (ICJ)) are more exhaustive.
The representativeness of a dataset will often be compromised due to the fact that not all cases are present in the database or to the public. For instance, the Council of Europe recommends that the highest courts of each jurisdiction publish all their cases. However, other courts only publish those cases that are of clear legal or societal interest.23 European Union Member States have developed distinctly different selection policies for case publication at the national level.24 In the legal discipline, non-exhaustiveness can be an unavoidable reality, but necessitates reflections about the limitations that the lack of representativeness imposes on the research results.
Even when all the cases are available, the number of cases may need narrowing down, either for the purposes of the research, or because their sheer number is unworkable. The creation of a sampling frame implies a selection method determining how to select a representative sample. Such frames can be identified by random, systematic or quota sampling,25 or more qualitatively by purposive sampling such as by means of search terms. These selection mechanisms must be made explicit in order to inform other researchers as to possible bias or weakness in the data set as well as to ensure that the research can be replicated.26
The quality and reliability of primary documents is considered a fundamental risk to content analysis.27 In legal research pertaining to case-law analysis, this appears to be a less specific concern, because judgments are exhaustively compiled thanks to advances in digitization. Naturally, research may still lack certain reflections on the institutional characteristics, court practices and deliberations leading to the final court decision. Thus, the dangers of attributing intention to written judgments continue to apply (as they would in traditional legal analysis). CAQDAS programs do not directly support the identification stage of relevant primary documents through data gathering but help to collect and organize primary documents.
2.3 The coding process
In the process of coding, the researcher assigns a code to the data: this can be a category, description, or a direct quote from the source. Codes are “classification devices” that ‘[…] create sets of related information units for the purpose of comparison.’28 Typically, codes are key themes, concepts, processes or contexts based on theories from the respective discipline.29
The different types of code need to be matched with the research questions and it is necessary to determine which codes will provide the data to answer the research questions. A few coding techniques of particular relevance in legal analyses are the following: (1) Attribute coding which refers to descriptive coding whereby some information in the dataset that may later be important for the quantitative and/or qualitative analysis is encoded. For instance, in the case of court judgments, such information could be the year in which a given judgment was rendered, or the jurisdiction or country where the judgment was given. (2) Magnitude coding codes for the intensity, frequency, weight or importance, presence or absence of a variable. Moreover, it offers evaluative markers (e.g. positive, negative, neutral, mixed) that are useful in defining a common metric for evaluations. (3) Simultaneous coding can be used when the content of a segment of a primary document justifies more than one code (e.g. overlapping or nested codes). This is particularly useful for conceptual analysis, and therefore, to determine and analyze legal concepts. (4) Structural coding covers a certain topic and links it to a specific segment of the data. This kind of code structures the source, typically in larger text segments. Structural codes in law can be used, for instance, to distinguish party submissions from the court holdings. Lastly, (5) descriptive or topic coding is used to mark a specific data segment, identify content, and in the later stage of analysis, it can be used for hypothesis building and theorization.30
Generically, the process of coding involves the creation of a set of codes based on the given research question that aids in defining codes and establishing the protocol for coding. Usually, two cycles of coding are regarded as necessary: an initial exploratory coding phase and a systematic one. A codebook is used to keep a compilation of the code, content description, brief data examples for reference (this is usually allowed in the programs). Collaborative coding is possible, but researcher collaboration requires special care for establishing a protocol that more than one person can follow.
One of the core challenges when coding is found in deploying content analysis meaningfully since it risks misrepresenting subjective interpretation as an objective fact. While the coding process is often represented as one of the severe limitations of content analysis, the author believes that it forces researchers to be more explicit about decisions taken in the interpretation process. Therefore, it may help to remove biases of the research, such as cherry-picking data. The consideration and selection of coding techniques also add to the systematic approach to taking research design choices.
2.4 Analysis – is a law-specific approach necessary?
One of the primary goals of coding is to identify patterns (or the lack thereof) in the data, characterized by, for instance, similarity, difference, frequency, sequence, correspondence or causation. The purpose is to explore theoretical concepts and provide support for theoretical developments by using a research tool that shows patterns or relationships within the data. Arguably, all lawyers, when analyzing legal documents, go through a process of implied “mental coding”, noting the development of courts’ reasoning over time, or how the interpretation of a concept is different in different judgments, for example. Therefore, software supported content analysis, and coding in particular, does not offer a radically new way for legal data analysis, but it can make research more systematic and methodologically consistent.
Although content analysis has been accused of being a-theoretical by some,31 others recommend qualitative content analysis for theory-guided text analysis.32 In fact, the main CAQDAS programmes, ATLAS.ti and NVivo, were developed with the capacities to cater for grounded theory approaches. Data can also be analyzed by means of theories, where hypotheses are initially formulated based on theory and subsequently tested. Both approaches are useful for legal analysis, and may co-occur in the coding process depending on the research issue.
3 Proof of concept: using the ATLAS.ti software in legal research
In the following section, we illustrate how the selected high-level research guidelines can be translated into effective coding in the ATLAS.ti software. We conducted a proof of concept that deployed ATLAS.ti to research a legal gap brought to our attention.33 The proof of concept project (the SPS Jurisprudence Project) studied the jurisprudential development of Article 5.6 of the Sanitary and Phytosanitary Agreement (SPS) under WTO law. This was a gap in the literature, the significance of the provision was underappreciated in doctrine and theory, and a number of new judgments on Article 5.6 SPS disputes had been rendered. The project was regarded as a suitable test case, due to the limited number of primary sources, which would allow manual coding as well as software coding side-by-side.
3.1 The research design: “epistemological fit”
A priori, doctrinal studies of case-law are a suitable type of legal research that can be conducted by using software supported content analysis because doctrinal questions are answered by interpreting texts, such as doctrinal literature or legal texts. Therefore, there is an epistemological fit between the research question and the methods employed.34 To this effect, the goal of our prototypical study was to examine how the Dispute Settlement Body of the WTO applied Article 5.6 SPS in its case law by using software that supports qualitative content analysis.
The research questions were simple and doctrinal: (1) what is the outcome of cases dealing with Article 5.6 SPS, and has that changed over time? (2) Article 5.6 SPS is applied by using a test with three cumulative conditions. How far are these three conditions determinant to the outcome? (3) How does the court test the individual condition?
All of these questions can be answered on the basis of textual interpretation.
3.2 Selecting cases
ATLAS.ti does not support the collection or download of documents. So-called Primary Documents (i.e. source documents) must be imported into ATLAS.ti, while the database remains external to the programme. The ATLAS.ti search functions, however, can help to narrow down a large number of datasets. For data management and analysis purposes, Primary Documents can be organized in different Primary Document Families. This is useful when one needs to, for example, capture differences in the nature of documents such as court hierarchies or groups of documents belonging to one case.
In the study, all disputes citing the SPS agreement were identified through the WTO dispute database. The database is exhaustive as all decisions rendered are accessible. In addition, the disputes are already categorized by Agreement (in our case SPS), and by which provisions were disputed (we looked for Article 5.6 SPS). Alternatively, the ATLAS.ti “auto-coding” function would have enabled importation of all SPS dispute documents and separate coding for whether Article 5.6 SPS was cited. Then, the documents in the Article 5.6 Primary Document Family were filtered based on whether the court effectively dealt with the provision in the findings or whether the court discussed Article 5.6 without applying it to the case necessarily.
3.3 Coding cases
ATLAS.ti provides a feature to break down Primary Documents into segments, so-called “quotations” that become independent objects such as “free quotations”, which can be linked to codes.35 This is usually regarded as a distinguishing feature of the software,36 as it provides an independent and unique identifier linked to parts of a Primary Document. Next to this, standard coding processes are available, for example coding-as-you-go, semi-automated keyword coding, or selecting codes from pre-established lists. Having an unlimited number of codes is possible as well as having several software features that facilitate code booking (e.g. attaching definitions to codes). Additionally, codes can be managed and organized through the creation of code families.
In a given research process, code creation and coding can be organized in a strictly separate and sequential manner. Alternatively, codes can be altered throughout the coding in an iterative process. These can be respectively termed top-down and bottom-up coding. The two methods have been attached to specific approaches. One argues that in order to scientifically test and falsify hypothesis, codes must be predefined and based on a theory, and therefore cannot be changed in the coding process. Grounded theory, on the other hand, is used in order to infer theory during the coding process and in interaction with the data. ATLAS.ti can be used in both ways. The SPS Jurisprudence Project used both approaches depending on the research questions (see Section 3.4.2).
In order to capture and later analyze the relevance of specific attributes, the entirety of Primary Documents was assigned with pre-defined attribute codes. These indicate specific information about Primary Documents, such as the year in which a judgment was rendered for the historiographic study of the data, countries involved or type of proceeding (i.e. Panel, Appellate Body, Follow-up Panel Report).
Coding case outcomes (Research Question (RQ) 1) was done using the following categories: violation, no violation and no conclusion. The latter category was included as the Dispute Settlement Body sometimes declined to rule on Article 5.6 for various, but mainly evidentiary, reasons. This categorical coding was later refined during analysis through a feature called “relationship analysis” that allows the creation of hyperlinks between different sources in order to account for outcomes that had later been overturned in subsequent proceedings.
To analyze judicial argumentation in the judgments (RQ 2 and RQ 3), a predefined structural coding was devised. Layer 1 marked the entire judgment to identify the case; Layer 2 identified the DSB’s argument about Article 5.6; Layer 3 identified arguments relating to the three conditions of the Article 5.6 test. This was the first cycle of coding during which, in addition to coding the pre-defined codes, new codes were elaborated through in-vivo and/or code-as-you-go-coding where there appeared to be potential relevance for the research (inductive method). These were later processed in the second coding cycle.
Two researchers collaborated in the coding process. Each coder was randomly assigned half of the documents and these were later exchanged during the second cycle. On the basis of the first cycle reading, we identified certain code additions. These were horizontal themes often linked to theories, for instance “the relevance of international standards”, “arguments relating to protectionism” or specific legal questions, for instance what is the relationship between Article 5.6 SPS and other provisions. For RQ 3, we proceeded to identify the types of arguments (such as “market access based test” or “trade gain effect based test” in relation to trade restriction) that were used to test the three conditions.37 These additions were assigned in the second code cycle.
3.4 Analyzing cases
ATLAS.ti offers several analytical features, which are discussed below together with a review of its potential application in legal research. It is also possible to export data to external programs with much vaster analytical capacities, such as Statistical Package for the Social Sciences (SPSS) for statistical analysis.
3.4.1 ATLAS.ti capabilities
In ATLAS.ti, the most important tools to facilitate the analysis are the following. A first, relatively crude, way of analysis is the word cruncher that counts how many times words appear in a document. This may lead to useful insights in adjudication, and it will for example be able to demonstrate how often a court makes reference to certain concepts.
Second, through the use of hyperlinks, different codes or free quotations can be linked across documents. This is done by introducing a start and target anchor and choosing a relation tag from the default tags (continued by, contradicts, criticizes, discusses, expands, explains, justifies, supports) or by editing relations in the relations editor. This mode of analysis is particularly useful for legal analysis; the potential applications are plentiful, for instance to examine inconsistencies in disagreements of judicial opinion and court decisions contributing to the interpretation of legal concepts or identification of precedents.
Third, the query, co-occurrence explorer and co-occurrence table allow analysis as to which codes occur with other codes at the same time.38 Different types of queries are possible when analyzing how codes are embedded, overlapping, co-occurring, proceeding and following. For grounded theory-based coding, this feature is the most useful, but is also highly dependent on designing the codes in a way to leverage the features’ analytical possibilities. As Franzosi and others emphasize in their analysis of software options: ‘CAQDAS query tools dictate the code naming criteria’.39
Fourth, the codes-primary documents table also shows the number of times a code was used in a document or document family, thus enabling a systematic analysis of variables and their spread through the source documents. A recurring legal question will be the analysis of certain concepts in case law. Linked with primary documents sorted by time, such codes-primary documents tables can be used to demonstrate whether there is a tendency of the court to resort to specific concepts.
Lastly, ATLAS.ti provides a graphical support tool that allows network representation of codes to structure the information and support the theory building process through visualization of codes and their linkages in networks.
3.4.2 Analysis in the SPS Jurisprudence Prototype
The SPS Jurisprudence Project prototype covered three common types of legal inquiry: case outcome overviews (RQ 1); identifying the influence of legal tests on the outcome (RQ 2); and categorizing types of arguments (RQ 3).
The research design to address RQ 1 (outcome overview) deployed co-occurrence coding and hyperlinking. The co-occurrence of the judgment year attribute code and the outcome codes allowed an overview of how often the courts found a violation of the SPS provision and the development thereof over time.
In order to account for judgments later being overturned, hyperlinking supported the analysis of procedural history, as it specifically indicates which parts of a case have been appealed and/or overturned through the so-called “relationship coding” function. A Primary Document Family of Final Rulings (the last instance dealing with a given dispute) was then compiled to account for final outcomes only.
The data showed, first, that many appealed cases resulted in the outcome being overturned. Second, it highlighted a specific condition of the test as particularly susceptible for this (i.e. in judgments with the attribute code “overturned”, the specific text segment coded with the relationship code “overturned” appeared in Layer 3 (that identified where a court’s arguments on Article 5.6 related to condition numbers 1, 2 or 3)). The high number of overruling cases can be seen as an indicator of legal uncertainty in the interpretation of that part of the test. Therefore, one of the recommendations of the SPS Provision Jurisprudence Project is a doctrinal clarification on this point. The content analysis software supported the analysis in several ways: attribute coding was useful in quantifying the number of overturned judgments, and relationship coding enabled the systematic identification of the specific conditions susceptible to being overturned. The interpretation of why a specific condition was overturned, and how to improve the legal test, however, could only be answered by traditional legal analysis.
For RQ 2 (identifying the influence of legal tests on the outcome) all Layer 3 quotations for each specific condition (condition numbers 1, 2 and 3) were compiled. Assessing the respective weight or importance of one of the conditions for the overall outcome is a technical legal operation, with some room for interpretation – it is a subjective judgment call. Here, a magnitude code with a metric for assessment of importance can be used. This step in practice necessitated a third cycle of horizontal coding of the Layer 3 quotations for conditions numbers 1, 2, and 3 respectively. Effectively, this step is mostly traditional legal interpretation, only that it becomes formalized and systematically documented through the coding process.
Finally, good coding also shows omissions: when comparing the Layer 2 quotations (so the entire argument) with the Layer 3 code about the three conditions (condition numbers 1, 2 and 3), we noted that large parts of quotation parts from Layer 2 did not have a Layer 3 code. Apparently, the court repeatedly dealt with an argument that could not be coded under the umbrella of the conditions. When we coded for this argument separately (in case the question whether alternative measures had been identified), it appeared sometimes as a sub-code in both condition number 1 and condition number 2. Based on this analysis, we suggest that the three-prong test should in fact be extended to include an additional condition.40
Our approach in RQ 3 (typology of arguments) was the most inclined towards grounded theory. During the third horizontal coding cycle of the conditions, descriptive or topic codes were assigned to the arguments used in testing the conditions 1, 2 and 3). Using the co-occurrence explorer, these arguments were compared and where possible grouped in higher level code families. Through this process, a taxonomy of arguments emerged for each condition, based on which we were able survey which arguments were used most often and with the greatest force of conviction. Much like the evaluations undertaken for the purpose of RQ 2, many coding decisions made during the analysis were subjective, and fell into the realm of the traditional interpretative analysis. However, the advantages of greater scientific rigor hold in this case, as the coding process forces the researcher to take a conscious decision and document it.
4 Evaluating the future of ATLAS.ti in legal research
One of the main advantages of using ATLAS.ti lies in reaping the benefits of content analysis, making ‘[…] legal scholarship more consistent with the basic epistemological underpinnings of other social science research.’41 The proof of concept of using ATLAS.ti for legal research confirmed the usual advantages cited, namely that CAQDAS make research more explicit and transparent, helping research audiences to evaluate quality, and positively influence validity, robustness and trustworthiness of research results.42 The utility of ATLAS.ti excels where the number of judgments exceeds the cognitive ability of a researcher to keep track of them.
Less tangible benefits also arise when using the software. For example, even when sticking to traditional interpretation of entire text documents, the layout of ATLAS.ti is reader friendly, and coding the documents enables easy navigation and quick orientation. There are plenty of small steps that support a systematic approach to research. For instance, all quotations on a given topic and/or code can be exported in one document, which can be further analyzed manually. While not fully exploiting the potential of ATLAS.ti, such semi-automation tremendously enhances the robustness of analysis and takes out potential bias in prior case selection. Most importantly, the mere necessity to take formal decisions about the research design – which source documents to use and why, which codes to create and use, or how analytical capabilities of the program can be captured – amplifies the general rigor of thinking about the project and brings the research questions into sharper focus.
Despite our overall positive assessment, one of the most important limitations of the methodology is highlighted by Hall and Wright:
[S]ystematic coding can increase internal validity by removing elements of researcher bias and improving thoroughness and accuracy. However, to the extent that content analysis ignores aspects of legal interpretation that are impossible to code objectively (for example, nuances relating to infrequent or highly complex factual and procedural patterns), content analysis loses relevance, or external validity.43
The authors experienced limitations when it comes to the objectivity of reading legal sources, and acknowledge that despite the pretence to objectivity, our coding involved a strong legal-technical and interpretative element. Coding remains, therefore, a “judgment call”.44 It gives rise to a risk of misrepresenting data, for instance, if the coding is unable to capture important nuances or matters of degree. However, this criticism is at least equally valid for traditional interpretative scholarship, while the formal nature of the coding process during systematic content analysis challenges researchers to explain and disclose choices.
The ATLAS.ti software also has technical limitations; notably the restricted number of queries that are available and the coding functionality for primary document attributes is unnecessarily laborious. Moreover, the analysis interfaces are not intuitive and the output of data and the visualization options for results appear rather limited.
In conclusion, qualitative analysis programs, such as ATLAS.ti, can reduce the cognitive overload that comes with an increasing number of source documents due to the pervasive digitalization of society. That said, content analysis software only supports legal doctrinal analysis and cannot replace a trained legal researcher due to the necessary coding decisions and subsequent interpretation.
The quality of the research conducted with ATLAS.ti depends entirely on the research design, i.e. whether research questions are suitable and codes and matched queries appropriate. Hall and Wright observed that: ‘In project after project, legal researchers reinvent this methodological wheel on their own.’45 The present article clarified the advantages of using content analysis software, formulated propositions about using ATLAS.ti in legal research and validated them in a proof of concept. This was done in the hope of making a marginal contribution to the overall debate on what are suitable legal research methodologies, such as systematic content analysis, and further the inquiry into the tools to implement these most effectively.