Some Thoughts on Case Law Selection and Why it Is OK to Make Mistakes (as Long as You Learn from Them)

By Stijn Smet

In this post, I aim to make two fairly straightforward points. First: methodology is crucial in any type of (academic) research. This is obviously the case for legal research as well, even if legal scholars have traditionally been less concerned with methodological questions than scholars in most other disciplines (I am painting with a broad brush here). Legal scholars are particularly prone to remain obscure on their methodology in their writings. But something seems to be moving in legal academia. Many of today’s PhD researchers are keenly aware of the centrality of their methodology. They seem to pay exceedingly careful attention to the selection of and proper application of their methodology. Previous posts by PhD researchers Dorothea Staes and Laura Van den Eynde, as well as the very organisation of this blog post series, are a testament to this welcome evolution.

I have written this post with an audience of such dynamic PhD researchers in mind. This post is meant, first and foremost, for them. Which brings me to my second point: it is OK to make (some) mistakes as a PhD researcher, as long as you learn from them. Doing a PhD is, after all, part of one’s education. It is supposed to be a learning process, from which you emerge as a (much) better scholar than the one you were when you just started. It thus seems only natural, to me, that you are likely to make mistakes along the way. And that there should be room for such mistakes. As long as you learn from them.

With those two thoughts in mind, I will share some of the methodological questions I struggled with in my PhD research, and the (sometimes imperfect) answers I came up with. My PhD concerned conflicts between human rights, perhaps the most notorious example being conflicts between freedom of expression and the right to private life in relation to news stories in the media. More particularly, my PhD research aimed – in relevant part, for the purpose of this post – to critically evaluate the ECtHR’s judicial reasoning in conflicting human rights cases.

The first challenge I faced was the case law selection. Where do I start? Do I examine all judgments? Do I aim for exhaustive case law analysis? Or will that not be possible? As it turned out, it was not possible. But at the outset I thought it was. At the start, I aimed to analyse each and every judgment (and decision!) the ECtHR had delivered in cases involving a conflict between human rights. A preliminary problem soon emerged, however. How do I define a conflict between human rights? This question was key, since it would determine my case law selection. But I could not satisfactorily answer it without having analysed a large enough sample of relevant cases first, and without having conducted a literature study. I decided to bracket the definitional question and focus on the case law analysis first, assuming that the former would flow organically from the latter. Which is precisely what happened. I started off with some rough idea of what constituted a conflict between human rights (after all, freedom of expression versus right to private life was a fairly obvious example; there were bound to be more along those lines). Throughout the case law analysis, I was then able to refine the definition into the one proposed in my PhD dissertation. For those interested, this is the final definition (it speaks of “Convention rights” instead of “human rights”, given the strict focus of my research on ECHR law):

A genuine conflict between Convention rights arises whenever the State is under   incompatible duties to protect/respect the Convention rights of two or more identified or identifiable individuals and/or entities, provided that their Convention rights are actually and sufficiently at stake.

Explaining the different elements of the definition would take me beyond the bounds of this post, which is supposed to focus on methodology and case law selection. So let me get back on topic. I still had to tackle the problem of selecting the relevant cases for my analysis of the Court’s reasoning in conflicting human rights cases. In the end, I used a combination of methods. I was, first of all, fortunate enough to be able to build on the work of another (former) PhD researcher, Peggy Ducoulombier, who had investigated a similar question to mine (in French) and had finalised her PhD shortly before I started mine. This allowed me to use her extensive case law selection as a basis for my own research. I complemented this firm basis with three other methods. Firstly, key word and other searches in the ECtHR’s HUDOC database. I searched for a plethora of possible key terms and expanded those as I went along (for instance, while reading some of the selected case I noticed the Court sometimes uses an expression I had not yet included in my search terms: “enter into conflict”). This allowed me to select a large amount of cases, but still caused me to end up (well) short of exhaustive case law selection. That is where the second method – the ‘snowball’ method – came in. My already selected cases often contained key references to earlier Court judgments and decisions, which I looked up and – where relevant – added to my case law selection. The last, and perhaps most important, method I used was a collaborative one. At our research unit, we had set up by-weekly meetings in which we discussed all ECtHR judgments released during the reporting period (we called this – and still do, because it still exists – the ‘Strasbourg Club’). Through a rotation system, it allowed us to bundle our forces in establishing a central ECtHR case law database that we all benefited from. The benefit for me was that I could – quite easily – pick out those cases that entailed a conflict between human rights, based on the summaries prepared by my colleagues (and myself).

Initially, the first half year or so, I was (somewhat naïvely) convinced that the combination of the four listed methods would give me access to an exhaustive database of ECtHR judgment on conflicts between human rights. I was wrong. Although the selection was certainly extensive (I ended up with 400+ judgments and decisions), it fell short of being exhaustive. I learned this fairly early on, while writing a paper on the conflict between freedom of expression and the right to reputation (published here) in the Court’s case law. Although I had included 120+ judgments and decisions in the analysis, I kept finding new judgments and decisions (some through completely random events) throughout the writing and publication process. I was forced to admit the inevitable shortcomings of both the Court’s HUDOC system and my own case law selection methodology. Exhaustive selection, it turned out, would not be possible. Fortunately, by that time I had come to realise that there was no need for exhaustive selection, since many of the relevant Court judgments and decisions were purely repetitive (i.e. they did not contain any new strands of judicial reasoning). I was also convinced – and remain convinced – that I had selected a sufficiently large amount of judgments and decisions, including all important or ‘leading’ cases, to be able to conduct and present my critical analysis of the Court’s reasoning. So I let go of the unattainable objective of exhaustive analysis and focused on what started to matter more and more: my own substantive contribution to the existing scholarship (which has taken the form of the structured framework for the resolution of conflicts between human rights proposed in my PhD dissertation).

So far for what concerns the question of case law selection. But I announced I would write something about making mistakes as well. And about how it is OK to do so as a PhD researcher (as long as you learn from them). What better way to bring this point across than by sharing a mistake I made during the early stages of my PhD, and which I avoided repeating at a later stage. It concerns chronological case law analysis. At the start of my PhD, case law analysis came across as quite a daunting challenge. So daunting that I at some point lost sight of one factor: time. For the paper (later published article) on freedom of expression and the right to reputation I mentioned above, I conducted a detailed case law analysis in which – I thought – I included all relevant factors. I wrote the paper during the first six months of my PhD research. I was focused on getting everything right: including all important cases, grouping them into relevant categories, analysing all judgments across the different categories, mixing categories to obtain correct results, selecting representative, but interesting illustrations; etc. One thing I did not do was analyse the judgments and decisions chronologically. I learned of this mistake when my supervisor asked me “Have you analysed the cases chronologically?”. I had not. As a result, one thing I could not offer in my paper was insight into whether – and if so, how – the Court’s case law on the conflict had evolved over time. So I made no claims – grand or otherwise – on this aspect in the paper. Since the paper focused on (i) critiquing the coherence and lack of transparency of the Court’s reasoning and (ii) evaluating the Court’s case law in light of a first, rudimentary version of my framework, this did not pose serious problems for the argument. But it could have.

Fortunately, in my case at least, mistakes we make are often much more prominently present in our memories than our successes. As a result, I was later given the opportunity to learn from my mistake when I analysed, during the later stages of my PhD research, a specific problem I had located in the Court’s reasoning in conflicting rights cases. The exact nature of the problem is not important here (but if you are interested, I warmly invite you to read my PhD dissertation!). What is important is that I attempted to demonstrate the existence of the problem through a – rather rudimentary – statistical analysis of the Court’s case law. It was at this stage that I avoided repeating the mistake I had made earlier. This time, I analysed the case law chronologically. In doing so, I realised that there were two defining moments in the Court’s case law, both of which impacted greatly on the statistical analysis I intended to conduct and present. The first moment was one in which the Court, through its case law, introduced a ‘new’ human right in the ECHR catalogue. From that moment onward, there arose a conflict – as I defined it – between the ‘new’ right and an ‘old’ right, firmly entrenched in the text of the ECHR itself (the rights were, once more, the right to reputation and freedom of expression, respectively). The second defining moment in the Court’s case law was when the margin of appreciation doctrine started dominating the particular area of case law I was investigating. Both moments were pivotal and would have completely skewed the outcome of my statistical analysis, had I not recognised them. As matters stood, I did recognise them, and split the analysis into three different time periods. This is just one example of how I learned from my earlier methodological mistakes. Learning from those mistakes is what, I think, the PhD process is partly about. It should be OK to make mistakes, as long as you learn from them.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s