<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:media="http://search.yahoo.com/mrss/"><channel><title><![CDATA[Mark's Blog]]></title><description><![CDATA[Personal blog of Mark Amery]]></description><link>https://markamery.com/</link><image><url>https://markamery.com/favicon.png</url><title>Mark&apos;s Blog</title><link>https://markamery.com/</link></image><generator>Ghost 5.42</generator><lastBuildDate>Sat, 18 Apr 2026 05:00:34 GMT</lastBuildDate><atom:link href="https://markamery.com/rss/" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title><![CDATA[Simple uses of quantifiers can make regexes take quadratic time unnecessarily]]></title><description><![CDATA[<!--kg-card-begin: markdown--><p>Suppose I want to extract all substrings of a text that consist of one-or-more letter <code>a</code>s followed by a letter <code>b</code>. This is obviously both:</p>
<ol>
<li>trivial to do with a single short regex</li>
<li>trivial to do in linear time</li>
</ol>
<p>but if you try to do it <em>with a regex</em></p>]]></description><link>https://markamery.com/blog/quadratic-time-regexes/</link><guid isPermaLink="false">67ac7ea86df73303218f77ba</guid><dc:creator><![CDATA[Mark Amery]]></dc:creator><pubDate>Wed, 12 Feb 2025 13:17:34 GMT</pubDate><content:encoded><![CDATA[<!--kg-card-begin: markdown--><p>Suppose I want to extract all substrings of a text that consist of one-or-more letter <code>a</code>s followed by a letter <code>b</code>. This is obviously both:</p>
<ol>
<li>trivial to do with a single short regex</li>
<li>trivial to do in linear time</li>
</ol>
<p>but if you try to do it <em>with a regex in linear time</em>, you may well screw it up and never even realise. There is a nasty trap here!</p>
<p>Until recently, I would have solved this problem with code something like this (example in JavaScript):</p>
<pre><code>function extractABs(text) {
  return text.match(/a+b/g);
}
</code></pre>
<p>... and I would have <em>assumed</em> that this would execute in linear time. After all, you can easily envisage the linear-time algorithm - you just need to make a single pass over <code>text</code>, keeping track as you go of the index at which each sequence of consecutive <code>a</code>s starts, and checking at the end of each one whether it is followed by a <code>b</code> or by another character. But the regex-based solution above does <em>not</em> execute in linear time in the worst case! Chuck the following code into your browser console or Node shell and behold the results:</p>
<pre><code class="language-javascript">function extractABs(text) {
  return text.match(/a+b/g)
}

const a50000 = &apos;a&apos;.repeat(50000);
const a100000 = &apos;a&apos;.repeat(100000);
const a150000 = &apos;a&apos;.repeat(150000);
const a200000 = &apos;a&apos;.repeat(200000);
const a250000 = &apos;a&apos;.repeat(250000);

function timeRegex(text) {
  console.time();
  extractABs(text);
  console.timeEnd();
}

timeRegex(a50000);
timeRegex(a100000);
timeRegex(a150000);
timeRegex(a200000);
timeRegex(a250000);
</code></pre>
<p>The times I got were 2.112s for 50000 <code>a</code>s, 8.451s (4x) for 100000 <code>a</code>s, 19.009s (9x) for 150000 <code>a</code>s, 33.757s (16x) for 200000 <code>a</code>s, and 52.856s (25x) for 250000 <code>a</code>s. In other words, execution time is clearly increasing quadratically with the size of the input.</p>
<p>Why is this happening? This is <em>not</em> an example of what many sources call <a href="https://www.google.com/search?q=catastrophic+backtracking">&quot;catastrophic backtracking&quot;</a>, which typically happens when you write regexes with nested quantifiers (like <code>(a+)+</code>) and results in <em>exponential</em> time complexity. In fact, the cause of the non-linear time complexity here doesn&apos;t involve any backtracking at all.</p>
<p>Instead the problem is simply that, after trying to match the regex at index 0 (the start) of our length-<em>n</em> string, consuming <em>n</em> characters, and failing, the regex engine then moves on to trying to match the regex at index 1 of the string, consumes <em>n-1</em> characters and fails, then tries to match at index 2 of the string, consumes <em>n-2</em> characters and fails, and so on. The total number of characters that need to be consumed this way is of course the <em>n</em>th triangular number<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup>, (<em>n</em>&#xB2; + <em>n</em>) / 2, and thus we have quadratic time complexity.</p>
<p>There&apos;s a simple fix: use a <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Regular_expressions/Lookbehind_assertion"><em>lookbehind assertion</em></a> in your regex that says that the sequence of <code>a</code>s, <code>a+</code>, must not be preceded by an <code>a</code>:</p>
<pre><code class="language-javascript">function extractABs(text) {
  return text.match(/(?&lt;!a)a+b/g)
}
</code></pre>
<p>Now the pathological case with 250000 <code>a</code>s, that was previously taking a minute to execute, takes a millisecond instead (on my machine). This works because the lookbehind short-circuits the regex engine&apos;s attempts to match the pattern in the <em>middle</em> of a substring of <code>a</code>s. After failing to match at index 0, the regex engine will still ask &quot;does the pattern match at index 1?&quot;, but instead of having to consume 249999 characters to get an answer to that question, it will instead see that there&apos;s an <code>a</code> directly before index 1, immediately answer &quot;no&quot;, and move on. Thus each such question requires only constant time to answer. And thus we&apos;re back to linear time complexity for the whole matching operation.</p>
<p>This tactic seems broadly applicable. In situations where you have a regex that starts with an indefinitely repeated expression (<code>a+</code>, <code>(foo)*</code>, <code>[0-9]+</code>, whatever), if you want to keep the regex&apos;s execution time linear, you almost certainly want to precede it with a negative lookbehind (<code>(?&lt;!a)a+</code>, <code>(?&lt;!foo)(foo)*</code>, <code>(?&lt;![0-9])[0-9]+</code>, etc.).</p>
<p>But until today, I didn&apos;t know this. I suspect most programmers I&apos;ve worked with didn&apos;t know this. How many quadratic-time regexes have we unwittingly written, to perform what ought to be linear-time operations, due to this trap? I&apos;m sure I&apos;ve authored a few...</p>
<hr class="footnotes-sep">
<section class="footnotes">
<ol class="footnotes-list">
<li id="fn1" class="footnote-item"><p>Actually, that&apos;s kind of an oversimplification, since there <em>is</em> backtracking involved here which I would guess roughly doubles the time taken. It just doesn&apos;t increase the time <em>complexity</em>. <a href="#fnref1" class="footnote-backref">&#x21A9;&#xFE0E;</a></p>
</li>
</ol>
</section>
<!--kg-card-end: markdown-->]]></content:encoded></item><item><title><![CDATA[CPS guidance misstates the law on defence of property?]]></title><description><![CDATA[<p><em>[Disclaimer: I am a layman, NOT a qualified lawyer of any kind.]</em></p><p>Something curious I noticed today: <a href="https://www.cps.gov.uk/legal-guidance/self-defence-and-prevention-crime">https://www.cps.gov.uk/legal-guidance/self-defence-and-prevention-crime</a> currently claims that:</p><blockquote>Self defence and the prevention of crime originates from a number of different sources. Defence of the person is governed by the common</blockquote>]]></description><link>https://markamery.com/blog/defence-of-property/</link><guid isPermaLink="false">667304336df73303218f7731</guid><dc:creator><![CDATA[Mark Amery]]></dc:creator><pubDate>Wed, 19 Jun 2024 17:04:29 GMT</pubDate><content:encoded><![CDATA[<p><em>[Disclaimer: I am a layman, NOT a qualified lawyer of any kind.]</em></p><p>Something curious I noticed today: <a href="https://www.cps.gov.uk/legal-guidance/self-defence-and-prevention-crime">https://www.cps.gov.uk/legal-guidance/self-defence-and-prevention-crime</a> currently claims that:</p><blockquote>Self defence and the prevention of crime originates from a number of different sources. Defence of the person is governed by the common law. Defence of property however, is governed by the Criminal Damage Act 1971. Arrest and the prevention of crime are governed by the Criminal Law Act 1967.</blockquote><p>This appears to me to be incorrect as it pertains to defence of property. There <em>is</em> a &quot;defence of property&quot; defence in the Criminal Damage Act 1971, specifically <a href="https://www.legislation.gov.uk/ukpga/1971/48/section/5">in section 5(2)(b)</a>, but it&apos;s a defence only to offences defined in that act (i.e. criminal damage and assorted related offences like threats of damage). So that defence would be entirely irrelevant when someone uses force <em>against another person</em> to protect their property.</p><p>If you take the CPS&apos;s word for it that defence of property is governed by the Criminal Damage Act and <em>not</em> by common law, you would logically conclude from this that there is only a very limited &quot;defence of property&quot; defence in English law and that it does not entitle you to use force against <em>people</em> to protect property (even if, perhaps, some other defence such as prevention of crime might give you such an entitlement). I&apos;m pretty sure, however, that this is wrong. &#xA0;Contra what the CPS says, there <em>is </em>a common law defence of &quot;defence of property&quot;.</p><p>This defence is described in the 2003 judgment <a href="https://www.bailii.org/ew/cases/EWHC/Admin/2003/2567.html">DPP v Bayer &amp; Ors</a>, where the judge says:</p><!--kg-card-begin: markdown--><blockquote>
<ol start="24">
<li>This line of authority is concerned with the defence of &quot;private defence&quot; or &quot;protective force&quot; when unlawful force is used or imminently threatened against a person who may use proportionate force to defend persons or property. It is to be distinguished from the line of authority which is concerned with a similar defence against trespassers. In the Law Commission&apos;s most recent attempt, following very widespread consultation, to articulate the relevant principles of law in a simple codified form (see <em>Offences Against the Person and General Principles</em> (1993) Law Com No 218, pp 106-110), these defences are set out (so far as they relate to defence of property) in the following terms:</li>
</ol>
<blockquote>
<ul>
<li>&quot;27(i) The use of force by a person for any of the following purposes, if only such as is reasonable in the circumstances as he believes them to be, does not constitute an offence &#x2013;
<ul>
<li>(c) to protect his property &#x2026; from trespass;</li>
<li>(d) to protect property belonging to another from &#x2026; damage caused by a criminal act or (with the authority of the other) from trespass &#x2026;<br>
...</li>
</ul>
</li>
</ul>
</blockquote>
</blockquote>
<!--kg-card-end: markdown--><p>Additionally, the &quot;defence of property&quot; common law defence is mentioned, and clarified, in statute! The <a href="https://www.legislation.gov.uk/ukpga/2008/4/section/76">Criminal Justice and Immigration Act 2008 &#xA7;76</a> defines &quot;Reasonable force for purposes of self-defence etc.&quot;, and subsection (2)(aa) of the act as amended says this applies to &quot;the common law defence of defence of property&quot; - i.e. the common law defence that the CPS states does not exist!</p><p>Unless I&apos;m missing some crazy development in the law since 2008 that I can&apos;t find any mention of when I google this issue, the CPS guidance on this is therefore very, very wrong! I&apos;ve duly filled in the feedback form on their website and submitted the following feedback:</p><blockquote>It appears to me that there is an error in your guidance about &quot;defence of property&quot; at https://www.cps.gov.uk/legal-guidance/self-defence-and-prevention-crime. There you claim that:<br><br>&gt; &quot;Defence of the person is governed by the common law. Defence of property however, is governed by the Criminal Damage Act 1971.&quot;<br><br>I believe this to be a misstatement of the law. Although there is a defence of property defence defined in the Criminal Damage Act 1971, it is a defence ONLY against charges of criminal damage (and as such is relevant only in situations where a person damages property to defend some other property, not in situations where a person uses force against another person to defend property). For any other charge, the defence of &quot;defence of property&quot; instead comes from common law, as described in e.g. https://www.bailii.org/ew/cases/EWHC/Admin/2003/2567.html. The existence of a common law defence of defence of property is also mentioned in statute, in the Criminal Justice and Immigration Act 2008 section 76(2)(aa).<br><br>I suggest reviewing the claims about defence of property on that page and correcting them if you agree that they are in error.</blockquote><p>Let&apos;s see if anything comes of it!</p>]]></content:encoded></item><item><title><![CDATA[Spiked is talking nonsense about the Laurence Fox libel ruling]]></title><description><![CDATA[<!--kg-card-begin: markdown--><p>Spiked magazine have published <a href="https://www.spiked-online.com/2024/01/30/the-laurence-fox-ruling-is-a-disaster-for-freedom-of-speech/">an article</a> by their chief political writer, Brendan O&apos;Neill, about <a href="https://www.judiciary.uk/wp-content/uploads/2024/01/Blake-and-Seymour-v-Fox-Judgment.pdf">the ruling against Laurence Fox in his libel case</a>. I&apos;ve seen it shared by a bunch of Twitter users I follow and <a href="https://freespeechunion.org/the-laurence-fox-ruling-should-horrify-anyone-who-cares-about-freedom-of-expression/">by the Free Speech Union</a>. It&apos;s a bad</p>]]></description><link>https://markamery.com/blog/laurence-fox-libel-ruling/</link><guid isPermaLink="false">65bbb3142f17df1693fb9d35</guid><dc:creator><![CDATA[Mark Amery]]></dc:creator><pubDate>Mon, 05 Feb 2024 11:54:48 GMT</pubDate><content:encoded><![CDATA[<!--kg-card-begin: markdown--><p>Spiked magazine have published <a href="https://www.spiked-online.com/2024/01/30/the-laurence-fox-ruling-is-a-disaster-for-freedom-of-speech/">an article</a> by their chief political writer, Brendan O&apos;Neill, about <a href="https://www.judiciary.uk/wp-content/uploads/2024/01/Blake-and-Seymour-v-Fox-Judgment.pdf">the ruling against Laurence Fox in his libel case</a>. I&apos;ve seen it shared by a bunch of Twitter users I follow and <a href="https://freespeechunion.org/the-laurence-fox-ruling-should-horrify-anyone-who-cares-about-freedom-of-expression/">by the Free Speech Union</a>. It&apos;s a bad article.</p>
<h3 id="background">Background</h3>
<p>The case is certainly interesting if you care about libel law, freedom of speech, or the culture war. If you&apos;re unfamiliar, a quick summary of the background to the case is as follows:</p>
<ol>
<li>in October 2020, apropos of nothing in particular, Sainsbury&apos;s posted <a href="https://twitter.com/sainsburys/status/1311672756010917889">a bizarrely combative tweet about Black History Month</a>, inviting customers who don&apos;t &quot;want to shop with an inclusive retailer&quot; to shop elsewhere</li>
<li>Laurence Fox responded to that post with a <a href="https://web.archive.org/web/20201005124030/https://twitter.com/LozzaFox/status/1312690460654219265">now-deleted Tweet calling on people to boycott Sainsbury&apos;s</a> for promoting racial segregation. Fox&apos;s post included a link to a separate announcement by Sainsbury&apos;s that they had introduced &quot;safe spaces&quot; for black employees, which was the basis for Fox&apos;s accusation of segregation.<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup></li>
<li>three woke Twitter users responded by calling Fox a racist</li>
<li>Fox responded by calling those three critics paedophiles. (It&apos;s <a href="https://www.theguardian.com/technology/2018/jul/15/elon-musk-british-diver-thai-cave-rescue-pedo-twitter">the done thing</a>...)</li>
<li>the three woke critics responded to that by suing him for libel</li>
<li>... and finally <em>Fox</em> responded by countersuing <em>them</em> for libel for having called him a racist in the first place.</li>
</ol>
<p>Even without digging further into the factual details, this summary raises, I think, some interesting questions of principle:</p>
<ul>
<li>How should the law evaluate accusations of &quot;racism&quot;, given that our society&apos;s current understanding of what that even means is bitterly divided? Should the contested nature of &quot;racism&quot; in the context of western society&apos;s current culture war over race mean that such a claim basically lacks any substantial meaning and can&apos;t be defamatory? (One can imagine similar questions over all sorts of vague, opinion-based accusations - for instance, what if someone were to publicly denounce you as &quot;immoral&quot;, or &quot;intellectually dishonest&quot;, or &quot;a pervert&quot;?)</li>
<li>How should <em>rhetoric</em> be handled? It seems obvious that a false accusation of paedophilia could be libellous in principle, and typically would be, but how should the law deal with the scenario where the speaker says that accusation wasn&apos;t actually intended to be believed by the reader? (For this was Fox&apos;s contention here: that the accusations of paedophilia were meant to be <em>obviously</em> false, and thus to highlight the nastiness and unfairness of flinging baseless accusations at people - which is, of course, how he sees the accusations of racism against him.)</li>
<li>Should there be any kind of principle that retaliation is fair game, and if so how far should that go?</li>
</ul>
<p>Most coverage I&apos;ve seen of the case, including <a href="https://www.bbc.co.uk/news/entertainment-arts-68132377">that of the BBC</a>, is disappointingly shallow, not exploring the legal details or implications for free speech at all. Brendan O&apos;Neill&apos;s article in Spiked is the sole exception. The trouble is, it repeatedly makes seriously false claims about what the judgment actually says.</p>
<h3 id="no-defence-of-rhetoric-in-uk-libel-law">No defence of rhetoric in UK libel law?</h3>
<p>A key implication in the Spiked article is that the courts simply ignored entirely Fox&apos;s claim to have been using the counter-accusation of paedophilia as a rhetorical device, and instead, robotically, took it as literally true. Consider the following passages, spread over a few paragraphs in the Spiked article:</p>
<blockquote>
<p>It was a rhetorical game. A verbal stunt. A drawing of linguistic daggers. ...</p>
<p>And yet, the High Court has found against Fox. Taking literalism to dizzying new heights, the judge criticised him for failing to show that his allegations were true &#x2013; yes, because they weren&#x2019;t! I know judges are out of touch, but surely they&#x2019;ve heard of rhetoric and humour. ...</p>
<p>... At an earlier stage of the Fox libel clash, the Court of Appeal &#x2018;did not accept that the ordinary reasonable reader would obviously understand Fox&#x2019;s complex rhetorical point that he was no more a racist than they were paedophiles&#x2019;, as one legal observer summarised it. ...</p>
<p>&#x2018;The law affords few defences to defamation of this sort&#x2019;, decreed the High Court. And that&#x2019;s the problem. That the defence of rhetoric is not permissible under our libel laws is dreadful.</p>
</blockquote>
<p>I will grant Spiked that they <em>do</em> at least note that the court opined on whether an ordinary reader would understand Fox&apos;s point is rhetorical. My beef is with the assertions that the judge acted as if she had not &quot;heard of rhetoric and humor&quot; and that &quot;the defence of rhetoric is not permissible under our libel laws&quot;. That&apos;s just objectively <em>not</em> what happened.</p>
<p>In fact, though you&apos;ll find no hint of this in the Spiked article, Fox actually <em>won</em> his defence against one of the three claimants, Nicola Thorp, on precisely the grounds that the accusation against her was rhetorical! It was not the case that the courts acted as if it had not &quot;heard of&quot; rhetoric; rather, they simply judged that the &quot;natural and ordinary meaning&quot; of two of the tweets was to literally accuse the claimant of paedophilia. From part 44 of the judgment, summarising previous findings from the court of appeal:</p>
<blockquote>
<p>i) The &#x2018;single natural and ordinary meaning&#x2019; of Mr Fox&#x2019;s tweets responding to Mr Blake and Mr Seymour (&#x2018;Pretty rich coming from a paedophile&#x2019; and &#x2018;Says the paedophile&#x2019;) was that &#x2018;each of these Claimants was a paedophile, someone who had a sexual interest in children and who had or was likely to have engaged in sexual acts with or involving children, such acts amounting to serious criminal offences&#x2019;. This was an allegation or imputation of fact. The imputation was &#x2018;of defamatory tendency at common law&#x2019; &#x2013; that is, in the meaning determined, it would &#x2018;substantially affect in an adverse manner the attitude of other people towards a claimant, or have a tendency to do so&#x2019; ...<br>
ii) Mr Fox&#x2019;s tweet responding to Ms Thorp was different. He had quote-tweeted her allegation, and reproduced it simply substituting &#x2018;paedophile&#x2019; for &#x2018;racist&#x2019;. &#x2018;Mr Fox was not using the word &#x2018;paedophile&#x2019; literally, to accuse Ms Thorp of being a paedophile; he was using that word rhetorically as a way of expressing his strong objection to being called a racist. Used in that way it was not defamatory&#x2019;.... Ms Thorp had originally claimed in libel against Mr Fox on the basis of his tweet to her, but since it was found not to have any defamatory tendency, no tort could have been committed, and her claim was dismissed on that basis.</p>
</blockquote>
<p>Later, when commenting more on how readers might have interpreted Fox&apos;s tweets, the judgment has more to say about this:</p>
<blockquote>
<p>As a rhetorical device &#x2013; even to the extent Mr Fox subsequently took pains to spell it out as being one &#x2013; it was not well-calculated to be effective. It relied on a reader recognising both imputations &#x2013; &#x2018;racist&#x2019; and &#x2018;paedophile&#x2019; &#x2013; as immediately and equally incredible. But the context, for the readership, was first Mr Fox&#x2019;s &#x2018;boycott Sainsbury&#x2019;s&#x2019; tweet, then the &#x2018;racist&#x2019; tweets, and only then the (unexpected and unexplained) &#x2018;paedophile&#x2019; tweets. The call to boycott Sainsbury&#x2019;s was itself startling but evidently intended to be taken wholly literally, and the &#x2018;racist&#x2019; responses were apparently heartfelt; any reader pondering the exchanges (and especially if they did not click through to the website) might be as likely to think (a) the &#x2018;racist&#x2019; jibe had hit home and been met with an equivalently devastating counterblow against these particular individuals, since others had questioned or protested the boycott tweet without getting the same response, as they were to think (b) both jibes were patent nonsense.</p>
</blockquote>
<p>You can certainly reasonably disagree with this, but hopefully this makes clear that Spiked is not merely steamrolling over nuances but peddling some outright falsehoods. Contra Spiked, the defence of rhetoric <em>is</em> available in our libel laws; the court simply wasn&apos;t persuaded that <em>these particular statements</em> were correctly understood as rhetorical, and outlined why this was so, in great detail!</p>
<p>Less consequential, but also untrue, is Spiked&apos;s claim that the court &quot;criticised&quot; Fox for failing to show that his accusation was true.<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup> There&apos;s simply nothing like this in the judgment. The judgment does <em>acknowledge</em> the fact that Fox didn&apos;t attempt to show the accusation was true; for instance, in paragraph 166, near the end:</p>
<blockquote>
<p>Mr Fox did not attempt to show these allegations were true, and he was not able to bring himself on the facts within the terms of any other defence recognised in law.</p>
</blockquote>
<p>... but it strikes me as nonsense to characterise that passage, or any of the similar ones in the judgment, as &quot;criticising&quot; Fox for &quot;failing&quot; to show that the claimants were paedophiles.</p>
<h3 id="why-no-ruling-on-whether-fox-is-a-racist">Why no ruling on whether Fox is a racist?</h3>
<p>O&apos;Neill distorts other elements of the judgment, too. When it comes to Fox&apos;s counterclaim of libel for the claimants calling him racist in the first place, the Spiked article has this to say:</p>
<blockquote>
<p>The court refused to make a &#x2018;determination&#x2019; as to whether the accusation that Fox is racist is &#x2018;substantially true&#x2019;. That is not a question that can be &#x2018;resolve[d] within the framework of this litigation&#x2019;, the ruling weirdly says. What does this mean? Why can the High Court decree that it is defamatory to refer to three non-paedophiles as paedophiles, but it cannot decide if it&#x2019;s defamatory to refer to a non-racist as racist? Do our libel laws only protect certain people against certain accusations? Seems iffy to me.</p>
<p>It looks to some of us as though this ruling implicitly upholds the right to call people racist. Or at least, the courts will not treat an unprovable accusation of racism as seriously as they will an unproveable accusation of paedophilia.</p>
</blockquote>
<p>Contra O&apos;Neill&apos;s implication here, the reason that the court doesn&apos;t evaluate whether Fox is racist is in fact clearly explained in the ruling. For a statement to be defamatory, it needs to both <a href="https://www.legislation.gov.uk/ukpga/2013/26/section/1/enacted">cause serious harm</a> to the claimant&apos;s reputation and <a href="https://www.legislation.gov.uk/ukpga/2013/26/section/2/enacted">be untrue</a>. The court found that Fox, whose <a href="https://www.standard.co.uk/news/uk/laurence-fox-question-time-meghan-racism-a4336741.html">views on race were already controversial</a> and who had already been called a racist on Twitter before October 2020 (I observe that simply searching for the complete phrase <a href="https://twitter.com/search?q=%22laurence%20fox%20is%20a%20racist%22%20until%3A2020-10-01&amp;src=typed_query&amp;f=top">&quot;laurence fox is a racist&quot;</a>, filtered to results before October 2020, finds dozens of tweets), did not suffer any serious reputational harm from a few extra tweets callng him a racist. Therefore it was moot whether or not he <em>is</em> a racist; there was simply no need for the court to rule on that question.</p>
<p>There are a few further things the judge noted in her ruling that I&apos;d like to note here, though. One is that, had things gone differently, one of the defendants against Fox&apos;s counterclaim might indeed have faced the task of proving that Fox is a racist. From paragraph 58:</p>
<blockquote>
<p>Because of the preliminary issues ruling that subsection (3) was not satisfied in her case, [the defence of honest opinion] is not available to Ms Thorp. Instead, she relies, if necessary, on being able to prove that the &#x2018;imputation conveyed by the statement complained of is substantially true&#x2019; (Defamation Act 2013, section 2). ... It requires her to establish that it is substantially, objectively, true that Mr Fox is, in fact, a racist.</p>
</blockquote>
<p>Also, the choice of what order to evaluate the elements of the tort of libel in is, to some degree, at the discretion of the court. Paragraph 163:</p>
<blockquote>
<p>Cases turn on their facts. There are certainly examples in the authorities of cases disposed of on alternative bases of failure to establish serious harm and success on a defence. It may for example be that a case can be swiftly and efficiently disposed of where a strong and straightforward defence appears, but where the serious harm element is more complex, by concentrating on the former, even though the latter is a logical precedent stage. But this is not a case of that sort.</p>
</blockquote>
<p>And finally, the court should, in the opinion of the judge, avoid opining on culturally controversial matters where it&apos;s irrelevant to the ultimate resolution of the legal matter before the court. From section 164:</p>
<blockquote>
<p>I am very much aware that Mr Fox would have liked to leave court with a clear determination that he &#x2018;is not a racist&#x2019;, Ms Thorp with a determination that it is substantially true that he is, and Mr Blake and Mr Seymour with an endorsement that at least they genuinely thought so and an honest person could have thought so too. But the entire case is, in that sense at least, all about contested views of what does and does not amount to being &#x2018;a racist&#x2019;. ... Courts do not shy away from difficult assessments of contemporary cultural standards where the law requires them to. But where, as I have concluded, the law does not so require, because, by operation of statute and application of the serious harm test, an opinion on such a matter must in law be regarded as &#x2018;not defamatory&#x2019;, then courts must be properly circumspect about wading unnecessarily into such territory.</p>
</blockquote>
<p>I&apos;ve not seen it articulated before, but this seems like a good principle, to me! If it&apos;s not going to affect who wins the case, a judge shouldn&apos;t use a judgment as a soapbox from which to decree officially correct and incorrect positions on matters disputed in the culture war (such as whether Laurence Fox&apos;s stridently anti-woke views constitute &quot;racism&quot;). Brendan O&apos;Neill bemoans this, but would he really want it otherwise?</p>
<p>I found it interesting that the judge chose to note the court&apos;s discretion about the order in which to consider each element of the definition of libel immediately before this paragraph. It seems to me that she means to imply - though she stops short of outright saying it - that where the opportunity arises, the courts should even evaluate elements of a tort or crime in an unconventional order if doing so will potentially allow them to dodge the need to directly rule on the merits of opposing views in the culture war.</p>
<p>But anyway, the answer to O&apos;Neill&apos;s rhetorical question about why Fox and his opponents were treated differently is right there in the ruling: the court <em>would</em> have evaluated whether Fox was a racist, and issued a ruling one way or the other, if the case had depended upon that. But it didn&apos;t, essentially because the pre-existing controversy around Fox meant that no additional harm to his reputation from a few more tweeters calling him a racist was possible. None of that constitutes a &quot;right to call people racist&quot; or a different standard for accusations of racism versus paedophilia, as alleged by O&apos;Neill.</p>
<p>(Aside: it&apos;s not a falsehood, but I want to note the irony of the fact that the paragraph discussed above, bemoaning a supposed judge-created &quot;right to call people racist&quot;, appears immediately after a paragraph praising the more liberal libel laws in the USA, and asserting that due to our lack of a defence of rhetorical hyperbole, we <em>&quot;lag in liberty behind our American cousins&quot;</em>. Unlike the UK, the US really does have a right to call someone racist! Such an accusation will <a href="https://californiaslapplaw.com/2020/02/is-it-defamatory-to-call-someone-racist/">basically always constitute &quot;opinion&quot; under US law</a>, and statements classified as &quot;opinion&quot; cannot be libellous in the US. This protection of &quot;opinion&quot; statements is absolute; unlike in the UK, there are no further elements to the defence. You can call anyone you like a racist over there, with absolutely no factual basis for doing so whatsoever, even if you don&apos;t actually believe they are racist at all and you are consciously lying, and it&apos;s still perfectly legal. To the American mind, the right to do this is part of the fundamental right to free speech guaranteed by the First Amendment. I guess O&apos;Neill is particular about which liberties he wants to copy from our American cousins. In fairness, on this point, I <em>partially</em> agree with him.)</p>
<h3 id="was-there-injustice-here-then-where-was-it">Was there injustice here, then? Where was it?</h3>
<p>All the above said, I think you can still very reasonably take issues with aspects of the ruling. The complaints I&apos;d potentially make are just different to, or at least more nuanced than, those made in Spiked.</p>
<p>Despite my annoyance at the misrepresentations of the judgement that went along with it, I am at least sympathetic to one part of O&apos;Neill&apos;s (and Fox&apos;s lawyer&apos;s) critiques of the judgments against Fox: the contention that the courts were far too ungenerous to the intelligence of the average Twitter user in finding that an &quot;ordinary reasonable reader&quot; would not understand Fox&apos;s tweets to be rhetorical. I certainly wouldn&apos;t expect many readers to actually <em>believe the accusation</em> or think that Fox had evidence substantiating it. Even so, I don&apos;t think it&apos;s <em>obvious</em> that the courts got this wrong; even if readers didn&apos;t believe the accusations were <em>true</em>, it doesn&apos;t mean they would understand them to be rhetorical, and I can easily imagine that many readers would instead simply think that Fox - perhaps in some kind of unhinged state - was carrying out a deranged act of retaliation by responding with baseless <em>but literal</em> accusations of paedophilia against his enemies. I think it was a genuinely hard call. I think <em>I</em> would&apos;ve been unsure what the hell was going on if I had read Fox&apos;s tweets at the time that he posted them, which in turn makes it hard for me to confidently suggest, in hindsight, what an &quot;ordinary reasonable reader&quot; would&apos;ve thought about them.</p>
<p>I am even <em>more</em> sceptical of the judgment&apos;s assessment of whether the claimants&apos; reputations were caused &quot;serious harm&quot; by the paedo accusations. I note that in the TPI (&quot;Trial of Preliminary Issues&quot;) where the courts first decided the &quot;single ordinary meaning&quot; of Fox&apos;s tweets, Justice Nicklin <a href="https://www.bailii.org/ew/cases/EWHC/KB/2022/3542.html">noted that it still mattered how people had <em>actually</em> intepreted them</a>, since this would dictate whether &quot;serious harm&quot; had been done to anyone&apos;s reputation:</p>
<blockquote>
<p>the Court has not resolved the issue of whether the Claimants in their claim, or the Defendant in his counterclaim, can satisfy the requirement to establish serious harm to reputation as required under s.1 Defamation Act 2013. This may well be a significant issue at any trial. For example, if the Defendant can establish that in fact a significant number of readers of his Tweets did understand them simply to be making a rhetorical comment about the baselessness of the Claimants&apos; claims of racism against him, then the Claimants may struggle to demonstrate that they have been caused serious harm to their reputation.</p>
</blockquote>
<p>This makes it sound like Fox would win the case as long as he could establish, basically, that nobody believed that what he said was true. Importantly, it seems to me that this could be <em>either</em> because they thought he was being rhetorical <em>or</em> just because they thought he was an unhinged lunatic lashing out with baseless (but literal) claims.</p>
<p>The final ruling against Fox, by Justice Collins Rice, caveats this somewhat:</p>
<blockquote>
<p>I observe in this connection that, contrary to some of the submissions made to me on this point, it is not necessary for the claimants to establish the probability of readers being immutably convinced of the truth of an allegation. That is not how reputation works. Serious reputational harm can be caused by a change of view some considerable way short of that. It is often the insidious creation of a &#x2018;bad odour&#x2019;, together with the difficulty of establishing a negative, that does the most reputational harm. That is particularly apposite to an allegation of paedophilia. But the test does require that people&#x2019;s minds were probably changed because of these tweets, and to a degree meriting the description of serious harm.</p>
</blockquote>
<p>Okay, fair enough; if everyone goes from taking for granted that I&apos;m not a paedo to not knowing who to believe and feeling like it&apos;s an open question whether I&apos;m a paedo or not, I think it&apos;s still fair to call that serious reputational harm. But then, as evidence that <em>anyone at all</em> gave <em>even partial credence</em> to the accusation, the court relies heavily on the claimants getting abused on Twitter by trolls. The crucial paragraphs are 84-86:</p>
<blockquote>
<p>There is certainly evidence of an adverse reaction to both men on Twitter, with paedophilia being cast back in their faces. It was put to me that, carefully read, many of these abusive responses either (a) indicate that the allegation was not in fact taken literally or seriously, or (b) can largely be dismissed as the utterances of deep-dyed Twitter trolls or homophobes who need little excuse.</p>
<p>I am unpersuaded that &#x2018;careful&#x2019; reading is necessarily the right approach to the former category ... The probability &#x2013; and the evidence as I read it &#x2013; is that published reactions within the readership (at least from those who did not know the claimants) spanned the full spectrum from credulous to dismissive. That is usual in mass publication cases of any sort. I have not been given sufficient reason, from context or from evidence of reaction, to find enough apparent or likely scepticism, as a proportion of the whole readership, to be able to conclude it more probable than not that the seriously harmful potential of these tweets simply failed to be realised.</p>
<p>As regards the trolls and homophobes, those are labels that might be attached to the particularly credulous, hostile and/or responsive. But however little excuse they needed, it is obvious that Mr Fox provided one and set them an example. Online abuse is at least a possible signifier of serious reputational harm... I accept the evidence of it here.</p>
</blockquote>
<p>I have not seen all the evidence of what Twitter users said that was put before the judge, and indeed I cannot find any of it on Twitter nor find the accounts of Simon Blake or Colin Seymour. So I concede that I am commentating while in possession of less evidence than the judge. But I note that pretty much any highly-viewed adversarial interaction on Twitter results in abuse; I am a nobody on Twitter and from time to time I trigger a flurry of insulting replies and quote-tweets for expressing even mildly controversial opinions. A vast torrent of abuse directed at the claimants was thus the inevitable consequence of getting into any kind of hostile interaction with Laurence Fox, even if not a single person found the paedophilia accusations even remotely credible. This seems to me like it no more constitutes evidence that anyone believed the paedophilia accusations than does the fact that the sun came up the next day; both are simply something that was inevitably going to happen either way. The court, though, seems to have started from a default presumption that mere existence of online abuse constituted evidence that the paedo accusation was given credence. My instinct is that this presumption is terribly unsound. (It might make more sense if the abuse on Twitter followed an accusation of paedophilia in a totally different venue, like a newspaper article, or even just if the abuse had persisted after the initial exchange between Fox and the claimants, but it doesn&apos;t sound like that&apos;s the case here. Rather, per the judge&apos;s own wording, we&apos;re talking about <em>reactions</em> to the Tweet - i.e. people on Twitter piling on to the fight that the parties had just got into <em>on Twitter</em>.)</p>
<p>The judgment gets worse, in my view, in paragraph 86 about &quot;trolls and homophobes&quot;. Perhaps I am parsing her words wrongly, but it seems to me that in this paragraph the judge is no longer even using the Twitter abuse as evidence of anyone believing the paedophilia accusation, but rather is saying that people merely using Fox&apos;s post as an &quot;excuse&quot; to abuse the claimants can still constitute a serious reputational harm <em>even if they didn&apos;t put any credence in Fox&apos;s accusation at all</em>. That seems to me like a nonsensical interpretation of the concept of <em>reputational</em> harm; if the mechanism is simply that a bunch of Fox&apos;s followers saw Fox being nasty to his critics and thereby perceived that those critics were acceptable targets for further abuse, then - however ignoble that may be and however much Fox may be to blame for it - what does any of it have to do with anyone&apos;s <em>reputation</em>? Nothing at all, I would think, yet this scenario is somehow contemplated as a &quot;signifier&quot; of reputational harm in the judgment.</p>
<p>These dubious bases for inferring reputational harm seem to me like bigger problems with the judgment than the interpretation of Fox&apos;s meaning. Contra Spiked, declining to treat Fox&apos;s tweet as rhetorical doesn&apos;t strike me as the core of the unfairness here; certainly it seems to me it was poorly-conceived enough that many readers will have taken it as a literal accusation made out of spite, even if this wasn&apos;t <em>intended</em>, and it&apos;s fair enough for such a miscalculation to put you at risk of a libel lawsuit <em>if the accusation actually harms the other party&apos;s reputation</em>. But to me, common sense would seem to dictate that even people failing to see the rhetorical point in Fox&apos;s tweet would still realise it obviously <em>wasn&apos;t true</em>; the dilemma of interpretation any sensible reader faced was not between the accusation being a rhetorical device and the accusation being <em>true</em>, but between the accusation being a rhetorical device and the accusation being literal but baseless, cast in spite and anger by Fox in a moment of madness. Nobody reads a totally unexplained accusation of paedophilia cast during a slapfight on Twitter and infers that the accuser must have some secret basis for the claim and that it&apos;s totally true - they simply assume the accuser is nuts!</p>
<p>The judge doesn&apos;t seem to have contemplated this point - that even people who didn&apos;t understand Fox&apos;s rhetorical point <em>still generally would not have believed the accusation to be true or even credible</em>. I view that as a huge mistake of reasoning, and suspect that it, in combination with the dubious approach of treating a pile-on on Twitter as evidence that people believed Fox&apos;s accusation, has led the judge to hallucinate reputational harm to the claimants that simply doesn&apos;t exist, seems rather implausible, and doesn&apos;t really have any evidence to support its existence.</p>
<h3 id="elements-of-how-the-counterclaim-got-handled-seem-problematic-too">Elements of how the counterclaim got handled seem problematic too</h3>
<p>Though Fox was the overall loser of the case (all his counterclaims failed and two out of three of the original claims against him succeeded), there was one detail in the courts&apos; handling of his counterclaims that I think is worth highlighting. Two out of three of the people who called Fox a racist did so in quote-tweets; the other, Nicola Thorp, did not. As a consequence of this, she lost her ability to use <a href="https://www.legislation.gov.uk/ukpga/2013/26/section/3/enacted">UK law&apos;s &quot;honest opinion&quot; defence</a>. The trouble is that the statute codifying the defence requires that your statement indicate the <em>basis</em> of your opinion in order for the defence to apply. So you can say &quot;based on this tweet, Laurence Fox is a racist&quot;, and the defence will apply (as long as you also convince the court of some other, easier elements about holding the opinion honestly). You can <em>quote-tweet</em> something race-related that Laurence Fox has posted and say &quot;Laurence Fox is a racist&quot; and the defence will likewise apply. But if you <em>just</em> say &quot;Laurence Fox is a racist&quot;, without indicating your basis for this belief, then you can&apos;t use this defence. This is why, had the court ruled that Fox&apos;s reputation <em>was</em> damaged by the tweets calling him racist, Thorp - and Thorp alone - would&apos;ve been dragged into the farce of having to establish:</p>
<blockquote>
<p>that it is substantially, objectively, true that Mr Fox is, in fact, a racist.</p>
</blockquote>
<p>This is probably a correct application of the law as it is written. But it seems manifestly silly. The basis for Thorp&apos;s opinion was not <em>indicated</em> in her tweet but could be easily guessed at from the context; who cares that she didn&apos;t explicitly indicate it?</p>
<p>I understand the motivation behind the requirement to indicate your basis. Forget Fox, for a moment, and suppose that you were to label some random person who has never publicly said anything contentious about race in their life a &quot;racist&quot;. The accusation - even if you think it is properly understood as a statement of opinion - surely implies the existence of <em>some</em> kind of scandalous non-public facts that serve as a basis for the accusation - it&apos;s just not clear exactly what they are. If you&apos;re not required to articulate any basis for such an opinion, you can damage reputations by <em>insinuating</em> that <em>some</em> such scandalous facts exist, through statements of opinion that the reader assumes surely must have <em>some</em> basis, all without ever alleging any <em>specific</em> defamatory facts as a justification for your opinions. US libel law, as I understand it, permits precisely this tactic; I&apos;m not persuaded that&apos;s a good thing, or that it&apos;s wrong for us to want to be stricter in this area.</p>
<p>But it seems to me that we&apos;re <em>too</em> strict, and the results can easily be absurd when the target is a public figure like Laurence Fox. If I say that Boris Johnson is dishonest, that Jeremy Corbyn is a terrorist sympathizer, or that Tiger Woods was a bad husband, people with some basic knowledge of those individuals are aware of facts that might lead me to say such a thing, and are capable of inferring that my opinion (which they may or may not agree with) is probably based on those widely-known facts. The same is true if I say that Laurence Fox is a racist; he has notoriously said controversial things about race and you will implicitly understand these to be the basis of my opinion, whether you agree with it or not. (This is <em>even more true</em> if I call him a racist minutes after he&apos;s said his latest controversial thing about race on Twitter, like Thorp did!)</p>
<p>If we don&apos;t want our libel law to be <em>quite</em> as liberal about opinion as that of our American cousins, we ought to at least tweak our more limited opinion defence to also cover cases like this - where the factual basis for the opinion isn&apos;t <em>indicated</em> but rather is already widely known due to the claimant being a public figure. But, unfortunately, no such defence currently exists in UK law.</p>
<h3 id="conclusions">Conclusions</h3>
<p>My thoughts on all this, in summary:</p>
<ul>
<li>Brendan O&apos;Neill&apos;s article in Spiked is bad. It misleads, and sometimes states outright untruths, about what the judgment said.</li>
<li>Inferring that an adversarial exchange with a high-profile account on Twitter caused someone <em>reputational</em> harm because it triggered an influx of abuse on Twitter seems completely unreasonable to me, yet the judgment against Fox seemed to rely on such an inference.</li>
<li>The rules protecting expression of opinion should be more robust. If well-known facts about a public figure provide an obvious, salient basis for some negative opinion about them, it should simply be legal to voice that opinion - even without spelling out your reasoning. Right now, this is not necessarily so.</li>
</ul>
<p>Even though the culture war is a societal cancer, reporting on cases like this <em>ought</em> to be an opportunity to leverage it for good; they are a good opportunity for journalists to educate the public about how our laws work, and for activists to argue for improvements to them. Alas, nobody in a position to put such an analysis before a mass audience seems to have seized the opportunity.</p>
<hr class="footnotes-sep">
<section class="footnotes">
<ol class="footnotes-list">
<li id="fn1" class="footnote-item"><p>That safe spaces announcement, read plainly, seemed to state that Sainsbury&apos;s had introduced blacks-only areas or events in their physical, real-life workplaces, and lots of angry Twitter users besides Fox condemned Sainsbury&apos;s on this basis. Sainsbury&apos;s later disavowed this interpretation, though, and said the &quot;safe spaces&quot; were actually online support groups for employees to discuss their experiences of racism. I don&apos;t think they ever clarified whether those <em>online</em> spaces were for blacks only. Anti-woke blogger Stephen Knight <a href="https://www.gspellchecker.com/2020/10/sainsburys-and-argos-promote-racial-segregation-in-the-name-of-inclusion/">has a good writeup</a> that includes a screenshot of the announcement, which Sainsbury&apos;s has since deleted. <a href="#fnref1" class="footnote-backref">&#x21A9;&#xFE0E;</a></p>
</li>
<li id="fn2" class="footnote-item"><p>This claim is accompanied by a link to <a href="https://www.bbc.co.uk/news/entertainment-arts-68132377">https://www.bbc.co.uk/news/entertainment-arts-68132377</a>, which I looked at to see if it somehow corroborated the claim in Spiked, but it doesn&apos;t; the closest it gets to being relevant to the claim it&apos;s cited in support of in Spiked is when it notes that the judge said that Fox <em>&quot;did not attempt to show the court that these allegations were true&quot;</em>. There&apos;s no mention of the judge criticising him for this. <a href="#fnref2" class="footnote-backref">&#x21A9;&#xFE0E;</a></p>
</li>
</ol>
</section>
<!--kg-card-end: markdown-->]]></content:encoded></item><item><title><![CDATA[Don't trust the first item in the X-Forwarded-For header]]></title><description><![CDATA[<!--kg-card-begin: markdown--><blockquote>
<p>Any security-related use of X-Forwarded-For (such as for rate limiting or IP-based access control) must only use IP addresses added by a trusted proxy. Using untrustworthy values can result in rate-limiter avoidance, access-control bypass, memory exhaustion, or other negative security or availability consequences.</p>
<p>-- <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For#security_and_privacy_concerns"><em>MDN&apos;s X-Forwarded-For article</em></a></p>
</blockquote>
<h3 id="short-version">Short</h3>]]></description><link>https://markamery.com/blog/dont-trust-the-first-item-in-the-x-forwarded-for-header/</link><guid isPermaLink="false">64567cd32f17df1693fb99b0</guid><category><![CDATA[programming]]></category><category><![CDATA[web-development]]></category><category><![CDATA[infosec]]></category><dc:creator><![CDATA[Mark Amery]]></dc:creator><pubDate>Sun, 07 May 2023 09:35:24 GMT</pubDate><content:encoded><![CDATA[<!--kg-card-begin: markdown--><blockquote>
<p>Any security-related use of X-Forwarded-For (such as for rate limiting or IP-based access control) must only use IP addresses added by a trusted proxy. Using untrustworthy values can result in rate-limiter avoidance, access-control bypass, memory exhaustion, or other negative security or availability consequences.</p>
<p>-- <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For#security_and_privacy_concerns"><em>MDN&apos;s X-Forwarded-For article</em></a></p>
</blockquote>
<h3 id="short-version">Short version:</h3>
<ul>
<li><em><strong>Do not</strong></em> take the first IP address listed in an <code>X-Forwarded-For</code> header, assume it&apos;s the public IP of the end user who sent you a request, and use it for rate limiting. An attacker can usually set it to whatever they like, and bypass your rate limiting.</li>
<li>If you&apos;re going to use the end user&apos;s IP address for anything security-related, like rate limiting, and you want to get that IP address from <code>X-Forwarded-For</code>, then your application needs to know how many reverse proxies it is running behind (and there mustn&apos;t be a way for the end user to bypass any of those reverse proxies).</li>
<li>If your web server framework provides some function that purports to get the end user&apos;s IP address by parsing the <code>X-Forwarded-For</code> header, but doesn&apos;t require you to tell it how many reverse proxies it&apos;s running behind, don&apos;t trust it. It is basically guaranteed to be vulnerable to spoofing.</li>
<li>If you&apos;re behind a single reverse proxy, you want to use the <em>last</em> IP address listed in <code>X-Forwarded-For</code>. If you&apos;re behind two reverse proxies, you want the second-last IP, and so on.</li>
</ul>
<h3 id="long-version">Long version:</h3>
<p>There is an irksome mistake I&apos;ve seen colleagues make a couple of times in my career now: adding an IP-based rate limiter to a web application that any attacker can trivially bypass by simply adding an <code>X-Forwarded-For</code> header with a randomly-generated IP address to each HTTP request they send. I&apos;m writing this blog post to explain the problem and to give me something to link to next time I inevitably see the same mistake.</p>
<p>Suppose you have a web application running on servers behind some sort of load balancer (or perhaps behind multiple layers of load balancers and reverse proxies, but for the sake of simplicity, let&apos;s say there&apos;s just one). You want to implement an IP-based rate limiter in the application itself<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup>. For instance, let&apos;s say you want to make it so that a given IP address can only make 10 login attempts per 5 minute interval, as a way to make it harder for attackers to brute-force users&apos; passwords.</p>
<p>Since the requests to your application are coming via the load balancer rather than directly from the user&apos;s computer, the client IP address as seen by the application will be the IP address of the load balancer. Obviously, that&apos;s no good. Load balancers and reverse proxies try to solve this problem for you by sticking the the client&apos;s original IP in a header in the HTTP request, which is almost always called <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For"><code>X-Forwarded-For</code></a>.<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup></p>
<p>Because some applications have multiple reverse proxies in front of them (e.g. requests might first go through a load balancer, then through a caching reverse proxy like <a href="https://varnish-cache.org/intro/index.html#intro">Varnish</a>, and only then reach the actual web application), the <code>X-Forwarded-For</code> header can actually contain a <em>comma-separated list</em> of IP addresses, and when a proxy receives a request that already has an <code>X-Forwarded-For</code> header it&apos;s supposed to append the IP address it received the request from to the end of the header. That means that <em>in typical circumstances</em>, where the end user sends a request from their browser to your domain with no <code>X-Forwarded-For</code> header, the first (leftmost) element in that comma-separated list will be the public IP of the end user&apos;s computer. But you can&apos;t <em>trust</em> this! A malicious user can send a request that already has an</p>
<pre><code>X-Forwarded-For: 123.123.123.123
</code></pre>
<p>header, and if your load balancer behaves in the normal way, it will <em>append</em> the actual public IP after that, and your application will receive something like this:</p>
<pre><code>X-Forwarded-For: 123.123.123.123, 94.6.194.169
</code></pre>
<p>If you&apos;re naively treating the first element of the list as the user&apos;s IP address, and using that for rate limiting, then you&apos;ve just allowed an attacker to trick you into seeing their address as 123.123.123.123 when that&apos;s not really the address the request came from. By changing the <code>X-Forwarded-For</code> header to a random IP address on each successive request, an attacker can totally circumvent your rate limiting.</p>
<p>What you need to do <em>instead</em>, if you have exactly one reverse proxy (such as a load balancer) in front of your application, is look at the <em>last</em> element of the <code>X-Forwarded-For</code> list, since that will be the one your reverse proxy set. If you have two layers of reverse proxies, you want the second-last IP address in the list (and if you have three layers of reverse proxies, you look at the third-last, and so on).</p>
<p>Note that this fundamentally requires that your application <em>knows how many reverse proxies are sitting between it and the public internet</em>.<sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup><sup class="footnote-ref"><a href="#fn4" id="fnref4">[4]</a></sup> If your application does not know this, it <em>cannot</em> determine, in a way that an attacker can&apos;t spoof, the public IP address from which the request was originally sent.</p>
<p>A corrolary of this is that if a web application framework purports to expose some method of getting the end user&apos;s IP address that magically parses headers like <code>X-Forwarded-For</code>, <em>without</em> having first to be configured in some way to tell it how many layers of reverse proxies it&apos;s running behind, then <em>you should view that method with extreme scepticism, because it is almost guaranteed to be vulnerable to spoofing</em>.</p>
<p>For example, consider Micronaut&apos;s <a href="https://docs.micronaut.io/latest/guide/#clientIpAddress"><code>HttpClientAddressResolver</code></a>, recent use of which by a colleague of mine inspired this blog post. At the time that I write this post, its docs say:</p>
<blockquote>
<p>You may need to resolve the originating IP address of an HTTP Request. Micronaut includes an implementation of HttpClientAddressResolver.</p>
<p>The default implementation resolves the client address in the following places in order:</p>
<ul>
<li>
<p>The configured header</p>
</li>
<li>
<p>The <code>Forwarded</code> header</p>
</li>
<li>
<p>The <code>X-Forwarded-For</code> header</p>
</li>
<li>
<p>The remote address on the request</p>
</li>
</ul>
</blockquote>
<p>If you&apos;re planning on using the client&apos;s IP address for rate limiting or some other security-related purpose, then as soon as you read the documentation I quote above, alarm bells should go off! There&apos;s no hint of a way to configure Micronaut to know how many reverse proxies it&apos;s behind, so we can only assume that when it looks at the <code>X-Forwarded-For</code> header, it simply takes the first element in the list - the one the attacker can control - and sure enough, <a href="https://github.com/micronaut-projects/micronaut-core/blob/v4.0.0-M2/http-server/src/main/java/io/micronaut/http/server/util/DefaultHttpClientAddressResolver.java#L65">it does</a>. (The fact that it will try multiple headers is <em>also</em> a problem, of course! If your reverse proxies only set <code>X-Forwarded-For</code>, an attacker can set <code>Forwarded</code> and your Micronaut app will give that priority over the <code>X-Forwarded-For</code> set by your reverse proxies.)</p>
<p>For an example of a framework that at least makes it <em>possible</em> to get this right using its built-in functionality, see Werkzeug, which requires you to <a href="https://werkzeug.palletsprojects.com/en/2.3.x/deployment/proxy_fix/">configure it with an <code>x_for</code> parameter that tells it how many proxies it&apos;s behind</a>. That&apos;s okay to use when you need an IP for rate limiting. Anything that just magically gets an IP address without needing such configuration categorically <em>isn&apos;t</em> okay to use.</p>
<p>A final note: although I&apos;m writing this blog post in response to multiple colleagues getting this wrong at work, I want to be clear that it wasn&apos;t <em>unusual</em> for them to make that mistake. If you do a Google search for <code>bypass rate limit x-forwarded-for</code>, you will find multiple pages of results listing blog posts, articles, and LinkedIn posts by pentesters and other people whose job is to hack web applications, all advising that you try setting an <code>X-Forwarded-For</code> header to bypass rate limits... and you&apos;ll also find lots of publicly disclosed reports on HackerOne from bounty hunters who discovered such bypasses in applications they were testing. I find the existing writeups I&apos;ve seen a bit frustrating because they just present setting <code>X-Forwarded-For</code> as a magical incantation you should try as an attacker and don&apos;t explain what mistake the defender must have made to be vulnerable to it, nor what the defender should&apos;ve done instead; hopefully this post, written from a developer&apos;s perspective instead of a hacker&apos;s, can fill that void. But those search results do make a couple of important facts clear: developers of web application are screwing this up <em>constantly</em> in exactly the same way my colleagues did, and the bad guys are absolutely aware of it and will routinely try exploiting precisely this weakness to bypass your rate limits.</p>
<p>Stop letting them!</p>
<hr class="footnotes-sep">
<section class="footnotes">
<ol class="footnotes-list">
<li id="fn1" class="footnote-item"><p>Some will doubtless object here that such rate limits should be implemented in your load balancer, ideally by simply setting some config parameters that the load balancer offers to enable rate limiting, and that ordinary web developers should not usually end up reinventing the wheel by implementing IP-based rate limiters themselves. They probably have a point, but I don&apos;t think this is <em>always</em> true. Sometimes you have <em>particular</em> bits of functionality in your web application that warrant strict rate limits, like login, but you <em>don&apos;t</em> want to apply those strict rate limits across the entire application. In that case, trying to configure them in your load balancer may simply not be something the load balancer software you&apos;re using supports at all or else may be possible but create a maintenance nightmare where there&apos;s a list of endpoint paths or regexes configured in your load balancer to tell it which requests to rate limit, and you have to keep that config in sync with changes to your application. I&apos;d rather roll my own rate limiter than deal with that. <a href="#fnref1" class="footnote-backref">&#x21A9;&#xFE0E;</a></p>
</li>
<li id="fn2" class="footnote-item"><p>Note that <code>X-Forwarded-For</code> doesn&apos;t have an official spec anywhere, unlike the <code>Forwarded</code> header specced in <a href="https://datatracker.ietf.org/doc/html/rfc7239">RFC 7239</a>. In practice, though, I don&apos;t know of <em>any</em> software that supports only <code>Forwarded</code> and not <code>X-Forwarded-For</code>, and I <em>do</em> know of software - like <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/x-forwarded-headers.html">Amazon&apos;s ALB</a> - that supports only <code>X-Forwarded-For</code> and not <code>Forwarded</code>. <code>Forwarded</code> may be the official standard, but <code>X-Forwarded-For</code> is the de facto standard - even today, 9 years after RFC 7239 was published. <a href="#fnref2" class="footnote-backref">&#x21A9;&#xFE0E;</a></p>
</li>
<li id="fn3" class="footnote-item"><p>An import unstated premise here is that there needs to be a <em>fixed</em> number of reverse proxies between the application and the public internet! If going through the reverse proxies is optional - that is, if the application itself or any of the reverse proxies besides the outermost one are themselves accessible from the public internet - then an attacker will still be able to trick your rate limiter by bypassing the outermost reverse proxy and setting an <code>X-Forwarded-For</code> header. <a href="#fnref3" class="footnote-backref">&#x21A9;&#xFE0E;</a></p>
</li>
<li id="fn4" class="footnote-item"><p>An alternative strategy in theory would be to have your outermost reverse proxy discard any incoming <code>X-Forwarded-For</code> header and set a new one with a single IP in it, instead of appending to the incoming <code>X-Forwarded-For</code> header. Then your application would be able to safely use the first item in the <code>X-Forwarded-For</code> as the user&apos;s IP address, and better yet, the logic would be robust against adding or removing additional intermediate reverse proxies. This is definitely possible sometimes; for instance, you can configure, Nginx to act in this way by writing <code>proxy_set_header X-Forwarded-For $remote_addr</code> in your config. If this is an option for you, great! In practice, though, you may discover to your frustration that this simply isn&apos;t a behaviour your reverse proxy supports! For instance, <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/x-forwarded-headers.html">AWS load balancers</a> support three different ways of handling the <code>X-Forwarded-For</code> header but <em>none</em> of them is this desired behaviour of discarding the incoming header and setting a new one based on the IP of the incoming request. Given that much of the web runs on AWS these days, that limitation likely means a lot of web developers for whom this alternative approach simply isn&apos;t an option right now. <a href="#fnref4" class="footnote-backref">&#x21A9;&#xFE0E;</a></p>
</li>
</ol>
</section>
<!--kg-card-end: markdown-->]]></content:encoded></item><item><title><![CDATA[I set up a blog. It was annoying.]]></title><description><![CDATA[<p>Today I created this blog. It&apos;s hosted in AWS and runs on the free and open source version of Ghost - although if you take a glance at the <a href="https://ghost.org/">Ghost homepage</a>, you might not even realise that there <em>is</em> a free and open source version, given the prominent</p>]]></description><link>https://markamery.com/blog/i-set-up-a-blog-it-was-annoying/</link><guid isPermaLink="false">643564502f17df1693fb95be</guid><dc:creator><![CDATA[Mark Amery]]></dc:creator><pubDate>Tue, 11 Apr 2023 17:39:24 GMT</pubDate><content:encoded><![CDATA[<p>Today I created this blog. It&apos;s hosted in AWS and runs on the free and open source version of Ghost - although if you take a glance at the <a href="https://ghost.org/">Ghost homepage</a>, you might not even realise that there <em>is</em> a free and open source version, given the prominent mentions of 14 day free trials and the slightly creepy focus on business...</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://markamery.com/content/images/2023/04/Screenshot-from-2023-04-11-14-48-39.png" class="kg-image" alt loading="lazy" width="2000" height="1488" srcset="https://markamery.com/content/images/size/w600/2023/04/Screenshot-from-2023-04-11-14-48-39.png 600w, https://markamery.com/content/images/size/w1000/2023/04/Screenshot-from-2023-04-11-14-48-39.png 1000w, https://markamery.com/content/images/size/w1600/2023/04/Screenshot-from-2023-04-11-14-48-39.png 1600w, https://markamery.com/content/images/2023/04/Screenshot-from-2023-04-11-14-48-39.png 2366w" sizes="(min-width: 720px) 720px"><figcaption>But I don&apos;t have an audience! I don&apos;t want to turn anybody into <em>anything</em>! I just want a blog.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://markamery.com/content/images/2023/04/Screenshot-from-2023-04-11-14-50-50.png" class="kg-image" alt loading="lazy" width="2000" height="1490" srcset="https://markamery.com/content/images/size/w600/2023/04/Screenshot-from-2023-04-11-14-50-50.png 600w, https://markamery.com/content/images/size/w1000/2023/04/Screenshot-from-2023-04-11-14-50-50.png 1000w, https://markamery.com/content/images/size/w1600/2023/04/Screenshot-from-2023-04-11-14-50-50.png 1600w, https://markamery.com/content/images/2023/04/Screenshot-from-2023-04-11-14-50-50.png 2354w" sizes="(min-width: 720px) 720px"><figcaption>Look, I&apos;m sure Ali is very successful, whoever he is, but I don&apos;t want 25 million dollars. I just want a blog.</figcaption></figure><h3 id="why-ghost">Why Ghost?</h3><p>I considered a few possible alternatives to Ghost, and ruled them out:</p><ul><li>I considered Substack, given its current popularity, but at least when I played around with Substack a few months ago, it had absolutely no support for syntax highlighting within code blocks. Since I&apos;m a web developer, I&apos;m probably going to want to post code on my blog at some point, and I&apos;d like to use a system that just gives me nice syntax highlighting out of the box.</li><li>I considered Wordpress, but decided against it due to its terrible reputation for security vulnerabilities. Maybe this is an outdated prejudice on my part and things are much better now. <em>Maybe</em>. But I preferred to give a competitor a chance.</li><li>I considered a self-hosted Jekyll website. Jekyll is somewhat unlike typical blogging platforms in that it&apos;s a static site generator and doesn&apos;t use a database. Rather than providing a WYSIWYG editor in an admin section of the site where you draft and publish posts, instead you write Markdown files in an editor, push them to a Git repo or something, and let Jekyll generate the site HTML from them. This is an elegant paradigm that has the nice side effect that if you put your blog in a public GitHub repo, you get a publicly-accessible revision history on your posts for free.<br><br>However, the thing I dislike about this paradigm is that it&apos;s inherently incompatible with allowing <em>comments</em> on your posts, at least as a properly integrated part of the blog; for that, you absolutely need some sort of database for the comments to get stored in, and some kind of admin panel (to let you log in and moderate comments, unless you fancy doing that via direct database edits or just doing without moderation and letting your comment section be a free-for-all for spammers, pornographers, and bitcoin scammers). Jekyll bloggers who want to allow comments get around this by doing something like <a href="https://disqus.com/admin/install/platforms/jekyll/">including a Disqus widget on their blog</a>, but I kind of hate that; it means that your comments are being hosted on a third-party platform separate from the rest of your blog content and might vanish one day based on that third party&apos;s whims and moderation policies. Seems bad. I&apos;d like the ability to host my own comments (eventually, when I get round to turning that on), and that means I need a traditional blogging platform with a database - like Ghost.</li></ul><p>After ruling out the options above, Ghost seemed like the next-most-famous option - at least, the only remaining option I could recall having heard of - and I didn&apos;t see any major problem with it, so I settled for Ghost. I decided to set up Ghost myself in an AWS instance instead of signing up for Ghost Pro, the paid version that Ghost (the company) will manage for you, because, for goodness&apos; sake, I&apos;m a web developer, and it&apos;d be embarrassing not to self-host. Besides, how hard could it be?</p><p>The answer, it turned out, was &quot;somewhat harder than I expected&quot;.</p><h3 id="fighting-with-mysqland-with-terrible-documentation">Fighting with MySQL - and with terrible documentation</h3><p>First I registered the domain markamery.com in <a href="https://aws.amazon.com/route53/">Route 53</a>, created an Ubuntu t2.small <a href="https://aws.amazon.com/pm/ec2">EC2</a> instance to run the blog (I tried a t2.micro first, but it was so feeble it hung forever when <code>ghost-cli</code> was trying to install Ghost&apos;s dependencies), created an <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html">Elastic IP</a> and associated it with the EC2 instance, and set up an A record to point markamery.com at that IP address.</p><p>Next I set about simply getting Ghost running, both on my local machine (by following the instructions at <a href="https://ghost.org/docs/install/local/">https://ghost.org/docs/install/local/</a>, and on my newly-created EC2 instance by following the instructions at <a href="https://ghost.org/docs/install/ubuntu/">https://ghost.org/docs/install/ubuntu/</a>. The local setup went fine. The <em>production</em> one derailed frustratingly when trying to figure out how to configure MySQL. Here are the pertinent bits of what Ghost&apos;s docs, linked above, currently say:</p><!--kg-card-begin: html--><blockquote>
    <h2>Prerequisites</h2>
    <p>...</p>
    <ul><li>...</li><li>MySQL 8</li><li>...</li></ul>    
    <h3>Install MySQL</h3>
    <p>Next, you&#x2019;ll need to install MySQL to be used as the production database.</p>
    <p>...</p>
    <pre><code># Install MySQL
sudo apt-get install mysql-server</code></pre>
    <p>...</p>
    <h2>Run the install process</h2>
        <p>Now we install Ghost with one final command.</p>

        <pre><code>ghost install</code></pre>
    <h3>Install questions</h3>
<p>During install, the CLI will ask a number of questions to configure your site.</p>
    <p>...</p>
    <h4>MySQL hostname</h4>
    <p>This determines where your MySQL database can be accessed from. When MySQL is installed on the same server, use localhost (press Enter to use the default value). If MySQL is installed on another server, enter the name manually.</p>

    <h4>MySQL username / password</h4>
    <p>If you already have an existing MySQL database, enter the the username. Otherwise, enter <code>root</code>. Then supply the password for your user.</p>

    <p>...</p>

    <h4>Set up a ghost MySQL user? (Recommended)</h4>
    <p>If you provided your root MySQL user, Ghost-CLI can create a custom MySQL user that can only access/edit your new Ghost database and nothing else.</p>
</blockquote><!--kg-card-end: html--><p>So - the <em>recommended</em> approach is to specify <code>root</code> as your MySQL username when asked by the CLI&apos;s setup wizard, and then tell it to create a custom MySQL user that Ghost will use. But what are we supposed to enter as the <em>password</em>? The docs say to <em>&quot;supply the password for your user&quot;</em> - for the <code>root</code> user, presumably - but what is that? They don&apos;t say.</p><p>Well, actually, there <em>isn&apos;t</em> one. Since <em>at least</em> as far back as MySQL 5.7, released in <em>2015</em>, the <code>root</code> user created by the MySQL installer gets configured with the <code><a href="https://dev.mysql.com/doc/refman/8.0/en/socket-pluggable-authentication.html">auth_socket</a></code> &quot;plugin&quot; (authentication methods, even default ones, are &quot;plugins&quot; in MySQL lingo, for some reason). The <code>auth_socket</code> plugin lets you authenticate over a <a href="https://en.wikipedia.org/wiki/Unix_domain_socket">Unix socket</a> instead of over the network, and simply confirms that the Unix user you&apos;re connecting as has the same name as the MySQL user you&apos;re trying to authenticate as. So, if the <code>root</code> MySQL user has <code>auth_socket</code> auth set up, only the <code>root</code> Unix user (or someone with <code>sudo</code> privilege who runs <code>sudo mysql</code>) will be able to authenticate as the <code>root</code> MySQL user.</p><p>There doesn&apos;t seem to be any way to get the Ghost CLI to attempt to connect to MySQL using the root user when you run <code>ghost install</code> (which you&apos;ll note the docs do <em>not</em> instruct you to run with <code>sudo</code>!). Leaving the password blank doesn&apos;t do it and there&apos;s no way to skip the question. So if you follow Ghost&apos;s docs as written, you&apos;re guaranteed to hit an error like this:</p><pre><code>ubuntu@ip-172-31-25-1:/var/www/marks-blog$ ghost install

Love open source? We&#x2019;re hiring JavaScript Engineers to work on Ghost full-time.
https://careers.ghost.org



&#x2714; Checking system Node.js version - found v16.19.1
&#x2714; Checking current folder permissions
&#x2714; Checking memory availability
&#x2714; Checking free space
&#x2714; Checking for latest Ghost version
&#x2714; Setting up install directory
&#x2714; Downloading and installing Ghost v5.42.2
&#x2714; Finishing install process
? Enter your blog URL: https://markamery.com
? Enter your MySQL hostname: localhost
? Enter your MySQL username: root
? Enter your MySQL password: [hidden]
? Enter your Ghost database name: marks_blog_prod
&#x2714; Configuring Ghost
&#x2714; Setting up instance
+ sudo chown -R ghost:ghost /var/www/marks-blog/content
&#x2714; Setting up &quot;ghost&quot; system user
? Do you wish to set up &quot;ghost&quot; mysql user? Yes
&#x2716; Setting up &quot;ghost&quot; mysql user
Nginx configuration already found for this url. Skipping Nginx setup.
&#x2139; Setting up Nginx [skipped]
Nginx setup task was skipped, skipping SSL setup
&#x2139; Setting up SSL [skipped]
Systemd service has already been set up. Skipping Systemd setup
&#x2139; Setting up Systemd [skipped]
+ sudo systemctl is-active ghost_markamery-com
+ sudo systemctl reset-failed ghost_markamery-com
? Do you want to start Ghost? Yes
+ sudo systemctl start ghost_markamery-com
+ sudo systemctl stop ghost_markamery-com
&#x2716; Starting Ghost
One or more errors occurred.

1) CliError

Message: Error trying to connect to the MySQL database.
Help: You can run `ghost config` to re-enter the correct credentials. Alternatively you can run `ghost setup` again.


2) GhostError

Message: Ghost was able to start, but errored during boot with: Access denied for user &apos;root&apos;@&apos;localhost&apos;
Help: Unknown database error
Suggestion: journalctl -u ghost_markamery-com -n 50

Debug Information:
    OS: Ubuntu, v22.04.2 LTS
    Node Version: v16.19.1
    Ghost Version: 5.42.2
    Ghost-CLI Version: 1.24.0
    Environment: production
    Command: &apos;ghost install&apos;

Additional log info available in: /home/ubuntu/.ghost/logs/ghost-cli-debug-2023-04-11T11_45_48_981Z.log

Try running ghost doctor to check your system for known issues.

You can always refer to https://ghost.org/docs/ghost-cli/ for troubleshooting.</code></pre><p>Aggravating! Okay, I thought: I&apos;ll try rerunning the installer, but <em>first</em> I&apos;ll explicitly set a password for the <code>root</code> MySQL user. How do I do that?</p><p>A quick google finds <a href="https://dev.mysql.com/doc/refman/8.0/en/resetting-permissions.html">the MySQL docs page on resetting the <code>root</code> user&apos;s password</a>, which warns that the <code>root</code> user having no password is insecure and links to <a href="https://dev.mysql.com/doc/refman/8.0/en/default-privileges.html">another page</a> that suggests setting an initial password with:</p><pre><code class="language-sql">ALTER USER &apos;root&apos;@&apos;localhost&apos; IDENTIFIED BY &apos;root-password&apos;;</code></pre><p>So I tried that. But wait; nothing changes, and I can&apos;t connect using the password I just &quot;set&quot;. What&apos;s going on?</p><p>The answer is that in order to set a password on a MySQL user, you need to change the auth <em>plugin</em> used by the <code>root</code> user, as well. Otherwise, MySQL just totally ignores your command. The <em>right</em> incantation to use to give the <code>root</code> user a password isn&apos;t the one indicated by the docs pages <em>specifically about giving the <code>root</code> user a password</em>; rather, it&apos;s <em>this</em>:</p><pre><code class="language-sql">ALTER USER &apos;root&apos;@&apos;localhost&apos; IDENTIFIED WITH mysql_native_password BY &apos;my-new-password&apos;;</code></pre><p>The <code>IDENTIFIED WITH</code> clause there changes the auth plugin to one that actually supports password auth. Once we do <em>that</em>, Ghost is finally able to complete its setup.</p><p>Naturally, this is infuriating! The MySQL docs I link to above are outdated to the point that their very existence is harmful; purging them outright would be a good thing. I suspect I am not the only one that they have confused; I have written <a href="https://security.stackexchange.com/a/269655/29805">a related answer on Security Stack Exchange</a>, since these docs currently give wholly incorrect advice about securing the <code>root</code> account.</p><p><em>(12 April 2023 edit: it turns out this isn&apos;t entirely fair. If you install MySQL the way the MySQL docs tell you to, instead of just running <code>sudo apt-get install mysql-server</code> like the Ghost docs tell you to, then <code>root</code> doesn&apos;t get given the <code>auth_socket</code> auth method and the docs make more sense! The overall situation is still crappy since an ordinary user who just installs MySQL from their OS&apos;s default repos is likely to be left confused by the docs, but they&apos;re not outright objectively wrong in the way I believed them to be when I wrote this yesterday.)</em></p><p>Bah! How did I end up wrestling with lying MySQL docs and writing SSE answers? I just wanted a blog!</p><h3 id="the-rest-of-the-setup-process">The rest of the setup process</h3><p>Given my grumbling above, I should give Ghost its due: once I had won my fight with MySQL, I found setting up Ghost to be a fairly painless experience.</p><p>I found it pretty neat that the <code>ghost install</code> wizard has an option to set up Let&apos;s Encrypt for you to manage your SSL certificates; you simply press <code>Y</code> &#xA0;when asked if you want this, it works, and you don&apos;t need to screw about with SSL certificates at all. Maybe other server software also supports this, these days, but I&apos;ve never seen it before now; I expected to have to cron <code>certbot</code> manually. As someone who has been in web development just long enough to remember the days <em>before</em> Let&apos;s Encrypt - when there was no automated generation of SSL certificates for sites on the public internet, and every year or so you&apos;d have to fork out a load of money to some certificate authority and manually swap out the certificates on your servers for new ones - it&apos;s a remarkable improvement in developer experience!</p><p>Here&apos;s a list of further steps I took, for the sake of anyone interested in replicating my setup (perhaps my future self):</p><ul><li>Visited <a href="https://markamery.com/ghost">https://markamery.com/ghost</a> - the URL of the admin UI - and created a user with a password.</li><li>Went to <em>Settings</em> -&gt; <em>Design</em> and selected the <em>Journal</em> theme. (Its styling is the most appealing to me of the build-in themes, although it&apos;s ostensibly a theme for a &quot;newsletter&quot; rather than a &quot;blog&quot; which leads to some annoying issues I address below...)</li><li>Went to <em>Settings</em> -&gt; <em>Membership</em> and changed <em>Subscription access</em> from &quot;Anyone can sign up&quot; to &quot;Nobody&quot; (&quot;Disable all member features, including newsletters&quot;). I might turn this back on at some point - perhaps to allow commenting - but disabling it was the easiest way to turn off this annoying pink banner asking people to &quot;subscribe&quot;:</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://markamery.com/content/images/2023/04/image-1.png" class="kg-image" alt loading="lazy" width="2000" height="511" srcset="https://markamery.com/content/images/size/w600/2023/04/image-1.png 600w, https://markamery.com/content/images/size/w1000/2023/04/image-1.png 1000w, https://markamery.com/content/images/size/w1600/2023/04/image-1.png 1600w, https://markamery.com/content/images/2023/04/image-1.png 2302w" sizes="(min-width: 720px) 720px"><figcaption>No! Don&apos;t subscribe! This isn&apos;t a newsletter! There are no &quot;members-only issues&quot;! I just wanted a blog.</figcaption></figure><ul><li>Went to <em>Settings</em> -&gt; <em>Navigation</em> and deleted the &quot;Sign up&quot; link from the &quot;Secondary navigation&quot; section; otherwise, this shows up in the site footer, and is just a broken link with <em>Subscription access</em> turned off.</li><li>Deleted the autogenerated <em>&quot;Coming soon&quot; </em>post that comes with the blog, as well as the <em>News</em> tag, and updated the <em>About this site page, </em>which by default begs people to subscribe to allow the site to continue to exist.</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://markamery.com/content/images/2023/04/image-2.png" class="kg-image" alt loading="lazy" width="2000" height="1482" srcset="https://markamery.com/content/images/size/w600/2023/04/image-2.png 600w, https://markamery.com/content/images/size/w1000/2023/04/image-2.png 1000w, https://markamery.com/content/images/size/w1600/2023/04/image-2.png 1600w, https://markamery.com/content/images/2023/04/image-2.png 2370w" sizes="(min-width: 720px) 720px"><figcaption>What?! None of this after the first line is true. I just wanted a blog.</figcaption></figure><ul><li>Modified the theme to change the word &quot;topics&quot; to &quot;tags&quot; and &quot;issues&quot; to &quot;posts&quot;. (The docs I&apos;ve been able to find on this aren&apos;t great, but the process is fairly simple: you run Ghost locally and configure it to use the Journal theme, which creates a copy of the theme in the <code>/content/themes/journal</code> directory, and then you modify the HTML templates in there, export your own modified theme via <em>Settings</em> -&gt; <em>Design</em> -&gt; <em>Change theme</em> -&gt; <em>Advanced -&gt; <em>journal</em></em> -&gt; <em>Download</em>, then import it on the live site via the <em>&quot;Upload theme&quot;</em> button on that same <em>Design</em> page.</li></ul><p>Finally, I set up a <em><a href="https://markamery.com/tag/diary/">Diary</a> </em>tag, which I&apos;ll put on posts that are just my record of stuff I&apos;ve worked on but which I don&apos;t expect to be of interest to most of the world. I wanted these to be hidden from the homepage, and found some promising-sounding advice on how to achieve this on Ghost&apos;s forum at <a href="https://forum.ghost.org/t/filter-out-posts-by-tag-on-homepage/14913/4">https://forum.ghost.org/t/filter-out-posts-by-tag-on-homepage/14913/4</a>. The instructions weren&apos;t quite complete as given, but I figured out how to get them working - see <a href="https://forum.ghost.org/t/filter-out-posts-by-tag-on-homepage/14913/6">my post</a> there.</p><p>And that&apos;s it! I have a working blog, and here is my first post! There might be some more tweaks I want to make to it in due course, like enabling comments and setting up backups somehow, but this should get me started. Along the way, I&apos;ve found several things broken in the world that need fixing:</p><h3 id="things-to-fix">Things to fix<br></h3><p>Below is my To Do list of things I&apos;ve discovered along the way to creating this blog that I&apos;m going to go and either fix or ticket.</p><h4 id="1-the-mysql-docs">1: The MySQL docs</h4><p>As I describe above, the docs at <a href="https://dev.mysql.com/doc/refman/8.0/en/resetting-permissions.html">https://dev.mysql.com/doc/refman/8.0/en/resetting-permissions.html</a> and <a href="https://dev.mysql.com/doc/refman/8.0/en/default-privileges.html">https://dev.mysql.com/doc/refman/8.0/en/default-privileges.html</a> are &#xA0;almost a decade out of date and need either deleting outright or updating to reflect the fact that the <code>root</code> user on a new MySQL install uses <code>auth_socket</code> authentication these days. I&apos;m also a little suspicious about the whole <code><a href="https://dev.mysql.com/doc/refman/8.0/en/mysql-secure-installation.html">mysql_secure_installation</a></code> utility, now. None of the four things the docs say it does need doing at all on a fresh install of MySQL on Ubuntu!</p><p><em>(12 April 2023 edit: I dug into this more. See <a href="https://markamery.com/diary/mysql-docs-bugs/">MySQL docs bugs weren&apos;t really bugs</a> in my diary.)</em></p><h4 id="2-ghosts-handling-of-the-root-user-using-authsocket">2: Ghost&apos;s handling of the <code>root</code> user using <code>auth_socket</code></h4><p><code>ghost-cli</code> should be made compatible with the <code>root</code> MySQL user being created with <code>auth_socket</code> auth, since that&apos;s MySQL&apos;s default behaviour, and the docs at <a href="https://ghost.org/docs/install/ubuntu/">https://ghost.org/docs/install/ubuntu/</a> should be updated to indicate what you should enter as the MySQL password if your <code>root</code> user uses <code>auth_socket</code> auth.</p><p><em>(12 April 2023 edit: I ticketed this at <a href="https://github.com/TryGhost/Ghost-CLI/issues/1759?ref=markamery.com">https://github.com/TryGhost/Ghost-CLI/issues/1759?ref=markamery.com</a>)</em></p><h4 id="3-ghosts-wysiwyg-editor-should-support-headings-inside-quotes">3. Ghost&apos;s WYSIWYG editor should support headings inside quotes</h4><p>An annoyance I discovered in Ghost&apos;s editor: if you try to format some text as a heading inside a blockquote (either by selecting text and formatting as a heading via the formatting bar that appears when you select text, or by writing a <code>#</code> symbol at the start of a line within a quote), the blockquote disappears, and similarly if you try to quote an existing heading, it ceases to be heading. The only way to quote content that includes headings - like I did earlier when quoting from the Ghost docs - is to insert a &quot;raw HTML card&quot; and write in HTML instead of using the WYSIWYG editor. This seems crappy and might be easily fixable.</p><p><em>(12 April 2023 edit: per <a href="https://forum.ghost.org/t/paragraphs-inside-blockquotes-koenig-editor/4351/2">https://forum.ghost.org/t/paragraphs-inside-blockquotes-koenig-editor/4351/2</a>, this is working as designed, and the Ghost devs suggest inserting a Markdown &quot;card&quot; to do blockquotes. I hadn&apos;t noticed those; that would&apos;ve been more convenient than HTML. I still think this is sucky UI, but it&apos;s tolerable.)</em></p><h4 id="4-the-initial-request-to-ghostapiadminauthenticationsetup-appears-to-fail">4. The initial request to <code>/ghost/api/admin/authentication/setup/</code> appears to fail</h4><p>Specifically, the request that gets made when you click the &quot;Create account &amp; start publishing&quot; button during setup times out after 1 minute with a 504 response from Nginx. The setup still actually <em>works</em> and a user gets created you can log in is, but something at the end of the process is hanging for a minute.</p><p>I didn&apos;t capture a screenshot of this when I was setting up this blog, so I made another one just to show this:</p><figure class="kg-card kg-image-card"><img src="https://markamery.com/content/images/2023/04/image-3.png" class="kg-image" alt loading="lazy" width="2000" height="1104" srcset="https://markamery.com/content/images/size/w600/2023/04/image-3.png 600w, https://markamery.com/content/images/size/w1000/2023/04/image-3.png 1000w, https://markamery.com/content/images/size/w1600/2023/04/image-3.png 1600w, https://markamery.com/content/images/size/w2400/2023/04/image-3.png 2400w" sizes="(min-width: 720px) 720px"></figure><p>(My best guess - without having looked at the code at all yet - would be that it&apos;s trying to send an email to the address I entered, and failing because there&apos;s no email provider configured yet - that&apos;s not part of the setup process at <a href="https://ghost.org/docs/install/ubuntu/">https://ghost.org/docs/install/ubuntu/</a>.)</p><p><em>(12 April 2023 edit: My guess above was right. See <a href="https://github.com/TryGhost/Ghost-CLI/issues/1760">https://github.com/TryGhost/Ghost-CLI/issues/1760</a>.)</em></p>]]></content:encoded></item></channel></rss>