• Flies and Smoke

    A guard looks out into the distance, and sees a black cloud. “Ah, smoke!” he thinks, “There’s a fire there.”. He remarks as much to his friend, who, slightly more eagle-eyed, identifies the cloud as flies. “Ah,” he thinks, “Then there is not a fire there after all.”

  • Avoiding blank lines in Terraform templates

    You can use Terraform templates to include or omit lines of configuration (or scripts) based on some condition:

    this is always present
    %{ if condition }
    this is present
    %{ endif }
    this is also always present
  • Migrating deprecated aws_s3_bucket in Terraform AWS Provider 4 and above

    Terraform AWS Provider 4.0 introduced a series of new, smaller resources intended to replace the monolithic aws_s3_bucket resource, which was deprecated. While it hasn’t been removed even in 5.0, it’s nice to migrate to the new subcomponents under the assumption it will eventually prevent major version updates.

  • Reflections on a Back Injury

    I injured my back on Wednesday afternoon, picking up a tennis ball from the ground too quickly. This is some notes about the experience.

  • Fixing broken characters in WSL

    By default in WSL, certain characters used by particular applications (e.g. ⚠, the warning sign), show up as a question mark in a box. This is because the default font used, Consolas, does not support these characters.

  • Sometimes language does affect performance

    A common refrain is that if rewriting your application in a different language improves the performance, this is mostly due to being able to rearchitect to remove technical debt, instead of due to the language itself.

  • Choosing the language for the job

    If you’re a developer who knows multiple languages and you’re working on a new task, you may, depending on how modular the codebase is, have the flexibility to pick a new language for it. I think this is something which should be strongly considered.

  • Looking for subdomains

    It’s useful to have a tool that can find subdomains of a given domain. I expect this is generally used for security testing, but I use it to look for development sites that other developers haven’t documented for projects I’m working on.

  • Using Containers for development without DevContainers

    DevContainers are a formal specification for adding JSON config to a Dockerfile to run applications and editors inside a container, and to be able to share the configuration that allows you to do this with others. But even without that, it’s still valuable (and somewhat easier or more flexible) to use containers in your local development.

  • Classifying ChatGPT Exploits

    ChatGPT was recently released to the general public and promptly exploited in a variety of ways. This post serves a brief overview of exploits known to me, sorted into particular categories. Exploits may belong to multiple categories. As OpenAI trains the bots, a broken exploit might be “fixed” by adding a technique from another category.

  • Curling ElasticSearch

    When trying to debug ElasticSearch index issues, it’s generally nicest use Kibana to provide a graphical interface to assist. But sometimes you can’t forward the Elastic instance to a place where you can run Kibana. Sometimes you just have a CLI.

  • Variable shadowing is good

    Variable shadowing is when you give a variable the same name as a variable that already exists. Linters and tools generally complain when you do this (e.g. no-shadow on eslint, mypy or TypeScript will report errors), and the language may forbid in entirely. Despite that, I find that it often makes code easier to read, instead of harder.

  • Azure App Service can fail to deploy due to free plan

    I recently failed to deploy a project to an Azure App Service with a variety of mysterious errors. The project ran fine locally, but:

  • Deliberate Forgetting

    In my third year of university, I tried to forget a specific thing, and succeeded.

  • Coding in Wiki Templates

    My first major programming projects, before university, involved coding using wiki templates.

  • Moving from Json.NET (Newtonsoft) to System.Text.Json

    In mid-2019, Microsoft released the System.Text.Json library, intended to be a faster and lower memory JSON serializer / deserializer than the existing and still most popular Newtonsoft.Json library. At the time, there were many features missing compared to Newtonsoft.Json, and it was difficult to migrate. Now, 3 years on, it is easier.

  • Installing private GitHub packages from CI

    While developing microservices or other projects, it can be useful to extract shared functionality into a library that can be used across all projects. To avoid copying or vendoring, it’s convenient to put this library somewhere external where it can be installed – often an external registry. In the absence of a registry, a library can be installed from source from a GitHub repository.

  • 100% coverage is not sufficient

    I recently posted a pull request that I was certain would work correctly as I had 100% code coverage. I was mistaken.

  • Two Factor is more secure than One Factor

    HSBC sent me an email at the start of February stating that they were simplifying the login process, and that I no longer needed to enter a password (which they call “an answer to a memorable question”), only my Secure Key (a probably time-based one time password).

  • Azote

    Those who have adopted these opinions, represent the atom of azote by the number 1.75. We consider the 5 compounds of azote and oxygen, as composed of 1 atom azote, united with 1, 2, 3, 4, and 5, atoms of oxygen.

    1831, Thomas Thomson, A System of Chemistry of Inorganic Bodies, Volume 1, page 133
  • Questing through the tax office

    Last year, I signed up to file self assessment. I wanted to claim tax relief on charitable donations and VCT investments. The website is tells you what to do in those cases: for charitable donations, either file self assessment or ask HMRC directly, and for VCTs, file self assessment. I knew a few people who filed self-assessment already, so I was expecting the process to be straightfoward. I registered as someone who isn’t self-employed, filled in form SA1 mentioning VCTs and charity in the “other reasons” box, and waited.

  • Moving Kubernetes resources across namespaces

    If you have a series of yaml files without namespaces in their metadata, and add one and reapply the files, an entirely new series of resources will be created in the new namespace. Moving a series of resources, then, involves spinning up a new set of resources, migrating the data across, and taking down the old set.

  • Fixing a hanging git pull

    Occasionally git will hang on a push or pull using an SSH key until it times out. Using HTTP(S) mode instead is a workaround but undesired because it requires faffing about with passwords.

  • Moving from Docker Compose to Kubernetes

    We were running a Task Definition generated from a Docker Compose file in AWS’ ECS when we encountered an issue: we needed to run 11 containers in a single task (as they all needed to talk to each other), but the limit was 10. Looking at the documentation, it seemed that putting in a network they could communicate on in the ECS infrastructure was about as hard as migrating everything to Kubernetes, which would offer other benefits, so we decided to go for that instead.

  • Statement injection in C#

    We have a system that allows a user to write a single statement to be evaluated with a subset of C#: something like

  • Fridays, after the Sprint Ends

    Our sprints start on Monday and last for two weeks, and we release at the end of each one. This means we have some dead time on Friday: we want to start the release early to give us some time to recover if it goes wrong or we find an issue after release, but if it all goes well we don’t want to start the next sprint early, to give us some time off the treadmill.

  • Show complicated filtered data in EntityFramework Core

    We had a requirement for an API to, for a given user, return only records that user had permissions to see. We didn’t want to manually filter in every case, because the filters were quite complicated, and there was no guarantee the developers would remember to call them.

  • GitHub Copilot

    GitHub has recently released a product called Copilot, an advanced form of autocomplete that can complete blocks of code when prompted with a filename, comments, and a function contract (name, inputs and outputs), and potentially any other code above or even below the item being completed.

  • Storing encrypted Ansible vault keys safely with GPG

    Ansible vault keys are required when running Ansible playbooks to decrypt vault secrets. Because of this, it’s useful to store them somewhere accessible from the machine running the playbooks (on disk or in an environment variable as a secret, for example). If you don’t have such a centralised machine, one possibility is to store the keys in the same git repository as the playbooks, encrypted so that only the dev team can access them.

  • COVID and the Manchurian Plague

    After being linked a book describing the response to the Manchurian plague in Moukden, I read a few chapters and found that there are parallels between the Manchurian plague and COVID. Some paragraphs could easily have been written in the modern day. I took the book from the Internet Archive; the Google Books link didn’t work for me.

  • Mathematics at Warwick

    I got my degree (MMath) from the University of Warwick, which I attended from 2010 to 2014. I decided on there because having looked around Cambridge I thought I probably wouldn’t do too well there (in retrospect, likely correctly) and it had a good reputation for maths, in my mind.

  • Collapse EntityFramework Migrations

    On every major version release (or any other time you don’t think you’re ever going to want to roll back to a given version), it’s nice to collapse migrations into a single file so they all run faster and don’t do needless work (e.g. deleting and recreating data, creating and dropping tables, etc.).

  • Migrating an IE7 App

    I recently had the experience of updating a web app designed for IE7 to additionally work in modern browsers (users use predominantly Internet Explorer or Edge). Here I note some changes I needed to make – I couldn’t find any solid documentation on differences in old versions of IE, and much was trial and error.

  • Ansible Patterns

    I’ve been using Ansible for a while now, and there are certain common practices I’ve observed, recorded here in the spirit of Design Patterns.

  • Running PHP scripts targeting localhost for MySQL in Docker

    When targeting localhost as the host for a MySQL connection, PHP will attempt to connect using the configured socket instead of TCP. This can prove a problem when the socket does not exist, such as when you’re running inside a container. A simple solution is to target instead (or alternate appropriate hostname), but this is can involve a big change to an existing codebase, particularly if it wasn’t written to use external or obvious configuration files.

  • Running SQL scripts in dockerized MySQL

    You can redirect a SQL script through a mysql client by running:

  • Debugging ancient PHP scripts

    Modern browsers have a “feature” where if a page takes too long to send anything (headers / body) it times out. I can’t find a way to stop this on the frontend, but it’s annoying for debugging old, slow, half-broken scripts designed for IE7 (in PHP 5.4/Apache, where I am at the moment).

  • Reducing the Number of Colors in a PNG

    My dad recently asked that I reduce the number of colors in a clip-art icon of a server, so he could recolor it to red (downloading the image shows it’s made of stripes of slightly different colors, probably because it’s been saved as a JPG).

  • Continuing to Use Flash in 2021

    Adobe Flash becomes end-of-life in Janurary 2021. Despite that, it’s still used for business apps and tools. Most have upgrades, but perhaps you weren’t able to convince your boss to let you schedule one. This post gives some options on what you can do.

  • Typing Express' res.locals in TypeScript

    TypeScript is a good tool to use to prevent bugs in the inconsistent use of variables by ensuring that they’re all of the same type, at least. By default, this restriction does not apply to Express’ res.locals, which is a map of string to any. This can lead to issues, as res.locals can be set in middleware or earlier routers quite distant from where it is used, which can be hard to notice, and units tests often mock the data, and so it can’t be noticed that way either. This post shows a way to add types to res.locals.

  • Interpreting 'Us' Metaphorically

    Having enjoyed Jordan Peele’s “Get Out”, I checked out his next film “Us”, which I was rather less impressed with. I found it had too many plot holes, required the characters behave foolishly to propel the plot, and like “Get Out” delivered its alternative messages (e.g. on privilege) in a way that was undermined by other parts of the film.

  • Digital Drill Data Systems vs Petrolink: A Hacking Case

    I occasionally enjoy reading legal judgements for interesting interpretations of the law and crazy behaviour companies and people manage to get up to. Digital Drilling Data Systems, LLC v. Petrolink Services, Inc., No. 19-20116 (5th Cir. 2020) does not disappoint. I first looked into this after it was linked off the youtube-dl GitHub DMCA reversal by the EFF.

  • A Successful Dogwhistle, or Inciting Outrageous Outrage

    A billboard was recently put up in Vancouver, Canada with the message “I ❤️ JK Rowling”

  • Getting code coverage for manual or integration tests in NodeJS

    It can be useful to get code coverage for a manual test set in order to check that you’ve covered all the cases for the code you’ve written. Getting this is similar to the process for getting coverage for an integration test.

  • In the COVID lockdown without a smartphone

    Not having a smartphone, I’m more aware of the changes brought about by COVID that assumes everyone has one (that’s fairly recent, and that’s on a data plan or can access local Wi-Fi). My local bus stop no longer posts times, instead having a QR code / NFC which presumably sends you to a website with the times. This is going to be more certain to be accurate – I expect they’re changing them more frequently due to lower use – and can be enhanced with the standard “when will the bus get here” approximations which are common online or on the text screens in more central bus stops.

  • Testing without Dependency Injection: NodeJS

    NodeJS offers the ability to overwrite methods and values in imported modules. This can be useful for testing without dependency injection, for example if you have a large amount of legacy code, and you don’t want to refactor the architecture before the unit tests are in place.

  • Testing without Dependency Injection

    Dependency Injection is a software pattern where you pass dependencies to an object, instead of creating them inside the object. Around a decade ago, it became very popular to use dependency injection for everything – not only for swapping out dependencies at runtime (the problem it was the solution for initially) but for creating a dependency with a particular lifecycle (e.g. singleton or once per session for a webapp), replacing dependencies for testing, and even for instantiating dependencies which were always going to be the same, which were “injected” as the concrete class and “registered” as the same.

  • Importing modules into a browser REPL

    It can be useful to import a module for a single session without saving it as a scratchpad, or to be able to use it against a webpage.

  • Using JavaScript's Proxy to add methods to a data object

    Assume we have a document obtained from a database as a JSON object, and we want to add additional properties, or override existing properties, preferably without modifying the original object.

  • Changes in the Definition of Racism

    From until the 1970s until about the 2010s, the “right” way to oppose racism was to be colour-blind – to avoid considering race as much as possible, and perhaps to even argue that it didn’t exist. Nowadays, the concern is systemic racism: the observation that even our attempts at colour-blindness haven’t lead to equal outcomes, and so the institutions (and the people in them) may be inherently anti-black in a way we didn’t notice before. Recently, “anti-racist” has become suggestive of “pro-black” – treatment like affirmative action, or quota systems, or treating minority students in a way that reflects their unique life experiences are the ways to go. Unfortunately, this conflicts with the legacy definition of racism as “treating people differently depending on their race”.

  • Using RxJS's switchMap to display a loading spinner after a delay

    The problem is to display a loading spinner while requests are in-flight. The complexity is that everything’s event-driven, so the components themselves don’t know when this happens. A solution to this complexity is to use RxJs or some other event management system.

  • Removing boilerplate in REST API tests

    While converting our backend REST API integration tests from Cucumber to JUnit, I saw the chance to simplify our rollback of test data.

  • Notes on Jenkins Declarative Pipelines

    At Lhasa on the Kaptis project we used Jenkins declarative pipelines to build, test and create artifacts after every commit. Here I note down some things I’ve learned.

  • Useful snippets from a reusable Angular Typescript input component

    Angular allows creation of reusable components to simplify development. I created an input component in order to simplify how the HTML form pages looked (it allowed a large block of code to be abstracted and replaced; later it allowed all the error texts to be put in one place instead of being repeated for each field).

  • Notes on Spring Data

    I find the Spring documentation generally comprehensive, but most useful after you already know the right answer to your questions. Here I note down some things I’ve learned this past project.

  • Aborting a Cucumber rerun of failed tests if too many failed

    Cucumber allows you to rerun failed tests by defining a test runner that specifies the features to run to be those with failed last time, as output by the built-in rerun plugin. We wanted to fail the build entirely if too many tests failed the first run, as we found tests weren’t being fixed (they would always fail the first time, refresh the database, and pass the second time: some earlier tests weren’t properly cleaning up after themselves).

  • Using JUnit5 Extensions to allow pushing tests for code not yet implemented

    Edit 18th May 2020: I’d now recommend just using @Disabled("Ticket XX-111") instead of having an entirely separate annotation with different behaviour. This means that you can search your codebase for the presence of disabled tags with closed tickets, and I think the tradeoff of this not being automated is worth the gain of not having another annotation and not running tests every build that aren’t expected to pass. You could automate this with a nightly job, for example: run through, find all references to tickets, look on JIRA to see if they’re closed, maybe make some metrics as to how long tests stay in this state.

  • Incomplete documentation is better than no documentation

    I’ve encountered many systems that were poorly documented (or entirely undocumented) that I’ve had to figure out in the course of my work. Sometimes I get lucky and there’s somebody still in the company who understands what’s going on and can explain it to me. Sometimes there isn’t.

  • Getting DataTables to map to classes in Cucumber version 4 and above

    In Cucumber 2 and below, you could reference classes containing primitive elements in step definitions, and data tables would be mapped automatically.

  • Testing an Angular AsyncValidatorFn using Jasmine

    The Angular documentation on validation mentions that there exists an AsyncValidatorFn counterpart to ValidatorFn, but doesn’t give any details in implementing it other than the function prototype. You can guess that it’s similar to ValidatorFn in the same way that AsyncValidator is similar to Validator, and this turns out to be correct as far as I can see.

  • Debugging by example: Build log liquibase error

    At work on Thursday, we encountered a particularly thorny issue. I think it would be beneficial to write up a log of how I attempted to solve the problem, what went right, what went wrong, and what I could have done better.

  • I lost a day to relative URLs

    We’d been having problems with this feature branch already. It was supposed to change the behaviour on logging in to redirect you to where you’d been just before. It worked fine locally, but failed on the build server.

  • Web Development upskilling: Creating a minimap for an SVG

    I find it easier to learn new skills if I have a project to work on. I knew that for my current project, I want to learn more CSS, vanilla JavaScript and SVG. I thought a project that could involve all three was creating a minimap: take a large SVG image, duplicate it in miniature, and allow the user to drag a square around the miniature replica to move the view around the larger one. Somewhat like the minimap in an RTS like Age of Empires (although that involves a simplification in the smaller map I don’t think I’d go for).

  • On Paul Graham's Bus Ticket Theory of Genius

    I read Paul Graham’s essays when they get linked on the other blogs I read, which is semi-regularly. The Bus Ticket Theory of Genius has just showed up on Hacker News, wherein Graham argues that to do great work requires natural ability, determination, and an obssessive interest in a particular topic, with the last one serving as a good proxy for either of the first two.

  • Basic CSS: Opacity on overlapping components

    Last week, I designed an icon using two separate overlapping SVG icons (a flask with a line through it). This was to represent whether to show or hide assays (on a graph of assays and key event relations), and I wanted to fade out the icon if assays were hidden. I tried this using color: rgba(255, 255, 255, 0.5) for both icons. This didn’t work:

  • Using Git over HTTPS through an SSL briged proxy

    I’ve recently joined an organisation that uses Forcepoint to monitor and SSL bridge all HTTPS connections. This means that connections to GitHub over HTTPS fail, as the GitHub certificate is replaced with Forcepoint’s, with the error message

  • Using Firefox at work

    I use Firefox at work. Unlike the prime focus of their advertising campaigns, I’m not interested in the privacy protections. I use it primarily for two features that haven’t yet made it to Chrome – containers (cookie isolation) and tab hiding.

  • A Use-Case for Monkeypatching Python's stdlib

    Some languages, such as C# and Kotlin, allow the declaration of extension methods – new methods callable on existing classes, without needing access to the original source code. Some languages, such as Python, go further and allow you to modify the behaviour of existing function calls (known as “monkeypatching”).

  • Learning other people's productivity tools

    One of the most important things I learn from demos and show & tells are frequently not what the presenter meant to put across, but all the minor things that show up when they’re setting up their machine, and moving around it to show what they need to. Windows has a way to disable notifications? What’s that editor? Woah, how’d you format that file? Outlook has a dark mode??

  • GitHub tech tests

    Many companies put tech challenges (to be completed when applying for the job) on the public web. If there’s a challenge on GitHub, there’s a good chance that people will put their solutions on GitHub as well, which can be useful to compare your own solution against / take inspiration / blatantly cheat.1

subscribe via RSS