Posts on David Chan - Today I Learned (TIL)

January 22, 2026 - Using .casefold() for Case-Insensitive String Comparison in Python

Thu, 22 Jan 2026 09:43:11 -0800

When you need to compare string in Python, a lot of us reach for .strip().lower() by default to pre-normalize the strings. But it turns out that .lower() doesn’t really work when you start thinking about non-ASCII characters. For example, the German letter “ß” (Eszett) should probably be equivalent to “ss” in a case-insensitive comparison, but .lower() doesn’t handle that correctly. So how can we do better? Python has a method .casefold() that is specifically designed for caseless matching of strings. It takes .lower() a step further by handling more complex cases, including those involving special characters and different alphabets:

July 29, 2024 - Running CRON Jobs on Modal

Mon, 29 Jul 2024 11:31:22 -0700

Recently, I’ve been working to migrate my own personal infrastructure off of Kubernetes, given that it has become quite expensive to run for just a single hosted website. For example, on Linode, the smallest cluster (3 nodes), along with a load balancer, and several disks, was approaching 50$ a month. Indeed, what did I actually use my cluster for? I used it for hosting one-off websites for a couple of days, and then tearing them down, I used it for my own personal website, and I used it for a host of CRON jobs that have to run every hour or every couple of days. The one-off websites I could easily migrate to vercel, and my personal website I migrated to Cloudflare Pages + static site generation (11ty), but the CRON jobs were a bit trickier. Luckily, I found Modal, a service for serverless compute which allows running compute on a schedule (with a bit of configuration).

June 11, 2024 - Using K9s to Manage Kubernetes Clusters

Tue, 11 Jun 2024 08:48:43 -0700

I use Kubernetes (K8s) to manage both my personal website, and some client-facing projects. Recently, however, I’ve discovered something that is making my life a whole lot easier: K9s - a terminal-based UI (think htop) for managing Kubernetes clusters.

Some of the awesome features:

I can actually see all of my pods running at once, their CPU/memory usage, and what their internal IPs are
I can tail logs using a UI, and I don’t have to remember all of the kubectl commands to do so
I can see if things are dead, or died, at a glance.

If you use Kubernetes, I highly recommend checking out K9s. It’s made my life a lot easier.

March 18, 2024 - Adding a tag to an older commit

Mon, 18 Mar 2024 11:02:34 -0700

To add a tag to an older commit, you can use the following command:

git tag -a {{ Tag Name }} {{ Commit Hash }} -m "Message here"

For example, if you wanted to add a tag to the commit with the hash a1b2c3d4, you would use the following command:

git tag -a v1.0 a1b2c3d4 -m "Version 1.0"

This will add a tag called v1.0 to the commit with the hash a1b2c3d4, and the message “Version 1.0” will be associated with the tag.

January 29, 2024 - Appending content to a database using Notion's API

Mon, 29 Jan 2024 15:32:04 -0800

Sometimes, I want to visualize some of my experimental results, or share them with others. One of the easiest ways to do this is using notion: which is capable of handling a variety of data types, and can be published to a relatively nice looking website, with minimal effort.

Step 1: Creating a notion integration

To do this, you first have to create a notion integration, which you can do here. Creating an integration will give you access to an API token, which we can use in the python script to interact with our notion database.

January 24, 2024 - Using UUID for Unique Identifiers in Python

Wed, 24 Jan 2024 15:21:20 -0800

In lots of situations, we may need a unique identifier for an object, for example, when running a database transformation, we may want to create a unique key for each record, or when creating a database, we might want a unique key for each object in the database. In these situations, I’ve seen a lot of people use some variant of the following code:

import random
import string

def generate_random_unique_identifier(length=10):
    return ''.join(random.choice(string.ascii_letters) for _ in range(length))

I’ve certainly been guilty of this myself. However, there are a few problems with this approach:

January 23, 2024 - Detecting COCO Objects with Detectron2 API

Tue, 23 Jan 2024 15:57:00 -0800

Sometimes you need to run an object detector as part of a larger system. One of the best tools for bringing in a pre-trained detector may be the detectron2 framework from Meta.

Installation

To install detectron2, it’s as easy as a pip install:

python -m pip install git+https://github.com/facebookresearch/detectron2.git

Detectron2 API

It’s amazingly easy to being to extract objects from images using their pre-trained models. First, you get a model from the model zoo:

January 22, 2024 - Handling basePath in next.js client components

Mon, 22 Jan 2024 13:32:16 -0800

It turns out that hosting a next.js application in a subdirectory isn’t as easy as it should be. There are two major issues that I’ve run into when doing this: (1) Handling links to local pages and (2) Handling links to static assets.

The right way to handle this is to set the basePath in the next.config.js file. This will cause the next.js router to prepend the basePath to all links:

January 19, 2024 - Cloudfront Egress is pretty cheap!

Fri, 19 Jan 2024 12:53:46 -0800

This isn’t really a TIL, but it turns out that Amazon provides 1TB of free egress from S3 buckets with a CloudFront CDN front-end (instead of the standard 100GB available for free with S3 alone). To get started, I followed this tutorial – but it assumes a pretty high level of prior understanding of AWS, so if you’ve never used CloudFront, AWS, or S3 before, it might be best to google around for another tutorial.

January 11, 2024 - Configuring AWS CLI Endpoint URL

Thu, 11 Jan 2024 10:09:38 -0800

So, for a long time, it wasn’t possible to configure the AWS CLI to use a custom endpoint url by default. Since I usually use Wasabi for my S3 storage, I had to use the --endpoint-url flag every time I wanted to use the CLI, pretty annoying, right? Well, turns out at some point Amazon added a set of useful configs to their CLI. To configure the CLI to use a custom endpoint url, you can add the following to your ~/.aws/config file:

May 12, 2023 - Testing for Multivariate Normality with a Henze-Zirkler Test (In Python)

Fri, 12 May 2023 11:31:55 -0700

If you’re just interested in the code, it’s super easy to use pingouin to do this:

import numpy as np
import pingouin as pg
data = np.random.normal(size=(100, 3))
output = pg.multivariate_normality(data, alpha=.05)
print(output.hz, output.pval, output.normal)

Sometimes, it’s useful to know when a sample looks a lot like a normal distribution. In a single dimension, we can use the function from scipy scipy.stats.normaltest to test whether a sample is normal. This test combines tests from D’Agostino and Pearson which measure the skew and kurtosis of the sample, and report when those differ from a similar normal population. Unfortunately, this test isn’t immediately generalizable to multiple dimensions, which makes a new test necessary. One example is a Henze-Zirkler test, which is based on a non-negative functional $D$ which measures the distance between two distribution functions, and has the property that $D(N_d(0, I_d), Q) = 0$ if and only if $Q$ is a multivariate normal distribution with identity covariance matrix. In practice, the Henze-zirkler test computes a weighted integral of the difference between the empirical characteristic function (ECF) and it’s pointwise normal approximation (in the limit).

December 07, 2022 - How to make a tree with a parent pointer in Rust

Wed, 07 Dec 2022 10:49:03 -0800

I was working on the Advent of Code (day 7) today, which I’m using to learn a bit more about Rust, and while I was able to quickly solve the problem with python (thanks to the fact that python allows references to everything everywhere), I had a hard time solving the problem with Rust, thanks to the inability to keep around references to a “parent” in the tree. This inconvenience is by design: Rust tries to ensure memory safety by forbidding you from doing things that might potentially be unsafe.

December 02, 2022 (2) - Encoding secrets for Kubernetes

Fri, 02 Dec 2022 11:18:25 -0800

Kubernetes uses base64 encoded secrets, but it’s important to make sure that when encoding the secrets, you don’t include any extra newlines - tldr; use echo -n instead of echo

# Works
> echo -n "my secret" | base64
bXkgc2VjcmV0

# Doesn't work
> echo "my secret" | base64
bXkgc2VjcmV0Cg==

Otherwise, you could be in for a world of pain :)

December 02, 2022 - How to use python-dotenv to easily manage env vars

Fri, 02 Dec 2022 10:21:41 -0800

So, I had a project recently where I have a bunch of API keys, but I didn’t want to accidentally commit them to a repo (since that would be bad…). But I also wanted the convencience of just in-lining the code, and not have to worry about sourcing environment variables every time I ran the code (since this would inevitably cause annoyances). To solve this, I found a handy tool - python-dotenv, a package which handles all of the complicated bits.

November 29, 2022 - PyPI Doesn't Allow Git Repo Dependencies

Tue, 29 Nov 2022 09:09:11 -0800

So, we have a package, VDTK which we’re planning to release in a new major version - there’s only one problem… We depend on several packages which do not publish builds to the PyPi repository. In our pyproject.toml file, they are specified as:

dependencies = [
    "...",
    "clip @ git+https://github.com/openai/CLIP.git",
    "mauve-text @ git+https://github.com/krishnap25/mauve.git",
    "en_core_web_lg @ https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.4.1/en_core_web_lg-3.4.1-py3-none-any.whl",
]

Unfortunately, when you upload a package built with dependencies like this, you get the error:

November 18, 2022 - Creating the perfect archetype in Hugo

Fri, 18 Nov 2022 16:17:02 -0800

This TIL site is powered by Hugo, but it’s currently a lot of effort to create the title and name for one of these posts… It would be great if we could have a single command line tool which does this! Each post right now has the following header:

# post-header.yaml
title: 'November 18, 2022 - Creating the perfect archetype in Hugo'
date: 2022-11-18T16:17:02-08:00
showToc: true
TocOpen: false
draft: false
hidemeta: false
comments: false
disableShare: false
disableHLJS: false
hideSummary: false
searchHidden: false
ShowReadingTime: true
ShowBreadCrumbs: true
ShowPostNavLinks: true
ShowWordCount: true
ShowRssButtonInSectionTermList: true
UseHugoToc: true

Most of this information is super easy to deal with - everything after showToc is static, so our template can just write those directly. This means we have only to deal with the title and the date.

November 17, 2022 - Running Single-Server MySQL on Kubernetes

Thu, 17 Nov 2022 16:41:25 -0800

Getting anything running on Kubernetes is a bit of a challenge, but today I was working on deploying MySQL so I could migrate my Ghost blog from v4.x to v5.x. This means creating a mysql instance, a user (for ghost) and any other data that we need to run the deployment. The first thing that we need to do is add a secret which will define the root user password and the password for ghost user: