If you use Brave as your browser and you're also using a Surface Book (or some other Windows detachable with multiple GPUs) then this message is going to spoil your detaching fun:

Please resolve before detaching
The following may need to be closed in order to detach.
Any unsaved changes will be lost.

So why does it happen?

By default, Brave uses the primary GPU which is the one in the base and as it's dependent on that while running then Windows won't let the screen be detached from the base.

And how do I fix it?

In short, tell Brave to use the other (power saving) GPU instead.

In a bit more detail:

Open Settings > Display > Graphics.

From the Add an App dropdown select Desktop app and click Browse, navigate to the brave.exe executable, and select it. By default, it'll be at C:\Program Files\BraveSoftware\Brave-Browser\Application\brave.exe. Now click Options on the Brave Browser card that's added to your options.

Finally select Power saving, save everything and restart Brave.

And now you can detach your screen without closing your browser and as an added bonus your laptop should run cooler and the battery will last for longer too!

Part 2 of a 2-part post on the FTS features available in Postgres and how you might use them to build a simple search API. If you want to follow along head over to the GitHub repo, grab the source data and spin up the included docker config, we'll be searching a database of 50,000 movies. Check out part 1 and the accompanying GitHub repository.

Searching multiple columns

Now what if we want to search across multiple columns, for example our database has an original_title and a title column for each movie. We can concatenate each column tsvector so we have one tsvector to search.

select to_tsvector(title) || ' ' ||  to_tsvector(original_title)
from movies

But what if I want to prioritise search results in certain columns, for example I want matches in title to be ranked higher than original_title.  Of course, Postgres has this covered with the setweight function which we can use to weight the results of any tsvector from A, the highest weighted, to D, the lowest.

select setweight(to_tsvector(title), 'A') || ' ' || setweight(to_tsvector(original_title), 'B')
from movies

Make querying easier

Now we can, for example, search for the movie Hero by its original title.

select title, original_title, 
	ts_rank(to_tsvector(title) || ' ' ||  to_tsvector(original_title), websearch_to_tsquery('英雄')) as rank
from movies
where to_tsvector(title) || ' ' ||  to_tsvector(original_title) 
	@@ websearch_to_tsquery('英雄')
order by rank desc 

This is pretty powerful, but our SQL is starting to get a bit cumbersome, and it would be nicer, and more performant, if we didn't have to build our vectors for every query. We could create a tsvector column on the table and store the weighted vectors and then use triggers to keep that column updated on any change to the table data. Yuck... 🤢

So instead let's take advantage of another awesome Postgres feature Generated Columns.

A generated column is a special column that is always computed from other columns. Thus, it is for columns what a view is for tables.

alter table movies 
add title_search tsvector 
generated always as	(
	setweight(to_tsvector('simple', coalesce(title, '')), 'A') || ' ' || 
	setweight(to_tsvector('simple', coalesce(original_title, '')), 'B') :: tsvector
) stored; 

We've changed the way we call to_tsvector slightly here because our generated column must be immutable so we use coalesce to make sure any null values will be represented as an empty string, and we also specify that we're using the 'simple' text search configuration (more on that in just a moment).

For now, let's just enjoy how quick and easy it is to query our weighted vectors.

select title, original_title, ts_rank(title_search, websearch_to_tsquery('hero')) as rank
from movies
where title_search @@ websearch_to_tsquery('hero')
order by rank desc

A brief discussion of Text Search Configuration

A text search configuration specifies all options necessary to transform a document into a tsvector

We can see a list of the available Text Search Configurations (TSC) for our Postgres instance by querying pg_ts_config.

SELECT * FROM pg_ts_config

The TSCs available by defaults are each focused on a language and are configured to use a language specific dictionary. So far, we have always used the simple TSC when calling to_tsvector and *to_tsquery. This TSC 'simply' converts the input tokens to lower case and discards common stop words.

select to_tsvector('simple', 'To fish the fishes a fishy fisher fished')

If we use a language specific TSC (english) instead then many more, English, stop words are discarded and each word is normalized so that, for example, the plural of fishes would be normalized to fish.

select to_tsvector('english', 'To fish the fishes a fishy fisher fished')

This has many benefits, first it saves space because fewer normalized values (lexemes) are stored. Second by normalizing words to lexemes based on the dictionary we are able to take advantage of stemming so that, for example, a search for fishes would match with fish, fishes, fished. Thirdly it means that all of this can be done in a way that supports whichever language our documents may be stored and searched in.

You can customize TSCs or build your own to support synonyms, thesauruses and much more, I'd highly recommend reading the Text Search Configuration documentation if you'd like to learn more.  

Will it index?

OK back to searching our movie database and at this point we need to start thinking about performance. Our database only contains a few thousand records, so our query timings are all measured in milliseconds. But what would happen if we were querying hundreds of thousands or millions of rows? Put simply, performance would suck because right now we're doing a sequential scan for every query which requires iterating through every row of the table. If we analyse our query, we can see a clear warning about this.

If only Postgres had some kind of special index designed for speeding up full text search that would be awesome, and of course it does ... GIN indexes.

create index idx_search on movies using GIN(title_search)

You can see the huge reduction in the Timings above, we've gone from around 7.5 ms to less than 0.04 ms, and at this point we have a search query that can be fast enough for a real production environment.

Addendum

Just one more thing before we're done. Intuitively if I was to run a title search for 'rush' I would expect the highest ranked result to be the movie with the title 'Rush' but, by default, the ts_rank function doesn't consider document or query length.

select title, ts_rank(title_search, websearch_to_tsquery('rush')) as rank
from movies
where title_search @@ websearch_to_tsquery('rush')
order by rank desc

If we pass a normalization option to ts_rank we can specify how the document's length should impact its rank and make our results more intuitive.

select title, ts_rank(title_search, websearch_to_tsquery('rush'), 1) as rank
from movies
where title_search @@ websearch_to_tsquery('rush')
order by rank desc

-- The End

Everyone needs Full Text Search (FTS) at some point and the most common way to get it is to push your data to Elastic/OpenSearch, Solr, or something else built on top of Lucene. It's a solution that works but adds extra complexity, cost, and consistency challenges. So, what if I told you that, if you're using PostgreSQL, powerful FTS is available to you out of the box?

Play along

If you want to follow along head over to the GitHub repo and spin up the docker file and included source data.

GitHub - admcpr/postgres-full-text-search-is-better-than: A PostgreSQL database and all the data needed to follow along with the `Postgres Full Text Search is better than ...` blog posts

A PostgreSQL database and all the data needed to follow along with the `Postgres Full Text Search is better than ...` blog posts - GitHub - admcpr/postgres-full-text-search-is-better-than: A Postgr…

GitHubadmcpr

Postgres text search the wrong way️

We're going to build the SQL for a hypothetical API endpoint that searches movies. Let's start by trying to search for a well-known film by its title the wrong way™️. We can use like to do basic pattern matching so a simple solution might be to split the search input into individual words and wrap them with %.

select title from movies where title like '%star%' or title like '%wars%' 

Two results, but not Star Wars though because of course I've forgotten that like is case sensitive.

select title from assets where title ilike '%star%' or title ilike '%wars%' 

Now I get 36 results and Star Wars is in there, but the basic pattern matching provided by ilike doesn't rank our results and is never going to help us with more complicated concepts like stemming.

Help me tsvector you are my only hope

So enough of what won't work let's look at some tools that will. We're going to start with tsvector which, to quote the Postgres documentation:

A tsvector value is a sorted list of distinct lexemes, which are words that have been normalized to merge different variants of the same word.

We can use the to_tsvector function to parse text and turn into a tsvector for us.

select to_tsvector('I am altering the deal. Pray I don''t alter it any further!');

Now let's do that to some of our movie titles.

select title, to_tsvector(title) from movies limit 20;

Now we can use the to_tsquery function to query the tsvector of the title instead of querying the title directly. Let's start simple.

select title 
from movies
where to_tsvector(title) @@ to_tsquery('Star')

It would be nice if we could order the results by how well they match our query so let's use the ts_rank function to return a value for that.

select title, ts_rank(to_tsvector(title), to_tsquery('Star')) as rank
from assets
where to_tsvector(title) @@ to_tsquery('star')

So, let's find Star Wars.

select title, ts_rank(to_tsvector(title), to_tsquery('star wars')) as rank
from movies
where to_tsvector(title) @@ to_tsquery('star wars') 

Oh, come on! It was all going so well but a restriction of tsquery is that it:

must consist of single tokens separated by the tsquery operators & (AND), | (OR), ! (NOT), and <-> (FOLLOWED BY)

In short, no spaces allowed. We could fix this simply by changing 'star wars' to 'star<->wars' but then if we were, for example, building a search API endpoint we'd need to parse the input, split and rejoin it. Instead let's just use plainto_tsquery which will generate a tsquery from plain text and now we can find Star Wars 🎉

select title, ts_rank(to_tsvector(title), plainto_tsquery('star wars')) as rank
from movies
where to_tsvector(title) @@ plainto_tsquery('star wars')
order by rank desc

More powerful search syntax

As tsquery supports operators it would be nice to expose this functionality via our API so that a user could execute more complicated searches. We could either expect our users to learn the tsquery syntax or we could parse the incoming query and build a valid tsquery for them. Fortunately, though we don't have to do either of those because the websearch_to_tsquery function lets us support the following popular syntax for free.

• "quoted text": text inside quote marks will be converted to terms separated by <-> operators. Also, stop words are not simply discarded, but are accounted for by inserting <N> operators rather than <-> operators.
• OR: the word “or” will be converted to the | operator.
• +: a plus will be converted to the & operator.
• -: a dash will be converted to the ! operator.

For example, select websearch_to_tsquery('"star wars" -clone') would give us a query of 'star'<->'war'&!clone' to search for results that include Star Wars but exclude Clone. Or if the user was only interested in Star Wars movies about Clones.

select title, ts_rank(to_tsvector(title), websearch_to_tsquery('"star wars" +clone')) as rank
from movies
where to_tsvector(title) @@ websearch_to_tsquery('"star wars" +clone')
order by rank desc 

Part 2

In the second, and final, part of this post we cover searching multiple columns, simpler queries, performance and more.

Postgres Full Text Search is better than ... (Part 2)

In which I show how ElasticSearch, Solr or another search specific database is probably unnecessary for a lot of PostgreSQL users ... Part 2.

/admcprAdam Cooper

I used to hate public speaking; I still do but I used to too (apologies to Mitch Hedberg).  But as I have spent less of my work time building and deploying software and more of it sharing why and how software is built and deployed, I've needed to get better at communicating to a room, or Zoom, full of people.

So, as it's something I must do I've tried to learn how to be better at it which helps make me feel more comfortable and even sometimes enjoy it. The single most helpful resource I've ever come across as I worked on that was this lecture given by Patrick Winston at MIT. It's entertaining, interesting and offers insights and tips on how to give a good presentation while giving a great presentation. A couple of choice snippets that I always try to remember:

• Tell people what they're going to know at the end of your talk that they didn't know at the beginning of your talk
• Cycle over your subject to make sure you cover it clearly
• Ask questions of your audience
• Sum up information within your talk sometimes to help listeners re-engage
• Questions are the worst way to end a talk
• Don't start with a joke

I considered trying to sum up the video in this blog post but honestly there's so much that's useful I would never do it justice and it's actually a very entertaining watch. Instead, I'll advise that you invest an hour of your life watching the video and in return, to quote from that video:

By the end of the next 60 minutes, you'll have been exposed to a lot of ideas, some of which you'll incorporate into your own repertoire, and they will ensure that you get the maximum opportunity to have your ideas valued and accepted by the people you speak with.

If you visit the MIT OpenCourseWare page which has the video available to watch or download plus transcripts and more.  

How to Speak | How to Speak | Supplemental Resources | MIT OpenCourseWare

This page includes the full How to Speak video recorded in January 2018, along with related content and commentary

MIT OpenCourseWare

Alternatively, you can watch the video on youtube.

Is a hippopotamus a hippopotamus... or a really cool opotamus?
- Mitch Hedberg

Recently I bought a bundle of graphic novels written by the fantastic Neil Gaiman, author of The Sandman, Coraline, many more.

Those novels were delivered as gigantic pdf files most of which were well over 1GB and that meant the old iPad I wanted to read them on was just not able to open them. Obviously, the ideal solution in this scenario is to buy a new iPad, yay!

But what if I could solve this problem without spending almost a thousand euros... perhaps I could just optimise these pdf files. Now I know that Acrobat can do that but then I need to spend hundreds of euros on an Adobe subscription and, honestly, I'd rather buy a new iPad. I did try opening the pdf files in every browser and printing to pdf, but that just killed the pdf printer.

So, I did some searching for open-source pdf tools and I was reminded of ghostscript which is 'an interpreter for the PostScript® language and PDF files' that has been around for over 30 years. And not only is it great at converting pdf files, but it has a specific switch, dPDFSETTINGS=/ebook, that can be used to output PDF files optimised for an e-reader. Here's what I did.

Install ghostscript

winget install ArtifexSoftware.GhostScript

brew install ghostscript

sudo apt install ghostscript

Convert the PDF files

gswin64c.exe -sDEVICE=pdfwrite -dPDFSETTINGS=/ebook -dNOPAUSE -dQUIET -dBATCH -sOutputFile="output.pdf" input.pdf

ghostscript -sDEVICE=pdfwrite -dPDFSETTINGS=/ebook -dNOPAUSE -dQUIET -dBATCH -sOutputFile="output.pdf" input.pdf

And once the conversion process completed, it takes a while, I had gone from having a 1.3GB pdf that my iPad couldn't even open to a 68MB pdf that opens in seconds.

There are a lot of switches available for the conversion process if your pdf is more complicated than mine were.

The wonderful Let's Encrypt makes it easy to easy to generate and install SSL certificates for free using the certbot ACME client on a webserver. But what if, like me, you want to generate and install your certs on a server behind a firewall so that you can use subdomains in your home network while keeping that network private.

Yep, they make that easy too because certbot supports dns challenge as a method of verifying your ownership of a domain and provides dns plugins for every major DNS provider.

How DNS challenge works with certbot

The certbot docs provide a much better explanation of this than I ever could but TLDR:

1. Certbot requests a certificate from the Let's Encrypt servers
2. Let's Encrypt servers return challenge contents
3. Plugin creates a TXT DNS record containing the challenge contents
4. Let's Encrypt servers validate the TXT DNS record
5. Let's Encrypt servers issue certificate

Let's use docker

In my case I use Cloudflare as my DNS provider and I'm going to generate the cert on my trusty Synology NAS. Now I could manually install certbot, it's dependencies and the Cloudflare plugin, but the Synology has Docker installed and there's a Docker image for the Cloudflare plugin so that's much simpler.

Getting started

The docker image needs to match a couple of letsencrypt volumes inside /var/lib and /etc/ so we'll start by creating them.

sudo mkdir /etc/letsencrypt
sudo mkdir /var/lib/letsencrypt

Set up Cloudflare credentials

To enable certbot to create TXT records via the Cloudflare API we'll need to create an API token via the cloudflare dashboard. We choose to create a token and select the Edit zone DNS template and create the token for my domain.

Now we have the token let's create a cloudflare.ini file that certbot will read the token from.

sudo vi /etc/letsencrypt/cloudflare.ini

The file contents should have the following format.

# Cloudflare API token used by Certbot
dns_cloudflare_api_token = 0123456789abcdef0123456789abcdef01234567

Now we can run the docker container passing in the location of the ini file containing the api token and the name of the domain to generate a certificate for. I'm using *.admcpr.com to generate a wildcard cert so I can use this cert for lots of different subdomains in my local network.

sudo docker run -it --rm --name certbot \
            -v "/etc/letsencrypt:/etc/letsencrypt" \
            -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
            certbot/dns-cloudflare \
            certonly --dns-cloudflare \
                     --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \
                     -d *.admcpr.com

Once this command completes the cert can be found in /etc/letsencrypt/live/$domain where $domain is the name of the domain the cert was generated for, so for me that's /etc/letsencrypt/live/admcpr.com. Now we can grab that cert and install it wherever we want to have SSL on a subdomain.

Next steps

Let's Encrypt certs are valid for 3 months so I'll need to setup some way to automate renewal and find a more secure place to store the Cloudflare API key.

#TLDR use PowerShell to install an extension.

If, like me, you're both and iPhone and a Windows user you've probably come up against a friendly message telling you that.

The HEVC Video Extension is required to display this file in full resolution. Download and Install it now.

High Efficiency Video Coding aka HEVC aka H.265 aka MPEG-H Part 2 is a video compression standard from those nice people at the MPEG-H standards group. It's the default video compression used when you're filming video on iOS and as a modern standard is of course natively supported on macOS, iOS, Android, and Windows not Windows. Yes, that's right despite making a profit of more than 20 billion dollars in 2021 cash strapped Microsoft can't afford to bundle the ability to play videos inside it's plucky upstart operating system. Or perhaps it can afford to, but it was too busy stuffing junk like Candy Crush into my start menu.

You may have clicked that download link and been disappointed to find that although you can 'Download and install' the extension Microsoft expects you to pay for the privilege. You used to be able to get around this by just installing the 'HEVC Video Extensions from Device Manufacturer' but conveniently those extensions are currently unavailable in the Microsoft Store.

PowerShell to the rescue

You can install the extensions from a PowerShell session without needing to redeem a code with the following command.

start ms-windows-store://pdp/?ProductId=9n4wgh0z6vhq

This will pop open the Microsoft Store page for 'HEVC Video Extensions from Device Manufacturer' and allow you to hit the Install button.

Disclaimer

Using PowerShell seems to skip straight to installation ignoring the store's requirement for you to have a code to redeem, I don't know if it violates any licensing.

The cost of the HEVC extensions in the store is about 99 cents was it really worth the effort for me to write this post and you to read it? Probably not, but one thing that is for sure is that Microsoft could easily eat this cost, keep its users happy and stop forcing them to jump through hoops to have basic functionality.

How to jazz up your WSL shell with fish and oh-my-fish.

Install a fancier shell

Bash (Bourne Again Shell) is the default shell in most Linux distributions so when you fire up WSL, you'll find yourself at a bash prompt. While it's a great default, it's pretty dull to look at and out of the box is missing features that I really love such as case insensitive autosuggestion and tab completion.

Fortunately, fish shell comes with these features, and plenty more so let's start by installing it. I'm using Ubuntu here, but you can find installation instructions for most distros on the fish homepage.

sudo apt install fish

Once apt's done its thing when I run the fish command, I should jump into a friendly new fish shell complete with helpful auto complete and even some nice git integration.

There are of course plugin/package managers for fish that extend the core functionality. Personally, the only extra functionality I need is easy prompt theming so let's install oh-my-fish for that.

curl https://raw.githubusercontent.com/oh-my-fish/oh-my-fish/master/bin/install | fish

Now I can list and select available themes (I can also preview them on github) and I can install my favourite, sashimi, as well as the dracula colour scheme.

omf install sashimi
omf theme sashimi
omf install https://github.com/dracula/fish

And now I have my shell just the way I like it.

Finally, I'll set fish as my default shell, so it loads whenever I open this wsl distro, if fish was installed to a different location by your distro, you can always find it using the whereis fish command.

chsh -s /usr/bin/fish

The fish shell is amazing

I’ve been lurking the fish shell for a couple of years now (and thenushell but it is another story foranother time). Not so long ago, I decided to try it, and it’s simply… amazing. If I had to state one feature that makes me like to use it, it’ll be theautocompletion, hands down. It’s the first t…

Rufus Mairo

It used to be the case that acquiring royalty free stock images or illustrations meant paying hundreds of dollars. Fortunately, thanks to the world of open source and public domain art and design, those times are behind us and there are now hundreds of thousands of high-quality images freely available for both personnel and commercial use. This is a list of some of my favourite sites for finding photos, illustrations and videos that are free to use for personal or professional projects.

Pexels

License: Free to use, attribution not required

Types: Photos, Videos

Pexels is a site where contributors upload their photos and videos to share with the community and the world at large. To encourage contributions Pexels runs regular challenges with cash or sponsored prizes. Their catalogue contains over two hundred thousand photos along with thousands of short video clips.

Any drawbacks?

Not really.

Artvee

License: Free to use, attribution not required

Types: Artworks, Illustrations

Artvee aggregates tens of thousands of scanned art and design images that have been made available by museums and libraries around the world including the Smithsonian and the Rijksmuseum. You can browse by category as well as search by artist or keyword.

Any drawbacks?

Copyright for public domain art is complicated and country specific, everything on artvee is public domain in the US and EU but not necessarily globally.

rawpixel Public Domain

License: Free to use, attribution not required (CC0)

Types: Artworks, Illustrations

Rawpixel is a marketplace for stock imagery and design resources who also provide a collection of public domain art and design resources like Artvee.

Any drawbacks?

Copyright for public domain art is complicated and country specific, everything on rawpixel Public Domain is public domain in the US and EU but not necessarily globally.

You must sign up for an account to download images.

Unsplash

License: Free to use, attribution not required

Types: Photos, 3D Renders

The elder statesperson of the open-source image world Unsplash boasts over two million photos contributed by the site's users.

Any drawbacks?

Not really.

Undraw

License: Free to use, attribution not required

Types: Vector Illustrations

Launched in 2017 Undraw is an incredible collection of more than 1200 vector illustrations single handedly created by the designer Katerina Limpitsouni. You can customise the colours of the illustration live on the website and download them in .svg or high resolution .png format.

Any drawbacks?

Nope.

ManyPixels Illustration Gallery

License: Free to use, attribution not required

Types: Vector Illustrations

ManyPixels is an on-demand graphic design service that also provide a free illustration gallery. You can customise the color of the illustrations and download them in .svg or .png format.

Any drawbacks?

Not really.

PixaBay

License: Free to use attribution not required

Types: Photos, Illustrations, Vector Illustrations, Videos, Music, Sound Effects  

Pixabay is a community similar to Pexels and Unsplash where almost two and a half million images, videos, music and sound effects have been uploaded by creators around the world.

Any drawbacks?

You have to log in to download the highest resolution images.

DrawKit

License: Free to use, attribution not required

Types: Vector Illustrations, Icons

DrawKit is a project from designstripe that offers both free and premium collections of icons and vector images. Their free kits include more than two hundred illustrations and three hundred icons each available in 4 distinct colour styles.

Any drawbacks?

You must sign up for an account to download.

There are some frustrating prohibitions in the license e.g. downloads can't be used in web templates or app templates.

Pixeltrue - Free Illustrations

License: Free to use attribution not required

Types: Vector Illustrations, Vector Animations

Pixeltrue is a small commercial illustration team who also make more than five hundred illustrations and animations available for free.

Any drawbacks?

There are some frustrating prohibitions in the license e.g. downloads can't be used in web templates or app templates.

When a docker build is failing for me I find that a great way to understand what's causing the failure is to jump into the container in an interactive terminal and have a poke around. This used to be easy because Docker would build an image and cache it for each step in your dockerfile and output the id for that image. Since  BuildKit became the default build tool those images are no longer made available and you just see an output like this.

Now you could disable BuildKit in your docker config but that would make all your builds worse™ so instead what I do is temporarily disable BuildKit by setting the DOCKER_BUILDKIT environment variable to 0. You can easily do this on the same line as your build command whatever shell you're using.

Powershell

$env:DOCKER_BUILDKIT=0; docker build .

Linux/macOS

DOCKER_BUILDKIT=0 docker build .

Windows CMD

set DOCKER_BUILDKIT=0& docker build .

This give will tell docker to use the legacy build tooling and so I'll get the id of each  image for each step.

So for the example above I can see that the last successful step is 1/2 and it produced an image 14119a10abf4. I can run that image and connect to an interactive shell using the following command and debug away.

docker run -it 14119a10abf4 sh

A Windows ISO

There's plenty of other places you can find one but personally I would do one of the following.

1. If you have a Microsoft account that's enrolled in the insiders program you can download an insider preview ISO.
2. If you just want a stable current version of you can download the media creation tool for Windows 10 and use it to generate an ISO.

Installing the new version

With the ISO downloaded let's start by mounting it, you could just right-click and choose Mount or you could use a powershell command instead.

Mount-DiskImage -ImagePath "c:\Windows-Insider.iso"

Now the ISO is mounted let's have a quick look at my filesystem looks.

Get-PSDrive -PSProvider FileSystem

You can see I've got Windows installed on the C: drive and F: is the mounted Windows install ISO. D: is where I'm going to install my second version of Windows and E: is a spare.

We're going to use the Deployment Image Servicing and Management (dism ) to do this and the first job we'll use it for is finding out which images are available in the ISO using the Get-WimInfo command. Depending how that ISO was created that info is stored either in a .wim file or a .esd file.

If you used option 1. above it will be in a install.wim file so we use the comman

 # List available editions for .wim based ISO
 dism /Get-WimInfo /WimFile:F:\Sources\install.wim

If you used option 2. above it will be in an install.esd file so we use the command:

 # List available editions for .esd based ISO
 dism /Get-WimInfo /WimFile:F:\Sources\install.esd

This will print a list of all the editions of windows available within the ISO and the index of each, here's a snippet of the output I get.

I'm going to install the image at Index 6 (Windows 11 Pro) from the F drive to the D drive.

dism /Apply-Image /ImageFile:F:\Sources\install.wim /Index:6 /ApplyDir:D:\

This will take a while, but you should see a nice progress bar like this:

Once complete we just need to add the new installation to the boot menu.

bcdboot D:\Windows

And, optionally, set its description.

bcdedit /set {default} description "Windows 11 Insider Preview"

Now when you reboot, you'll get the choice of which Windows version to boot.

By default, and for reasons that I'm sure make sense to someone, wsl distributions launch at your windows user's home path which for me is /mnt/c/Users/adamc.

I definitely don't want that,  the performance of windows shares in wsl right now is not great and it's never going to be as fast as working inside the distro's file system. So I always aim to work inside the distro's filesystem and I want to start in my users home directory. WSL gives the option to specify the startup path when launching so running wsl ~ will start the default installed distro in it's home directory.

If you have multiple distros installed you can specify the distro as well as the path when you start for example wsl ~ -d Debian will open my Debian distro in my Debian user's home folder.

With Windows Terminal you can configure the command that launches your distro via the commandLine parameter of a profile. I have my terminal configured to launch Ubuntu and Debian like below which means when I launch either I always start where I want to be.

{
	"guid": "{2c4de342-38b7-51cf-b940-2309a097f518}",
	"hidden": false,
	"name": "Ubuntu",
	"source": "Windows.Terminal.Wsl",
	"commandline": "wsl ~ -d Ubuntu"
}
{
	"guid": "{58ad8b0c-3ef8-5f4d-bc6f-13e4c00f2530}",
	"hidden": false,
	"name": "Debian",
	"source": "Windows.Terminal.Wsl",
	"commandline": "wsl ~ -d Debian"
}

You can also specify the starting directory when configuring a profile. This has to be a windows path but because wsl exposes every distro under the windows network path \\wsl$\  you can access any path on any distro. So another way for me to configure my profiles would be the config below.

{
	"guid": "{2c4de342-38b7-51cf-b940-2309a097f518}",
	"hidden": false,
	"name": "Ubuntu",
	"source": "Windows.Terminal.Wsl",
	"startingDirectory": "\\\\wsl$\\Ubuntu\\home\\adam"
}
{
	"guid": "{58ad8b0c-3ef8-5f4d-bc6f-13e4c00f2530}",
	"hidden": false,
	"name": "Debian",
	"source": "Windows.Terminal.Wsl",
	"startingDirectory": "\\\\wsl$\\Debian\\home\\adam"
}

I often clone a repo on GitHub or GitLab make some changes to it locally and then think I'd like to contribute my changes back to the repo. At that point I always wish I could remember the steps to switch to a fork, so for the benefit of my future self this time I wrote down the steps.

Step 1 - Clone a repo

Most recently I thought I'd have a look at how to contribute to the new winget package manager.

git clone https://github.com/microsoft/winget-pkgs

Step 2 - Make some changes

I updated the version of QuickLook and was ready to push and create a PR when ...

Step 3 - Realise I should have forked first

At this point I remember I have no permissions on this repo, and I should have started from a fork 🤦‍♂️

Step 4 - Fork

So off to GitHub and hit the Fork button which gives me a forked repo at https://github.com/adam7/winget-pkgs

Step 5 - Rename my origin repo to upstream

I'm going to want my fork to be origin so I can push there so let's rename the current origin.

git remote rename origin upstream

Step 6 - Make my fork the origin

Now I can just point origin at my newly forked repo via SSH:

git remote add origin git@github.com:adam7/winget-pkgs

or via HTTPS:

git remote add origin https://github.com/adam7/winget-pkgs

Step 7 - Fetch from my new origin

git fetch origin

Step 8 - Set origin main (or master)

Set the upstream of my main branch to point to my fork.  

git branch --set-upstream-to origin/main main

Or if it's an older repo with master instead of main as the default branch.

git branch --set-upstream-to origin/master master

Step 9 - Push to my fork

And now I can push to my forked repo and create a PR.

git push

Step 10 (Optional)

Write this blog post

Running Docker on Windows has been easy for a long time, but it has always needed to run inside a Hyper-V virtual machine. There are at least two, and probably more, problems with this approach:

1. Hyper-V requires Windows Professional, a ludicrous limitation, which made docker (and lots of other great tools) unavailable to Windows Home users.
2. The CPU and memory resources allocated to the Hyper-V VM and therefore to your docker containers must be managed by you.  Assign too many and you'll lose access to those resources from your Windows OS, assign too few and you'll see performance issues in your containers.

Thanks to the Windows Subsystem for Linux 2 though it's now possible to run docker in a WSL distribution and avoid the need for Hyper-V altogether.

Stuff you'll need

1. Windows 10 Build 19041 or later.
2. Docker desktop version 2.2.0.5 or later.
3. WSL 2 installed.

Before using WSL

By default, docker desktop currently uses Hyper-V which means if you install it today and start it up, behind the scenes it will spin up a VM. If you run Get-VM from PowerShell you can see that VM has the rather un-mysterious name DockerDesktopVM.

In my case docker has assigned 2GB of memory to the VM and if I run Get-VMProcessor I can see that my docker VM has two processors assigned.

And if I run docker info I can see that those are the resources available to my docker server.

If I open up docker desktop Settings->Resources tab, I'll see the same settings and from here I can manually configure those available resources.

Enable Docker WSL

So that's a quick overview of how things work running docker in Hyper-V, now let's see how things are different with WSL. To enable the WSL engine, I just need to go to Settings -> General and turn on Enable the experimental WSL 2 based engine.

Once I Apply & Restart docker will restart and my DockerDesktopVM will no longer be running.

Now instead I have two WSL distros running which I can see by running wsl -l -v.

Two distros ... that's seems strange, why does docker use two WSL distros when it only needs one Hyper-V VM? The name is giving us a clue, the docker-desktop-data distro exists as a storage for images and configs, as well as the Kubernetes data store. Those are used by the docker-desktop distro, the same result is achieved when docker is run under Hyper-V by mounting a VHD (Virtual Hard Disk) in the Hyper-V image but mounting a VHD isn't possible with WSL2 yet.

But moving on to more exciting news let's run docker info.

Now I can see that docker has access to 8 CPU cores and 13GB of the 16GB installed in my laptop. Because docker is now running in a WSL distro it gets access to all the CPU cores and by default 80% of the RAM. This is the shared across all distros running under WSL 2 and the allocation of resources will be handled dynamically for you by WSL.

If you want to modify config settings for WSL 2 to explicitly limit the available resources you can do that by creating a .wslconfig file inside your user profile directory and setting any/all of the parameters below. Personally, though I've found that just leaving WSL alone to manage these for me has worked pretty smoothly.

[wsl2]
kernel=<path>              # An absolute Windows path to a custom Linux kernel.
memory=<size>              # How much memory to assign to the WSL2 VM.
processors=<number>        # How many processors to assign to the WSL2 VM.
swap=<size>                # How much swap space to add to the WSL2 VM. 0 for no swap file.
swapFile=<path>            # An absolute Windows path to the swap vhd.
localhostForwarding=<bool> # Boolean specifying if ports bound to wildcard or localhost in the WSL2 VM should be connectable from the host via localhost:port (default true).

# <path> entries must be absolute Windows paths with escaped backslashes, for example C:\\Users\\adamc\\kernel
# <size> entries must be size followed by unit, for example 8GB or 512MB

Accessing docker from another WSL distribution

Navigating to Settings -> Resources in docker desktop now that WSL integration is enabled I see can see that there's no longer an Advanced tab for configuring docker resources, which makes sense given what we've just talked about. I also see that there's a new setting available for WSL Integration.

In this case I've installed an Ubuntu distro from the Windows store, but you might see different distros listed or even none. If you don't see any listed even though you have a distro installed, you'll need to upgrade your distro to WSL 2. Toggling this switch and then hitting Apply & Restart install the docker client tools in my Ubuntu distro and give it access to the docker server running inside my docker-desktop distro.

If I launch my Ubuntu distro and run docker info I can see that I have access to the same server that's available to me from Windows.

For a fun way to prove that I can run a container from inside Ubuntu.

Switch to Powershell in Windows and when I list my docker containers there I'll see the exact same container running.

Wrapping up

In summary, for me the new WSL backend for docker makes docker so much nicer to work with on Windows. I get all the performance I need along with sweet integration across Windows and any Linux distributions I have installed. On top of this docker now gets the ability to run on Windows Home, Microsoft have made that impossible for far too long but this a very welcome change.

If you want to find out more about the details of the docker desktop WSL 2 backend I thoroughly recommend Introducing the Docker Desktop WSL 2 Backend on the docker blog.

Ghost provides an easy way to back up the content of your website, from the Labs page you can choose export your content which will 'Download all of your posts and settings in a single, glorious JSON file'. However, the export doesn't include a vital part of your blog content, specifically any files that you've uploaded. Frustratingly there is also no straightforward way to download all those files that I'm aware of. If you have ssh access to the server hosting your Ghost blog however you can do this easily using scp.

The Secure Copy Protocol is a network protocol, based on the BSD RCP protocol, which supports file transfers between hosts on a network. What that translates to in practice is a simple way to do file transfers over ssh and since Windows now bundles  ssh like a real operating system you're pretty much guaranteed to have scp available whatever os you're currently using.

By default, Ghost stores images in the content/images folder of your installation organised into sub-folders by year and month. If, like me, you're hosting on a Digital Ocean Droplet then your images will be stored under /var/www/ghost/content/images/. So, to download all my images I simply create a local folder and scp all the files into that folder like this:

mkdir ghost-images
scp -r myusername@46.101.125.10:/var/www/ghost/content/images ghost-images

Specifying the -r flag tells scp to run recursively so all sub-folders are also downloaded and, depending on how many images you have this might take a while.

I was doing this download so that I could migrate from one server to another, so the next step was to upload all those images. To do that you just have to reverse the order of arguments you pass to scp:

scp -r ghost-images myusername@46.101.125.10:/var/www/ghost/content/images 

Happy copying :)