Evan Shortiss

Quarkus, Podman, and Testcontainers on macOS

Thu, 26 Jan 2023 00:00:00 +0000

Recently, I wanted to use Podman for running Testcontainers with a Quarkus application on my Mac. I followed this helpful post by Stephen Nimmo, but my containers were not being run in the Podman VM. It turns out that running both Podman and Docker on the same machine can make things less than straightforward!

If you find yourself in the same situation it’s probably because you started the Docker VM before the Podman VM (via podman machine start or Podman Desktop). This means that Docker is still running and listening at /var/run/docker.sock and therefore Testcontainers is talking to it instead of Podman.

To get your containers running in the Podman VM you can stop the Docker VM by exiting Docker Desktop using this helpful snippet from Stack Overflow and restarting the Podman machine:

# Stop the running Podman machine VM
podman machine stop

# Kill the running Docker VM and Docker Desktop
ps ax|grep -i docker|egrep -iv 'grep|com.docker.vmnetd'|awk '{print $1}'|xargs kill

# Restart the Podman machine VM
podman machine start

Once the podman machine start command has completed it should report that it’s listening on /var/run/docker.sock, since Docker no longer has a hold on it.

You can now start your Quarkus application that uses Testcontainers, and they’ll be run using Podman!

NOTE: I tried setting the DOCKER_HOST and TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE variables to force Quarkus and Testcontainers to bypass the Docker socket and use my Podman socket at /Users/$USER/.local/share/containers/podman/machine/podman-machine-default/podman.sock but this always reported errors about missing containers.

BlueJeans Primetime & AirPods Play/Pause

Thu, 01 Sep 2022 00:00:00 +0000

This a quick, oddly specific, tip for those of you that find yourself using Apple AirPods during a BlueJeans Primetime call.

I have the “Automatic Ear Detection” setting enabled on my AirPods. This means that my AirPods automatically pause audio/video playback on my iPhone or MacBook when I remove one of them from my ears. It also restarts the audio/video playback when I place the AirPod back in my ear. Neat!

The “Automatic Ear Detection” feature is fantastic, but it’s not ideal when combined with BlueJeans Primetime. Despite BlueJeans Primetime being a “live” stream, it will pause when you remove an AirPod from your ear! BlueJeans Primetime doesn’t expose playback controls because it’s live. Placing an AirPod back in your ear is the only way to resume playback! Or is it?

A quick fix for this, without refreshing your BlueJeans Primetime tab/browser after removing the AirPod is to open up the developer tools in your web browser (usually using Ctrl + Shift + I on Windows and Linux, or Cmd + Opt + I on macOS) and run this JavaScript in the console.

document.querySelector('video').play()

I assume it might also be possible to use media controls if your keyboard/device is equipped with them, but I haven’t tested that.

CodeReady Containers Everywhere and Anywhere with Tailscale

Tue, 15 Mar 2022 00:00:00 +0000

This post serves as a quick summary of a setup I’m using to access my OpenShift cluster, based on CodeReady Containers (CRC). The TLDR is that I’m using Tailscale to create a mesh network so I can access my devices anywhere, and it’s awesome! Read on to learn more about the setup.

Why CodeReady Containers?

Working at Red Hat means that I frequently need access to an OpenShift (enterprise Kubernetes) cluster. Using OpenShift via CodeReady Containers (CRC) is a great option for me, since I don’t always need a production grade cluster with multiple nodes for high-availability.

CRC isn’t perfect though; it requires 9 GB of memory and 4 CPU cores to run. My MacBook Pro can handle this, but I found that I have a better experience running CodeReady Containers by booting Fedora on my gaming PC and deploying CRC there. This way I can assign more memory to CRC, and my MacBook doesn’t burn my thighs! 🔥

Of course, running CRC on my PC means I have to deal with some mildly painful networking and DNS issues. It also means I can’t access my OpenShift cluster when I’m away from home. However, a fellow Red Hatter mentioned Tailscale to me recently, so I figured I’d try using it as a solution to these networking and DNS challenges.

Networking

Turns out Tailscale is a perfect solution to managing a small mesh network. Tailscale provides a zero config option for connecting all of my devices via a VPN. Basically, Tailscale enables a peer-to-peer mesh of encrypted connections between registered your devices using the WireGuard protocol. This means that my MacBook can communicate with my Fedora machine in my office, even when I’m out and about.

Here’s how to get started:

Sign-up for Tailscale (it’s free!)
Setup the Tailscale client on your machines (that’s macOS and Fedora in my case)
Start CRC on the Fedora PC and expose it via HAProxy (all explained in sections 3.2 and 5.4 of the CRC docs)

In the image below you can see both of my machines listed in the Machines section of the Tailscale UI, so I knew they were configured correctly. I further verified connectivity by SSH’ing from my MacBook Pro to the Fedora machine using the Tailscale interface IP address, i.e ssh evan@100.80.245.58 (enable SSH on Fedora if you want to do this!).

DNS

The CRC documentation describes how to configure dnsmasq on your development machine to resolve the CRC addresses if you’re deploying CRC on a remote system like I am, but in my case I’m taking a different approach. Technically this approach uses dnsmasq too, but it’ll work for all devices connected to my Tailscale network thanks to Pi-hole!

I run Pi-hole on my Raspberry Pi for DNS on my home network, since it provides a network-wide ad/tracker blocking. Pi-hole also supports configuring custom DNS and CNAME rules for your network. I went ahead and setup Tailscale on my Raspberry Pi to extend the Pi-hole capabilities to devices on my Tailscale network. You can see my Raspberry Pi in the Machines list in my Tailscale dashboard.

Tailscale provides a MagicDNS feature that automatically configures DNS entries for devices in my Tailscale network, so you can connect to devices using their hostname instead of their Tailscale IP address. Another neat feature that Tailscale supports is setting a global nameserver. I enabled MagicDNS and set my Raspberry Pi as the global nameserver.

The last step, was to update my Pi-hole deployment to accept a custom dnsmasq config to handle address resolution for the CRC cluster. To do this I created a folder named etc-dnsmasq.d on my Raspberry Pi and created a file named 02-pihole-custom.conf within that folder. I then added the following lines to the file:

address=/apps-crc.testing/100.80.245.58
address=/api.crc.testing/100.80.245.58

After this I recreated my Pi-hole container (I run Pi-hole using Docker on my Raspberry Pi) and made sure to pass the custom etc-dnsmasq.d folder as a volume to docker run using -v "/home/pi/etc-dnsmasq.d/:/etc/dnsmasq.d/".

NOTE: It is possible to manually define the routes in the Pi-hole admin Local DNS Records UI, but this is tedious since you need to add wildcard routes individually, i.e Pi-hole does not support entering *.apps-crc.testing. If you prefer to use this option you need to at the very least define api.crc.testing, console-openshift-console.apps-crc.testing, and oauth-openshift.apps-crc.testing to use the OpenShift cluster. Any OpenShift Routes you create for your applications will need to be added to the Pi-hole too, e.g my-java-app.apps-crc.testing. Personally, I prefer the dnsmasq config option, since it’s one and done!

Verification

At this point, everything was in place. I was able to navigate to https://console-openshift-console.apps-crc.testing/ in the web browser on my MacBook Pro, accept the self-signed TLS certificate warnings, and access my OpenShift cluster powered by CodeReady Containers!

I created a Project and deployed a sample Quarkus application. This created a HTTP Route that I was able to access and use to verify that wildcard domains such as *.apps-crc.testing/ were resolved correctly too. Now I can access my OpenShift cluster running on my Fedora PC from anywhere!

How I fixed my AMD Driver Timeout Issues and retained 8/10 Bits per Color

Mon, 26 Jul 2021 00:00:00 +0000

Summary/TLDR

My driver timeouts are related to my 144 Hz monitor and AMD GPU driver not playing nice, as far as I can tell. This YouTube video has an even quicker fix.

Note: I think this fix is just a trivial config change, but follow this guide at your own risk. If something goes wrong use Safe-Mode and the CRU reset script to fix it, etc.

First, confirm your driver timeout has a similar cause to mine. You can verify this by checking your VRAM frequency. If the VRAM frequency is constantly maxed out, then you might have the same or similar issue. This frequency should be able to drop to a few hundred MHz when the PC is idle, but mine was locked at 1750MHz which is the default max VRAM speed for my 5600 XT GPU. Remember to close all other programs when checking this value using GPU-Z or the Radeon Software Metrics, as shown.

There are two straightforward solutions to this problem. The easiest option is to change your refresh rate to 120Hz in the Advanced display settings of Windows, but what if you’ve payed for 24Hz more that your consciousness can barely perceive? 😉

Refresh Rate	Frametime
144Hz	6.944ms
120Hz	8.333ms
60Hz	16.666ms

If you really want 144/165Hz then you need to correct your high refresh rate to use correct/standard VESA timings with CRU (Custom Resolution Utility) like so:

Download CRU and unzip it.
Start the CRU program by double-clicking the blue icon.
Double-click your 144/165Hz resolution from the Detailed resolutions list. This should be the same as the resolution you’ve set in your Windows Advanced Display settings.
Change the Timing setting from Manual to Automatic (PC). Don’t change anything else.
Click OK to save the new timing values.
Close CRU by clicking OK.
Run the restart64 script included with CRU.
Wait for your monitor to refresh.

Your VRAM frequency should now be idling at a lower value than it was prior to adjusting the timing using CRU. You can see that my 5600 XT VRAM idles at 200MHz after applying this fix.

This completely resolves the crashes on my PC. I don’t entirely understand why, but I won’t question it. I can even overclock my VRAM and it will happily run constantly at over 1800 Mhz when gaming, so it’s not the memory frequency.

Using the automatic CRU timings at 144Hz supports 8-bit colour on my monitor. The highest refresh rate I can obtain with 10-bit colour via CRU is 138Hz. I also found that I could leave all timing settings in CRU at the defaults and simply increase the Vertical Blank from the default value. Increasing it from the default of 56 to 98 works for me, but the exact values will probably vary across monitors.

That’s all folks! Hopefully fixing your monitor timings and vertical blanking resolves your AMD driver timeouts like it did for me.

No need to read beyond here unless you’re bored and want some background information!

The Issue

Does the following image make your blood boil? It sure as shit had an affect on me this past year. The idea that your GPU might be toast sucks, but it’s a complete nightmare in the current PC hardware market. Thankfully, for some of us, it’s just a software issue related to our GPU and monitor not playing nice!

For some people this dialog isn’t displayed, but they get a warning in their Windows logs that states “driver amdkmdag stopped responding and has successfully recovered”.

There are many potential causes for this issue. Poor RAM compatibility, unstable RAM timings, unstable overclock, and a weak PSU seem to be common causes. In my case it appears to be a specific AMD Driver issue due to my GPU and 144Hz monitor combination.

I’m the owner of an MSI Radeon RX 5600 XT GAMING X. It’s a fantastic GPU for the price, considering it gives an RTX 2060 Super a run for it’s money, assuming you can find one at MSRP these days. Its cooler is an absolute unit, so it has a nice feature that allows it to disable the fans when not gaming. Unfortunately the AMD drivers combined with my monitor have a bit of an issue when gaming at 1440p 144Hz in Windows; the driver crashes occasionally. And yes, based on my limited testing, Fedora Linux 34 does not suffer from this issue when gaming at 1440p 144Hz.

This GPU had a bit of rough a launch, so I cut it some slack when this issue started occurring. After multiple driver updates I started to lose hope though. The awful state of the GPU market made matters even worse. Returning it would result in credit/cash towards a new massively overpriced GPU. Worse again, I bought it from the UK so I wasn’t sure how the Brexit shenanigans might affect the return process. Another matter of concern is that this appears to be a software issue when using certain monitors and refresh rates, so the return would probably have been rejected!

After trying ~~almost~~ literally every fix posted online, I eventually found that I could fix the problem by lowering my refresh rate to 120Hz. I wasn’t totally satisfied with this so I kept digging, as you’ll see in this post.

Is this the definition first world problem? Sure is. But on the other hand I paid for the whole monitor and GPU, so I’m gonna use the whole monitor and GPU!

The Clue

When AMD released the 21.7.1 drivers I decided to see if the 144Hz crashing issue was fixed. Spoiler alert: it’s not fixed, at least not for all of us. During my latest test I noticed that my VRAM was constantly locked at 1750Mz when I set my refresh rate to 144Hz. This is fine when gaming, but why would it be locked so high when the PC is idle? As soon as I changed my refresh rate back to 120Hz the VRAM was able to fluctuate between 200Mhz and 1750Mhz, depending on workload. Why is it fine pinned to 1750Mhz gaming at 120Hz, but not fine when pinned at 1750Mhz and gaming at 144Hz? I still don’t fully understand, but I knew that I had found a lead. Another interesting aspect to this bug is that my VRAM can overclock to more than 1800Mhz just fine, it’s only when 144Hz is being used that I experience the random driver timeouts.

Using the VRAM clock speed clue leads us to a fantastic solution by Hard Reset on YouTube. Their video demonstrates that creating a Custom Resolution solution solves the issue. I created a 1440p custom resolution with a 142Hz refresh rate, and my crashing problems were solved. So, is this it? Case closed? Not entirely!

Image quality seemed different when using my new custom resolution. It wasn’t very noticeable, but what has been seen cannot be unseen. Every time I opened a new tab in my web browser I could see a grain on the blank page if I looked closely. I think this was because the custom resolution was limiting colour depth. Radeon Software listed 8 and 10 bit colour options, but selecting either would fail and revert back to 6 bpc after using the technique from that YouTube video.

The My Solution

I’m not a video expert, but I assume the custom resolution configuration was maxing out the available bandwidth on my DisplayPort. Now, I know for a fact that I can run at 1440p 144Hz with 10 bit colour via DisplayPort on my hardware since the default resolutions support this, albeit with an occasional driver crash when gaming. Knowing this, I spent a few minutes tweaking the custom resolution settings in Radeon Software to try get it to expand the colour depth while retaining a high-refresh rate. After 5 minutes I accepted defeat 😢

This is when I turned to CRU (Custom Resolution Utility). It was mentioned in one of the threads I found online when researching this issue, so I figured it was worth a try.

Here’s how I used CRU to create a high-refresh rate that supported the full colour depth of my monitor:

Edit: This solution is for 10 bpc. Use the one from the top of the article if 144Hz is more important to you than having 10 bpc.

Start the CRU program.
Take note of the Pixel Clock listed beside the default entry for 144Hz (or whatever resolution/refresh rate combination is causing trouble for you).
Add a new entry under the Detailed Resolutions.
Double click this new entry to open the Edit dialog.
Select Automatic (PC) from the Timing dropdown.
Set the Refresh Rate to a value that results in a Pixel Clock that’s equal to, or less than, the default value.
Click the OK button to save the custom resolution.

At this point you will have two resolutions listed like so. The Pixel Clock of the new resolution does not have to match the original perfectly, just be sure it’s equal or lower.

I managed to get them to align at 568.72Mhz while keeping a 136Hz refresh rate somehow by setting the timing back to Manual and editing the Pixel Clock, but can’t get it to happen when I try again when testing it for this blog! I’m too lazy to spend more time trying to squeeze the refresh rate higher, and the difference between 144Hz and 136Hz is less than half a millisecond so who cares. Wait, couldn’t you make a similar argument for 120Hz vs 136Hz and that I shouldn’t have wasted my time on this first world problem? 🤔

Refresh Rate	Frametime
144Hz	6.944ms
136Hz	7.353ms
120Hz	8.333ms
100Hz	10.000ms
60Hz	16.666ms

No, this effort is for “science”! Anyways, click the “OK” button to close CRU. Run the restart64 executable included with your CRU download. Your screen will go blank for a moment, then come back to life. Now, head over to the Windows display settings.

Verify that the selected resolution matches that one you created in CRU, then sroll down and select Advanced display settings.

Take note of the listed Bit depth, e.g 10-bit in my case. Select your custom refresh rate. The screen will go blank briefly, and should return to normal after a second. If everything seems fine click the Keep changes button. If the screen remains blank don’t panic! It just means your custom refresh and resolution isn’t supported, but Windows will revert the change after 30 seconds.

If you were able to select Keep changes, verify that the listed Bit depth matches what it was previously. You should also verify that your monitor is displaying the refresh rate correctly. Many monitors include an overlay that displays the active refresh rate. For example, my monitor has it built into the settings overlay like this. This verification is important, since I found that Windows might accept the custom resolution and refresh rate, but in reality your monitor will only output half of the refresh rate. For example, I had a 140Hz refresh rate configured, but my monitor was only displaying at 70Hz which was pretty noticeably slower than 140Hz, and also easily confirmed via the monitor settings!

You can see that my custom refresh rate of 136Hz retains 10 bit depth. It might be possible to push the refresh rate higher and maintain the 10 bits, but I’ve been unable to find a setting that does so with my monitor and GPU. A few tutorials state that increasing the blanking on the default resolution profile does the trick too, so it might be worth trying.

Excluding files from local Source-to-Image (s2i CLI) Builds

Tue, 29 Jun 2021 00:00:00 +0000

TLDR; Use the flag --exclude "(^|/)\.git|node_modules|some_other_folder(/|$)"

Source-to-Image (s2i) provides a convenient solution for developers looking to easily create OpenShift-ready container images from their code.

A simple way to build and deploy an application on OpenShift using s2i is via the oc new-app command.

export BUILDER=registry.access.redhat.com/ubi8/nodejs-14
export REPO=https://github.com/sclorg/nodejs-ex.git

oc new-app $BUILDER~$REPO

It’s also possible to use s2i to build code in your development environment using the s2i CLI.

When building a Node.js application locally you might have .env, node_modules/, and other files that you’d rather not pass to the s2i build. These might be files that are typically included in your .gitignore or a .dockerignore.

I recently learned that s2i supports an --exclude flag to support this behaviour. The help output for this flag is surprisingly, well, helpful:

Regular expression for selecting files from the source tree to exclude
from the build, where the default excludes the '.git' directory
(see https://golang.org/pkg/regexp for syntax, but note that ""
will be interpreted as allow all files and exclude no files)
(default "(^|/)\.git(/|$)")

Now, I don’t know about you, but I immediately assumed this would be pain in the ass to configure. 99% of the time I write RegEx I do so in JavaScript, and I use the Node.js REPL to experiment and get it right.

Much to my surprise though, I got lucky with this one on the first try. Seriously. I know the RegEx ninjas out there think I’m worthless but sometimes I get lucky! 🤷‍♂️

Here’s how you can use --exclude to exclude node_modules and two other directories:

# Output image name
export IMAGE_NAME=quay.io/evanshortiss/rhosak-nodejs-sbo-example

# Base image used to create the build
export BUILDER_IMAGE=registry.access.redhat.com/ubi8/nodejs-14

# Build the local code into a container image, but do not pass
# the .git, node_modules, or .bindingroot directory to the build
s2i build -c . --exclude "(^|/)\.git|.bindingroot|node_modules(/|$)" \
$BUILDER_IMAGE $IMAGE_NAME

Pretty Print for Pino Logs with Stern

Thu, 13 May 2021 00:00:00 +0000

Posting this almost as a “note to self”, so I’ll keep it short.

Here’s how to pretty print pino logs if you’re using stern to view them from a Node.js Pod running on Kubernetes or OpenShift:

stern $POD_SELECTOR -o raw | npx pino-pretty -t

I use npx pino-pretty, but running npm install -g pino-pretty will allow you to do the following:

# Install pino-pretty globally
npm install -g pino-pretty

# Use it to view the logs
stern $POD_SELECTOR -o raw | pino-pretty -t

Hosting CodeReady Containers on DigitalOcean

Mon, 21 Sep 2020 00:00:00 +0000

Note: The official documentation for CodeReady Containers now includes a section similar to this post. You can find the official documentation at this link.

This is a slightly customised version of Jason Dobies’ post on the Official OpenShift blog. That blog was inspired by Trevor McKay’s GitHub Gist found here. That means I’m being incredibly derivative with this one, but I had fun and I hope you find it interesting!

Like Jason, I sometimes want admin access to a full, but lightweight OpenShift development environment. Running CodeReady Containers on my Macbook works well, but it’s not fun when Firefox, Chrome, Visual Studio Code are also in competition for my 16 gigs of memory. I’d also rather my Macbook’s fans didn’t spin at 100% speed all the time.

I have a gaming PC (look, I’d be fooling no one if I said it’s primary purpose is productivity) that could easily run CodeReady Containers, but it only has a single 1TB drive with Windows 10 installed. Partitioning my drive, or running CodeReady Containers on Windows didn’t sound like exciting prospects, and waiting for an SSD delivery would require patience, so I figured I’d try something else.

This post outlines how I setup a DigitalOcean Droplet to run OpenShift 4 via CodeReady Containers, and how I configured my Macbook’s DNS to communicate with it.

DigitalOcean Droplet Setup

I’m going to assume you have a DigitalOcean account with a credit card attached to it. If you don’t, go ahead and set that up before reading any further. They’re not going to let you run a Droplet with 8 vCPU and 16GB of RAM for free!

Get started by creating a new project on DigitalOcean. Name it something boring, like “CRC OpenShift 4”, and give it a similarly sensible description.

Creating a project for CodeReady Containers on DigitalOcean.

Next, create a Fedroa 32 Droplet in that project you just created. You’ll want something beefy since the CodeReady Containers documentation states that it requires:

4 vCPUs
8GB RAM (I’ve found it uses closer to 9GB)
35 GB of disk space

A Basic Droplet type with 8 vCPUs and 16GB RAM does the trick. This costs approximately 12 USD cents an hour, or 80 USD/month. Even though this is a throwaway single node OpenShift cluster, make sure you configure an SSH Key for authentication, because professionals have standards (warning: cartoon violence)!

The Droplet I chose has 8vCPUs and 16GB RAM. Nice!

While the Droplet is being created, head to the Networking section and click Create Firewall in the Firewalls section. Add the following rules:

SSH
HTTP
HTTPS
TCP 6443 (use the Custom option in the New Rule dropdown)

Your rules should look similar to the screenshot below. Don’t forget to actually apply the rules to your new Fedora 32 Droplet!

Software Versions and Other Notes

The bash commands throughout this guide reference a $DROPLET_IP variable. As you may have guessed, this is the IPv4 address that’s listed alongside your shiny new Fedora 32 Droplet in the DigitalOcean UI. Anywhere you see $DROPLET_IP, replace it with your Droplet’s IPv4 address.

Also, take note of this list. If you encounter issues recreating my setup, it might due to a version mismatch* with these:

Fedora 32
CodeReady Containers version: 1.16.0+bf72d3a
HA-Proxy version 2.1.7 2020/06/09
NetworkManager 1.22.10-1.fc32
firewalld 0.8.3
libvirtd 6.1.0

* and hopefully not an error in my instructions!

User Setup

SSH into your Droplet once it finishes provisioning. Read this DigitalOcean How-To for SSH if you’re having trouble.

CodeReady Containers should not be run as the root user, so the first thing to do is create another user, e.g crc-user.

ssh root@$DROPLET_IP
useradd crc-user
passwd crc-user
sudo usermod -aG wheel crc-user

Now that the user is added, you can copy add your SSH key to the authorised keys for the new user. Run this from your development/client machine:

ssh-copy-id crc-user@$DROPLET_IP

There’s more you can do to secure your Droplet, but that’s outside the scope of this guide. Step 2 in this guide for Fedora is a start!

Install Dependencies and Confgure firewalld

# Note that you SSH as crc-user this time!
ssh crc-user@$DROPLET_IP

sudo dnf -y install NetworkManager haproxy firewalld policycoreutils-python-utils
sudo systemctl start libvirtd
sudo systemctl start firewalld
sudo firewall-cmd --add-port=80/tcp --permanent
sudo firewall-cmd --add-port=6443/tcp --permanent
sudo firewall-cmd --add-port=443/tcp --permanent
sudo systemctl restart firewalld
sudo semanage port -a -t http_port_t -p tcp 6443

Before moving on, verify that a valid nameserver entry is set in /etc/resolv.conf. The primary nameserver needs to be set to 127.0.0.1 so OpenShift Routes can be resolved. I used 1.1.1.1 as my secondary nameserver. The updated /etc/resolv.conf will look like this:

# Generated by NetworkManager
nameserver 127.0.0.1
nameserver 1.1.1.1

Download and Run CodeReady Containers

I’ve included a URL in the first step here for completeness, but if it fails you can check cloud.redhat.com/openshift/install/crc/installer-provisioned to find a working URL.

# Download, unpack, and move CRC the binary onto the PATH
curl https://mirror.openshift.com/pub/openshift-v4/clients/crc/latest/crc-linux-amd64.tar.xz > crc-linux-amd64.tar.xz
tar -xf crc-linux-amd64.tar.xz
sudo mv crc-linux-1.16.0-amd64/crc /usr/local/bin/

# Verify the binary is on the PATH. It should print version information.
crc version

# If the version command worked, run this setup command
crc setup

The crc setup command takes a minute to run, so now is a good time to get a “pull secret” prepared for the next step.

Obtain the Pull Secret under the cluster creation UI. You can also find the CodeReady Containers download URL here by right-clicking the “Download Code-Ready Containers” button and choosing “Copy Link Location”.

Your pull secret is found on cloud.redhat.com:

Login to cloud.redhat.com/openshift.
Select Clusters from the side menu.
Click the Create button on the clusters listing page.
Choose Red Hat OpenShift Container Platform as the cluster type.
Choose the Run on Laptop option.
Click the Copy Pull Secret button.
Return to your SSH session.
Paste your pull secret into a file, e.g vi ~/crc-pull-secret.json and paste the contents.
crc start -p ~/crc-pull-secret.json

CodeReady Containers will take a minute or two to start, so this is a good time to mindlessly browse Reddit or Twitter.

Configure HAProxy

HAProxy will route HTTP and HTTPS traffic to CodeReady Containers:

Backup the default HAProxy configuration: sudo cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.original
Obtain the CodeReady Containers VM IP using the crc ip command.
Paste the following configuration into /etc/haproxy/haproxy.cfg, being sure to replace the CRC_IP strings with the IP obtained using crc ip:

defaults
    mode http
    log global
    option httplog
    option  http-server-close
    option  dontlognull
    option  redispatch
    option  contstats
    retries 3
    backlog 10000
    timeout client          25s
    timeout connect          5s
    timeout server          25s
    timeout tunnel        3600s
    timeout http-keep-alive  1s
    timeout http-request    15s
    timeout queue           30s
    timeout tarpit          60s
    default-server inter 3s rise 2 fall 3
    option forwardfor

frontend apps
    bind 0.0.0.0:80
    bind 0.0.0.0:443
    option tcplog
    mode tcp
    default_backend apps

backend apps
    mode tcp
    balance roundrobin
    option tcp-check
    server webserver1 192.168.130.11 check port 80

frontend api
    bind 0.0.0.0:6443
    option tcplog
    mode tcp
    default_backend api

backend api
    mode tcp
    balance roundrobin
    option tcp-check
    server webserver1 192.168.130.11:6443 check port 6443

Start HAProxy using the sudo systemctl start haproxy command.

If HAProxy fails to start, use haproxy -f /etc/haproxy/haproxy.cfg -db to view logs and diagnose the issue.

Note that the CodeReady Container VM IP can change on reboots/restarts. If the IP changes you’ll need to update the haproxy.cfg with the new IP and run sudo systemctl restart haproxy!

Configure DNS on your Development Machine

As Jason said in his post, there’s no right or wrong way to do this, but you’ll need to tell your development machine how to resolve the CodeReady Container URLs, i.e to resolve URLs with the testing suffix to your $DROPLET_IP.

This is pretty easy to configure for the static OpenShift routes via /etc/hosts on Linux and macOS. Add the following line to your /etc/hosts file, and don’t forget to replace $DROPLET_IP with the correct value:

# CodeReady Containers (running on DigitalOcean)
$DROPLET_IP api.crc.testing oauth-openshift.apps-crc.testing console-openshift-console.apps-crc.testing default-route-openshift-image-registry.apps-crc.testing

DNS resolution for deployed application Routes is a little less straightforward. This is because they are wildcard routes, i.e *.apps-crc.testing. I had hoped my PiHole would help with this, but it doesn’t seem to support setting a redirect for wildcards. I used dnsmsaq instead.

These commands will configure dnsmasq to resolve your CodeReady Containers installation on DigitalOcean on macOS:

brew install dnsmasq

# Commands based on: https://asciithoughts.com/posts/2014/02/23/setting-up-a-wildcard-dns-domain-on-mac-os-x/
echo "address=/.testing/$DROPLET_IP" > /usr/local/etc/dnsmasq.conf

# Copy the daemon config plist
sudo cp -fv /usr/local/opt/dnsmasq/*.plist /Library/LaunchDaemons
sudo chown root /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist

# Approve the request to "accept incoming" connections if asked
sudo launchctl load /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist

# Redirect DNS queries for "testing" to localhost
sudo mkdir -p /etc/resolver
sudo sh -c 'echo "nameserver 127.0.0.1" > /etc/resolver/testing'

Finally, in the Network settings for your device make 127.0.0.1 your first nameserver, then follow it up with another, e.g 1.1.1.1 and/or 8.8.8.8.

macOS Network settings to resolve the testing domain via dnsmasq, and 1.1.1.1 for everything else.

Revel in Your Success

Once you have DNS configured, navigate to console-openshift-console.apps-crc.testing in your web browser. You’ll receive some very exciting SSL warnings. Yes, these are exciting because it means you’ve been successful!

The most exciting self-signed SSL certificate error you’ll see this week.

Add an exception for the self-signed SSL certificates, and voila! You’ll be brought to the OpenShift login page. You can login using credentials found by running crc console --credentials on your Droplet - hell yeah!

The OpenShift Projects list. You can see I created a Project to test the basic cluster functionality.

Create an OpenShift Project, and deploy a template. I went with the Node.js application because I’m a bit of a JavaScript fanboi. You can see my Node.js application running in the Developer Topology view in the screenshot below.

A Node.js application running in my Project.

Clicking the application node presents a details pane on the right. Open the URL under Routes and briefly marvel at the wonders of cloud technology.

Configure OpenShift User Logins

Last but not least, you’ll want to change the default login details. Those are the ones that can be obtained via crc console --credentials. The developer user it provides has a default password that should be changed.

Here’s how you can update it:

ssh crc-user@$DROPLET_IP

# This will make the OpenShift CLI (oc) available
eval $(crc oc-env)

# Installs htpasswd, among other things
sudo dnf install httpd-tools

# Get the kubeadmin credentials 
crc console --credentials

# Login as kubeadmin (you can omit the -p flag so it prompts you for the password)
oc login -u kubeadmin -p $KUBEADMIN_PASSWORD https://api.crc.testing:6443

# Save the current htpasswd file that OpenShift is using to a local user.htpasswd file
oc get secrets/htpass-secret -n openshift-config -o jsonpath="{.data.htpasswd}" | base64 -d > users.htpasswd

# Update the password for the "developer" user. You can also use htpasswd
# to add more users if you want
export DEVELOPER_PASSWORD="newsafepassword"
htpasswd -c -B -b users.htpasswd developer $DEVELOPER_PASSWORD

# Output the htpasswd in base64 format
cat users.htpasswd | base64

Visit console-openshift-console.apps-crc.testing/k8s/ns/openshift-config/secrets/htpass-secret/yaml and replace the existing htpasswd value in the YAML with the new one, then click Save. You can also do this via oc patch or oc edit commands.

Verify that the new password is working by logging in as the developer user:

oc login -u developer

That’s a Wrap

Don’t forget to shutdown the Droplet when you’re not using it!

Create React App on OpenShift

Thu, 17 Sep 2020 00:00:00 +0000

Create React App is described as “the best way to start building a new single-page application in React” on reactjs.org. It’s no surprise so many developers start building a modern web application using this toolchain.

This post outlines a straightforward method to deploy an NGINX container that serves the contents of a Create React App application, or any modern web application, on an OpenShift cluster.

A custom source-to-image (s2i) builder that I maintain at github.com/evanshortiss/s2i-nodejs-nginx is part of the magic that simplifies this process. This builder image was originally authored by the folks at jshmrtn!

Using this method allows you to deploy code that’s stored in a Git repository, or to upload your local codebase directly to a Build running on OpenShift. Both approaches are covered below.

Requirements

An OpenShift 4.x cluster
OpenShift CLI v4.x
Node.js v12 or newer
GitHub Account (optional)

Git Method

Note: I’m going to assume you’re starting from scratch, but if you have an existing repository with a React application that was generated via Create React App or that uses react-scripts you can skip to the bash script below.

To use this approach head over to github.com/new and do the following:

Enter a Repository name
Skip the Initialize this repository with section
Click the Create repository button.

Once the repository is created, execute the commands below to get your web application hosted by NGINX on OpenShift.

# The builder image is used by OpenShift to build the application
export BUILDER_IMAGE="quay.io/evanshortiss/s2i-nodejs-nginx"
export APPLICATION_NAME="cra-application"

# REPLACE THIS WITH YOUR REPOSITORY URL
export GIT_REPO="https://github.com/yourusername/your-react-app-repo"

# Create a React application template using CRA, and push to
# the empty GitHub repository you created
npx create-react-app $APPLICATION_NAME
cd $APPLICATION_NAME
git remote add origin $GIT_REPO
git push origin master

# Create an OpenShift project
oc new-project webapp

# Create a new BuildConfig on OpenShift based on the Node.js & NGINX builder
# --name        The name for the output container image stream
# --build-env   Environment variable indicating where the builder image should
#               look for the bundled web application. This is the result of
#               running "npm build" or "yarn build". For CRA this is a "build"
#               directory, but for a custom webpack config it might be "dist"
oc new-app $BUILDER_IMAGE~$GIT_REPO \
--name $APPLICATION_NAME \
--build-env BUILD_OUTPUT_DIR=build

# Expose the service via HTTP using an OpenShift Route
oc expose svc/$APPLICATION_NAME

# Check the build status
oc get builds

# Print the application URL (Route)
echo "http://$(oc get route/$APPLICATION_NAME -o jsonpath='{.spec.host}')"

To deploy future changes run oc oc start-build $APPLICATION_NAME. You could also set up a Build Trigger to automate builds using a Git hook.

Git-less Method

If you’d rather develop locally and upload the application assets for builds, then give this binary build solution a try.

export APPLICATION_NAME="cra-application"
export BUILDER_IMAGE="quay.io/evanshortiss/s2i-nodejs-nginx"

# Create a React application template using CRA
npx create-react-app $APPLICATION_NAME

# Create an OpenShift project
oc new-project webapp

# Create a new BuildConfig on OpenShift based on the Node.js & NGINX builder
# --binary      Indicates we'll trigger builds by uploading assets directly
# --to          The name for the output container image stream
# --build-env   Environment variable indicating where the builder image should
#               look for the bundled web application. This is the result of
#               running "npm build" or "yarn build". For CRA this is a "build"
#               directory, but for a custom webpack config it might be "dist"
oc new-build $BUILDER_IMAGE --binary \
--to $APPLICATION_NAME \
--build-env BUILD_OUTPUT_DIR=build

# Start a Build based on the BuildConfig. Pass the local CRA project folder
# as the build input
oc start-build $APPLICATION_NAME --from-dir=$APPLICATION_NAME

# Create a Deployment using the latest tag on the application ImageStream
oc new-app --image-stream $APPLICATION_NAME

# Expose the service via HTTP using an OpenShift Route
oc expose svc/$APPLICATION_NAME

# Check the build status
oc get builds

# Print the application URL (Route)
echo "http://$(oc get route/$APPLICATION_NAME -o jsonpath='{.spec.host}')"

To push new changes from your local machine to OpenShift and trigger a build run the oc start-build $APPLICATION_NAME --from-dir=$APPLICATION_NAME command again.

Result

Once oc get builds shows that the build is complete, you’ll be able to view the application in your web browser. The URL is obtained via the echo command above.

Creating a Windows 10 USB Installer using macOS in 2020

Wed, 13 May 2020 00:00:00 +0000

You’d think creating a USB-based installer for Windows 10 would be easy using macOS, or any modern operating system, but my experience following the guides that Google returns for macOS wasn’t great. Most guides are either outdated or contain what appear to be errors in the Terminal commands, so the steps below are what I used to create a working Windows 10 64-Bit USB installer in 2020.

Update: Two helpful readers reached out to me with alternative methods after I published this post. They are described under the Method 2 and Method 3 headings. Consider trying those easier options before you try my Method 1. Personally, I have only tested Method 1.

Method 1

This guide is mostly based on the one by Quincy Larson on freecodecamp.org, but it contains modifications that I found necessary to get it working.

Open a Terminal

We’ll be using the Terminal on your Mac to create the bootable USB. If you don’t know how to open the Terminal App, then read this - it only takes a second!

Install Homebrew & Wimlib

You might already have Homebrew installed, so check by running which brew. If the command prints not found then you need to run the following command to install it:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

Make sure to copy the entire command. This might require you to scroll horizontally.

Once you have Homebrew installed, quit the Terminal application (the shortcut is CMD + Q), then reopen it. Install wimlib by running brew install wimlib in the new Terminal you just opened.

Download Windows

Download a Windows 10 ISO. You can do so at this link on microsoft.com.

Get Tea or Coffee ☕

Since the Windows 10 ISO is ~5.5GB, maybe now is a good time to have a drink? It’ll take a few minutes to download, even on a reasonably fast internet connection. If you’re lucky enough to have gigabit internet, then I guess you can just wait the 50-ish seconds it’ll take?

Plug the USB Drive into your Mac

OK. I think you can figure this one out, but if you’re using a new Macbook Pro like me you probably need a USB-C to USB-A adapter, and that sucks.

Identify the USB Drive

Run diskutil list in Termainal and a list attached disk drives will be displayed. Your USB will probably be /dev/disk2 or /dev/disk3. We’ll store that in a variable for use later by running the command DRIVE_MOUNT=/dev/disk2.

I used /dev/disk2 in the example DRIVE_MOUNT, but make sure you change it to the correct path for your USB drive. The next step involves formatting the drive. That means it deletes everything on the drive, so you want to be sure it’s the right one!

Format the USB Drive

Format the drive using the following commands:

Note: The lines beginning with # are comments explaining what the command on the line below does. You don’t need to paste them since they do nothing.

# create a variable for the drive name - it'll be handy later
DRIVE_NAME="WINSTALL"

# format the disk so we can boot the Windows installer from it
diskutil eraseDisk MS-DOS $DRIVE_NAME MBR $DRIVE_MOUNT

Mount the ISO File

Enter hdiutil mount followed by a space character in the Terminal, then drag the ISO file you downloaded into the Terminal. The end result will be a command that looks similar to:

hdiutil mount /Users/your-username/Downloads/Windows.iso

Press enter to run the command. Once the ISO has mounted it will print the mount path and volume name. The volume name will be similar to /Volumes/CCCOMA_X64FRE_EN-GB_DV9, but varies depending on the version of Windows and the language. Save it to a variable like so:

WIN_VOLUME="/Volumes/CCCOMA_X64FRE_EN-GB_DV9"

Copy Files to the USB

Copy files to the USB by running the following command. Note that the install.wim must be excluded since it’s too large to copy, but we’ll deal with that next.

rsync -vha --exclude=sources/install.wim $WIN_VOLUME/* /Volumes/$DRIVE_NAME

Copy install.wim to the USB

Use the following wimlib command to copy the final install file to the USB drive:

wimlib-imagex split $WIN_VOLUME/sources/install.wim /Volumes/$DRIVE_NAME/sources/install.swm 4000

Make sure to copy the entire command. This might require you to scroll horizontally.

Unmount the USB & Install Windows

Eject the USB by running diskutil unmount $DRIVE_MOUNT and use it to install Windows!

Method 2

I managed to find the video shared by this helpful person via YouTube, but I must have deleted their email. Whoever you are, thank you!

Here’s a link to the YouTube video by George Holden.

Method 3

Tim King shared the following tip. I believe it’s the same as Method 2!

I hadn’t realised that the simplest method of all was to format the USB stick as ExFAT, thus removing the 4Gb file size limitation of FAT32. This method had the added advantage of taking just a few seconds to complete.

Automatic npm Registry Switching (npmrc daemon)

Sun, 08 Dec 2019 00:00:00 +0000

TLDR; My npmrcd package allows you to automatically switch from the public registry to a private registry. Switching occurs when the private registry can be resolved via DNS or your machine connects to a specific network.

Recently at work I had to switch from directly using the public npmjs.com registry to an internal mirror/proxy server we’re using. Doing this is pretty straightforward since it can be achieved with just one or two commands as shown below:

# Make your npm CLI use the registry at $CORPORATE_REGISTRY_URL
npm config set registry $CORPORATE_REGISTRY_URL

# Optional, but might be necessary depending on your org setup
npm config set cafile $CA_FILEPATH

Seems easy enough, but as someone who often travels for work, frequently works from home, and is generally a forgetful guy, I knew this would quickly become annoying. It was inevitable that I’d forget to switch between the corporate registry at work and public registry as I moved between networks, projects, and jumped on and off VPNs.

In my quest to conquer my forgetfulness, and this problem worthy of /r/mildlyinfuriating, I decided to make switching registries an automated process. The conditions were simple enough, as demonstrated by the pseudocode below:

/**
 * Only use the public npm registry if:
 *  - the corporate registry cannot be resolved
 *  - and we're not on corporate WiFi
 */
if (dnsResolvesUrl(CORPORATE_NPM_REGISTRY_URL) || isConnectedToCorporateWifi()) {
  switchToCorporateRegistry()
} else {
  switchToPublicRegistry()
}

Effectively what I was looking for was a way to have a “work” and “personal” profile for my npm configuration, and a mechanism that would automatically determine which to use.

Have no fear, npmrc is here!

The npm CLI manages your configuration by saving it in a file named .npmrc. This file is typically stored in your home directory on MacOS/Linux. Creating multiple .npmrc files and rotating them would allow me to achieve what I needed.

I figured this had to be a solved problem, and sure enough I found the npmrc package on npm. Using this package I could easily create a new profile using npmrc -c [profile-name], for example I could do the following:

# Create the work profile
npmrc -c work

# Switch to the work profile
npmrc work

# Configure settings for this profile
npm config set registry $CORPORATE_REGISTRY_URL
npm config set cafile $CA_FILEPATH

# Switch back to the default profile
npmrc default

The npmrc package worked perfectly despite not being updated in 5 years, and it has zero external dependencies!

Finding an existing tool to manage the profiles saved me some time and meant I could focus purely on the automation work.

Agents and Daemons

Automating the switch between profiles by running a script on a regular interval seemed like a reasonable solution, so I pursued that avenue. Since my primary development machine is a MacBook Pro, my search for setting up a daemon on macOS lead me to launchd.info. The website summarises how to create a daemon/agent as follows:

The behavior of a daemon/agent is specified in a special XML file called a property list (plist). Depending on where it is stored it will be treated as a daemon or an agent.

This seemed straightforward enough, until it wasn’t! I wrote a script that would use the npmrc CLI to switch profiles, created a plist and tried to load it using the launchctl CLI. I spent a while placing my plist in /Library/LaunchAgents and /Library/LaunchDaemons, running variations of sudo launchctl load -w my-plist.plist, and changing various properties in the plist to get around issues such as:

“why the hell can’t it find the node executable? It’s on my PATH!”
“why doesn’t the daemon run at all?”
“oh I needed to specify the user and group it should run as”
“wait, it’s still not working”

Eventually when I moved the plist into the folder ~/Library/LaunchAgents/ and ran launchctl load -w my-plist.plist it worked! This confusion is probably my own fault since I just looked at the summary on the homepage of launchd.info and didn’t really RTFM. Basically, I needed to place it in the Agents folder for my specific user.

The final plist was similar to this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>com.npmrcd</string>
    <key>EnvironmentVariables</key>
    <dict>
      <key>PATH</key>
      <string>my-path-string</string>
      <key>HOME</key>
      <string>/Users/my-username</string>
    </dict>
    <key>Program</key>
    <string>my-daemon-script.js</string>
    <key>RunAtLoad</key>
    <true/>
    <key>StartInterval</key>
    <integer>120</integer>
    <key>StandardOutPath</key>
    <string>/tmp/npmrcd.stdout</string>
    <key>StandardErrorPath</key>
    <string>/tmp/npmrcd.stderr</string>
  </dict>
</plist>

This plist tells the OS to run my-daemon-script.js once every 120 seconds. So, my Mac will check verify it’s connected to the correct registry every two minutes. Perfect!

Putting it all Together

Once I had the Agent, lookup, and npmrc pieces figured out, I needed to wire them together. To this end, I created a small Node.js script that would be executed by the macOS Agent every couple of minutes to:

Perform a DNS lookup to verify if the corporate registry could be resolved.
Check the WiFi SSID that my Mac is connected to.
Based on the DNS/WiFi results invoke the npmrc command to change registry.
Notify that the change has occurred.

I wasn’t sure how best to notify myself of the change since this would be a process running in the background, but then I recalled that various Node.js modules I’ve used trigger native desktop notifications. Research lead me to the conclusion that the node-notifier module seems to be the most common solution used by other modules for triggering a native notifications, and it worked perfectly for me too - just look at that glorious notification in the screenshot below.

The entire solution is wrapped together as a neat little module called npmrcd so others can easily use it. If you frequently switch between registries give it a try like so:

# install npmrcd
npm install -g npmrcd

# create a work profile with the following settings
npmrcd --registry=https://repository.acme.com/nexus/repository/registry.npmjs.org

If you need to use a CA file that option is supported. The CA file can point to a local file or HTTP/HTTPS endpoint that the file can be fetched from:

# create a work profile with the following settings
npmrcd \
--cafile=~/REGISTRY-CA.crt
--registry=https://repository.acme.com/nexus/repository/registry.npmjs.org

Finally, you can specify that certain WiFi SSIDs should trigger the switch, even if the registry is not accessible:

# even if the registry url does not resolve, switch if connected to
# the "Red Hat Guest" WiFi to prevent using the public registry
npmrcd \
--triggerssid='Red Hat Guest'
--cafile=~/REGISTRY-CA.crt
--registry=https://repository.acme.com/nexus/repository/registry.npmjs.org

Here’s a screenshot showing my registry switch when I first opened my laptop in the office at 10:25AM - hey, I was on calls at 7AM at home before getting there so give me a break on getting there at 10:25AM! I hadn’t plugged into the network via an Ethernet cable yet hence the “currently not accessible” message. You can see that when I opened my laptop at home later that day it switched back to the public npm registry.

I hope this post and module are helpful, or even mildly interesting!