This will walk you through setting up encrypted backups using Backblaze B2 and the Restic app on the client. All run through the terminal. Restic Docs Backblaze Sign In

Keep in mind that this is a file level backup, not a block level backup. Things like VM images, encrypted containers, etc will cause massive bandwidth usage and isn’t the appropriate solution for those. Really any large binary file that changes frequently shouldn’t use this method.

Setup Backup

Recommendation: Customize the Caps and Alerts page to your liking to avoid unexpected cost.

Create a B2 bucket on Backblaze, and get the Bucket ID.

Go to the App Keys section and create a new app key with access to your bucket.

Restic uses environment variables for account access. Export these environment variables (also known as keyID and applicationKey): export B2_ACCOUNT_ID={ID} export B2_ACCOUNT_KEY={KEY}

Initialize the Restic repository: restic -r b2:{BUCKET NAME}:{REMOTE REPO PATH} init

You will now enter an encryption key for this backup. Keep this safe. You can’t recover your data without this.

Begin the backup: restic -r b2:{BUCKET NAME}:{REMOTE REPO PATH} --verbose backup {LOCAL PATH TO BACKUP FROM}

Viewing Backup History

View snapshot history with: restic -r b2:{BUCKET NAME}:{REMOTE REPO PATH} snapshots

Restoring Backup

Note: I have not fully tested this

Use this to restore a remote backup to your local computer. Use the ID found with the above snapshots command.

restic -r b2:{BUCKET NAME}:{REMOTE REPO PATH} restore {ID} --target {LOCAL PATH TO RESTORE TO}

Viewing Backup Data / Mounting Backup

Use the mount option to restore only certain files, or view that the backup is working properly.

restic -r b2:{BUCKET NAME}:{REMOTE REPO PATH} mount {LOCAL MOUNT PATH}

This is a super basic usage of ffmpeg for changing video quality. Very useful for home media servers, or just people who own devices that record 4k in x264 or worse…

ffmpeg basics

Make video smaller (but still good quality) ffmpeg -i input.mkv -vcodec libx264 -crf 24 output.mkv

CRF values: x264: 0 (lossless) - 23 (default) - 51 (worst) x265: 0 (lossless) - 28 (default) - 51 (worst)

Multiple files

Change the file extension, and CRF value to whatever you need

mkdir ffmpeg_output
for file in *.mkv; 
	do ffmpeg -i "\$file" -vcodec libx264 -crf 20 "ffmpeg_output/${file%.mkv}".mkv
done

Use this one liner to easily find Raspberry Pi’s via ARP request on the current Subnet

Command

arp -na | grep -e "b8:27:eb" -e "dc:a6:32" -e "e4:5f:01"

On a server with many drive bays (hard drives, HDD, SSD) it can be hard to find which one the software is referring to. Use “ledmon” to identify the drive using the built-in LED’s

Steps

Install ledmon (it’s called that in Ubuntu repos)

Identify your drive: sudo ledctl locate=/dev/sdb

You should see the LED begin to flash

You can turn off the locate when done with: sudo ledctl locate_off=/dev/sdb

Use this script to list every file in the directory, and it’s video codec. Try sending the stdout to a file to view the info in a spreadsheet.

Script

#!/bin/bash

# Field seperator fix for file names with spaces
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")

# Loop through every file
for file in $(ls)
do
	echo $file "|" $(ls -l --block-size=G $file | awk '{print $5}')  "|" $(ffprobe -v error -select_streams v:0 -show_entries stream=codec_name -of default=noprint_wrappers=1:nokey=1 $file)
done

IFS=$SAVEIFS

SSHuttle works pretty well as a quick and dirty solution, but is really slow so use it sparingly.

Commands

To connect with sshuttle sshuttle -r USERNAME@SERVER_IP 0.0.0.0/0 -vv

Sudo should not be used if you’re using ~/.ssh private keys but sshuttle will request elevation Use 0.0.0.0/0 to route all traffic, or specify an IP/CIDR

If that doesn’t work, try this: sshuttle -r USERNAME@SERVER_IP -x SERVER_IP 0/0 -vv

To specify private keys add: --ssh-cmd 'ssh -i .ssh/NAME'

Note: More ‘-v’ makes it more verbose

This walks you through setting up Wireguard. It is focused towards a traditional Server/Client VPN structure, however if you understand how it works you can make other configurations.

Good unofficial docs Arch wiki, good info but I had issues with their configs

Understanding AllowedIPs

This is a confusing option, since it seems like it means two things depending on client or server. Really it’s one thing, but it’s complicated. On the client under [Peer], AllowedIPs is the IP’s that will be routed through this Wireguard tunnel. On the server, it is the IPs that the clients will be allowed to use

net.ipv4.ip_forward

Sometimes it’s necessary to enable this to actually connect to devices on the remote LAN. Temporarily enable with this command: sysctl -w net.ipv4.ip_forward=1 Permenantly enable by adding a file /etc/sysctl.d/98-wireguard.conf (name is for organization purposes) with contents: net.ipv4.ip_forward=1

Client config

[Interface]
Address = 192.168.30.12/24 (IP you want the client to have. Corresponds to server AllowedIPs but notice different CIDR)
PrivateKey = [key content]

[Peer]
PublicKey = [key content]
PresharedKey = [key content] (Optional - use wg genpsk)
AllowedIPs = 192.168.0.0/16 (See Above)
Endpoint = [Server IP]:51820 (Server IP with port to connect on)
PersistentKeepAlive = 25

Server config

[Interface]
Address = 192.168.30.1/24 (VPN gateway and subnet)
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o enp4s0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o enp4s0 -j MASQUERADE (Replace enp4s0 with your interface)
ListenPort = 51820
PrivateKey = [key content]

[Peer]
# Nickname
AllowedIPs = 192.168.30.12/32 (IP you want the client to have)
PublicKey = [key content]
PresharedKey = [key content] (Optional - use wg genpsk)
PersistentKeepAlive = 25

This guide is not complete, and should be re-written when I setup a ZFS Zpool again. However, it should capture some of the issues I had last time.

Finding WWN

The WWN is usually best to use as a unique identifier for the drives then partuuid, since UUID applies to partitions, but WWN is physical drives. You can use the /dev/sdaX however those may change on reboots, rendering the ZPool degraded.

Use this command to list drives and their WWN: lsblk -o name,size,mountpoint,wwn

Keep note of the WWN’s for the drives you want to use

Creating the ZPool

Create the relevant type of ZPool using this command, modified as needed. I’ll create a mirored vdev: zpool create {NAME} mirror /dev/disk/by-id/{WWN IDENTIFIER} /dev/disk/by-id/{WWN IDENTIFIER}

Viewing Status

Use these two commands to view the status: zpool status zpool list

Configure Scrub

It is a good idea to configure a regularly scheduled scrub, such as once or twice a month. The command is: zpool scrub {NAME}

The plan for the blog portion of this site has always been, in part, posting simple how-to documentation for friends to view. That continues to be the primary goal of the blog. I haven’t been very consistent with actually writing documentation in my personal life though. That’s changed this year with my personal discovery of Obsidian.

Obsidian

I highly recommend anyone who likes taking notes wander their way over to the Obsidian website, and see what makes it wonderful. I care about it for

• Vendor Tie-In: I never really have to worry about moving away from Obsidian, since it just uses locally stored markdown files. Moving to another platform is just a matter of copying the raw files over if it supports markdown, or converting them somehow otherwise.
• Portability: Due to the nature of the files, it’s completely up to the user how they want to handle the files. Locally store? Done. Sync between devices with Syncthing? Easy. Git? Cryptomator? Google Drive? OneDrive? All of these are easily implemented to fit in with whatever your workflow is. And if you really want it, they offer their own sync service.
• Graph view: Discoverability of notes, and visualizing how complex thoughts and documentation relate to each other is significantly better on Obsidian than any other note application I’ve used. This is particularly valuable for sharing documentation with someone new.
• HUGO: It goes without saying, but since it’s based on markdown, I can just copy the files over to HUGO and post to this site.

I’ve spent many, many hours documenting my existing personal infrastructure, and guides on how to replicate things I’ve done in Obsidian. If I think people I know could benefit from it I’ll copy them over to this site. Expect to have a lot more files joining the blog for the next little bit! Keep in mind, it won’t be everything, and it generally won’t be anything I wrote for work. So expect lower quality guides than I am capable of. These are all written pretty quick, and I just want to share them with people.

Final Thoughts

As always, I’m constantly learning new things! If I’ve made a mistake about something on the site, feel free to reach out to me on the contact page and let me know!

As a broad overview, RDRAND is a method of returning pseudo-random numbers from the CPU. Originally, it was developed by Intel, however it has since been implemented on AMD CPU’s, which is of course where this bug lies.

If you are a Windows user, you can still be affected by this bug, you’re just less likely to notice. The theoretical security implications will still affect you, and you should still fix it.

Further reading: Wikipedia - RDRAND

What is AMD Ryzen’s RDRAND Bug?

On some Ryzen CPU’s, whenever RDRAND is called it will return the extremely random value of 0xffffffff every single time. I would highly recommend reading Jim Salter from Ars Technica’s article on this bug since it will explain more about the bug, and for testing I will have you using his code later.

This has been a very irritating bug that has afflicted me for a while. Using a Linux system, I have found that Linux will soft lockup at random periods. Usually during low CPU usage such as browsing the internet. The GUI would lock up, and I would be unable to click, or use any UI elements. I could still switch to another TTY, however I was unable to recover the GUI.

According to Jim Salter, these soft lockups are related to SystemD’s handling of bad RDRAND output. Although I’m admittedly unsure as to what SystemD is specifically doing with RDRAND that causes this. If you have an idea, please email me and let me know!

While I felt that while Jim Salter’s article was fantastic, it is a little unclear for those unfamiliar with Linux on how to test it. This step-by-step guide should show you how to test for it yourself, and again I will be using same code Jim Salter provided.

Step-By-Step Testing Guide

If you are already using Linux, or know how to setup Linux bootable disks, skip to step 4 for testing

1. Download necessary files

   • We will be using Ubuntu for this tutorial, which can be downloaded here

   • Download and launch Rufus

2. Flash the iso to a thumb drive

   1. In Rufus, select your thumb drive under ‘Device’. Note that all data on this device will be destroyed!
   2. Under ‘Boot selection’ choose your Ubuntu .iso file
   3. ‘Partition scheme’ should be MBR, and ‘Target system’ should be BIOS or UEFI
   4. The rest of the settings can remain the same, click ‘START’ and wait for Rufus to finish

      Image from https://rufus.ie

3. Reboot into Ubuntu

   1. With the thumb drive plugged in, reboot to your computer’s BIOS. Exact steps are dependant on your motherboard manufacturer, but usually during boot you can keep pressing the F12 or Delete key
   2. Once in your BIOS, look for a boot menu and select your thumb drive to overwrite the boot order
   3. Ubuntu may prompt you to “Try Ubuntu without installing” or “Install Ubuntu”. You will choose Try Ubuntu without installing

      You should not encounter any partitioning at any time during this tutorial. If Ubuntu is asking you to partition, you may have chosen “Install Ubuntu” which is not what you’re looking for

   4. You should see Ubuntu loading, and eventually see a desktop
   5. Connect to the internet by using the WiFi button at the top right, searching “WiFi”, or by just using ethernet

4. Launch Terminal, and test for RDRAND bug

   1. Press your ‘Windows key’, or click the 9 dots at the bottom left to bring up your system search

   2. Search “Terminal” and open up the corresponding application

   3. Enter this command to download Jim Salter’s test application: wget https://cdn.arstechnica.net/wp-content/uploads/2019/10/rdrand-test.zip

   4. Enter this command to unzip the archive: unzip rdrand-test.zip

      If the unzip command is not found, enter this command (you will need an internet connection): sudo apt install unzip

   5. Run the amd-rdrand-bug application: ./rdrand-test/amd-rdrand-bug

   6. The output should tell you whether or not you are affected

Fixing the RDRAND Bug

In the unfortunate event that you are affected by this bug, you are at the whim of your motherboard manufacturer to patch it. Fortunately, this bug was patched by AMD about a year ago as of writing, so your manufacturer has likely added the fix to their firmware.

To install the patch you need to update your motherboard’s BIOS/firmware. Unfortunately I cannot write a step-by-step guide on how to do that, since the instructions very depending on which motherboard you own. Simply search “BIOS update” with your motherboard make and model online, and you should find your manufacturers website with instructions.

Warning: Be careful when updating your BIOS/firmware. If you restart your computer, or lose power during the update your motherboard may be bricked and have to be replaced

Follow the directions provided by your motherboard manufacturer carefully

Once your motherboard’s BIOS is updated to the latest version, be sure to go through the step-by-step guide once more (you can skip straight into booting into Ubuntu. No need to re-make the thumb drive)

Running the test program should now tell you that your do not have the AMD bug.

Closing Thoughts

This isn’t the type of tutorial I would normally like to put on this website. Most of the stuff will be more technical then this. However, I did feel that there was very few tutorials online about testing for this common bug, so hopefully this tutorial will be helpful to someone!

The simple answer is I’ve always wanted a website. I grew up during the Yahoo Geocities days, and back then everyone had made their own website. Sure, most of the time they were hideous, brightly colored, horrible animated messes, but they were also unique. They weren’t built using drag and drop utilies, or often times even templates. A websites content was solely determined by it’s creator. Every title, dropdown, and image was part of the vision of some 12 year old.

Nowadays, personal websites are uncommon, and those that do exist usually just hold resumes. The decentralized internet that once held people’s blogs, vacation photos, and even dancing dinasaur gifs has been replaced with corporate services like Facebook and Twitter. While there are advantages to using those services, the downside is a lack of individuality and independence I remember from ‘the internet of old’

Another reason is that I often times get asked a lot of questions about self-hosting, privacy, etc. And while I’m certainly not an expert in any of these topics, I’d like to scream my opinion and experiences into the air. After all, what else is the point of the internet?

Why not use Wordpress?

Every decision is a balancing act between the pro’s and con’s of each available option. For me, the con’s of Wordpress far outweigh the pro’s.

Don’t get me wrong, Wordpress is a fantastic service, and there are a lot of situations where I would recommend using. However, I have no plans to make my site very complex, have interactive content, a storefront, or anything else that Wordpress excells at.

There are a few downsides to using Wordpress on a simple site like mine. The main issue is added complexity. This complexity can make a website difficult to manage for someone who isn’t working on it every day. Complexity can make a website load slower, requiring me to pay for better hosting, or upgrade my self hosted equipment. As it stands, I pay for this website to be hosted but it’s practically free. Complexity can add security vulnerabilies that for many sites are completely unecessary. Most Wordpress vulnerabilities come from plugins and themes, but anytime you’re running a small application in a web browser you’re going to have more vulnerabilites than a simple page displaying formatted text.

To summarize, KISS

What do you do when you don’t want to fully write your own website, don’t want to have massive Javascript applications or Wordpress, but still want it to be easy on the eyes? Queue static site generators!

Experience with Hugo thusfar

So far Hugo has been fantastic. I spent a bit of time on Hugo’s website to find the right theme for me. After that it was just a matter of installing Hugo (which could be done through Ubuntu’s built in repo’s) and adding the theme. Now I have spent the majority of the time doing what I feel is most important: adding content to the site.

Writing this article was a great experience. With the Hugo server running in the background, all I needed to do was hit save in my text editor and my browser automatically updated the newest content.

If you’re thinking about setting up your own website, I highly recommend looking into Hugo and seeing if it will fit your needs, before moving to expensive options.