I’ve been running my own node now for nearly 9 months and when it was built, the build documentation recommended a 512GB SSD. At the time I had one laying around so I used it, but honestly I knew this day was coming as I watched the free space get eaten up by the blockchain growth over time. I’m also not alone in this either with forums filled with comments about needing to upgrade their storage as well.
The blockchain will only get bigger, not smaller and fortunately the cost of storage is also dropping: the 500GB drive cost about $300 AUD six years ago, and the 1TB same brand similar model today cost only $184 AUD. In future upgrading to a 2TB SSD will probably cost $100 or less in another five years or so time.
This update is going to take a few hours, so during that time obviously your node will be offline. It can’t be helped.
My goals:
- If possible, use nothing but the RaspiBlitz hardware and Pi 4 USB ports (SPOILER: Not so good it seems…)
- Minimal Risk to the existing SSD to allow an easy rollback if I needed it
- Document the process to help others
ATTEMPT 1
- Shutdown all services currently running on the RaspiBlitz
Extracted from the XXshutdown.sh script in the Admin Root Folder:
sudo systemctl stop electrs 2>/dev/null
sudo systemctl stop lnd 2>/dev/null
sudo -u bitcoin bitcoin-cli stop 2>/dev/null
sleep 10
sudo systemctl stop bitcoind 2>/dev/null
sleep 3
[Only use this if you're using BTRFS]: sudo btrfs scrub start /mnt/hdd/
sync
- Connect and confirm your shiny new drive
sudo blkid
The following is a list of all of the mounted drives and partitions: (not in listed order)
- sda1: BLOCKCHAIN Is the existing in-use SSD for storing the configuration and blockchain data. That’s the one we want to clone.
- sdb1: BLITZBACKUP Is my trusty mini-USB channel backup drive. Some people won’t have this, but really should!
- sdc1: Samsung_T5 Is my new SSD with the default drive label.
- mmcblk0: mmc = Micro-Memory Card - aka the MicroSD card that the RaspiBlitz software image is installed on. It has two partitions, P1 and P2.
- mmcblk0p1: Partition 1 of the MicroSD card - used for the boot partition. Better leave this alone.
- mmcblk0p2: Partition 2 of the MicroSD card - used for the root filesystem. We’ll also leave this alone…
If you want more verbose information you can also try:
sudo fdisk --list
- Clone the existing drive to the new drive:
There’s a few ways to do this, but I think using the dd utility is the best option as it will copy absolutely everything from one drive to the other. Make sure you specify a bigger blocksize - the default of 512bytes is horrifically slow, so I used 64k for mine.
sudo dd if=/dev/sda1 of=/dev/sdc1 bs=64k status=progress
In my case, I had a nearly full 500GB SSD to clone, so even though USB3.0 is quick and SSDs are quick, this was always going to take a while. For me it took about three hours but I finally got this error:
dd: writing to '/dev/sdc': Input/output error
416398841+0 records in
416398840+0 records out
213196206080 bytes (213 GB, 199 GiB) copied, 10896.5 s, 19.6 MB/s
Thinking about it, the most likely cause was a dip in power on the Raspiblitz. The tiny little device was trying to drive three USB drives and most likely there was a momentary power dip driving them all, and that was all it took to fail.
ATTEMPT 2
Research online suggested it would be much more reliable to use a Linux distro to do this properly. I had no machines with a host-installed Linux OS on it, so instead I needed to spin up my Virtual Box Ubuntu 19.04 VM.
It was safe enough to power off the RaspiBlitz at this point, so I do that then disconnect both drives from the Pi, then connected them to the PC.
To get VirtualBox to identify the drives I needed to enable USB 3.0 and then add the two drives to the USB filter, reboot the VM and then ran the above but now under Virtual Box.
499975847936 bytes (500 GB, 466 GiB) copied, 4783 s, 105 MB/s
7630219+1 records in
7630219+1 records out
500054069760 bytes (500 GB, 466 GiB) copied, 4784.58 s, 105 MB/s
This time it completed with the above output after about 1 hour and 20 minutes. Much better!
If you want to confirm all went well:
sudo diff -rq sda1 sdc1
An FDISK check now yields this error:
GPT PMBR size mismatch (976773167 != 1953525167) will be corrected by write.
The backup GPT table is not on the end of the device. This problem will be corrected by write.
- Resizing the new drive Step 1
In my case I started with a 500GB drive and I moved to a 1TB drive. Obviously you can use whatever size drive you like (presumably bigger) but to utilise that additional space, you’ll need to resize it after you clone it.
sudo gdisk /dev/sdb
x (Expert Menus)
e (Move GPT Table to end of the disk)
m (Main Menus)
w (Write and Exit)
Y (Yes - do this)
All this does is shift the GPT table away from the current position in the middle of the disk to the end - without doing this you can’t resize it.
- Resizing the new drive Step 2
There’s a few ways to do this step, but in Ubuntu there’s a nice GUI tool that makes it really simple. Starting from the Ubuntu desktop install GParted from the Ubuntu Software library, then open it.
Noting the maximum size and leaving the preceding space alone, I adjusted the New size entry to 953,838 leaving 0 free space following. Select Resize/Move then Apply all operations (Green Tick in the top corner) and we’re done.
- Move the new drive back to the RaspiBlitz and power it on.
Hopefully it starts up and works fine. :)
Conclusion
I left this far too long and far too late. Much later than I should have. My volume was reporting only 3GB free space and 100% utilisation which is obviously not the right approach. I’d suggest people think about doing this when they hit 10% remaining and not much more than that.
The Bitcoin/Lightning also hammers your SSD, shortening its life so swapping out for an identically sized drive would follow all steps except 4 & 5 and should work fine as well.
Whilst this whole exercise had my node offline for 36 hours end to end, there were life distractions, sleep and a learning curve inbetween. It should really only take about 2-3 hours for a similar sized drive.
Good luck!