#schmutz #kubernetes #homelab

As is tradition i have not been writing much. Not for any particular reason except lack of desire which i suppose is reason enough.

Homelab

i finally got my k3s homelab cluster working. It took a lot of trial and error and hardware. Initially i wanted to use the libre.computer raspberry pi clones but those were not beefy enough for all the services i wanted to run. that took awhile to figure out. Along the way i learned a bunch tho and i upgraded my hardware to some Lenovo thinkcentres. i was able to mount an additional ssd inside for a dedicated longhorn disk per node. i went with the HA k3s install. Another lesson learned: my cluster was really unstable and nodes kept going down but with embedded etcd there was sometimes split-brain behavior (pretty sure that is what was happening). i started again with external etcd and i haven’t run into that problem even with the instability.

That leads me to the instability: i couldn’t understand what was going on. i was using the Tailscale k8s operator for ingress cuz i was too lazy to learn how nginx ingress works. This was fine for awhile but then my services were having all kinds of network issues and i thought it was due to all the extra network hops from tailscale. i ended up learning nginx-ingress and using external-dns with my pihole and cert manager for an internal CA for https… wow it took a bit to figure that out but it wasn’t too bad in the end. So even after that, longhorn was constantly having degraded volumes. i turned to iperf after noticing the reason for the failed rebuilds on the volumes was a connection timeout. Turns out the switch i was using seems to be unable to handle 3 simultaneous GbE connections and it was auto-negotiating one of my nodes down to 10Mb/s!! i learned this via iperf and then changing some ports around on my switch/router then checking cat /sys/class/net/<interface>/speed. For now two nodes are into the switch and the third is directly into the router. Luckily everything is right underneath the router otherwise i’d need to wait for a new switch or move stuff. Eventually i’ll get a beefier switch but for now this is fine and i am super happy to have solved the mystery.

Now i am still migrating sleepycraftstudios.com over to be self hosted but it should be a lot smoother now that things actually work! The staging site is migrated over and it took a bunch of stuff to replace cloudflare and tbh i’ll probably still use cloudflare via the tunnel feature to expose the service to the web, but i won’t be reliant on cloudflare pages which had a less than stellar local-dev experience (depending on what you’re doing of course. For my needs it was bad).

Moar Services

Now that i am comfortable spinning up my own services on my k8s homelab, i want to move over my other non-self-hosted stuff over, including this blog which is currently hosted on the Vercel free plan, and the blog’s API which houses the content that is synced from obsidian using my fancy obsidian-diy-sync plugin! the server for that is currently hosted on Render.com but i want to move it too. Then i’ll only need to pay for storage backups 😊 (i realize the costs of homelab including hardware, power, and internet so yeah it’s not the only cost…).

Thanks for reading, see you next time!