Homelab Log 8: Builing harnesses is hard without my team

Shoutout to the heroes in DevOps out there.

My day job is an AI Engineer, and i've grown to appreciate my DevOps/Platform/Security team so much more through this process.

Currently building out the orchestration framework and security around the locally run homelab. By far the hardest part (minus that thermal issue lol) has been new model deployment rollouts, dealing with drivers for ROCm, docker/harbor, full failure recovery, and system monitoring and alerts.

This is stuff I never deal with in my day to day. I mainly work on testing the functionality of new models or harness design from a performance/outcome and deterministic vs probabilistic lens. I rarely have to deal with the headache of managing system-wide upgrades or booting up gracefully after any system interrupts or restarts.

....safe to say i'm buying the team doughnuts tomorrow when I get to work.