Nevermind, problem solved.
After troubleshooting this with Visual Studio Code and Putty, on various USB ports, with the USB Downloader board and an ESP-Prog board, and even rebooting my computer to check driver issues...
... it turns out I have a bad pin on the breadboard that I've been using to test these. It somehow failed after the last firmware update, direct connections to any of my boards works just fine.