skarnet.org will be down until further notice - it may take up to 10 days (!)
-
That was way faster than expected! Some person read my emergency ticket sent last night (from logged out), and I seem to have been convincing enough because they restarted the server for me, and now everything is working again. π₯³
Guess I won't have these 10 days of forced vacation.
Now to take measures so that it doesn't happen again...
@ska i guess "please restart this vps that is hang anyway" is a easier ask to "give me access to this account" -
@ska i guess "please restart this vps that is hang anyway" is a easier ask to "give me access to this account"
@navi Absolutely. I had asked to either send a recovery code to my gmail account, or reboot the server. They rebooted the server. Totally fair.

-
@navi Absolutely. I had asked to either send a recovery code to my gmail account, or reboot the server. They rebooted the server. Totally fair.

-
@lanodan @navi https://git.skarnet.org/cgi-bin/cgit.cgi/s6-linux-init/commit/ - not pasting the permanent URL to avoid people getting caught by my LLM blocker, the commit id is 43394581bb3106633c8802b378a7f2e6e53b352a
(Edit: add the commit id)
-
@lanodan @navi https://git.skarnet.org/cgi-bin/cgit.cgi/s6-linux-init/commit/ - not pasting the permanent URL to avoid people getting caught by my LLM blocker, the commit id is 43394581bb3106633c8802b378a7f2e6e53b352a
(Edit: add the commit id)
-
@lanodan @navi I never have problems booting thanks to the static guarantees of s6-rc, whenever I have an issue it is shutting down because after some living, the state may be different from what I assumed it would be.
But I'm learning, and incrementally fixing assumptions about state, this is only the latest case - and hopefully the last one.
-
@lanodan @navi I never have problems booting thanks to the static guarantees of s6-rc, whenever I have an issue it is shutting down because after some living, the state may be different from what I assumed it would be.
But I'm learning, and incrementally fixing assumptions about state, this is only the latest case - and hopefully the last one.
-
@navi Spreading your dependencies is clearly a more resilient setup and you're right to do so.
But the thing is, this incident also confirms me in my belief that it's a good thing for me to have control on as many things as possible, because as soon as I depend on one external entity, they fuck something up. So I want to be self-reliant even more.
And I guess my external line will now be gmail, for the sole reason that they're too big to fail. Which I don't like, but I'm going to use that property.
-
@lanodan @navi You can always force a shutdown until it's on a remote VPS and the shutdown worked just well enough to kill everything including your sshd and your emergency getty and leave you with a brick and you're logged out of the VPS administrative account and the 2FA codes are sent to the brick.

(Edit: accuracy)
-
@navi What's the alternative? Paying for a second VPS hosted elsewhere? Having to manage another e-mail address?
One of the points of skarnet.org is maximum independence. And really, it has worked pretty well so far: I could always recover from mistakes.
Now my new recovery address will be my gmail one. Lovely. I could change and ask for some hosted address somewhere else, but that defeats the purpose of being self-reliant.
Really, it all comes down to "don't cut my lifeline without telling me you fucking clowns".
@ska@social.treehouse.systems @navi@social.vlhl.dev Having a backup MX or a backup DNS server would give options. While mail often can't be run on residential lines, DNS usually can.
A backup MX will receive and store email to be forwarded, and you can just look directly in the mail queue to get the auth code, for instance.
I completely agree that companies' ideas about two factor are broken, are often added without asking and without configuration, and can be highly problematic during emergencies. At least you're here, sharing with others, so others can consider whether they might be in the same scenario should a problem occur.
-
@ska@social.treehouse.systems @navi@social.vlhl.dev Having a backup MX or a backup DNS server would give options. While mail often can't be run on residential lines, DNS usually can.
A backup MX will receive and store email to be forwarded, and you can just look directly in the mail queue to get the auth code, for instance.
I completely agree that companies' ideas about two factor are broken, are often added without asking and without configuration, and can be highly problematic during emergencies. At least you're here, sharing with others, so others can consider whether they might be in the same scenario should a problem occur.
@AnachronistJohn @navi that's the point of the thread: so others can learn from my misadventure

Setting an external secondary DNS server is something I have thought about, obviously, but the answer was always "not worth it" because only this server is under skarnet.org so losing the DNS means losing access to a server that is down anyway.

To have enough redundancy to avoid what happened to me, the external machine would also need to be MX and mail storage. I don't want to install that myself (duplicate work), I don't want to pay for that, and that's a little too much infrastructure to ask friends. I'd rather learn the lesson and not be an idiot next time

-
That was way faster than expected! Some person read my emergency ticket sent last night (from logged out), and I seem to have been convincing enough because they restarted the server for me, and now everything is working again. π₯³
Guess I won't have these 10 days of forced vacation.
Now to take measures so that it doesn't happen again...
Was the procedure to wait for a secret code via postal mail?
-
R relay@relay.mycrowd.ca shared this topic