site banner

Site Rollback to February 3rd

Very sorry about this one, guys.

For reasons that I'm currently unsure of, the database decided to eat itself. I'm giving it something like 40/40/20 "hacker", "postgres bug", or "host glitch".

The site currently has two different backup methods. The manual backup triggers whenever I do a site update; the last site update was on 1/31. The automatic backup is supposed to be daily, and I check it once in a while to make sure it's working. It's been working literally ever since before the main Motte site was launched . . . and it broke on 2/4. Good timing, thanks, system.

In theory, the quality-contribution system captured all reported quality contributions during this time. I'll try to retrieve those. Any opinions on whether I should just go ahead and repost them, or whether I should send them to the person so they can repost them?

I do have a dump of a bunch of text snippets that are all that was left of the database. If you remember some phrases you used I might be able to retrieve parts of lost posts. That said, someone's tried this with a few posts and got 0/2, so absolutely no promises here. Feel free to ask though! If you want to take data recovery more seriously, make a copy of your browser cache, which can then be pored over to find people's posts. I'm not totally sure how important this is, but I bet at least a few people will be sad to lose effortposts they made.

I've fixed the backup issue and set up better monitoring so it will yell at me if it fails again. I've also temporarily increased backup frequency to hourly, just in case there's some serious stability issue right now that I'm not aware of. The good news is that this shouldn't happen again, at least with as much lost data. But that doesn't really fix this one.

Apologies again.

This too shall pass.

17
Jump in the discussion.

No email address required.

That was me. I didn't do anything, I was just logged in as you when I went to the website.

Maybe shut things down until you figure this out? It doesn't seem like a good idea for people to automatically be logged in as head admin.

@ZorbaTHut

Oy. That doesn't surprise me, honestly, I should probably clear logins just in case weird stuff happened.

Offline again, back in a few :V

Can't tell if serious security bug, or Zorba is having a schizophrenic moment.

Like in Life of Pi, I choose to believe it is actually both because that's a much more interesting world to be living in.

I can think of a few ways the site could go kind of weird with regards to login tokens during an incident like that. Easiest way to solve it is just to reset the login signature code.

Now everyone gets to log in again also, which inevitably means I'll get like six emails from people who entered a random password and didn't bother to save it and also didn't set their email so they could recover it.

I don't know if you're all still on friendly terms with Aevann but this [site issue followed by people being logged in as the main admin] was common in the first six months or so of rdrama, there's likely documentation and a path to assistance.

Nah, this was honestly expected - the login system is based on user ID, and if the user IDs get reset and you get a token for a now-available user ID, that token is still valid once the site reboots. No particularly great mystery.

The session tokens identify logged-in user by user ID. This is cryptographically signed serverside. The first person to make an account on the empty site got the server to give them a valid signed session token for UID 9, which persisted after restoration. Anyway, changing the signing key like you did fixes it.

Yeah, it's a little weird because this implies that @loper happened to be the one who made the admin account, and I'm not convinced they're the one who made the admin account because I think they would have mentioned it.

(loper are you the one who made the admin account when the database melted a bunch of hours? if so, that answers that question, I guess)

(also, if so, I'm so curious what you thought was happening)

I might have. I tried to visit the site earlier today and couldn't view anything or log in so I tried creating an account but that didn't change anything. But this was like 10 hours ago. I still wasn't "let in" and I got a 503 service unavailable error until now, at which point I was logged in as you.

The only thing is that is perhaps a bit strange was that the empty site persisted for pretty long and I wasn't particularly quick starting to try different ways to get in. I've created test accounts previously when the site has had issues and this has never happened before.

I think that name was Kaldrop. And if not, what I remember is that a web search for the name should also return something about services for e-commerce sellers.

That was me. The name has no meaning. I usually just write something random that sounds like a word, which is probably the similar to how some people pick the name for their products.

What was the name of the account you created? I saw the new admin account, I can't remember its name exactly, but I can tell if something is that name or not.

Huh. That actually might've been you, then, yeah.