site banner

Site Rollback to February 3rd

Very sorry about this one, guys.

For reasons that I'm currently unsure of, the database decided to eat itself. I'm giving it something like 40/40/20 "hacker", "postgres bug", or "host glitch".

The site currently has two different backup methods. The manual backup triggers whenever I do a site update; the last site update was on 1/31. The automatic backup is supposed to be daily, and I check it once in a while to make sure it's working. It's been working literally ever since before the main Motte site was launched . . . and it broke on 2/4. Good timing, thanks, system.

In theory, the quality-contribution system captured all reported quality contributions during this time. I'll try to retrieve those. Any opinions on whether I should just go ahead and repost them, or whether I should send them to the person so they can repost them?

I do have a dump of a bunch of text snippets that are all that was left of the database. If you remember some phrases you used I might be able to retrieve parts of lost posts. That said, someone's tried this with a few posts and got 0/2, so absolutely no promises here. Feel free to ask though! If you want to take data recovery more seriously, make a copy of your browser cache, which can then be pored over to find people's posts. I'm not totally sure how important this is, but I bet at least a few people will be sad to lose effortposts they made.

I've fixed the backup issue and set up better monitoring so it will yell at me if it fails again. I've also temporarily increased backup frequency to hourly, just in case there's some serious stability issue right now that I'm not aware of. The good news is that this shouldn't happen again, at least with as much lost data. But that doesn't really fix this one.

Apologies again.

This too shall pass.

17
Jump in the discussion.

No email address required.

Test

Ehm... Should i be logged in as you?

That was me. I didn't do anything, I was just logged in as you when I went to the website.

Maybe shut things down until you figure this out? It doesn't seem like a good idea for people to automatically be logged in as head admin.

@ZorbaTHut

Can't tell if serious security bug, or Zorba is having a schizophrenic moment.

I can think of a few ways the site could go kind of weird with regards to login tokens during an incident like that. Easiest way to solve it is just to reset the login signature code.

Now everyone gets to log in again also, which inevitably means I'll get like six emails from people who entered a random password and didn't bother to save it and also didn't set their email so they could recover it.

I don't know if you're all still on friendly terms with Aevann but this [site issue followed by people being logged in as the main admin] was common in the first six months or so of rdrama, there's likely documentation and a path to assistance.

Nah, this was honestly expected - the login system is based on user ID, and if the user IDs get reset and you get a token for a now-available user ID, that token is still valid once the site reboots. No particularly great mystery.