site banner
ZorbaTHut  2mo ago (text post)   3604 thread views

Site Rollback to February 3rd

Very sorry about this one, guys.

For reasons that I'm currently unsure of, the database decided to eat itself. I'm giving it something like 40/40/20 "hacker", "postgres bug", or "host glitch".

The site currently has two different backup methods. The manual backup triggers whenever I do a site update; the last site update was on 1/31. The automatic backup is supposed to be daily, and I check it once in a while to make sure it's working. It's been working literally ever since before the main Motte site was launched . . . and it broke on 2/4. Good timing, thanks, system.

In theory, the quality-contribution system captured all reported quality contributions during this time. I'll try to retrieve those. Any opinions on whether I should just go ahead and repost them, or whether I should send them to the person so they can repost them?

I do have a dump of a bunch of text snippets that are all that was left of the database. If you remember some phrases you used I might be able to retrieve parts of lost posts. That said, someone's tried this with a few posts and got 0/2, so absolutely no promises here. Feel free to ask though! If you want to take data recovery more seriously, make a copy of your browser cache, which can then be pored over to find people's posts. I'm not totally sure how important this is, but I bet at least a few people will be sad to lose effortposts they made.

I've fixed the backup issue and set up better monitoring so it will yell at me if it fails again. I've also temporarily increased backup frequency to hourly, just in case there's some serious stability issue right now that I'm not aware of. The good news is that this shouldn't happen again, at least with as much lost data. But that doesn't really fix this one.

Apologies again.

This too shall pass.

17
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

Test

Ehm... Should i be logged in as you?

That was me. I didn't do anything, I was just logged in as you when I went to the website.

Maybe shut things down until you figure this out? It doesn't seem like a good idea for people to automatically be logged in as head admin.

@ZorbaTHut

Oy. That doesn't surprise me, honestly, I should probably clear logins just in case weird stuff happened.

Offline again, back in a few :V

Can't tell if serious security bug, or Zorba is having a schizophrenic moment.

Like in Life of Pi, I choose to believe it is actually both because that's a much more interesting world to be living in.

I can think of a few ways the site could go kind of weird with regards to login tokens during an incident like that. Easiest way to solve it is just to reset the login signature code.

Now everyone gets to log in again also, which inevitably means I'll get like six emails from people who entered a random password and didn't bother to save it and also didn't set their email so they could recover it.

I don't know if you're all still on friendly terms with Aevann but this [site issue followed by people being logged in as the main admin] was common in the first six months or so of rdrama, there's likely documentation and a path to assistance.

Nah, this was honestly expected - the login system is based on user ID, and if the user IDs get reset and you get a token for a now-available user ID, that token is still valid once the site reboots. No particularly great mystery.

The session tokens identify logged-in user by user ID. This is cryptographically signed serverside. The first person to make an account on the empty site got the server to give them a valid signed session token for UID 9, which persisted after restoration. Anyway, changing the signing key like you did fixes it.

Yeah, it's a little weird because this implies that @loper happened to be the one who made the admin account, and I'm not convinced they're the one who made the admin account because I think they would have mentioned it.

(loper are you the one who made the admin account when the database melted a bunch of hours? if so, that answers that question, I guess)

(also, if so, I'm so curious what you thought was happening)

I might have. I tried to visit the site earlier today and couldn't view anything or log in so I tried creating an account but that didn't change anything. But this was like 10 hours ago. I still wasn't "let in" and I got a 503 service unavailable error until now, at which point I was logged in as you.

The only thing is that is perhaps a bit strange was that the empty site persisted for pretty long and I wasn't particularly quick starting to try different ways to get in. I've created test accounts previously when the site has had issues and this has never happened before.

I think that name was Kaldrop. And if not, what I remember is that a web search for the name should also return something about services for e-commerce sellers.

More comments

What was the name of the account you created? I saw the new admin account, I can't remember its name exactly, but I can tell if something is that name or not.

Huh. That actually might've been you, then, yeah.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats