This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Social science research done well is not going to reach the levels of hard science, of course, but it is certainly not worthless if done well. There are various strategies to establish validity, reliability, etc. in quantitative measures, and credibility, transferability, etc in qualitative research. The problem is ideological capture, wrongthink, and institutional guardrailing. As well as a general lack of rigor, even when the methods to maintain such rigor and force accountability exist.
Can you give us some examples?
It will all depend on what you're measuring and how you're measuring it. Quantitative and qualitative analyses are two completely different paradigms (though they are often married in mixed-methods research.) They don't even use the same vocabulary, and the product of their aims is fundamentally different.
A quantitative look at divorce, for example, might have a questionnaire as its main instrument. You get a bunch of numbers and it turns out the main predictor of divorce (note: not cause. Predictor) is contempt (as one of my friends likes to say whenever the topic comes up--I think he read something by Gladwell.) A qualitative study wouldn't have a questionnaire at all. You'd probably use interviews as participant observation of a marriage would have intrusion and ethical issues involved. You might end up with several narratives where you have a man recounting how his wife would always take her phone in the bathroom and stay far too long, as if sending messages, or a wife would talk about how the husband suddenly started washing his own clothes when he got home. Or whatever. The write-up for each would be totally different, and you'd be learning and digesting different things. A lot of people hate qualitative research. It's my main choice. It can be done very, very sloppily, however. I'll get to that in a second.
Back to making a quantitative questionnaire--if that questionnaire has not been, for example, piloted for content/face validity (via relevant experts who know the topic, but also item correlation/factor analysis) its results are more or less just noise (You also cannot use, for example factor analysis on just any set of questions. There are various requirements [a reasonable sample population, get rid of outliers, variables have to be on I believe an ordinal/interval scale, there has to be some degree of correlation in variables, you really need a normal distribution, etc.] If you don't fulfill these prerequisites again you're producing noise.) Once you have an instrument (questionnaire/whatever) that is relatively polished you administer it, but then in order to determine whether your results will be generalizable you'll need an appropriate sample size (and even then depending on the randomness of your sample your results will only generalize to that population, not the entirety of humans in all cultures).
Research design is also important. What you want to measure will determine how you measure it. I do not mean that you cater your plan to find a conclusion you want. I mean that if you want to measure something you need the right tool. Statistics do not show causation. A research design using statistics can establish causation (as in the numerous studies on cigarette smoking and cancer.) There are also criteria establishing causation.. The old saw correlation does not prove causation is of course true, though often positive or negative correlation is the smoke where there is fire. But not always, or even most of the time, in my opinion (which is worth about five cents.)
There are numerous threats to quantitative validity. Regression to mean, compensatory rivalry (in the case of two groups), compensatory equalization (when a researcher interferes), history, maturation, etc. etc. I highly recommend the book The Research Methods Knowledge Database (<- this is a US Amazon link, which I changed from the Japan link, but I cannot recall where you are.) I am not sure how much of this book no longer holds but it was a great read when I first went through it.
Qualitative research (what I do) can seem wild and woolly to an outsider, as if it's just you talking to someone and writing down your thoughts. And that is probably not a bad description of it, though there are, again, all sorts of rules you need to follow, primarily but not only to set the groundwork for reproducibility. And that, as we have seen, is a problem in both types of research--the findings are not reproducible. Documenting exactly what you do at every stage is very important for this. One study requires mountains of documentation. If you have any interest in qualitative research (and I would say the median user on the Motte does not) I would recommend the book Naturalistic Inquiry.
I have probably not answered your question, but I have let this sit and I didn't want to write something fast with my thumb. And you may know all of the above and more. You also may not find any of this compelling. I am not particularly good at statistics, though I used to use SPSS with some degree of expertise when I was made to do so. I believe that software may be outdated, or at least not used as much as it was. The Bayesian wave of the last 20/30 years has also changed a lot and I am out of the loop. Note I have known many researchers in social sciences, most of them what I would consider well-meaning ignoramuses, some of them outright frauds, this in both quantitative and qualitative. I have known only one man who I thought was a genius psychometrician; he didn't rely on the software as he knew the formulas and why they were the way they were. A math guy, like many of you (maybe not you-you, but the general Motte You.) He died far too early of a brain tumor in one of life's cruel ironies.
I have read a lot of shoddy research. Far too much. Enough to at least doubt (if not completely dismiss) every study that I see, most seriously when the study confirms my own biases. But then I think that's the right way to live. I could be wrong.
edit for clarity and this is still not very clear. My parentheses runneth over.
More options
Context Copy link
Standardized methods is one way. This at least lets you compare between studies that use the same methods, and thus determine if you find the same thing. The best methods have clearly defined scopes that indicate what they are and are not looking for. So the authors should be well aware of limitations well before starting their studies.
You are also supposed to be aware of your biases and validity threats, and admit to them in your writing so that the reader can easily understand your limitations.
It's just that a lot of researchers don't do this. I remember reading feminist papers, cited by journalists and used in political arguments, that make conclusions not supported by their results or where the methods are clearly designed to create a bias which is never admitted. The opportunity for rigor was clearly there, but the authors and reporters decided to forego it.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link