I am not a psychologist either, but I have been following your work for a couple of years and as a student of social sciences, somewhat aware of methodological challenges we face for creating representative samples.

I have often wondered about the possibility of doing experiments (and I don’t mean just dictator games/ ethnographic work) in real world settings. Of course they are messy to carry out and resource intensive, but isn’t most of empirical work in economics also based on household surveys? I agree with Jesse that educated internet savvy individuals is in a group in itself (already exposed lots of ideas/interested in scientific research) – so yes, it’s good data but demographic variables are also correlated to psychological variables to an extent. It might be interesting to learn more about communities that are not studied very often by mainstream social scientists.

PC

Jesse, I agree with you too, but I do think Jon has an overall point that demographic representativeness is not the be all/end all that it is sometimes made out to be. Interactions don’t just occur on demographic variables, but also on psychological variables which are rarely measured. Being able to generalize results is a function of not having a weird sample, but also a result of good measurement….and these days, I would say there is response bias in any research. I guess that’s the point your were making with your science analogy, so maybe I’m not adding anything new. Personally, I think all research is imperfect and therefore I only believe any data (representative, well measured, well designed, or not) if it is replicated enough times (a la fivethirtyeight.com in the last election). Non-representativeness is but one of many ways to be imperfect and is often not the most important imperfection to address.

An outsider’s comment … I became a fan of MFT a couple of years ago after my son introduced me to your TED talk from back then. As someone who has found value in the self-awareness that understanding these morals and how they impact our opinions, I’d like to see more people become aware of the test and results.

I’m also a user of facebook and I notice how much people there enjoy taking short surveys and exploring themselves and comparing themselves with others. Some surveys/tests are silly, like “Which hollywood star am I most like?” but others are more interesting. For awhile all my friends were taking and reporting their result from a “Meyer-Briggs like” test you could take online through facebook.

I have no idea if you’d view facebook as a more useful cross section or not, but I ensure you, if you wanted to create a “buzz” around the concepts and theory, integrating the YourMoral.org test into facebook such that people can learn of and take the test would open a large audience. The facebook integrated tests/surveys I’ve seen usually make it optional as to whether your results are announced (to friends and/or family) or not … or whether “just the fact you took the test” is announced (with a link that lets your friends take it too). NOTE: I suspect if I heard “Sally” took the test, that would be one thing. If I learned “Sally” took the test and she was an XYZ … I might be influenced when I jumped in and took the test myself. So the sharing of individual results may pollute the study … I’ll let the experts judge that)

Integrating with facebook would serve two purposes: A) create a larger awareness of the theory which serves those how possess the knowledge and B) it would expand the sample base. I should also add that I’m a conservative and it bothers me a little that there are 5 times as many liberals as conservatives in your current sample base. Thats nothing like what I’ve read is the rough cross section of the population (USA?).

Anyway … 2 cents from an outside observer.

I disagree with Jon here. Psychologists may not be that interested in making claims about exact percentages of national or international populations, but we DO want to make claims like “liberals care more than conservatives about Harm and Fairness, and conservatives care more than liberals about Ingroup, Authority and Purity.” And if we’re only studying liberals and conservatives who are educated and interested enough to find academic websites, what evidence do we have that our findings apply to liberals and conservatives in general? We’re not psychologically interested in educated and engaged ideologues, they’re just the ones easiest to reach. Subsample analyses help (and I agree with Ravi about the importance of such replication in our science), but if those subsamples are characterizing different kinds of educated, internet savvy people, then we still don’t know if we can make this claim about libs/cons in general, or only the kind of libs/cons who come to our website. Saying “but it’s higher-quality data this way” seems like field-justification, as does the other common rejoinder from psychologists, “but I’m doing experiments, not correlational studies,” as we as a field temporarily forget about interactions and pretend that experimental effects reveal universal, invariable human processes, so we can go on doing all those experiments with college undergrads. Anyone interested in morality, or prejudice, or any number of psychological phenomena need to also know how these phenomena play out among people who are harder to reach and more likely to give us “low-quality” data. Otherwise we keep studying political ideology as the difference between very liberal and slightly liberal rich university students.

Saying measurement is better than sampling is like a worker in a jelly factory convincing herself that peanut butter sucks and we’re better off without it. Clearly the goal here isn’t all-jelly sandwiches, but workers in both factories stealing from their employers and coming together to make delicious science sandwiches. (In this analogy, statisticians bring the bread.)

- Give more research credit for replication, especially in interesting, diverse, or difficult to sample populations. As a research consumer, I’d rather see people build on existing theories/data rather than come up with something that is represented as entirely new, but is really a slight twist on what exists..unfortunately, the latter is rewarded more in my experience.
- More sub-sample cross-validation in internet research. These days, it’s easy to collect more data on the internet. I’m not sure psychologists are aware how simple it is to track participants found by different recruitment methods. It’s a natural and easy way to show that effects replicate and generalize to a larger sample. Maybe that generalizes to just internet users, but that’s not an insignificant sample of the world anymore, such that there are likely few populations it won’t generalize too if it replicates among all the different recruitment methods used.