Peer-review system failed in controversial gay parenting study

In early June a new social science study was published online ahead of print on the purported differences in children who were raised by gay parents as opposed to those raised by heterosexual parents ("How different are the adult children of parents who have same-sex relationships? Findings from the New Family Structures Study" by Mark Regnerus). Concerns were immediately raised about the study's sample, methodology, analysis, and therefore results. Most notably, the study characterized anyone who ever had a same-sex relationship after having a child as thereafter a gay father or lesbian mother regardless of whether they parented the child together as a couple. If you're studying the effects of same-sex relationships on the rearing of children, don't you think this would be an important detail to pay close attention to? This is just bad science.

And a new review by the journal that published the piece agrees. The editor of the journal assigned a member of the journal's editorial board to assess how such a "bullshit" paper got published in the first place. Where did our system break down? The report and the piece in The Chronicle of Higher Education is fairly damning and underscores how important our job as scientists is when we put on our reviewer hats:

In his audit, he writes that the peer-review system failed because of “both ideology and inattention” on the part of the reviewers (three of the six reviewers, according to Sherkat, are on record as opposing same-sex marriage). What’s more, he writes that the reviewers were “not without some connection to Regnerus,” and suggests that those ties influenced their reviews.

I've not yet been asked to put a reviewer's hat on - nor should I until I'm further along in my studies - but I've seen a few publications through the review process and I've seen my PIs, good, thoughtful PIs, calmly refuse to accept manuscripts because of conflicts of interest. I know it's tough sometimes; the academic nature of science is incestuous and we collaborate or have connections with just about everyone in our narrow fields who is qualified to give our manuscripts a thorough review. Or at least that's how it feels sometimes. But that's no excuse. If you're asked to review a manuscript of an author with whom you have a conflict of interest, kindly refuse to accept, simple as that.

In reality, only two respondents lived with a lesbian couple for their entire childhoods, and most did not live with lesbian or gay parents for long periods, if at all. The information about how parents are labeled is in the paper. Regnerus writes that he chose those labels for “the sake of brevity and to avoid entanglement in interminable debates about fixed or fluid orientations.” Sherkat, however, called the presentation of the data “extremely misleading.” Writes Sherkat: “Reviewers uniformly downplayed or ignored the fact that the study did not examine children of identifiably gay and lesbian parents, and none of the reviewers noticed that the marketing-research data were inappropriate for a top-tier social-scientific journal.” [emphasis mine]

And I suppose this is where it all comes down to: flawed peer-review in this case failed to identify severely flawed social science.