Monday, May 27, 2013

The Slaughter of Good Ideas (Warning! A bit crotchety)

I am amazed and saddened by the number of studies out there based on analysis that doesn't fit the data. It's like watching the slaughter of an army of good ideas, a slaughter that could have been avoided if only the messenger hadn't taken a wrong turn.

Every single statistical test is based on assumptions. This means that, when you write your proposal, you need one simple little clause and an extra (rather wordy) sentence for each analysis you plan to do. Instead of writing

"I will analyze the data using a linear regression analysis."

you should write

"I will analyze the data using a linear regression analysis if the data meet the assumptions for that test. If they do not meet the assumptions, then I will either transform the data so that they meet the assumptions or perform an equivalent non-parametric test, a test that is robust to unequal variances, or a test that otherwise fits the data."

"Non-parametric" is short for "test that works even on data that are not normal." And yes, you can quote me.

If you don't have these bits of writing in your proposal and your data doesn't meet the assumptions for your chosen analysis, then you are in trouble. Some committees actually require students to run a test that shouldn't be run simply because they said they would, resulting in the student not being able to answer the research questions. I have seen this happen. You may not be allowed to use the appropriate non-parametric equivalent of your chosen analysis. That means that you won't be able to find an effect that may actually be there. It also means that somewhere down the line, like when the dean's office reviews the dissertation, or when you want to become a famous researcher, someone may take a look at your dissertation and find out that the data didn't meet the requirements for the analysis you did—and disrespect you

So safeguard yourself. If you care about your future career, if you want to play with the big, famous, knowledgeable, ethical researchers out there in the real world, talk about assumptions in your proposal and provide a backup plan in case they are not met. 

Later, I'll provide a list of tests and the assumptions upon which they rest. You can also find such lists if you search the internet for them. 

No comments:

Post a Comment