Piloting your survey

Piloting your survey is very important for any number of reasons; because there are so many reasons I'm only going to talk about a few of what I feel are the most important reasons and give some real examples of how they've applied to me both in past and current research projects.  Basically, they boil down to three things:

1. Content:  Try to have your survey as perfect as you can before you pilot it.  In a perfect world, when you pilot your survey the pilot participants will return it to you, completed, with no thoughts on how it could be better understood or what might be missing.  Furthermore, in this perfect world scenario, you will realize that all of your questions cover what you your topic thoroughly and that no changes need to be made to the items before pushing forward to have your survey approved by your "friendly" IRB board.  This, however, is rarely the case--but with good reason.  In piloting your survey, you give yourself another chance to look over the questions with a critical eye.  Also, if participants are not answering questions in a manner that is easy for you to record, you can change the question to be more specific.  For example, an open ended question asking "Do you play sports?" might be interpreted as a yes or no question.  But, it could also be interpreted as wanting the participant to list any sports that they might play.  A better question might be "Please list any sports that you play.  If you do not play sports, please write N/A."  Now the participants know exactly how you want them to respond.  This all seems a bit elementary, but it does matter.  In my current research, had I not done my pilot study I would have had all of my participants listing their date of births in whatever manner pleased them.  However, because I did a pilot study, I was able to change the question from simply saying "Date of Birth:" to "Date of Birth MM/DD/YYYY."  Doing this will make it much easier for me when I am recording my participants' demographic information.  Even if you are doing a survey that someone else wrote, it is still good to pilot the study to make sure that most of the participants, as well as the researcher themselves, understand what each question is asking.  Do not be afraid to reword another person's question if you think that it will make the question clearer to your subject population. 
2. Subjects: Preferably you should do this with people who can relate to your research topic but would not be good candidates to participate.  For example, my current research questions is "Does being in a  Freshman Seminar (a.k.a. Gateway Seminar) at Ferrum College help the average freshman student to build stronger friendships with those who are in their Gateway Seminar?"  Notice that my question is about the average freshman student.  So when I piloted my study, I had the honors gateway seminar class complete my survey; these students still related to my research because they were freshmen, however because they are not considered to be average freshmen, they would not qualify to participate in my study when I ran it on a larger scale.  This goes for both a survey that you have written yourself as well as a survey that somebody else has published.
3. Scoring: After your pilot population has completed the survey, go through and score the questions to make sure that you either know how you want each item to measure (if you have written your survey) or that you understand the scoring system (if you are using an existing survey).  Regardless of whether you are using an online or paper based survey (but especially if you are using a paper based survey) you need to go through and make sure you understand which questions need are scored in which manner.  Make sure you know which questions are reverse-keyed/reverse-scored before you run your research so that you wont be kicking yourself for scoring a handful of questions that determine the overall results of your surveys wrong 13,500 times.  Trust me, I'm speaking from experience.

Here are some examples of items which I had to change on the survey that I wrote for my current research project, as well as the reasons for why I decided to change them.  Please keep in mind that my survey is on a 5 point likert scale which in which participants respond to statements based on how much they do or do not agree (1-strongly disagree; 5-strongly agree).  The main changes have been highlighted in red.

• "I am more comfortable in classes which where there are students who I know from other classes" became "I am more comfortable in classes in which I know students from other courses."

This question had too many typos.  It may seem trivial, but when you are writing your own survey, it is very important to look out for typos that can leave those responding to the survey confused.

• One question went through several changes before I was satisfied with it:
1. "I would not want to work on a group project with certain members of my gateway course." 
2. "I would rather do an individual project than a group project with members of my gateway course."
3. "I would rather do an individual project than work on a group project with members of my gateway class."
4. "I WOULD NOT enjoy working on a group project with members of my gateway class."

I had trouble with the wording of this question.  As you can see, this  item went from being a reverse-key question, to a normal key question, and back again.  The main reason for this was because I was trying to make sure that each page of the survey had enough reverse-key questions to make sure that the participants were paying ample attention.  Furthermore, I continued to change the question because I was unhappy with its wording.  I felt very much as though it was not getting at what I was trying to ask them, which was, "How mad would you be if you were stuck working on a group project with someone you did not like in your gateway class."  But this would have been too wordy and leading.  I did not think to change the question to the final version (version no. 4) until after I had piloted my survey.  I realized the wording of version no. 3 still seemed to be asking "Do you prefer to do individual or group projects?".  I felt that the final wording of the question most adequately pinpointed what I was looking to explore in this question.