Piloting your survey

Piloting your survey is very important for any number of reasons; because there are so many reasons I'm only going to talk about a few of what I feel are the most important reasons and give some real examples of how they've applied to me both in past and current research projects.  Basically, they boil down to three things:

1. Content:  Try to have your survey as perfect as you can before you pilot it.  In a perfect world, when you pilot your survey the pilot participants will return it to you, completed, with no thoughts on how it could be better understood or what might be missing.  Furthermore, in this perfect world scenario, you will realize that all of your questions cover what you your topic thoroughly and that no changes need to be made to the items before pushing forward to have your survey approved by your "friendly" IRB board.  This, however, is rarely the case--but with good reason.  In piloting your survey, you give yourself another chance to look over the questions with a critical eye.  Also, if participants are not answering questions in a manner that is easy for you to record, you can change the question to be more specific.  For example, an open ended question asking "Do you play sports?" might be interpreted as a yes or no question.  But, it could also be interpreted as wanting the participant to list any sports that they might play.  A better question might be "Please list any sports that you play.  If you do not play sports, please write N/A."  Now the participants know exactly how you want them to respond.  This all seems a bit elementary, but it does matter.  In my current research, had I not done my pilot study I would have had all of my participants listing their date of births in whatever manner pleased them.  However, because I did a pilot study, I was able to change the question from simply saying "Date of Birth:" to "Date of Birth MM/DD/YYYY."  Doing this will make it much easier for me when I am recording my participants' demographic information.  Even if you are doing a survey that someone else wrote, it is still good to pilot the study to make sure that most of the participants, as well as the researcher themselves, understand what each question is asking.  Do not be afraid to reword another person's question if you think that it will make the question clearer to your subject population. 
2. Subjects: Preferably you should do this with people who can relate to your research topic but would not be good candidates to participate.  For example, my current research questions is "Does being in a  Freshman Seminar (a.k.a. Gateway Seminar) at Ferrum College help the average freshman student to build stronger friendships with those who are in their Gateway Seminar?"  Notice that my question is about the average freshman student.  So when I piloted my study, I had the honors gateway seminar class complete my survey; these students still related to my research because they were freshmen, however because they are not considered to be average freshmen, they would not qualify to participate in my study when I ran it on a larger scale.  This goes for both a survey that you have written yourself as well as a survey that somebody else has published.
3. Scoring: After your pilot population has completed the survey, go through and score the questions to make sure that you either know how you want each item to measure (if you have written your survey) or that you understand the scoring system (if you are using an existing survey).  Regardless of whether you are using an online or paper based survey (but especially if you are using a paper based survey) you need to go through and make sure you understand which questions need are scored in which manner.  Make sure you know which questions are reverse-keyed/reverse-scored before you run your research so that you wont be kicking yourself for scoring a handful of questions that determine the overall results of your surveys wrong 13,500 times.  Trust me, I'm speaking from experience.

Here are some examples of items which I had to change on the survey that I wrote for my current research project, as well as the reasons for why I decided to change them.  Please keep in mind that my survey is on a 5 point likert scale which in which participants respond to statements based on how much they do or do not agree (1-strongly disagree; 5-strongly agree).  The main changes have been highlighted in red.

• "I am more comfortable in classes which where there are students who I know from other classes" became "I am more comfortable in classes in which I know students from other courses."

This question had too many typos.  It may seem trivial, but when you are writing your own survey, it is very important to look out for typos that can leave those responding to the survey confused.

• One question went through several changes before I was satisfied with it:
1. "I would not want to work on a group project with certain members of my gateway course." 
2. "I would rather do an individual project than a group project with members of my gateway course."
3. "I would rather do an individual project than work on a group project with members of my gateway class."
4. "I WOULD NOT enjoy working on a group project with members of my gateway class."

I had trouble with the wording of this question.  As you can see, this  item went from being a reverse-key question, to a normal key question, and back again.  The main reason for this was because I was trying to make sure that each page of the survey had enough reverse-key questions to make sure that the participants were paying ample attention.  Furthermore, I continued to change the question because I was unhappy with its wording.  I felt very much as though it was not getting at what I was trying to ask them, which was, "How mad would you be if you were stuck working on a group project with someone you did not like in your gateway class."  But this would have been too wordy and leading.  I did not think to change the question to the final version (version no. 4) until after I had piloted my survey.  I realized the wording of version no. 3 still seemed to be asking "Do you prefer to do individual or group projects?".  I felt that the final wording of the question most adequately pinpointed what I was looking to explore in this question.

Surveys

One of the easiest ways to collect a lot of data in a short amount of time is to do surveys.  As I've found from past experience, its most efficient to do these surveys electronically versus handing out paper versions for one main reason:  It saves time.

In a past research project that I did (and in my defense, the first research experiment that I ever did) I gave out paper based surveys for my participants to fill out during each of the three sessions that I held.  There were about 75 people who showed up for each research session (give or take) and around 60 items (that's savvy speak for questions) per survey.  Which I had to grade.  By hand.  Its easy to see why this wouldn't be very appealing to do, because if you do the math (hang on, let me grab my calculator) I graded a grand total of 225 survey packets.  Given, it was slightly less once I had weeded out the people who had skipped one of the sessions or, even more infuriatingly, one question from one week of their sessions.  On top of this, as each question on each survey was supposed to be scored very specifically (i.e. subcategories for items and reverse-key questions), there was always doubting myself as to whether I was scoring the my surveys properly.  And to add to this (as I haven't made this sound horrible enough already) I had to do all of the math for final scores on survey categories by hand or by calculator.  This may not sound bad, but when you remember that you're scoring 13,500 questions in all, you're bound to make a mistake.  If I haven't made this sound bad enough, email me.  I could go on and on.  But for the sake of brevity (well actually I think that boat has sailed), I'll stop here.

Having participants do research on an electronic platform allows for all of the answers to be collected and scored automatically, meaning that you only have to go back and pull questions apart by category and change the scoring on any reverse-key questions that you may have.

There are few ways to do a survey electronically.  One is to send out emails and have people reply to your questions.  But that takes a lot of sorting and typing and time on your part when people reply, not to mention that it junks up your mail box.  What I would recommend is to find a free website, such as surveymonkey.com, that will allow you to create and send out surveys online for free.  I used surveymonkey.com this year and its great.  This website allows people to create surveys that ask questions in any number of ways and also records results for the researcher for overall responses, as well as breaking down responses so that you can look at them on an individual level as well.

"But Emily," you ask, "master of research, knower of all that is to be known about research, what about my actual survey?"

To this I have two things to say:  1) flattery will get you every where in life, and  2) the content of your survey depends on what questions you are trying to answer. 

When deciding the content of a survey, you can do one of two things:

1. Find an established survey that has been proven reliable.  If you are able to find a survey that works within the topic of your survey that someone else has already written and found reliable, why on earth would you spend time writing your own?  Also, by using a established survey, your research will seem more reliable when you present it, as well as being appealing to people who have also used this survey in past research.
2. Write your own survey.  While using preexisting surveys is a lot easier for the researcher, sometimes there isn't a survey out there that covers all that you want to find out about your topic.  When this is the case, it is time to get dirty and write your own survey.  This is also a good course to take because you can ask the specific questions that you want to know about.

Regardless of which route you go, you have got, have got, have got to pilot your survey.  More about this in the next post.