Web-based surveys offer a quick and effective way to collect data. Several companies sell software-as-a-service which makes the construction of surveys quite easy using only a web browser. At the University of Tennessee, we currently have a system-wide site license for Qualtrics. Initial discussions suggested an intent from Qualtrics to more than double its price from the previous year. This article describes the process we followed to evaluate and select an alternative, in the interest of providing similar functionality at a reasonable cost.
In choosing the options to review, our first step was to search for online software reviews of tools that we knew were already in use by various groups across all UT campuses and institutes. If one of those provided similar features at a lower price, we could minimize our training and migration expenses. A summary of the reviews’ overall ratings is shown in Table 1.
|Source||Qualtrics||Question Pro||REDcap||Survey Gizmo||Survey Monkey|
Table 1. Summary of web survey ratings from online review sites. Some scores are rescaled to range from 1 through 5. While most scores are available directly from the Source link, to get a score on the Comparisons site you must first choose a comparison of two products, then click on one of them to get its full review.
The tools were all highly rated, but we wondered if the raters needed as many features as we use for academic research. A search using Google Scholar confirmed that these were the five survey tools most widely used in scholarly research.
We read all we could find regarding the financial state of each company, read about what employees said about them on Glassdoor.com, and searched for complaints of any sort regarding the companies or their products. We found no significant problems in any of those areas. The companies all seem to be growing quite rapidly.
We also considered using open source software. However, the most popular open source web survey tool is LimeSurvey, and our previous experience with it was not positive. We investigated reviews of LimeSurvey to see if it had improved since we used it last, but there were very few of them compared to the others.
We formed a committee of ten university faculty and staff, all with substantial survey design expertise. The committee compiled a list of 141 web survey features that we considered important. We then surveyed the university research community, including our current survey tool users, asking them to rate the importance of each feature on a 5-point Likert scale. The detailed table of features and importance ratings appears in the appendix.
We used the features list to write the specifications for a Request for Qualified Suppliers (RFQ-S), a type of Request for Proposal. We specified a 5-year contract including separate pricing for: individuals, groups (e.g. departments, colleges, or institutes), single campus, and multi-campuses; as well as internal use, external use with not-for-profit clients, and external use with for-profit clients.
External use is important because the University of Tennessee is a Carnegie Engaged University, which means one of its prime goals is to perform collaborative research with external organizations. The for-profit category was included because students like to solve the types of problems that companies provide. Our Institute for Public Service also occasionally does surveys for companies in Tennessee. Such use is often prohibited from academic software licenses.
The RFQ-S was sent to the five vendors shown in Table 1. To keep things comparable, REDcap Cloud was chosen as the vendor for REDcap. They offer the same type of software-as-a-service as the other venders, though REDcap is also available for on-premises installations for free.
The bids we received covered an extremely wide range, with the highest price more than ten times larger than the lowest. Each of the responding vendors specified which of the 141 features they offered. Only one vendor, Qualtrics, stated that it was not acceptable to use their software for the benefit of any type of external organizations. Qualtrics requires an expensive commercial license when third parties are involved.
We then tested each feature, verifying the companies’ claims. We considered rating each for ease-of-use and effectiveness, but found that if the feature was implemented at all, it was generally both easy to use and effective.
We then created a composite score which weighted each feature according to the ratings from the feature importance survey. Our purchasing department prefers 1,000-point scales, so we adjusted the scores accordingly. A perfect score of 1,000 would indicate software that offered every feature that a demanding person would describe as Very Important. The resulting scores are shown in Table 2. Readers can use the data in the appendix to develop their own scores. Since REDcap Cloud did not respond to our bid, we did not evaluate or score it. However, does offer an extensive feature set including some very advanced features for database use and clinical trials. We are already using the free on-site version for projects that require those features, but it’s not as easy to use as the other products.
(out of 141)
Table 2. Feature counts and scores weighted by feature importance. Each feature is listed in the appendix. See vendors for pricing.
QuestionPro and SurveyGizmo offered nearly identical feature sets to Qualtrics, with equivalent ease-of-use, at significantly lower prices. SurveyMonkey offered a smaller set of features, at a price that was much lower than Qualtrics, but still much higher than the others.
We called the three references that QuestionPro and SurveyGizmo had each provided. They all reported that the software worked well, had close to zero downtime, had technical support that was quick and competent, and they all reported planning to continue using their chosen vendor well into the future.
QuestionPro not only scored the highest on the 141 attributes we initially focused on, but it also currently offers advanced features that we had not even considered. The company’s future features roadmap is substantially more advanced than any other product currently on the market, and anything we heard regarding the other vendors’ future plans. As a result, we decided to go with QuestionPro.
We have changed web survey tools twice before, so we are well aware of the challenges involved. When moving to a new software platform, the software cost is important, but so are migration costs. For example, if we were to attempt a migration from SAS to R in one year’s time, the cost to train people and convert tens of thousands of programs would far outweigh the savings (at least at educational prices). However, survey software is relatively easy to learn, taking an hour to learn how to set up a typical survey. We estimate that a typical survey could be migrated in well under an hour. Complex ones might take much longer, especially those that must migrate longitudinal data along with the survey.
The Qualtrics administrative dashboard allows us to download extensive details about how our people have used the software over the last five years. The total number of accounts is intimidating, at just over 11,000. However, thousands of accounts contain only a single survey with a single response, indicating that the person was simply trying the software out. Thousands more accounts have not been logged into in years. We learned that 80% of surveys each year are created by new users who are starting from scratch and thus have no work to migrate.
We have developed preliminary estimates of migration effort from current usage, and they range between a total of 1,000 and 4,000 hours. Our support staff will be available to help users migrate their projects. We are developing a survey to get more details from current users to better assess migration needs. This will allow us to determine the number of projects that entail additional complexity such as multi-institutional collaboration or longitudinal studies that must maintain long-term data compatibility. We will also be recording the time it takes to move surveys and data to the new system. As we collect such data, we will be able to forecast how long the migration will take.
Our work resulted in the selection of a software package, QuestionPro, which is comparable to Qualtrics at a much lower price. Comparing our final expenditure to the initial price increase that our Qualtrics sales rep requested, the savings over the 5-year life of the contract is over $900,000. In addition, we now have a product that members of the university community can use to solve problems for companies, helping to engage UT more fully into the economy of Tennessee.
If you found this post useful, I invite you to check out many more on my website or follow me on Twitter.
Appendix: Web survey tool features and their importance ratings
The importance of each feature (1=very low importance…5= very important) was determined by current users of web survey tools at the University of Tennessee. A rating of zero indicates that the software lacks that feature. This information was current as of 12/1/2017; the vendors all regularly add new features so check their web sites for their latest information.
|Display Text (e.g. instructions)||4.46||4.46||4.46||4.46|
|Required Questions / Required Answers||4.64||4.64||4.64||4.64|
|Redirect Browser at end of survey||3.83||3.83||3.83||3.83|
|Anonymous Responses (separate survey data from distribution/contact information)||4.51||4.51||4.51||4.51|
|Embedded Data/Hidden Values/Custom Values||3.85||3.85||3.85||3.85|
|Export to CSV||4.69||4.69||4.69||4.69|
|Export to at least one statistics package (such as SAS, SPSS…)||4.51||4.51||4.51||4.51|
|Phone Support (for users and admins)||4.51||4.51||4.51||4.51|
|Integrated Question Design Methodology Advice||0.00||0.00||0.00||3.96|
|Machine Learning to Improve Response Rate||0.00||3.84||0.00||3.84|
|Randomization of Questions||3.71||3.71||3.71||3.71|
|Randomization of Responses||3.57||3.57||3.57||3.57|
|A/B Test Questions (Split respondents between scenarios)||3.82||3.82||3.82||3.82|
|Custom Survey Templates||4.23||4.23||4.23||4.23|
|Preview Survey for Standard Screen Size||4.58||4.58||4.58||4.58|
|Preview Survey for Phone Screen Size||4.58||4.58||4.58||4.58|
|Easy to Create Interface||3.95||3.95||3.95||3.95|
|Recode Values/Set Reporting Values (i.e. set Yes to 1 and No to 0 or recode on an unsual scale such as 0,1,3,5)||4.16||4.16||4.16||4.16|
|Loop&Merge / Page Piping (loop through the same set of questions a given number of times)||3.94||3.94||3.94||3.94|
|Insert Piped Text from question or custom value into question text||4.02||4.02||4.02||4.02|
|Insert Piped Text from question or custom value into question response||3.95||3.95||3.95||3.95|
|Carry Forward selected answers to populate future questions||4.11||4.11||4.11||4.11|
|Carry Forward response choices to populate responses on future questions||4.10||4.10||4.10||0.00|
|Soft-Require Question (request response)||4.41||4.41||4.41||0.00|
|Optional Progress Bar||4.21||4.21||4.21||4.21|
|Email Triggers/Action Alerts (trigger emails automatically to be sent based on survey responses)||4.17||4.17||4.17||4.17|
|Contact List Triggers (add people to contact list automatically based on how they answer a survey)||3.98||3.98||0.00||0.00|
|Edit Next, Submit and Close Button Text||4.23||4.23||4.23||4.23|
|Hide Next, Submit or Close buttons||4.03||4.03||4.03||0.00|
|Import Surveys Questions from Word||0.00||4.28||4.28||4.28|
|Export Survey Questions to Word||4.39||4.39||4.39||0.00|
|Import/Export Survey file (to create a copy that can be exported/imported)||4.61||4.61||4.61||4.61|
|Copy Survey within survey tool||4.64||4.64||4.64||4.64|
|Revert to previous versions||4.16||4.16||4.16||0.00|
|Password/contact list authentication within a survey (respondent must authenticate with credentials saved within the contact list to continue)||3.90||3.90||3.90||3.90|
|SSO authentication within a survey (respondent must authenticate with SSO credentials to continue)||3.78||3.78||3.78||3.78|
|Connect to Web Services (such as random number generator)||3.79||3.79||3.79||0.00|
|Survey Collaboration within the university/license/brand||4.30||4.30||4.30||4.30|
|Survey Collaboration outside of the university/license/brand||3.81||0.00||3.81||3.81|
|Side by Side||3.99||3.99||3.99||3.99|
|Closed Card Sort Questions||3.08||3.08||3.08||0.00|
|Image Heatmap Questions||2.88||2.88||2.88||0.00|
|Open Card Sort Questions||0.00||3.08||3.08||0.00|
|Semantic Differential Questions||3.28||3.28||3.28||3.28|
|Text Highlighter Questions||3.53||3.53||3.53||0.00|
|Choice Based Conjoint||3.35||3.35||3.35||0.00|
|Max Differential (Max Diff) Questions||3.39||3.39||3.39||0.00|
|Track Time Respondent Stays on a Question or Page||3.55||3.55||3.55||0.00|
|Audio/Video Sentiment Questions (slider recording response while video/audio plays)||0.00||3.45||3.45||0.00|
|Embed External Audio and Video Files||3.81||3.81||3.81||3.81|
|View Reports Online||4.67||4.67||4.67||4.67|
|Export report from within offline report (pdf or other format)||4.51||4.51||4.51||4.51|
|Import response data from Excel or other format||4.26||4.26||4.26||0.00|
|Manually enter response data collected externally||4.16||4.16||4.16||4.16|
|Export reports (Please attach list formats)||4.67||4.67||4.67||4.67|
|Export Individual Charts||4.39||4.39||4.39||4.39|
|Create reports from multiple data sources||4.35||4.35||4.35||4.35|
|Share Contact Lists||4.19||4.19||4.19||4.19|
|Social Media Distribution||3.88||3.88||3.88||3.88|
|Template Library of Messages||4.06||4.06||0.00||4.06|
|Embed First Question in Email Invite||3.29||3.29||3.29||3.29|
|Mobile First UX||3.95||3.95||3.95||3.95|
|Schedule email invites and reminders||4.41||4.41||4.41||4.41|
|Schedule survey to close||4.53||4.53||4.53||4.53|
|Resend Link after completion||3.31||3.31||3.31||3.31|
|Send email to Contact List without survey link||3.32||0.00||3.32||3.32|
|Optional opt out link in emails||4.07||4.07||4.07||4.07|
|Stand alone Mobile App||4.17||4.17||0.00||4.17|
|Change variable labels and/or response values before export||4.19||0.00||4.19||4.19|
|Select if data are exported as Response Text or Response Value||4.37||4.37||4.37||4.37|
|Export subset of data||4.43||4.43||4.43||4.43|
|Tag Text Themes Manually||3.60||3.60||3.60||3.60|
|Ability to log into user account through Admin account||3.95||0.00||3.95||0.00|
|Encryption at Rest||3.95||3.95||3.95||3.95|
|Email Support (for users and admins)||3.95||3.95||3.95||3.95|
|Chat Support (for users and admins)||3.95||3.95||3.95||0.00|
|Set up User Groups (for shared access)||3.95||3.95||3.95||3.95|
|Dedicated Account Manager||3.95||3.95||3.95||3.95|
|Migrate Survey and Respondent Data||4.25||4.25||4.25||4.25|
|Ability to set email limits per account||3.95||0.00||3.95||0.00|
|Ability to set access to different question types||3.95||0.00||0.00||0.00|
|Multiple Admin Accounts||3.95||3.95||3.95||3.95|
|Auto creation of accounts (via SSO)||3.95||3.95||3.95||3.95|
|ADA Compliance (Fully Accessible/508 Compliant)||3.95||3.95||3.95||3.95|
|SAS no. 70|
|Data Encryption in transit||3.95||3.95||3.95||3.95|
|Data Encryption at rest||3.95||3.95||3.95||3.95|
|Data Encryption on all backups||3.95||3.95||3.95||3.95|
3 thoughts on “A Review of Qualtrics, QuestionPro, REDCap, SurveyGizmo, & SurveyMonkey”
A few months in, how is QuestionPro working for you?
Excellent study, thanks for sharing your information.
I’m glad you found it useful. Since that was written, QuestionPro has released lots of new features and has is being rolled out by several major state university systems.