Testing different rank order question layouts for PC and smartphone respondents

We studied the impact of different layouts for rank order questions on respondent effort, data quality, and substantive results among PC and smartphone respondents, in an experiment in an opt-in online panel in Spain, using an order-by-click design. We experimentally varied the device, the number of columns, and, for smartphone respondents, the position of the ‘next’ button in questions on trust in institutions.

We found some evidence of lower data quality for smartphone users but no evidence that presenting ranking items in one column performs differently than two columns. We also find little evidence that these effects differ by the number of response options presented or the number to be ranked. The placement of the ‘next’ button had little effect on performance on ranking items. Overall, our findings suggest that the format and layout of order-by-click questions has little effect on data quality, regardless of device used.