Ranking: Is It Really Sequential Choice?

ABSTRACT - Preferences generated by rank order tasks are typically assumed to be equal to those generated by choice tasks. This exploratory study reports the findings of an empirical investigation of this assumption. The results indicate that ranking and choice tasks result in differences in resulting preferences. In addition, two manipulations, requiring a justification and altering the information format, were investigated to determine their impact on resulting preferences. Requiring a justification of resulting preferences had a differential impact on ranking and choice tasks.

Citation:

John A. Schibrowsky and James W. Peltier (1995) ,"Ranking: Is It Really Sequential Choice?", in NA - Advances in Consumer Research Volume 22, eds. Frank R. Kardes and Mita Sujan, Provo, UT : Association for Consumer Research, Pages: 71-77.

Advances in Consumer Research Volume 22, 1995      Pages 71-77

RANKING: IS IT REALLY SEQUENTIAL CHOICE?

John A. Schibrowsky, University of Nevada

James W. Peltier, University of Wisconsin-Whitewater

ABSTRACT -

Preferences generated by rank order tasks are typically assumed to be equal to those generated by choice tasks. This exploratory study reports the findings of an empirical investigation of this assumption. The results indicate that ranking and choice tasks result in differences in resulting preferences. In addition, two manipulations, requiring a justification and altering the information format, were investigated to determine their impact on resulting preferences. Requiring a justification of resulting preferences had a differential impact on ranking and choice tasks.

INTRODUCTION

Preference data is one of the primary tools employed in the development of marketing strategies. It is commonly assumed that each alternative has an underlying utility or expected value and that the consumer prefers the alternative with the highest value. This value maximization assumption (VM) is the cornerstone of the economic theory of the consumer, and is used extensively by theoretical and practical marketing researchers (Simonson and Tversky 1992). One principal ramification of the VM assumption is that preferences between alternatives are independent of the preference elicitation task employed. Tversky, Sattath, and Slovic (1988) refer to this proposition as procedural invariance: Normatively equivalent procedures should yield the same preferences.

One area of investigation of procedural invariance is the realm of response modes. To date, researchers investigating procedural invariance with regard to response modes have focused on the distinction between various rating and choice tasks (see Payne, Bettman, and Johnson 1992). The research suggests that rating and choice tasks result in differences in preferences (e.g., Einhorn and Hogarth 1981; Hansen 1976; Hogarth 1980; Huber et al. 1993; Slovic and Lichtenstein 1983; Tversky, Sattath, and Slovic 1988).

The relationship between preferences generated by ranking and choice tasks has received little attention. Ranking is typically assumed to be a method of eliciting sequential choices. The alternative that is ranked first is assumed to be the alternative that would be the first choice. The alternative ranked second is assumed to be the alternative that would be selected if the first choice is not available, etc. While the distinction between ranking and choice tasks to elicit preferences appears to be a simple substitution of sequential choices (ranking) for individual choices, research in the area of rating and choice tasks suggests that this assumption may be incorrect.

The research question addressed by this study was, "Do ranking and choice tasks result in similar preferences?" The experiment reported here had two purposes. First, it was designed to investigate the impact of ranking and choice tasks on resulting preferences. Second, it was designed to identify factors that affect ranking and choice tasks differently.

RANKING AND CHOICE

Individuals are often asked to rank order their preferences. For example, members of a marketing department are asked to rank prospective faculty candidates. A market researcher asks individuals to rank order their preferences for brands of laundry detergent. In each case, ranking is assumed to yield the same resulting preferences as those generated by an individual choice task. While recent findings (Ben-Akiva, Morikawa, and Shiroishi 1992) suggest that this assumption may not be accurate for lower ranks (e.g. third or fourth), the assumption that the first ranked alternative would also be chosen as the most preferred alternative has not been explicitly tested.

In consumer research, ranking tasks are often used to determine preferences, especially when decompositional models of preferences are employed (e.g. Green and Srinivasan 1978). In a ranking task, an individual must determine his/her order of preference. From a measurement perspective this requires an ordinal information response.

While little is written about the underlying mental processes associated with ranking tasks, Desoto (1961) found that individuals have a preference for single orderings. This suggests that ranking processes are likely to be top to bottom or bottom to top oriented based on one key attribute. As such, the highest priced, highest quality alternative or the lowest price, lowest quality alternative is likely to be the first ranked alternative.

Choice tasks require a selection from a set of alternatives. This response mode is often employed in marketing research studies, based on the argument that choice is the actual task performed by the consumer when preferences are used to make a purchase (Bither and Wright 1977). This "realism" is not without a cost. When this preference elicitation task is employed, less information is gathered per respondent. From a measurement perspective, this task requires only a nominal information response.

Choice tasks are associated with a wide array of choice heuristics, designed to simplify the process by employing cutoff strategies and other noncompensatory processes (Gensch and Ghose 1992; Johnson and Russo 1981, 1984; Klein and Bither 1987; Olshavsky and Granbois 1979; Tversky 1972). Recently, tradeoff contrast and extremeness aversion have been identified as choice strategies (Simonson and Tversky 1992).

A better understanding of the relationship between preferences generated by ranking and choice tasks is important for two reasons. First, both, ranking and choice data, are often used by marketing researchers to obtain preference data (Huber et. al. 1993). This is based on the procedural invariance assumption. Second, ranking seems to be a very natural task (Desoto 1961). As such, it is likely to be common consumer activity, and needs to be investigated.

FACTORS AFFECTING RANKINGS AND CHOICES

In 1981, Hogarth and Einhorn noted that "the conditions under which judgments and choices are similar as well as different need to be better understood if a descriptive theory of judgment and choice is to be developed." One of the purposes of this study was to operationalize this challenge in a consumer setting. Two factors that merited investigation were information formats and justification.

Information presentation formats and justification were selected to be investigated in this study based on three criteria. First, a review of the literature suggested that these conditions might affect rankings and choices differently. Second, they did not change the utilities of the alternatives or the consideration set in any way. Third, they were conditions that might be manipulated by marketing researchers gathering preference data.

Information Format Effects.

Bettman and Kakkar (1977), Bettman and Park (1980), Johnson, Payne, and Bettman (1988), and Jarvenpaa (1991) showed that preference formation was sensitive to the way information was presented. Kahneman and Tversky (1979) hypothesized that judgment tasks are affected by the order in which the alternatives are evaluated. They referred to this phenomenon as "anchoring and adjusting." Lynch, Chakravarti, and Mitra (1991) investigated this phenomena in a consumer setting. It seems plausible that anchoring and adjusting might affect ranking tasks in the same way as rating tasks.

Also, choices have been shown to be affected by the frame associated with the consideration set (e.g., Simonson, 1989; Ratneshwar, Shocker, and Stewart, 1987). Simonson (1989) and Simonson and Tversky (1992) extended these findings to explain the way individuals evaluate the advantages and disadvantages of the available alternatives. They provide evidence that decision makers tend to avoid extreme alternatives. This is referred to as "extremeness aversion" (Simonson and Tversky 1992).

In summary, the research suggests that changing the order of the alternatives and making the middle alternative easier to identify might have a differential impact on rankings and choices. Specifically, in the centered/ordered condition (see Appendix A) subjects assigned to the choice task should be less likely to prefer the end alternatives (extremeness aversion), while subjects assigned to the ranking task should be more likely to prefer the end alternatives (top-down or bottom-up processing).

Justification.

Hagafors and Brehmer (1983) provided evidence that justification of one's judgments leads to a more analytic evaluative process, while the lack of justification leads to a more intuitive evaluation process. "One conception (of choice) asserts that much of the deliberation prior to choice consists of finding a concise inherent set of reasons that justify the selection of one option over others," (Slovic, Fischhoff, and Lichtenstein 1982). Tversky (1972) and Slovic (1975) used an "ease of justification" reasoning to explain the use of particular choice rules. This finding was also supported by Simonson (1989), who concluded that justification results in the use of easy to explain choice rules. He reported that attraction and compromise effects tend to be stronger among subjects who knew they would be asked to justify their selections.

These studies reflect the importance of considering justification in ranking and choice. It is logical to suggest that requiring a justification will affect rankings and choices in different ways.

HYPOTHESES

The literature review lead to the following testable hypotheses:

H1: The preferences obtained from ranking and choice tasks will be significantly different.

H2: The preferences obtained from centered/ordered and not centered/ordered product by brand matrices will be significantly different.

H3: The preferences obtained from subjects that were asked to justify their responses will be significantly different from the preferences obtained from subjects that were not asked to justify their responses.

In addition to these main effect hypotheses, the following interaction effects were proposed:

H4: The preferences obtained from the ranking and choice tasks will be affected differently by the centering/ordering manipulation.

H5: The preferences obtained from the ranking and choice tasks will be affected differently by the justification manipulation.

METHOD

Design and Subjects

The experiment employed a 2 tasks (ranking and choice) x 2 information structures (centered/ordered and not centered/ordered) x 2 levels of justification (required or not) factorial, between subjects design. A total of 370 subjects were recruited from those students enrolled in the various upper division business classes at a midwestern university.

Procedure

Eight questionnaires were prepared for the study, one for each experimental condition. All booklets contained the same warm-up exercise followed by one of two product matrices comparing five fictitious fast food restaurants on five attributes (no dominated alternatives)(see appendix A for the two different matrices), one of two tasks (ranking or choice), and a justification required or not. This was followed by a page of questions asking for information pertaining to manipulation checks and product usage.

Subjects were assembled in a classroom, randomly assigned to one of eight treatments, and given general directions concerning the procedure they were to follow. Subjects were told that they were taking part in a marketing research project sponsored by a fast food company contemplating moving into the local market. They were given as much time as needed to complete the questionnaire. Following the data collection task, subjects were debriefed and thanked for their participation.

Independent Variables

The subjects were asked to either rank order or choose among the alternatives. The particular ranking task employed in this study was as follows:

Rank order the restaurants shown above in terms of your preference in having the various restaurants close to your residence (1=most preferred through 5=least preferred).

The choice task used in this study was as follows:

From the alternatives shown above, choose the one restaurant you would most prefer to have close to your residence.

The information structure manipulation was accomplished by arranging information about the five fictitious fast food restaurants in two different ways. In the centered/ordered condition, the information was presented by ordering the restaurants from highest price to lowest price, and noting the middle value on each attribute. ln the not centered/ordered condition, the information was presented in a random order with no indication of the middle alternative. Appendix A contains the two different product matrices.

The justification manipulation was accomplished with a question asking a portion of the respondents to explain the reasoning behind their rankings or choices. The subjects assigned to the justification condition, knew they would need to justify their responses before they made their ranking or choice.

TABLE 1

DATA AND MODEL FIT FOR H1

TABLE 2

DATA AND MODEL FIT FOR H2

Dependent Variable

The dependent variables in this study were the resulting preferences generated by the different tasks. As such, this study employed a behavioral decision making or structural perspective rather than an information processing perspective (Johnson and Puto 1987).

The results were reported in percentages or probabilities of each of the alternatives being ranked or chosen as most preferred. Using this approach, the dependent variable was conceptualized as a series of log odds ratios of the probability of a subject choosing or ranking each of the alternatives as most preferred.

RESULTS

Since the dependent variable was multinomial and interactions were predicted, the data were analyzed using a multinomial logistic modeling method. Critical to the investigation was the measure of goodness-of-fit. A chi-square fit statistic was used to compare the various models' ability to account for the data. As a model's ability to account for the data increases, chi square also increases. In order to directly compare the "fit" of models, a nested model method was used. Basically, a model provided a better "fit" when the improvement in chi-square (given the change in degrees of freedom) was large enough to be significant at the p<.05 level.

Main Effects

These hypotheses predicted main effects associated with the task, information format, and justification manipulations. To test each of these hypotheses, a constants only model was compared to a model containing the hypothesized main effect.

Task Effects. H1 predicted that ranking and choice tasks would result in significantly different preferences. The results of the test of H1 are presented in Table 1.

They indicated that the inclusion of the task variable significantly improves the fit of the model (chi-square increased by 11.38 while the degrees of freedom were reduced by 4). This indicated that the preferences obtained from the ranking task were significantly different from those obtained from the choice task. Specifically, subjects assigned to the ranking task were more likely to prefer the high priced alternative (N), while subjects assigned to the choice task were more likely to prefer the lower priced alternatives (J&L). This finding supports H1.

Information Format Effects. H2 predicted that the centered/ordered and not centered/unordered presentation formats would result in significantly different preferences. The results of the test of H2 are presented in Table 2.

The inclusion of the centered/ordered variable did not significantly improve the fit of the model. This indicated that the differences in preferences obtained across the two conditions were not statistically significant. This finding does not support H2.

Justification Effects. H3 predicted that the justification manipulation would significantly affect preferences. The test of H3 is presented in table 3.

The addition of the justification variable did not significantly improve the fit of the model. This finding indicated that the preferences obtained across the two conditions were essentially the same.

TABLE 3

DATA AND MODEL FIT FOR H3

TABLE 4

DATA AND MODEL FIT FOR H4

Interaction Effects

Hypotheses, H4 and H5, predicted interaction effects between the centering/ordering manipulation and task and between the justification manipulation and task. A nested model method was employed to test these hypotheses.

Task by Centering/ordering Interaction. H4 predicted that the centered/ordered manipulation would affect preferences from ranking and choice tasks in different ways. To test this hypothesis, the main effects model was compared to the interaction model. The test H4 is presented in Table 4.

The inclusion of the task by centering/ordering interaction variable did not significantly improve the fit of the model. This implied that the centering/ordering manipulation did not have a differential impact on ranking and choice. These findings do not support hypothesis H4.

Task and Justification Interaction. H5 predicted that the justification manipulation would affect preferences from ranking and choice tasks in different ways. To test this hypothesis, the main effects model was compared to the interaction model. The test H5 is presented in Table 5.

In the case of the justification manipulation, the inclusion of the task by justification interaction variable significantly improved the fit of the model (chi-square increased by 9.82, while degrees of freedom were reduced by 4). This indicated that the justification manipulation had a differential impact on ranking and choice. Specifically, the justification manipulation resulted in more ranking subjects (38% vs 33%), but fewer choice subjects (22% vs 44%) preferring the end alternatives. These findings support hypothesis H5.

DISCUSSION

The key finding of this study is that preferences obtained from the ranking and choice tasks were significantly different. Two points pertaining to these results need to be mentioned. First, it should be noted that the absolute magnitude of these differences was substantial. In the ranking task, 35% of the subjects preferred alternative N; while in the choice task, 26% of the subjects preferred alternative N. Second, the result contradicts the procedural invariance assumption. With both tasks, the constraints of the situation and the utilities of the alternatives were identical. As such, VM would predict no differences in the preferences generated by the two tasks.

TABLE 5

DATA AND MODEL FIT FOR H5

A review of Table 1 indicates that individuals asked to perform the ranking task were more likely to prefer alternative N (the high price) than the individuals performing the choice task (35% vs 26%). One appealing explanation is that subjects who were asked to rank the alternatives were more likely to use some type of "top to bottom" ranking rule.

The centered/ordered condition did not have a main or interaction effect on preferences. One possible explanation can be found in the manipulation check. In the centered/ordered condition 85% of the subjects were able to accurately identify the middle alternative, while in the not centered/ordered condition 66% of the subjects were able to identify the middle alternative. It is possible that the not centered/ordered condition was not complex enough to disguise the ordering of the alternatives. In addition, it should be noted that the two information structure manipulations were confounded in this study. It would have been interesting to have investigated them separately.

While the justification manipulation did not result in a main effect, the predicted interaction between task and justification was supported. For subjects assigned to the ranking condition, the justification manipulation increased the likelihood of preferring the extreme alternatives, N and L (33% vs 38%). For subjects assigned to the choice condition, the justification manipulation reduced the likelihood of selecting the extreme alternatives, N and L (45% vs 22%) and increased the likelihood of selecting the middle alternatives P, Q, and J (56% vs 78%). Perhaps the selection of these middle alternatives as most preferred in the justification condition was the result of some type of trade-off between quality and price. If so, this pattern of results supports Hagafors and Brehmer's (1983) findings that justification leads to more analytical and compensatory processing of evaluation information.

These results might also be explained by the "ease of justification" hypothesis forwarded by Tversky (1972) and Slovic (1975). For individuals assigned to the ranking task, the easiest rule to justify might have been a "top to bottom" ranking rule. For individuals assigned to the choice task, the easiest rule to justify might have been some sort of compromise or aversion to the extreme rule.

It should be noted that these "processing" explanations are conjecture, based on previous research and a qualitative review of the written justifications that subjects provided in the justification condition. However, the interesting and indisputable finding is that the justification manipulation resulted in more ranking subjects preferring the end alternatives and more choice subjects preferring the middle alternatives.

CONCLUSIONS

In summary, this study reports two critical findings for those researchers interested in ranking and choice. First, it was shown that ranking and choice tasks do not necessarily result in the same set of preferences. Second, asking individuals to justify their preferences affects ranking and choice tasks in different ways.

The results of this study have three important implications for marketing. First, this study sheds light on the distinction between rankings and choices. It enhances the work of consumer researchers that have shown that response modes impact resulting preferences.

Second, this study suggests that the mode of response used to elicit preferences and the degree to which individuals justify their preferences will impact their resulting preferences. This conclusion has implications for those researchers involved in preference research. The decision of the response mode used to reveal preferences is typically a function of the statistical methods to be employed. This study suggests that researchers need to recognize that this decision might affect the resulting preferences.

Third, this study has implications for new product development research. If consumers actually structure the evaluation of some new products as ranking tasks and others as choice tasks, predicting the success of new products might be improved if researchers take this distinction between ranking and choice into consideration in the product concept testing and product development stages of new product development.

APPENDIX A

CENTERED/ORDERED AND NOT CENTERED/ORDERED MATRICES

REFERENCES

Ben-Akiva, Moshe, Takayuki Morikawa, and Fumiaki Shiroishi (1992), "Analysis of the Reliability of Preference Ranking Data," Journal of Business Research, 24, 149-164.

Bettman, James R. and Pradeep Kakkar (1977), "Effects of Information Presentation Format on Consumer Information Acquisition Strategies," Journal of Consumer Research, 3, 233-240.

Bettman, James R. and C. Whan Park (1980), "Effects of Prior Knowledge and Experience and Phase of the Choice Process on Consumer Decision Processes: A Protocol Analysis," Journal of Consumer Research, 7, 234-248.

Bither, Stewart W. and Peter Wright (1977), "Preferences Between Product Consultants: Choices versus Preference Functions," Journal of Consumer Research, 4, 39-47.

DeSoto, Clinton B. (1961), "The Predilection for Single Orderings," Journal of Abnormal and Social Psychology, 62, 16-23.

Einhorn, Hillel J. and Robin M. Hogarth (1981), "Behavioral Decision Theory: Processes of Judgment and Choice," Annual Review of Psychology, 32, 53-88.

Gensch, Dennis H. and Sanjoy Ghose (1992), "Elimination by Dimensions," Journal of Marketing Research, 29, 417-429.

Green, Paul E. and V. Srinivasan (1990), "Conjoint Analysis in Marketing Research: New Developments and Directions," Journal of Marketing, 54, 3-19.

Grether, David M. and Charles R. Plott (1979), "Economic Theory of Choice and the Preference Reversal Phenomenon," American Economic Review, 69, 623-638.

Hagafors, Roger and Berndt Brehmer (1983), "Does Having to Justify One's Judgments Change the Nature of the Judgment Process? " Organizational Behavior and Human Performance, 31, 223-232.

Hansen, Fleming (1976), "Psychological Theories of Consumer Choice," Journal of Consumer Research, 3, 117-142.

Hogarth, Robin M. (1980), Judgment and Choice, John Wiley and Sons, New York.

Huber, Joel, Dick R. Wittink, John A. Fiedler, and Richard Miller (1993), "The Effectiveness of Alternative Preference Elicitation Procedures in Predicting Choice," Journal of Marketing Research, 30, 105-114.

Jarvenpaa, S.L. (1990), "Graphic displays in Decision MakingCThe Visual Salience Effect," Journal of Behavioral Decision Making, 3, 347-62.

Johnson, Eric J., John W. Payne, and James R. Bettman (1988), "Information Displays and Preference Reversals," Organizational Behavior and Human Decision Processes.

Johnson, Eric J. and J. Edward Russo (1981), "Product Familiarity and Learning New Information," in Advances in Consumer Research, Vol.8, Kent B. Monroe (ed.) Ann Arbor Michigan: Association for Consumer Research.

Johnson, Eric J. and J. Edward Russo (1984), "Product Familiarity and Learning New Information," Journal of Consumer Research, 11, 542-550.

Johnson Michael and Christopher P. Puto (1987). "A Review of Judgment and Choice," Review of Marketing, Michael Houston, ed., American Marketing Association: Chicago IL, 236-292.

Kahneman, Daniel and Amos Tversky (1979), "Prospect Theory," Econometrica, 47, 263-292.

Klein, Noreen M. and Stewart Bither (1987), "An Investigation of Utility Directed Cutoff Selection," Journal of Consumer Research, 14, 240-256.

Lynch, John G., Dipankar Chakravarti, and Anuaree Mitra (1991), "Contrast Effects in consumer Judgments: Changes in Mental States or Rating Scales?" Journal of Consumer Research, 18, 284-297.

Olshavsky, Richard W. and Donald Granbois (1979), "Consumer Decision Making: Fact or Fiction," Journal of Consumer Research, 6, 93-100.

Payne, John, James R. Bettman, and Eric J. Johnson (1992), "Behavioral Decision Research: A Constructive Processing Point of View," Annual Review of Psychology, 43, 87-131.

Puto, Christopher P. (1987), "The Framing of Buying Decisions," Journal of Consumer Research, 14, 301-315.

Ratneshwar, Srinivasan, Allan D. Shocker, and David W. Stewart (1987), "Toward Understanding the Attraction Effect: The Implications of Product Stimulus Meaningfulness and Familiarity," Journal of Consumer Research, 16, 411-420.

Simonson, Itamar (1989), "Choice Based on Reasons: The Case of the Attraction and Compromise Effects," Journal of Consumer Research, 16, 158-74.

Simonson, Itamar and Amos Tversky (1992), "Choice in Context: Tradeoff Contrast and Extremeness Aversion," Journal of Consumer Research, 16, 158-74.

Slovic, Paul (1975), "Choice Between Equally Valued Alternatives," Journal of Experimental Psychology: Human Perception and Performance, 1, 280-287.

Slovic, Paul, Barruch Fischhoff and Sarah Lichtenstein (1982), "Response Mode, Framing and Information Processing Effects in Risk Assessment," in R. M. Hogarth (ed.), New Directions for Methodology of Social and Behavioral Science: The Framing of Questions and the Consistency of Response, San Francisco: Jossey-Bass.

Slovic, Paul and Sarah Lichtenstein (1968), "The Relative Importance of Probabilities and Payoffs in Risk Taking," Journal of Experimental Psychology Monograph Supplement, 78 (3), part 2.

Slovic, Paul and Sarah Lichtenstein (1983), "Preference Reversals: A Broader Perspective," American Economic Review, 73 (4), 596-605.

Tversky, Amos (1972), "Elimination by Aspects: A Theory of Choice," Psychological Review, 79, 281-291.

Tversky, Amos (1988), "Context Effects and Argument-Based Choice," Paper presented at the Association for Consumer Research Annual conference, Maui, Hawaii.

Tversky, Amos, S. Sattath and Paul Slovic (1988), "Contingent Weighting in Judgment and Choice," Psychology Review, 95, 371-384.

Tversky, Amos, Paul Slovic, and Daniel Kahneman (1990), "The Causes of Preference Reversal," American Economic Review, 80, 204-217.

----------------------------------------