Q - Tips: Using Qualitative and Quantitative Techniques in Tandem to Assure Valid Manipulations

ABSTRACT - Multiple Discriminant Analysis (MDA) has received widespread use as a quantitative predictor equation for categorical group membership. Focus groups and other qualitative data generation methods have also been receiving widespread use as preliminary classificatory tools. This paper explores the feasibility of using these techniques in tandem to assure valid experimental manipulations in certain settings. Data were collected via survey questionnaires for a study concerning the product class involvement levels of four different product types.

Citation:

Don R. Rahtz and David L. Moore (1986) ,"Q - Tips: Using Qualitative and Quantitative Techniques in Tandem to Assure Valid Manipulations", in NA - Advances in Consumer Research Volume 13, eds. Richard J. Lutz, Provo, UT : Association for Consumer Research, Pages: 291-296.

Advances in Consumer Research Volume 13, 1986      Pages 291-296

Q - TIPS: USING QUALITATIVE AND QUANTITATIVE TECHNIQUES IN TANDEM TO ASSURE VALID MANIPULATIONS

Don R. Rahtz, College of William and Mary

David L. Moore, College of William and Mary

ABSTRACT -

Multiple Discriminant Analysis (MDA) has received widespread use as a quantitative predictor equation for categorical group membership. Focus groups and other qualitative data generation methods have also been receiving widespread use as preliminary classificatory tools. This paper explores the feasibility of using these techniques in tandem to assure valid experimental manipulations in certain settings. Data were collected via survey questionnaires for a study concerning the product class involvement levels of four different product types.

MDA was then used as a manipulation check on the qualitative involvement classification. The results of the MDA technique's application as a manipulation check seem to suggest a continuing use of the statistical technique when data configurations are compatible.

INTRODUCTION

Multiple Discriminant Analysis (MDA) has enjoyed fairly widespread use as a multivariate statistical technique in the marketing discipline (e.g., Wiley and Richard 1974). Dillon and Westin (1982) state that the availability of multiple discriminant analysis procedures in canned computer software packages has made the application of MDA very easy for the marketing practitioner. The use of MDA, however, has apparently been seen as being limited to that of a predictor equation for some given categorical criterion variable. For example, the probability of a given product or service strategy succeeding or failing to induce consumers to buy a product.

The purpose of this paper is to suggest a different application for multiple discriminant analysis. That role is that of a manipulation check technique. Since the MDA procedure is applicable when there is a categorical dependent variable and interval independent variables, many experimental situations created by marketers could lend themselves to data configurations appropriate for the use of MDA in such a context. Such use, under the correct conditions, could provide researchers with greater confidence concerning the success of the manipulation than would a simple t-test of means or similar techniques performed on the data.

Churchill (1979) has advocated the use of qualitative and quantitative techniques in tandem to produce richer constructs. This paper demonstrates an effective combination of a qualitative technique (i.e., the nominal group technique) and a quantitative technique (i.e., MDA). When research involves vague or loosely specified constructs (e.g., involvement), qualitative approaches to construct operationalization can reveal novel and insightful interpretations of the construct. HDA can then be employed, as this study demonstrates, to assess the validity of the resulting operationalizations of construct dimensions.

If marketing research is to be considered scientific, reliability assessment of constructs is essential (Peter 1979). It is also desirable to explore alternative techniques for assessing statistical significance in marketing research (Sawyer and Peter 1983). Clearly, there has been an increased sensitivity to questions surrounding methodological rigor in marketing research.

It is not the intent of this paper to delve into the many statistical issues surrounding the estimation and application of the linear discriminant function (LDF). Rather, the intent here is to suggest a possible expanded role for the technique in research being conducted in marketing. The remainder of this paper presents an example of the application of MDA as a manipulation check technique for nominal group technique generated construct dimensions.

THE STUDY

In the Spring of 1984 a study was conducted to assess the impact of product class involvement on individual's cognitive consistency. As part of that study, it was necessary to generate four products classes for use in the study. Two of these products classes had to be considered low involvement product types and two had to be considered high involvement product types by the target population.

The samples used throughout this study consisted of undergraduate students at two major mid-Atlantic universities. The use of a student sample was deemed appropriate since the products used were relevant to the population from which the sample was drawn (i.e., students). While some authors (Cunningham, Anderson, and Murphy 1975) question the assumption that students are "real people," others (Lamb and Stem 1980) argue that use of student samples is appropriate given that the students are relevant to the context of a given study. That is, those types of situations where students were not expected to be affected in their behavior by simply being students. Since the products utilized in the study were relevant to the student as a consumer, no threat to the external validity of the results was anticipated from selection-treatment interaction.

In order to provide high and low involvement product classes and brands which were more meaningful and relevant to the subjects used in the broader study's major procedure and hypothesis testing, a valid classification of involvement was crucial. Too often, involvement studies have relied strictly on a researcher's "feeling" that "this" is high involving and "that" is low involving. Manipulations of involvement are sometimes questionable at best, relying on other research from different populations, or post-hoc assignments to classify objects/situations as high or low involving. (See Moore and Rahtz 1984 for a methodological review of involvement studies.)

In the present research, care was taken to provide an involvement classification of product classes based on the population's own feelings, not a single researcher's contention. These initial product classes were later employed to generate relevant brands and product attributes used in purchase decisions by the population under study.

While some authors, (c.f., Lastovicka and Gardner 1978) have developed multi-item scales for measuring involvement, their use in this study was deemed unwarranted for two reasons. First, such scales are of a reactive nature. This could create a demand artifact (Sawyer 1975) in that products rated are selected by the researcher and not the subjects. That is, the product pool from which the specific products are selected is predetermined by the researcher. This then limits the possible responses to the a priori defined set. Second, from a logistical standpoint, the number of responses required (as many as 1,100 per subject for 50 products (22 scale items used by Lastovicka and Gardner, 1978 to define produce class involvement x 50 products = 1100) renders this approach untenable.

Given the unwieldiness and reactive nature of such a quantitative approach, the present study sought to explore an alternative qualitative approach to generate the needed product information. The qualitative approach chosen was the nominal group technique (NGT) (Huber and Delbecq 1972). The NGT is linked with the family of "brainstorming" groups discussed by Osborn (1957). The NGT operates in a manner such that individual creativity is not hindered by group processes, an often bothersome aspect of interactive groups (Collaros and Anderson 1969; Taylor, et. al., 1958), but enhanced and sharpened (Delbecq, Van de Van, and Gustafson 1975). That is, individual responses can be explained and interpreted more clearly through the group discussion session of NGT, allowing the individual to sharpen the response or make other related corrections to uncover more quality and a greater quantity of ideas. Claxton, Ritchie, and Zaichkowski (1980) support the method's use in a somewhat similar setting of attribute generation for conjoint studies. In this study, the nominal group technique procedure utilized in obtaining the high and low involvement product classes followed the procedure outlined by Delbecq, et. al. (1975). Tables 1 and 2 summarize the NGT procedures and benefits as adapted for this study.

TABLE 1

NGT PROCEDURES

TABLE 2

NGT BENEFITS

Assessment of Product Class Involvement

Eighty subjects from undergraduate business classes were enlisted to assess product class involvement. Half were assigned to eight NGT groups of five members each to obtain two high involvement product classes. The remaining half were assigned to eight NGT groups of five members each to obtain two low involvement product classes. Assignment to high and low groups was assumed to be random by computer assignment to various sections of the same class. Within these classes, individuals were randomly assigned to groups. Since Shaw (1981) had pointed out that single sex groups may operate more along conformist lines, thus limiting possible idea generation quality, groups included no more than three members of the same sex.

Students assigned to each involvement group were then instructed to proceed with the NGT process, see Table 1. The question presented to the respondents concerning involvement with the product class was derived from work by Lastovicka (1979a), Cialdini, Levy, Herman, Kozlowski, and Petty (1976), Ray Sawyer, Rothschild, Heeler, Strong, and Reed (1973), Ray and Webb (1976), Lastovicka and Gardner (1978), and DeBruicker (1979) which related involvement to a combination of commitment, importance, perceived product differentiation, and information processing levels (thinking) by the individual.

In order to provide a selection of product classes which would most likely be of high or low involvement to the entire population to be used in the broader study, an additional step was added to the NGT procedure at this stage. The products in both the high and low involvement procedures were rank ordered by the researcher. The procedure for rank ordering was to give ten points for a top ranked product, nine points for the second ranked product, on down to one point for the tenth ranked product for each subject. Those products not receiving a top ten ranking were not awarded any points. The points were then totaled across subjects for each product and the products ranked with respect to their overall point scores. From the top ten for each of the involvement conditions, two products were selected for inclusion in the study for a total of four products. These products were: (I) stereos and (2) clothes (jeans) for high involvement; and (3) bar soap and (4) paper towels for low involvement. Table 3 and Table 4 summarize the results of this ranking procedure.

TABLE 3

TOP TEN PRODUCTS

NGT High Involvement Product Sumnary

TABLE 4

TOP TEN PRODUCTS

NGT Low Involvement Product Summary

Selection of the products was based on ranking and perceived appropriateness for the broader study's contrived purpose. Participants in the broader study were asked to complete a product advertising survey concerning individual's feelings about randomly selected brands for sale in their area and advertising in general. For example, while a college education is viewed as high involving, it is hard to classify as a tangible "brand". Jeans were chosen as the specific clothing garment since a substantial portion of the clothes listed were jeans or designer jeans. In addition, earlier work by Lastovicka and Gardner (1978) had shown jeans (as well as stereos) were rated as being highly involving for college students.

The two low involvement products, paper towels and bar soap were selected on the basis of their rankings. Bar soap was ranked sixth out of ninety products listed. Those products finishing in front of bar soap, as Table 4 shows, were mostly grocery type products, often bought as unbranded or store branded products (e.g., milk, eggs). This was felt to be in conflict with the cover story of the research being concerned with advertising and brands.

THE ROLE OF MANIPULATION CHECKS

As Bagozzi (1980) points out, a definition of casualty relies on a "human agent" being about something while controlling for all other possible effects. In order to actually profess a cause (e.g., involvement) and effect (e.g., cognitive consistency of the individual) relationship, a manipulation check is necessary to control for extraneous interferences by showing the "agent" (e.g., involvement) did in fact cause the effect, or was present at all.

To determine if the product classes used in the study were correctly classified as to their involvement levels (high or low), a number of items in the broader study's scale served as manipulation checks. These items consisted of nine-point semantic differential scale statements (e.g., "strongly agree" to "strongly disagree") measuring four elements of involvement gleaned from the literature. As noted, involvement was operationalized as a combination of four components: (1) information processing levels; (2) importance; (3) differentiation of salient attributes; and (4) commitment. Host studies agree that a high involvement state elicits higher levels of information processing on the part of the individual than does a low involvement state. A majority of the work also alludes (implicitly or explicitly) to importance, whether it be internally or externally instigated. Involvement seemed to be related to the ability of the individual to differentiate relevant (salient) relationships between objects and beliefs. Finally, there is agreement on commitment by the individual. These four components are developed from the work on involvement by Lastovicka (1979a, b), Lastovicka and Gardner (1978), DeBruicker (1979), Mitchell (1981), Petty and Cacioppo (1979, 1981), Kiesler (1971), Robertson (1976), and Ray, Sawyer, Rothschild, Reeler, Strong, and Reed (1973) and others. Figure 1 illustrates the operational measurement model for the involvement indicators used in the multiple discriminant analysis manipulation check.

FIGURE 1

OPERATIONAL MEASUREMENT MODEL FOR THE MANIPULATION CHECK INDICATORS

RESULTS AND ANALYSIS

A reliability analysis was run on the four indicators of product class involvement using the internal consistency measure of Cronbach's Alpha (Cronbach, 1951). Each of the four indicators of product class involvement (importance, commitment, information processing, and perceived differentiation) were operationalized by three items from the main questionnaire. The analysis on each of the sixteen scale combinations (four products X four indicator scales) yielded reliabilities above or near the generally accepted level of 0.8 (Peter 1979). Only one scale, the commitment scale for the low involvement product class of paper towels, yielded questionable results. This scale had had problems for the same product in the pretest and had been modified and retested qualitatively on a small group with no apparent problems. It is speculated, however, that the nature of the product itself may have caused the inconsistency to occur. That is, since paper towels may have been such an obviously low involvement product type, subjects may have not cared to bother with being consistent at all. These results can be seen in Table 5.

TABLE 5

INTERNAL CONSISTENCY MEASURES (ALPHA) FOR INVOLVEMENT MANIPULATION CHECK SCALES

Following this overall reliability analysis, another set of reliabilities was calculated for the involvement indicators. This procedure generated reliabilities for the combined involvement scales for each product class. In the procedure a four item analysis was used with commitment, importance, information processing, and perceived differentiation as the individual items for each product class involvement scale. Once again, the reliabilities were above the 0.8 level. An item analysis conducted on the scales showed that there would be a slight decline in the Alpha if any of the items were removed. There appeared to be significant declines if the commitment or importance items were removed for the jean or paper towel product class involvement scales. These product class involvement scales were four item scales comprised of the four components of involvement presented earlier: (1) importance; (2) commitment; (3) information processing; and (4) perceived differentiation. Each of these scale items was an averaged index of the three items from the questionnaire used to measure each of the product class involvement components. Scale means are an averaged mean of the four items over a scale from zero to one.

Scale means for the four product classes were .60 for jeans, .643 for stereos, .176 for paper towels and .413 for bar soap. These means indicate a higher mean scale involvement for the two reclassified high involvement product classes than for the two low involvement product classes. However, while these results suggest a difference, it was felt further analysis would be beneficial to help verify a correct preclassification of the involvement product classes.

Using MDA

To verify the involvement levels, as high or low, multiple discriminate analysis (MDA) was used. Multiple discriminant analysis is a multivariate extension of the univariate t-test for differences between means under the assumption of equal variances. Since the design was a within design, the procedure was well suited for use here. Maximal separation for the means was accomplished by taking linear combinations of the original variables which maximized the ratio of the between to within group variance.

Multiple discriminate analysis was deemed appropriate in this setting due to the nature of the predetermined nominal groups (i.e., high and low involvement) and the interval type involvement manipulation check scales (Hair, Anderson, Tatham, Grablowsky 1979; Klecka 1980). Using the involvement groups (high and low) as the dummy coded dependent variable and the four scaled item responses to the product class involvement measures as the independent variables; allowed an examination of whether or not the respondents viewed the predetermined involvement levels as correct.

In this procedure, each of the four product measures for each subject was treated as a separate case. By doing so, a sample for analysis of 1000 was created. The use of the two high and the two low involvement products effectively created a preassigned grouping of 500 subjects per involvement category.

It is noted that the artificial inflation of sample size from 250 to 1000 significantly increases the degrees of freedom for the statistical procedure. However, due to the restriction of this analysis to use as a manipulation check, this inflation was not viewed as critical. An additional note is added here concerning the violation of the independence criterion of the observations (see Klecka 1980). Since the 1000 observations are four observations per individual, a possible violation of the statistical assumption of independence exists. However, it was not felt that individuals' responses to one product class would affect their responses to another. Therefore, the violation of the independence assumption was not viewed as crucial to the use of the procedure as a manipulation check. Use of these data beyond simple manipulation check, however, is advised only with clear recognition of their limitations.

The discriminant equation for the analysis was generated by using a sample of ninety-nine subjects from the base sample of 250. Since each of the respondents had generated four distinct product class responses, the actual "cases" used to generate the discriminate equation totaled 396 (four products X ninety-nine subjects). The canonical correlation for the discriminant function was .600736, significant at the p=.00 level. The eigenvalue was .56466. Table 6 summarizes the standardized canonical discriminant function coefficients, structure matrix and canonical discriminant function evaluated as the group centroids (means).

This EDA output provides the researcher with substantive information as to the effectiveness of the manipulation and the explanatory ability of the manipulation check operationalizations. As Table 6 shows a substantial portion of the variance was explained by the discriminant function. It is also possible to assess the magnitude of the influence of each of the operationalized components in the explanation. For example, the commitment portion of the scale offers the most explanation of the four operationalized elements of involvement.

TABLE 6

SUMMARY OF MDA ANAYLSIS

The classification table for this analysis is shown as Table 7. As can be seen in Table 8 the cross-validation MDA procedure using the discriminant functions from the prior sample gave support to the classification of the study's products into their respective involvement categories. Using the remaining 151 respondents to generate 604 "cases", the results in Table 6 show a large proportion of the cases were classified correctly (81.79 percent). Hair, Anderson, Tatham, and Grablowsky (1979) liken this "hit percentage" to a regression procedure's R-square. The MDA results support the contention that the NGT procedures had generated correct classificatory data for use in the main study.

TABLE 7

CLASSIFICATION RESULTS FOR CASES SELECTED FOR USE IN THE ANALYSIS

TABLE 8

CLASSIFICATION RESULTS FOR CASES USED FOR CROSS VALIDATION

CONCLUSIONS

Very often, when manipulations are employed one finds ANOVA's and t-tests being used as the statistical manipulation check technique. The authors are unaware of any studies where MDA has been used. Clearly, in cases where the data is appropriate MDA should be explored as a technique to use.

Often in marketing researchers may be led by the techniques they are familiar with rather than by the technique which may be best suited for the task at hand. It is hoped that this paper has contributed to a greater awareness of a different method (EDA) which may be appropriate for manipulation check techniques under certain data configurations.

In addition, it is hoped that researchers (especially those working in the involvement area) may be more predisposed to using combinations of qualitative and quantitative methods to assure valid manipulations. As noted, manipulations may be at times be simply based on a researcher's "notion", not "hard data". The HDA results here indicate that the NGT procedure appears to hold promise for marketers exploring nebulous constructs for which consumers subjective responses are salient to an explanation of the construct.

REFERENCES

Bagozzi, R. R. (1980), Causal Models in Marketing, New York: John Wiley and Sons.

Churchill, G. (1979) "A Paradigm for Developing Better Measures of Marketing Constructs," Journal of Marketing Research, 16 (February), 64-73.

Cialdini, R. B., A. Levy, P. Herman, L. Koslowski, and R. Petty (1976), "Elastic Shifts in Opinion: Determininants of Direction and Personality," Journal of Personality and Social Psychology, 34, 663-672.

Claxton, J. D., J. R. B. Ritchie and J. Zaichkowsky (1980; "The Nominal Group Technique: Its Potential for Consumer Research," Journal of Consumer Research 9 (December), 308-313.

Collaros, P. A., and E. F. Loftus (1975), "A Spreading-Activation Theory of Semantic Processing," PsYchological Review, 82, 407-428.

Cronbach, L. J. (1951), "Coefficient Alpha and the Internal Structure of Tests," Psychometrika, 16, 297-334.

Cunningham, W. H., W. T. Anderson, Jr., and J. H. Murphy (1975), "Are Students Real People?, Journal of Business, 47. 399-409.

DeBruicker, F. S. (1979), "An Appraisal of Low-Involvement Consumer Information Processing," in J. C. Maloney and B. Silverman, eds., Attitude Research Plays for High Stakes, Chicago: American Marketing Association, 112-130.

Delbecq, A. L., A. B. Van de Van, and D. H. Gustafson (1975), Group Techniques for Program Planning, Glenview, It: Scott. Foresman and Company.

Dillon, W. R. and S. Weston (1982), "Scoring Frequency Data for Discriminant Analysis: Perhaps Discrete Procedures Can Be Avoided," Journal of Marketing Research, 19 (February), 44-56.

Hair, J. F.. R. E. Anderson, R. L. Tatham, and B. J. Grablowsky (1979), Multivariate Data Analysis, Tulsa, OK: The Petroleum Publishing Company.

Huber, G., and A. L. Delbecq (1972), "Guidelines for Combining the Judgement of Individual Members in Decision Conferences," Academy of Management Journal, 15, 2 (June), 161-174.

Kiesler, C. A. (1971), The Psychology of Commitment, New York: Academic Press.

Klecka, W. R. (1980), "Discriminant Analysis, Sullivan, ed, Series: Quantitative Applications in Social Sciences, No. 19.

Lamb, C. W., and D. W. Stem (1980), "An Evaluation of Students as Surrogates in Marketing Studies," in J. Olson, ed. Advances in Consumer Research, Vol. 7, Ann Arbor: Association for Consumer Research, 796-799.

Lastovicka, J. L. (1979a), "Questioning the Concept of Involvement Defined Product Classes," in W. Wilkie, ed., Advances in Consumer Research, Vol. 6, Ann Arbor: Association for Consumer Research, 174-179.

Lastovicka, J. L., (1979b), "Are Attitude Models Appropriate for Mass TV Advertising?," in J. Eighmey, et., Attitude Research Under the Sun, Chicago: American Marketing Association, 151-170.

Lastovicka, J. L., and D. M. Gardner (1978), "Low Involvement Versus High Involvement Cognitive Structures," in H. K. Hunt, ed., Advances in Consumer Research, Vol. 5, Ann Arbor: Association for Consumer Research. 87-92.

Mitchell, M. A. (1981), "The Dimensions of Advertising Involvement," in K. B. Monroe, et., Advances in Consumer Re- search, Vol. 8, Ann Arbor: Association for Consumer Research, 42-47,

Moore, D. L., and D. R. Rahtz (1984), "Involvement Paradigms and Problems: A Critical Review," Working Paper, School of Business Administration, College of William and Mary, Williamsburg, Virginia, 23185.

Osborn, A. F. (1975), Applied Imagination, New York: Scribners.

Peter, J. P. (1979), "Reliability: Review of Psychometric Basics and Recent Marketing Practices," Journal of Marketing Research 16 (February), 6-17.

Petty, R. D. and J. T. Cacioppo (1979), "Issue Involvement Can Decrease Persuasion by Enhancing Message-Relevant Cognitive Response," Journal of Personality and Social Psychology, 37, 1915-1926.

Petty, R. D. (1981), "Issue Involvement as a Moderator of the Effects on Attitude of Advertising Content and Context," in K. B. Monroe, ed., Advances in Consumer Research, Vol. 8. Ann Arbor: Association for Consumer Research. 20-24.

Ray, M. L., A. G. Sawyer, M. L. Rothschild, R. M. Heeler, E. C. Strong, and J. B. Reed (1973), "Marketing Communication and the Hierarchy of Effects," in P. Clarke, ed., New Models_for Mass Communication Research, Beverly Hills, CA: Sage Publishing Co., 147-176.

Robertson, T. S. (1976), "Low-Commitment Consumer Behavior," Journal of Advertising Research, 16 (April), 19-26.

Sawyer, A. G. (1975), "Demand Artifacts in Laboratory Experiments in Consumer Research," Journal of Consumer Research, 1 (March), 20-30.

Sawyer, A. G. and H. O. Oeter (1983), "The Significance of Statistical Significance Tests in Marketing Research," Journal of Marketing Research, 20 (May), 122-133.

Shaw, J. D. (1981), Group Dynamics: The Psychology of Small Group Behavior, New York: McGraw-Hill.

Taylor, D. W., P. C. Berry, and C. H. Block (1958), "Does Group Participation, When Using Brainstorming, Facilitate of Inhibit Creative Thinking?," Administrative Science Quarterly, 3 (June) 23-47.

Wiley, J. B. and L. M. Richard (1974), "Application of Discriminant Analysis in Formulating Promotional Strategy for Bank Credit Cards," in M. J. Schlinger, ed., Advances in Consumer Research, Vol. 2, Urbana: Association for Consumer Research. 535-544.

----------------------------------------