A Procedure For Testing the Emotional and Attitudinal Effects of Design

ABSTRACT - This project presents the problem of testing decisions companies make about their visual appearances before investing heavily in a new design program. The test was developed on the basis of experiments that were inspired by advertising tests (ELAM). The experiments were based on visual material shown to respondents who used a questionnaire. The data has been analyzed by factor analysis and it is shown how the test can be conducted with a reduced questionnaire. We are able to distinguish between the emotions attitudes, preferences and liking for designs. The findings are valid and the test can be used in practice for diagnosis, pretesting and planning.

Citation:

Tore Kristensen, Gorm Gabrielsen, Flemming Hansen, and Jens Halling (2001) ,"A Procedure For Testing the Emotional and Attitudinal Effects of Design", in E - European Advances in Consumer Research Volume 5, eds. Andrea Groeppel-Klien and Frank-Rudolf Esch, Provo, UT : Association for Consumer Research, Pages: 300-309.

European Advances in Consumer Research Volume 5, 2001      Pages 300-309

A PROCEDURE FOR TESTING THE EMOTIONAL AND ATTITUDINAL EFFECTS OF DESIGN

Tore Kristensen, Copenhagen Business School, Denmark

Gorm Gabrielsen, Copenhagen Business School, Denmark

Flemming Hansen, Copenhagen Business School, Denmark

Jens Halling, Copenhagen Business School, Denmark

ABSTRACT -

This project presents the problem of testing decisions companies make about their visual appearances before investing heavily in a new design program. The test was developed on the basis of experiments that were inspired by advertising tests (ELAM). The experiments were based on visual material shown to respondents who used a questionnaire. The data has been analyzed by factor analysis and it is shown how the test can be conducted with a reduced questionnaire. We are able to distinguish between the emotions attitudes, preferences and liking for designs. The findings are valid and the test can be used in practice for diagnosis, pretesting and planning.

INTRODUCTION

The need for pre-testing corporate designs is evident. Often, these programs have been launched with a minimum of tests. The tests most frequently used are qualitative and partial. They focus on one issue, e.g. the corporate reputation, an advert or a product. Comprehensive tests are not common. We, therefore, aim at developing a test that is robust and easy to use for companies and design consultants who want to test a corporate design.

To develop a relatively simple test procedure, a large design test program was initiated. Some findings from this program have been published earlier. The discrimination power of the earliest part of the tests is demonstrated in Gabrielsen et al. (1999). In Hansen et al. (2000), the special role of color in design is explored, and in Kristensen et al. (2000) the applicability of the results from testing different design is discussed. On the whole, the discriminative power of the test items used is well documented. The test took about 50 minutes for the respondents to complete. This procedure is both time consuming and costly, which demonstrates a need for a simplified test that requires less time for respondents to complete and less data analysis.

In this presentation, a reduced, standardized research instrument for applied testing of designs is presented. The process of development included the following steps:

The original data consists of 6 different samples of contrast between the actual design of a travel agency and an artificial design. The artificial design may be seen as an attempt to change the existing design to effect strategic renewal in an existing company. It may also be seen as the attempt, from an entrant, to capture the market by positioning himself closely to the incumbent’s design. Then we obtained data from a bookstore, which was also presented in its original design and contrasted with the artificial design. The artificial design was essentially identical in both samples. This means that we got two samples with the same artificial design, disguised as a travel agency and a bookstore. In both cases, pictures of the fatade, brochures, stationary, ticket leaflets and 4 different paper qualities with the same stationary and 4 different colors of the logo represented the designs. Finally, we also tested 2 existing designs corresponding to the Danish railroad’s old and new designs (implemented in 1999 while our data collection took place). The respondents reported their emotions; attitudes and preferences according to words associations describing emotions and feelings based on Richins (1997) test battery. Furthermore, we added a few more words that we wanted to test too. The respondents were asked to indicate the suitability of the pre-selected appropriate words on a 5-point scale. In total, we had 4 different samples with single source data obtained from a monistic test. The data was analyzed in accordance to a factor analysis with varimax rotation. This resulted in a variety of significant factors explaining the discriminations obtained in the testing procedure. The findings are reported in Hansen et al. (1999), Kristensen et al. (2000) and Gabrielsen et al. (2000). Clearly, some word associations obtained a higher score, in the sense of being significant and discriminating factors, than others. The strongest indications were retested in a reduced test battery. We then proceeded to demonstrate how the reduced test might be used as a simplified test.

METHODOLOGY

Design testing involves two basic problems. First, the design must be presented to the respondents in a credible form and in a realistic context. It is seldom possible to do this in practice, either because the alternatives are too costly (car design) or because it is difficult to present the elements that are contained in a complete design line at the same time.

Secondly, unlike the testing of most other forms of communication, design testing deals with more diffuse, emotional, and thus less quantifiable effect measures.

This way of presenting the test elements is relevant when testing individual designs, as well as complete design lines. However, the presentation problem is most distinct in the testing of the design line. We chose to work with the design line problem in general, as well as in particular, e.g. paper quality and colors. We developed a test that would not force people to deliberate on the designs, but rather to entice them to give us their judgements in a natural way. With this kind of test, people usually focus on the company’s offers and not its design. Yet, we believe the design can influence people’s perception of a product or service when there is limited differentiation between competing products or there is quality uncertainty, similarly to corporate associations (Brown and Dacin, 1997).

An artificial design for a travel agency with the name "Subgate" was developed in the form of a logo, a letterhead, a brochure, a store facade, and a ticket envelope. This design was presented both as a new travel agency and as a bookstore. In addition, measures were obtained regarding existing travel agencies and an existing bookstore. Also, they were presented in the form of multiple design elements like those in Exhibit 1.

In this project, the focus is on the testing of the design line, as the experiences achieved here can be transferred to the testing of individual design elements; the reverse action is not possible. It was also chosen in order to work with an experimental set up in line with the research designs that are used in the testing of advertising, packaging, and other communication.

The study was conducted through interviews with student samples in five test rounds. The overall sample is composed of four subsamples consisting of respondents who were selected according to the matched sampling principle. The number of observations, for each of the four designs, is shown in table 1. In addition to completing the monadic part of the test, the respondents are asked to evaluate and compare alternative executions of the design (colors, paper quality, etc.). This data, however, is not considered here.

TABLE 1

NUMBER OF OBSERVATIONS

FIGURE 1

THE ELABORATION LIKELIHOOD ADVERTISING (ELAM) MODEL

In the actual data collection, each test group is confronted with the following seven steps in addition to being informed that the tests take about 1 hour to complete:

1. Motivation for participation.

2. First questionnaire (socio-demographical, behavioral, and other descriptive measures).

3. First exposure to test material (four different versions presented monadically).

4. Second questionnaire (evaluation and response to the design to which the respondent is presented).

5. Presentation of alternative design executions.

6. Third questionnaire (comparison of executions).

7. Conclusion, debriefing.

All measurements of importance for the evaluation of the whole design line are recorded in the second questionnaire, after exposure to one of each of the four full design lines.

MEASUREMENTS

The test design (second) questionnaire follows a standardized model using measures inspired by the standardized measures used in advertising testing: The (ELAM) Elaboration Likelihood Advertising Model (Petty and Caccioppo, 1986; Hansen, 1997).

In the ELAM test, measurements reflect central as well as peripheral information processing. The measurements are illustrated in Figure 1. In the first part, the central processing is represented, in the second part we find the peripheral processing. In central information processing, as in problem solving, the individual is concerned with evaluating alternatives and their merits. In advertising, alternatives and their merit is brand / product / company related information.

In the second part of Figure 1, the peripheral, less involved, more emotional information processing is reflected in measurements relating to the story in the advertisement, to aroused emotions and to the execution of the advertisement.

Measurements relating to these two processes are applied to the extent that they are meaningful in concept testing. Using the measures used in advertising testing makes it possible to compare findings with results from other test areas. Measurements concerned with peripheral information processing (Petty et al., 1986) are particularly in focus, but other measurements are also included.

In the complete design evaluation questionnaire, with 228 items, the measures are:

1. Pre/post awareness of the company behind the design being tested.

2. Evaluative effects, Attitudes towards designs (A-De). In the ELAM standard test a battery of eight questions is used. Here, this is expanded to a total of 16 questions.

3. Emotional responses, measured partly with the use of a standardized simple, binary coded battery taken from the ELAM procedure. Additionally, considering the importance of the emotional responses in connection with design, an alternative international standardized measurement instrument is also used (Richins, 1997).

4. In the test, respondents are asked to give verbal reactions as to what they like and dislike about the design they are shown. In a sequence of eight questions, they are asked things like what they think the company/design holder wants to communicate through the design, what they would emphasize if they were to describe the design to others, etc.

5. Especially developed questions are used for the purpose of quantifying the role of particularly relevant design elements, logos, colors, paper quality, etc.

6. Word associations. The respondents were asked to choose 4 out of 26 words, such as movie, graffiti, family, etc. These words reflect possible product design related associations, suggested by the designer, indicating what best described the design they had just seen. Since some of the respondents had seen one of the existing companies’ designs, and other had seen only the artificial Subgate design, we expect very different responses corresponding to the different conditions. In association with the product area/company for which it is presented, the words chosen reveal part of the meaning covered by the design.

7. Preferences. From the answers to two different preference questions, a total preference score can be computed.

8. Intentions. Finally, intentions were measured, partly as straight forward "purchase intentions" and partly by asking people whether they found that the design made them more or less likely to choose the supplier with whom the design was associated next time they were to buy books/travel tickets.

TABLE 2

THREE FACTO VARIMAX-ROTATED SOLUTION FOR ATTITUDES TOWARDS DESIGNS

The Reduced Design Battery

The original test battery, with its 228 items, is obviously far too complex for applied research.

From the measures employed in advertising pre-testing, using the ELAM battery, measures, numbered 2, 3, 5, 7, 8, 10 and 11, are included in the reduced design test battery. Additionally, in the proposed reduced test battery, it is suggested to include number 9, a measure of "design liking", as well as a self-rated measure of attention value (number 1).

Furthermore, some of the additional measures, which were included in the large questionnaire to reveal aspects of the perception of the design, are also included in the reduced questionnaire.

Brand Awareness (question)

By measuring brand awareness, early and late, relative to the exposure of the test material, we hoped that changes in awareness of the name of the designing company could be measured. Such changes did not occur, and consequently we only work with one measure of awareness of the designing company in the reduced battery.

Awareness is, of course, higher for known companies, but it is remarkable that the rated awareness for Subgate is higher in the travel agency condition, than in the bookstore condition.

Attitudes towards Design (A-De)

Attitudes towards design are measured along the same attitude-dimensions used for measuring attitudes towards advertising. This battery was improved by adding attitudinal items thought to be particularly relevant for the respondents’ judgement of attitudes towards design.

The answers to these questions were submitted to a factor analysis for each of the five design tests. The factor structure was quite similar for all of the designs, and consequently a joint analysis across the four designs was conducted.

Here, a three-dimensional varimax-rotated solution was found to be very useful. This solution accounted for 58 percent of the total variance, and adding a fourth factor only modestly increased the amount of variance.

The solution is shown in table 2. The first factor obviously suggests entertaining, lively, untraditional and acceptable aspects of the design. We labeled this factor "enjoyment". Similarly, in advertising testing, the same factor is also labeled "entertaining".

The second factor reveals the "credibility" of the design. Descriptions such as "informative, trustworthy, effective" are loading on this factor. The third factor combines negative sides associated with the designs. It can be seen as "dislike". For future analyses, the three highest loading descriptions on each of the three factors are included.

TABLE 3

ATTITUDES TOWARDS DESIGN INDEX

FIGURE 2

REDUCED EVALUATIONS OF DESIGN ITEMS

In Table 3, the scores are indexed for each of the three selected descriptions, for each of the three factors and for each of the four alternatives evaluated. Here, it appears that for the travel agencies, the Kilroy version of the design scores higher than the completely unknown Subgate version in all categories. It scores higher on the stimulating and informative factor, but lower on the negative, evaluative factor.

With the bookstores, a somewhat more complicated picture emerges. Here, the Subgate design wins over the same design in the "Samfundslitteratur" association with regard to the "enjoyment" dimensions, but "Samfundslitteratur" scores higher on the information value and negative evaluations.

The context in which the design is presented influences the evaluations, but an existing, known company does not always give added value to a given design. Looking at the travel applications, in 6 out of 9 cases, the Subgate evaluations come out more positive than the "amfundslitteratur" evaluations. The more positive evaluation of the artificial design, presented as a real travel agency in contrast to a real bookstore, also suggests a strong, positive interaction with the product area. There are only small differences between the Subgate versions.

Evaluation of Design Elements

A design line is composed of a smaller or larger number of elements through which the design is expressed. In the testing conducted here, each design line was represented with different items, as shown in Exhibit 1. Here, the artificial company "Subgate" was presented as a travel agency. In other conditions of the tests, it was presented as a bookstore. Furthermore, the same design elements were presented with the names of an existing travel agency (Kilroy) and an existing students’ bookstore ("Samfundslitteratur").

The evaluation of each of the design elements may differ, and some evaluative dimensions (attitudes) may be specific to single design elements, such as the paper quality of the letterhead, or the illustrations of the brochure. Others are more general, such as colors, use of font in logos etc.

To explore how the four tested design lines were evaluated for each of the five elements through which they were presented, respondents were asked to evaluate each of the five design elements for each of the five design lines, along 7-8 evaluative statements. Analyzing these statements, design line by design line and design element by design element, we found a remarkable stability in the items, which explains most of the variations in the judgements.

Most of the evaluations were of a more general type, such as "modern, dull, exclusive", etc. Consequently, it was suggested that we reduce this large element in the original questionnaire, (7 by 5 by 5 informational items). This was accomplished by having respondents associate the most general and most important test elements with the items representing the designs. This is shown in Figure 2.

TABLE 4

CONNECTION BETWEEN DESIGN ELEMENTS INDEX

TABLE 5A & 5B

THREE FACTOR VARIMAX-ROTATED SOLUTIONS FOR TRAVEL AGENCIES (a) AND BOOKSTORES (b)

Perceived Relationship among Test Elements

The perceived consistency of the design elements was rated on a single 5-point scale. This scale is maintained in the reduced questionnaire.

The average perceived relationship between the alternatives is shown in Table 4. It seems, for the similar design elements used here, that this relationship is not strongly influenced by the context in which the design elements are presented.

Word associations

To judge the extend to which the different design lines give rise to different associations in the target group, a number of items, such as "summer vacation", "film", "graffiti", "crime", were suggested by the designers, and the respondents were asked to select five of these. In table 5a and 5b, the varimax-rotated factor solutions is shown based upon the responses to the 26 items, for travel agencies and bookstores, respectively. Both result in three factor solutions affording reasonable interpretations. Obviously, the solutions are different for books and for travel agencies (and so are some of the statements). In applied use, it is recommended that you develop these statements with the tested product area in mind. However, it should be possible to work with less than the 26 items included here. If more information had been available, the three highest loading items for each of the three factors could basically have provided the same information as the larger selection of questions. Thus, it is recommended to work with a 10 +/- individually selected items battery with a particular product area in mind. To identify these items, preliminary research may be required.

In table 6, the percentage of respondents are given for the items loading highest in the first three factors for travel agencies and bookstores, respectively. It appears that these attitudes are different for Subgate and Kilroy Travels, and for "Samfundslitteratur" and Subgate Books.

If the purpose of the Subgate design is to generate an image of" techno-advanced" modernity, it is obvious that Subgate, as a travel agency, out-performs the existing Kilroy. On the contrary, Kilroy stands out as more traditional (homemade summer holiday, seaside holiday) than the Subgate Travel alternative. Similarly, with the bookstores, "Samfundslitteratur" is seen as much more traditional (80’ies, classic, Danish), and much more as a "bookstore" (Special literature bookstore/ library), but less "techno-advanced" (sex, "kings", freedom).

Preferences

Preferences are measured with two 6-point scales taken from the ELAM battery. These are used in the reduced version too.

As seen in table 7, there is little or no effect using this measure. These measures are primarily indicators of central information processing. Here they give a limited indication of such processing, but the real Bknown- companies, of course, score higher.

Purchase Intentions

Purchase intentions in the ELAM test are measured by distinguishing between a direct purchase intention and a self-rated change in purchase intention. The difference between the purchase intention, and the self-rated change in purchase intention may be seen as an effect measure of the exposure to the design. The data is shown in Table 8.

In this view, the real company designs generate more positive effects than the artificial Subgate design.

TABLE 6

ASSOCIATIONS

TABLE 7

PREFERENCES

TABLE 8

PURCHASE INTENTIONS

Emotions

Emotions were measured with the 12 item emotional battery used in advertising pre-testing. Factor analyses conducted on the emotional responses to the design tested here show that one factor accounts for more then 40 percent of the total variance, and covers the essence of these measures. This factor is shown in table 9a. Four statements, two positive and two negative, are recommended for use in the reduced version.

In addition, we included a selection of meaningful items from an alternative emotional measurement battery (Richins, 1997) based upon the assumption that emotional responses are extremely important in connection with design evaluations. 13 emotional items are rated according to their suitability with each of the four-design/product area combinations. Here, a varimax-rotated factor analysis identified two factors that account for 47 percent of the total variance in the data set. Additional factors account for only little more variance than any single question in the battery, and the two factors are easily identified as reflecting positive versus negative emotions. The solution is shown in table 9b. Thus, we recommend that the three highest loading positive and three highest loading negative items be included in a standardized battery. It is remarkable that the response to both batteries can basically be reduced to positive/negative emotional responses, rather than more elaborate feelings.

The percentage of the respondents who agree with the most important statements in each of the two emotional factor solutions are shown in table 10a. The remarkable sensitivity of these statements is obvious. Joy is definitely more related to the travel alternatives than to the bookstore alternatives. Among te travel alternatives, Kilroy gets the highest score. Grief is first and foremost associated with the bookstores, and so is anger. When the two Subgate versions are compared, the more positive evaluation of the travel version may be ascribed to its association with the travel business. As similar picture, with less dramatic differences in the scores, appears when the second emotional battery is analyzed in the same manner. The data is shown in table 10b.

Here, the Subgate designs for travel agencies do very well.

TABLE 9A & 9B

FACTOR SOLUTIONS-EMOTIONS

TABLE 10A & 10B

EMOTIONS

Recognizability

As a separate item of particular importance in design evaluation, the extent to which respondents judge the designs as easily identifiable, is measured by having respondents allocate 100 points to five alternative well-known designs, among which the test-design is one.

In table 11, the artificial Subgate designs score remarkably higher on average both as a travel agency and as a bookstore. The lowest score assigned to the real travel agency and bookstore may be ascribed to the new design being unfamiliar compared to the designs of well-known companies. The lack of perceived recognizability for the known companies may be a result of a lack of recognizability in relation to the existing designs of these companies, as they are known in advance.

In the reduced questionnaire, a measure of recognizability is maintained by the inclusion of a single 5-point rating scale.

Company Image

To test whether the different designs influence the perception of the companies behind the designs in different ways, a number of general image statements were included in the battery. While analyzing these perceptions, it turned out, as expected, that different factor structures are found for bookstores and travel agencies, respectively. It turns out that in all four cases, three factor solutions provide meaningful interpretations.

TABLE 11

RECOGNIZABILITY

TABLE 12

THREE FACTOR SOLUTION FOR SUBGATE TRAVELIMAGE ITEMS

As an example, the Subgate Travel solution is shown in Table 12. Here, the three factors explain 54 percent of the variance with the first factor that reflects "in-ness"; loading on "international", "modern" and "for young people", the second reflects "service", "good selection", "easy to communicate with" and "well-known".

The last factor reflects special "student"-related items, such as "student-friendly" and "long waiting time".

Once again, no standardized battery seems feasible. The questions chosen must be related to product/company/design. The most frequent answers for bookstores and travel agencies, among the statements used here, are: "not a good selection", "not friendly towards students", "not easy to communicate with".

Our research has shown that the different designs give rise to different evaluations along the dimensions used here, but in specific testing we recommended that you work with items selected for their particular relevance to the industry or company in question.

CONCLUSIONS

The purpose of this article is to investigate the possibility of a simple robust tool for design testing. Based on an experimental design test with a large questionnaire, we have been able to reduce the number of test elements by weeding out the less discriminative items of the experiments. Redundant information and noise has been removed. What is left is attitudes to the designs, which according to the factor analysis with varimax rotation yielded 3 factors that explained 58% of the variation. The Index Table strongly indicates that discrimination is possible, see Table 2. Also the attitudes towards individual design elements (see Figure 2) demonsrate an ability to distinguish between design elements using a reduced battery. The use of word associations as an additional method of testing reinforces the ability of the test to generate images of very different designs. The number of associative words can be reduced and still be able to represent the company’s images as, for instance, "techno-advanced" (Subgate) vs. home-made, summer seaside holiday (Kilroy). We strongly believe that such feedback provides important managerial information. It was possible to measure preferences and purchase intentions with the reduced test battery. The emotional reactions were measured adequately with 2 factors, indicating positive vs. negative. Additional factors yielded very limited information and a reduction to the 3 strongest positive/negative items in the test battery provided sufficient information. The test also indicated that recognizability strongly favored the artificial designs. Compared to the existing designs, this may indicate that changes in a design, to the extent Subgate differs form Kilroy and Samfundslitteratur, will prompt recognition. This, on the other hand, does not necessarily lead to preferences, because, as the next part of the test showed, incumbent firms (or existing designs) have a stronger foundation. We saw that Subgate was not perceived as credible.

The reduced test should become a useful, robust and practical tool. The reduction of information makes the data collection process faster, and the simpler test battery reduces the computational requirements. The next step will be to investigate other forms of design representation, such as pair wise comparison and the use of personal computers that the respondents would work with. When this is done, the process of transforming data from paper to a database will be eliminated and the test can be conducted over an even shorter period. Also, it will be possible to adjust the sampling during the process because the data can be analyzed along with the data collection. We predict that the use of a portable computer in hall tests and other places will provide feedback about designs from relevant samples faster. It should be possible to transfer such data to management input within a short period of time, so it becomes possible to experiment with more designs and get more detailed test data based on less data. More iterations of the test should support management learning in ways that are not possible when the test takes months to complete.

EXHIBIT 1

SAMPLE DESIGN LINE

REFERENCES

Brown, Tom J. and Dacin, Peter A.: The Company and the product: Corporate Associations and Consumer Product Responses. Journal of Marketing 61 (January 1997): 68B84.

Hansen, Flemming: Quantifying Creative Contributions: Advertising Pretesting’s New Generation. Proceedings from ESOMAR’s 50^th Congress (September 1997).

Kristensen, T. and Sverdrup-Jensen, Jonas: Testing Customer encounters. Design Management Journal 10 (1999):

Petty, R.E., Cacioppo, J.T. and Schumann, D: Central and Peripherical Rates to Advertising Effectiveness: The Moderating Role of Involvement. Journal of Consumer Research 10 (1986):

Richins, M.L: Measuring Emotions in the Consumption Experience. Journal of Consumer Research 24 (1997):

Gabrielsen, G: Corporate Design: A Tool for Testing. Corporate Communications International 5/2 (2000):

Gabrielsen, G, Kristensen, T., Hansen, F and Halling, J.: The meaning of Colours in Design. Research Paper no 2, Copenhagen Business School (2000):

----------------------------------------