# A Meta-Analysis of the Diffusion of Innovations Literature

^{[ to cite ]:}

David F. Midgley (1987) ,"A Meta-Analysis of the Diffusion of Innovations Literature", in NA - Advances in Consumer Research Volume 14, eds. Melanie Wallendorf and Paul Anderson, Provo, UT : Association for Consumer Research, Pages: 204-207.

^{[ direct url ]:}

http://acrwebsite.org/volumes/6687/volumes/v14/NA-14

The 1840 published studies relating to 20 of Rogers and Shoemaker's generalizations on the diffusion of innovations are meta-analyzed by way of assumptions and simple techniques addressing sampling error. It is found that most of the discrepancy between studies supporting and not supporting these generalizations can be explained as statistical artifacts, and that there are likely to be correlations of order 0.20 between most independent variables and time of adoption. Such correlations are capable of improving our success rate in predicting earlier/later adopters to 60% (for one independent variable).

PURPOSE OF THE PAPER

The purpose of this paper is to begin the task of meta-analyzing the empirical literature concerning the diffusion of innovations. This beginning is achieved by first deriving a set of reasonable assumptions about research methodologies from a convenience sample of the diffusion literature. These assumptions are then used to guide a meta-analytic thought experiment on twenty of the major generalizations advanced by Rogers and Shoemaker (1971), generalizations which are based on a total of 1840 empirical studies, and which represent the variables most frequently researched by diffusion scholars. The thought experiment thus encompasses a major portion of the literature and essentially allows us to infer upper and lower bounds on the strength of the bivariate relationships postulated by Rogers and Shoemaker. The experiment allows us to test some of the basic tenets of diffusion theorists - namely that time of adoption is related to various independent factors. We seek to determine which of these relationships are spurious artifacts of small-sample surveys, and which represent valid correlations between time of adoption and some underlying characteristic of the individual, the innovation or the process by which the two are brought together. If we can establish some valid correlations then it may be worthwhile investing substantially greater efforts into a deeper and more systematic meta-analysis. Alternatively if we fail to find robust relationships then we are unable to choose between two competing (but equally pessimistic) conclusions. Firstly, that our basic tenets are somewhat flawed or overly simplistic. Secondly,that relationships exist but are obscured by some unspecified set of moderator variables. Indeed the contingency theories of innovative behavior advanced by Rogers (1983) and Midgley and Dowling (1978) would suggest that communication and situational moderators are necessary to explain relative time of adoption well. However, since these moderator variables have not been measured in the past this second conclusion would be tantamount to saying that 30 years of research had been to some degree wasted.

Background

Science progresses by cumulating and organizing empirically verifiable knowledge. As the knowledge base of a discipline increases so our ability to falsify earlier theories improves, as does the opportunity for advancing more sophisticated frameworks. However, in the social sciences it is widely recognized that progress has been relatively slow. This is undoubtedly due to a variety of factors, amongst which are:

1. the difficulty of measuring mental processes,

2. a reliance on correlational methodologies (which provide relatively weak tests of causality),

3. the difficulty of designing sensible experiments on social behavior,

4. the widely differing measures and methods used even within a set of studies on the same topic (the lack of replication),

5. the low statistical power of the generally small sample surveys employed by most researchers.

More relevantly to this paper one major factor retarding progress has been a forced reliance on the literature review as a means of cumulating the knowledge in any one area. At best, a written review of empirical findings is a weak approach to synthesis, particularly when the review relies on simple counts of the studies for and against a postulated relationship. Indeed it will be argued later that this procedure (termed 'vote counting') has the potential to combine with point 5 above in a dangerously misleading manner.

Recently quantitative techniques for cumulating results across studies have become available. These 'meta analysis' techniques hold out the promise of providing a rigorous approach to the cumulation problem, an approach which should yield more substantive insights as well as more precise estimates of the strength of relationships. There are two forms of 'meta-analysis'. The one, following Glass(1976), seeks to explain apparent variation in the strength of a relationship across studies by examining numerous characteristics of the methodologies used by those studies. The other form follows Hunter et al(1982) and assumes such variation ('conflict in the field') is largely the product of sampling error, range restriction and measurement unreliability. Hunter et al. (1982) report that in 152 applications of their techniques these three statistical artifacts account for 72% of the apparent variation in effect sizes (strength of relation ships). Given the purpose of this paper we will concentrate on sampling error here. Such errors are likely to be the most important artifact in the diffusion literature. It would also require substantial effort to handle other artifacts/study characteristics, particularly as adoption studies tend not to include much detail of the methodologies used. Furthermore, since most were conducted in the 1960's and 70's they tend to use somewhat unsophisticated methodologies (for example reliability is seldom reported). Thanks to the work of Hedges and Olkin (1980) we can assess the effects of sampling error by simply ascertaining the average sample size of the studies and the proportion of them that support the postulated relationship (under certain assumptions detailed below)

To digress, it might be argued that a meta-analysis of such an early literature is of limited value. Surely research on the diffusion of innovations has become more sophisticated since the 1970's? While this may be true such an argument misses the point on at least two counts. Firstly, that much of the subsequent conceptual elaboration is a direct result of early researchers' perceptions of apparent 'conflict' between the findings of comparable studies. Secondly, that much normative prescription on new product marketing implicitly assumes that individual characteristics are related to time of adoption. Now the methodology to re-examine the early literature is available it is surely worthwhile questioning how firm the foundations of the field are.

The reason sampling error is important relates to a common misconception of the meaning of significance levels. This misconception being that the use of a 5% significance level guarantees an error rate of 5% or less. In fact it only does so if the null hypothesis is true. If the null hypothesis is false then the error rate can go up to 95%. To take a simple illustration from Hunter et al(1982, p21) who show that if the true population correlation was 0.20 and all studies were done with a sample size of 40 then 2 studies out of every 3 would find the correlation to be not significant. Furthermore, as demonstrated by Hedges and Olkin(1980), such a situation would get worse the more studies that were done - leading reviewers to incorrectly assume the relationship did not exist. Indeed these authors highlight the fact that for the magnitude of effects we expect in the social sciences, and even for sample sizes of 100 or more, the power of vote-counting procedures can be asymptotically zero. Note we are only examining bias that might have been brought about by the low statistical power of early diffusion studies . This paper does not address the 'pro-innovation bias' of the field (Rogers, 1976), or biases brought about by the failure to publish non-significant findings (the 'file drawer problem' - Rosenthal, 1979).

Turning to the diffusion literature then it could be argued that we would expect relatively small correlations between (say) the characteristics of individuals and their subsequent time of adoption. This is because of the various product, communication and situational effects we might expect to moderate the relationship between the measurable characteristic of a particular individual and his/her behavior toward a specific innovation. This contingent view of adoption being advanced by Midgley and Dowling(1976), amongst others. Midgley and Dowling argue that correlations of around 0.25 would seem reasonable under most circumstances - just the magnitude of effect that would be hard to detect without large sample sizes.

Methodology

From a convenience sample of 95 published adoption studies we estimated the geometric mean sample size of the adopter and non-adopter groups used in these studies. Most adoption studies employ the two group methodology, contrasting the criterion variable between the groups. Here the values were 74 adopters and 173 non-adopters in a 'typical' sample of 247. 72 of the studies used a measure of effect size (most often that statistic) while 23 were correlational. 47 reported tests at the 5% level, but the remainder reported no tests of significance - simply stating whether the results supported the supposed relationship or not.

Hedges and Olkin(19807 present three meta-analysis techniques appropriate to this situation of little information, but which are adequate to allow a meaningful thought experiment here. The three techniques differ primarily in what is assumed or observed about the level of reporting in the literature. Firstly, we can assume simply that the studies report whether they support the relationship or not (termed 'positive results') without citing statistical tests. Secondly, we can assume that the studies report whether or not the relationship is in the postulated direction using significance tests at the 5% level. Finally we can assume that both positive and negative significant results are reported in the literature. In addition Hedges and Olkin methods assume that the population effect size is the same for all studies. Here we will be making such assumptions because we have estimated the typical sample size of a diffusion study from our convenience sample of 95 studies. This estimate is then applied to the wider population of the 1840 studies concerning the 20 generalizations of interest. The only other data needed for the thought experiment are Rogers and Shoemaker's tabulation of the number of studies supporting each generalization (1971, pp347-376) . Indeed as Rogers and Shoemaker do not say whether these studies employed statistical tests or not this is the only information available without individually examining all 1840 studies. By inputting the proportion of studies supporting the relationship, together with the geometric means of the group sizes, we can compute confidence bounds for the strength of the relationship. These confidence bounds are generally wider than those obtained by other meta-analysis approaches but then the Hedges and Olkin techniques require far less information on the knowledge base. Here we also assume that all studies are effect size studies (i.e. use t tests) because this dramatically simplifies the computations, and also because the Hedges and Olkin technique has not been extended to all statistics. This assumption places restrictions on the validity of our results, fortunately as noted above the majority of the i studies in our sample used effect size measures. Finally, Q at the end of our analysis we convert the bounds for the ; effect sizes into their equivalents in terms of point-biserial correlations (we also make a minor correction to adjust these correlations to those for equal group sizes). j We make this conversion because we feel most people can interpret correlations better than effect sizes.

Results

In the first column of Table 1 we list the names of the independent variables which are postulated by Rogers and Shoemaker to be related to time of adoption. We have stated these so that all are postulated to be positively related to time of adoption. We have also prefixed each with a letter indicating whether they are generalizations relating to attributes of the innovation(I), socioeconomic characteristics of the adopters(S), personality traits of the adopters(P), attitudes of the adopters(A), motivations of the adopters(M), or their integration with social communication networks(C). The second and third columns detail the number of studies which have been conducted on each generalization (up to 1971), and the proportion of studies which support the positive relationship with time of adoption. It should be remembered that these studies cover a wide range of Fields, from rural sociology to education to consumer behavior. The two proportions indicated with question marks are those where Rogers and Shoemaker feel there is no relationship. The fourth and fifth columns present the lower and upper 90% confidence bounds on the implied correlation between the independent variable and time of adoption assuming that the literature reports no statistical tests (a pessimistic assumption). Finally the sixth and seventh column present similar bounds assuming that all studies report results at the 5% significance level (an optimistic assumption if the findings of our convenience sample are a guide). The 'Positive Results' columns present a depressing picture of the field with the highest upper bound being 0.10! However, we suspect that this picture is too pessimistic as the results contained in the 'Positive Significant Results' columns are more in line with those found in other areas of the social sciences. That is, persistent low correlations between theoretical variables of interest and dependent measures (Cohen, 1977). If the convenience sample is a reasonable representation of the field (with 47 out of 95 studies reporting statistical tests) then the 'truth' obviously lies between the two extremes. For the moment, and until more data are available, we will concentrate on the 'Positive Significant Results' findings. If the sixth and seventh columns are examined then it is apparent that the sub-areas of the field where most work has been done have (as would be expected) the narrowest confidence bounds. Hence we are reasonably certain about the strength of relationships for the socioeconomic(S) and social communication(C) variables, and somewhat less certain about the remaining variables. We also see that the two variables Rogers and Shoemaker thought unrelated to time of adoption probably do have significant positive correlations (complexity and age). Indeed for age we can improve the estimate since Rogers and Shoemaker report studies with negative correlations as well as those with positive correlations (1971, pp352-354). Thus we can employ Hedges and Olkin's third technique to produce a more precise confidence bound of 0.10 to 0.14 as opposed to 0.08 to 0.10 (assuming again that these are positive and negative significant results). It appears that earlier adopters are more likely to be older than later adopters. Another interesting aspect of Table 1 is that the personality and attitudinal variables appear equally powerful to the other measures in predicting time of adoption. Some authors have argued against generalized personality measures as useful predictors of consumer behavior (Kassarjian and Sheffet, 1975), whilst the controversy as to whether attitudes predict behavior is well known. The Table also suggests where more research might usefully be done. For example trialability, the ability to deal with abstractions and rationality all appear variables with reasonably good predictive power and which make good theoretical sense - but which have had little attention in the past.

IMPUTED 90% BOUNDS ON CORRELATION COEFFICIENTS

CONCLUSIONS

It could be argued that correlations between 0.13 and 0.18 (which are the averages of the Table 1 results) are a relatively tenuous basis on which to built theory. However, it should be remembered that we are probably unable to make corrections for two other statistical artifacts, one of which would make the population correlations higher, and the other of which can have the same effect in some situations. These are the reliability of the measures and range restriction in the independent variables. Any reliability less than 1.0 in either the dependent or independent variables attenuates the correlation markedly. For example if we assumed the dependent variable (time of adoption) was relatively well measured (reliability of 0.8) but the independent less so (reliability of 0.6), and that our estimate of the population correlation was 0.14 then adjusting for attenuation the true population correlation would be 0.20. Range restriction is a little harder to assess. If diffusion studies have less variation in their independent variables than the population then the observed correlations will be reduced. Equally if they have greater variation (range enhancement) then the correlation will be increased. Range restriction is more common than range enhancement so we suspect again that the population correlations are higher than we state above. As neither reliability or range tend to be reported in past studies we will be unable to do much more than speculate about these effects.

It should also be noted that a correlation of 0.20 provides a useful amount of predictive power. The work of Rosenthal and Rubin(1982) shows that a correlation of 0.16 yields a 58% chance of predicting earlier/later adopters correctly. A correlation of 0.20 yields a 602 success rate. So even variables accounting for a very low proportion of the variance can increase our predictive power substantially.

There is also a question of how much these results might change if we added in the studies done since 1971. Unfortunately Rogers(1983) does not update the generalization analysis, possibly because the volume of studies has approximately doubled and it is increasingly difficult to keep track of them.

However, if experience in other areas is any guide more recent studies will not help us that much. As fields 'mature' they tend not to do the basic research exemplified by Table 1 but prefer more elaborate theoretical frameworks and sophisticated statistical analysis. Partly this is because of peer pressure for scientific 'contribution' but also it is an attempt to resolve the apparent conflict in earlier studies, or perhaps to find 'better' predictors. If the earlier studies were done with small samples (and meta-analysts argue that small means anything less than several thousand respondents - Hunter et al, 1982) then this increasing sophistication may be misguided. Clearly diffusion research would be well served by several large scale studies using a set of well-measured independent variables. We need the set because very few studies have utilized a comprehensive list of variables, and we are therefore unsure as to the interactions between the independent constructs shown in the Table. The construction of this set can obviously be guided by theory since we have made some progress in the last 30 years. Such studies should also include the various moderator variables which have emerged in the literature. Above all they should be well-reported. This means complete details of all results (significant and non-significant), including means, standard deviations, correlations and reliabilities. Hunter et al(1982) contains specifications for well-reported studies - specifications which are not followed by most journals as yet.

Finally there is a little more that can be done with the past literature. Most studies do report which innovations were examined and in which cultural setting the research was conducted. The author is currently working on an meta-analytic investigation of the effects of these moderator variables - specifically on the relationship between social status and time of adoption.

REFERENCES

Cohen, J. (1977), Statistical Power Analysis for the Behavioral Sciences, New York: Academic Press

Glass, G.V. (1978), 'Integrating Findings: The meta-analysis of research', Review of Research in Education, 5, 351-379.

Hedges, t.V. and I. Olkin (1980),'Vote-counting Methods in Research Synthesis', Psychological Bulletin, Vol. 88, No. 2, 359-369.

Hunter, J.E. Schmidt, F.L. and G.B. Jackson (1982), Meta- Analysis: Cumulating Research Findings Across Studies, Beverley Hills: Sage Publications

Kassarjian, H.H. and M.J. Sheffet (1975) 'Personality and Consumer Behavior. One More Time', in E.M. Maze (ed), Combined Proceedings of the Spring and Fall Conferences of the American Marketing Association, 197-200.

Midgley, D.F. and G.R. Dowling (1978).'Innovativeness: The Concept and Its Measurement', Journal of Consumer Research, 4, 229-242.

Rogers, E.M. (1983), Diffusion of Innovations, Glencoe: Free Press

Rogers, E.M. (1976) 'Sew Product Adoption and Diffusion', Journal of Consumer Research, 2, 4, 290-301.

Rogers, E.M. and F.F. Shoemaker (1971), The Communication of Innovations, Glencoe: Free Press.

Rosenthal, R. and D.B. Rubin (1982),'A simple, general purpose display of magnitude of experimental effect', Journal of Education Psychology, 74, 166-169.

Rosenthal, R. (1979), 'The file drawer problem", Psychological Bulletin, 86, 638-641,

----------------------------------------

Tweet
window.twttr = (function (d, s, id) { var js, fjs = d.getElementsByTagName(s)[0], t = window.twttr || {}; if (d.getElementById(id)) return; js = d.createElement(s); js.id = id; js.src = "https://platform.twitter.com/widgets.js"; fjs.parentNode.insertBefore(js, fjs); t._e = []; t.ready = function (f) { t._e.push(f); }; return t; } (document, "script", "twitter-wjs"));