The New Statistics
.@neuromusic Great paper, out of my field. Wouldn’t have read if @sapinker hadn’t endorsed. The New #Statistics
http://pss.sagepub.com/content/early/2013/11/07/0956797613504966.abstract 1/3
.@neuromusic @sapinker Fig 1 shows how much more variable p-values are than CIs across simulated replicates, virtually no replication. (2/3)
.@neuromusic @sapinker Article argues strongly for meta-analysis & that we can get by with calculations for AIPE instead of power (3/3).
Admit that I’m inclined to the position taken by the medical literature to report CIs & p-values.
QT:{{”
Second, in response to renewed recognition of the severe flaws of null-hypothesis significance testing (NHST), we need to shift from reliance on NHST to estimation and other preferred techniques. The new statistics refers to recommended practices, including estimation based on effect sizes, confidence intervals, and meta-analysis.
…
I describe here a major problem of NHST that is too little recognized. If p reveals truth, and we replicate the experiment–doing everything the same except with a new random sample–then replication p, the p value in the second experiment, should presumably reveal the same truth. We can simulate such idealized replication to investigate the variability of p. Figure 1 depicts the simulated results of 25 replications of an experiment with two independent groups, each group having an n of 32. The population ES is 10 units of the dependent variable, or a Cohen’s δ of 0.50, which is conventionally considered a medium effect. Statistical power to find a medium-sized effect is .50, so the experiment is typical of what is published in many fields in psychology (Cohen, 1962; Maxwell, 2004).
…
The 95% CIs bounce around as we expect; they form the dance of the CIs. Possibly surprising is the enormous variation in the p value–from less than .001 to .75. It seems that p can take almost any value! This dance of the p values is astonishingly wide! You can see more about the dance at tiny.cc/dancepvals and download ESCI (Cumming, 2013) to run the simulation. Vary the population ES and n, and you will find that even when power is high–in fact, in virtually every situation–p varies dramatically (Cumming, 2008).
“}}
http://pss.sagepub.com/content/early/2013/11/07/0956797613504966.long