Ram Bajpai writes:

I’m an early career researcher in medical statistics with keen interest in meta-analysis (including Bayesian meta-analysis) and prognostic modeling. I’m conducting a methodological systematic review of Bayesian meta-analysis in the biomedical research. After reading these studies, many authors presented both Bayesian and classical results together and comparing them and usually say both methods provide similar results (trying to validate). However, being a statistician, I don’t see any point of analysing data from both techniques as these are two different philosophies and either one is sufficient if well planned and executed. Consider me no Bayesian expert, I seek your guidance on this issue. My basic question is do we really need data to be analysed by both methods?

My quick answer is that I think more in terms of methods than philosophies. Often a classical method is interpretable as a Bayesian method with a certain prior. This sort of insight can be useful. From the other direction, the frequency properties of a Bayesian method can be evaluated as if it were a classical procedure.

This also reminds me of a discussion I had yesterday with Aaditya Ramdas at CMU. Ramdas has done a lot of theoretical work on null hypothesis significance testing; I’ve done lots of applied and methodological work using Bayesian inference. Ramdas expressed the view that it is a bad thing that there are deep philosophical divisions in statistical regarding how to approach even the simplest problems. I replied that I didn’t see a deep philosophical divide between Bayesian inference and classical significance testing. To me, the differences are in what assumptions we are willing to swallow.

My take on statistical philosophy is that all statistical methods require assumptions that are almost always clearly false. Hypothesis testing is all about assumptions that something is exactly zero, which does not make sense in any problem I’ve studied. If you bring this up with people who work on or use hypothesis testing, they’ll say something along the lines of, Yeah, yeah, sure, I know, but it’s a reasonable approximation and we can alter the assumption when we need to. Bayesian inference relies assumptions such as normality and logistic curves. If you bring this up with people who work on or use Bayesian inference, they’ll say something along the lines of, Yeah, yeah, sure, I know, but it’s a reasonable approximation and we can alter the assumption when we need to. To me, what appears to be different philosophies are more like different sorts of assumptions that people are comfortable with. It’s not just a “matter of taste”—different methods work better for different problems, and, as Rob Kass says, the methods that you use will, and should, be influenced by the problems you work on—I just think it makes more sense to focus on differences in methods and assumptions rather than frame as incommensurable philosophies. I do think philosophical understanding, and misunderstanding, can make a difference in applied work—see section 7 of my paper with Shalizi.

Ron Bloom wrote in with a question:

The following pseudo-conundrum is “classical” and “frequentist” — no priors involved; only two PDFS (completely specified) and a “likelihood” inference. The conundrum however may be interesting to you in its simple scope; and perhaps you can see the resolution. I cannot; and it is causing me to experience something along the lines of what Kendall says somewhere (about something else entirely) about “… the problem has that aspect of certain optical illusions; giving different appearances depending upon how one looks at it…”

Suppose I have p(x|mu0) and p(x|mu1) both weighted Gaussian sums with stipulated standard deviations and stipulated weights; for definiteness say both are three term sums; moreover all three constituent Gaussians have the common mean named in the expression p(x|mu). so they look like “heavy tailed” Gaussians at least from a distance.

Suppose mu0 < mu1 are both stipulated too; in fact everything is stipulated; so this is *not* an estimation problem; nothing to do with "EM" or maximum likelihood. Just classical test between two simple alternatives. A single datum is acquired: x. The classical procedure for deciding between "H0" and "H1" is to choose the test "size". Put down the threshold cut T on the right tail of p(x|mu0) so the area above that cut is the test size; the power of that test against the stipulated alternative H1 is of course the area above T under p(x|mu1). When the PDFs are Gaussian or in an exponential family or when "a sufficient statistic is available" this procedure above is identical to what one does if he uses the Neyman-Pearson likelihood criterion: which amounts to putting a cut with the same "size" on the more complicated random variable L(x) = p(x|mu0)/p(x|mu1). When the PDFS are nicely behaved or more generally *monotonic* the probability statement about a rejection test on the variate L(x) translates into a a statement about a rejection test on the variate (x) simpliciter. But in the case of this "nice" Gaussian mixture I discover that for mu1 sufficiently close to mu0 (and certain combinations of weights and standard deviations) that the likelihood ratio L(X) is *not* monotonic and so I am suddenly faced with an unexpected perplexity: it seems (to the eye anyway) that there's only one way set up a right-tailed rejection test for such a pair of simple hypotheses: and yet the Neyman Pearson argument seems to say that making that cut using the PDF of L(x) and making that cut using p0(x|mu0) itself will not yield the same "x" --- for the same test size. Can you see the resolution of this (pseudo)-conundrum?

I replied: Yes, I can see how this would happen. Whether Neyman-Pearson or Bayes, if you believe the model, the relevant information is the likelihood ratio, which I can well believe in this example is not a monotonically increasing then decreasing function of x. That’s just the way it is! It doesn’t seem like a paradox to me, as there’s no theoretical result that would imply that the ratio of two unimodal functions is itself unimodal.

Bloom responded:

I finally was able to see what is obvious. That indeed there are many alternative “rejection regions of the same size” and if the PDF of the “alternative” is bumpy (as in this example) or more generally if the likelihood ratio is not monotone (and this is *not* “easy to see” for ratios of “simple” Gaussian mixtures all of whose kernels have common mean) then indeed the best (most powerful) test is not necessarily the upper tail rejection test. See my badly drawn diagram. This by the way can be filed under your topic of how the Gaussianity ansatz sufficiently well-learned can really impede insights that would otherwise be patently obvious (to the unlearned).

Article URL: https://www.ams.org/notices/201010/rtx101001303p.pdf

Comments URL: https://news.ycombinator.com/item?id=40890847

Points: 179

# Comments: 143

Kiran Gauthier writes:

After attending your talk at the University of Minnesota, I wanted to ask a follow up regarding the structure of hierarchical / multilevel models but we ran out of time. Do you have any insight on the thought that probabilistic programming languages are so flexible, and the Bayesian inference algorithms so fast, that there is a balance to be struck between “simple” hierarchical models and more “complex” hierarchical models that augment the simple frameworks with more modeled interactions when analyzing real data?

I think that a real benefit of the Bayesian paradigm is that (in theory) if the data doesn’t converge my uncertainty in a parameter, then the inference engine should return my prior (or something close to it). Does this happen in reality? I know you’ve written about canary variables before as an indication of model misspecification which I think is an awesome idea, I’m just wondering how to strike that balance between a simple / approximate model, and a more complicated model given that the true generative process is unknown, and noisy data with bad models can lead good inference engines astray.

My reply: I think complex models are better. As Radford Neal put it so memorably, nearly thirty years ago,

Sometimes a simple model will outperform a more complex model . . . Nevertheless, I believe that deliberately limiting the complexity of the model is not fruitful when the problem is evidently complex. Instead, if a simple model is found that outperforms some particular complex model, the appropriate response is to define a different complex model that captures whatever aspect of the problem led to the simple model performing well.

That said, I don’t recommend fitting the complex model on its own. Rather, I recommend building up to it from something simpler. This building-up occurs on two time scales:

1. When working on your particular problem, start with simple comparisons and then fit more and more complicated models until you have what you want.

2. Taking the long view, as our understanding of statistics progresses, we can understand more complicated models and fit them routinely. This is kind of the converse of the idea that statistical analysis recapitulates the development of statistical methods.

Summary

p‐Values are commonly transformed to lower bounds on Bayes factors, so‐called minimum Bayes factors. For the linear model, a sample‐size adjusted minimum Bayes factor over the class of g‐priors on the regression coefficients has recently been proposed (Held & Ott, The American Statistician 70(4), 335–341, 2016). Here, we extend this methodology to a logistic regression to obtain a sample‐size adjusted minimum Bayes factor for 2 × 2 contingency tables. We then study the relationship between this minimum Bayes factor and two‐sided p‐values from Fisher's exact test, as well as less conservative alternatives, with a novel parametric regression approach. It turns out that for all p‐values considered, the maximal evidence against the point null hypothesis is inversely related to the sample size. The same qualitative relationship is observed for minimum Bayes factors over the more general class of symmetric prior distributions. For the p‐values from Fisher's exact test, the minimum Bayes factors do on average not tend to the large‐sample bound as the sample size becomes large, but for the less conservative alternatives, the large‐sample behaviour is as expected.