Given a parameter or parameters fit to data according to a model:
Multi-dimensional confidence region:
The endpoints of a confidence interval (or the boundaries of a region) are determined by data, and therefore are random variables.
The p.d.f.s of the data in the model hypothesis together with the procedure used to choose the region determine the coverage level.
In general, confidence regions for parameter(s) are defined by
where is the confidence level. [Larson]
An okay procedure for can have a CL that depends on the true values of the unknown parameter(s) of the model, but still meets the inequality above. A very clever procedure can have a CL that is equal to regardless of the true model parameter(s). One such procedure is due to Neyman, described in [PDG-Stat].
There are shortcuts for gaussian statistics, and some texts only describe those, but I'll keep it more general.
Consider an estimator for some constant . Suppose our model tells us that the is a gaussian random variable with mean and standard deviation .
What is the difference between the rms of the estimator and the region with 67% coverage?
(Draw picture.)
The following intervals could all have the same coverage and contain the best fit:
(Following section 32.3.2.1 in [PDG-Stat].)
Find intervals for each value of the parameter such that
Here is the p.d.f. of the estimator .
and should depend on monotonically. The functions are invertible: implies .
Then (see figure)
Suppose we toss a coin times. The coin may or may not be fair. We have an unknown probability of the coin coming up heads, tails.
The maxmimum likelihood estimator for is
where is the number of times the coin came up heads.
The probability distribution of is known:
We can find the sum
Use that to define the upper limit according to the Neyman procedure. (The lower limit of the interval is set to 0.)
(...picture...)
We can find the sum
Use that to define the lower limit according to the Neyman procedure. (The lower limit of the interval is set to N.)
(...picture...)
Suppose our procedure (perhaps not consiously decided) were to report a "95% CL" upper limit for if turned out very small, and a "95% CL" lower limit for if the is near 1.
What's wrong with that? Suppose the coin is fair, and really.
10% of the time, we make report implying that the fair coin is unfair "with 95% CL".
One common approach: use and to define the intervals.
The only problem is that sometimes this isn't possible. E.g., if we get tails all times, , the probability at one end is constrained. Then we have to adjust somehow,
Very similar to the two-sided limit, except we pick the edges of the region to have a given log-likelihood. If the parameter space has a boundary, no problem, just adjust the log-likelihood level to encompass enough space.
In general, the p.d.f. of the log-likelihood is generated via MC for each parameter.
Read sections 32.3.2.1 and 32.3.2.2 in [PDG-Stat].
Build the 90%-CL and 99%-CL confidence regions for the same exponential + background of the assignment from class 11 (aka class 0x0B), with the restriction that the background parameter must be in the range and the mean must be positive.
[KamLAND2008] | KamLAND Collaboration, "Precision Measurement of Neutrino Oscillation Parameters with KamLAND", Phys.Rev.Lett.100:221803,2008; arXiv:0801.4589v3 [hep-ex]. |
[DZero2010] | D0 Collaboration, "Evidence for an anomalous like-sign dimuon charge asymmetry", Submitted to Phys. Rev. D, 2010; Fermilab-Pub-10/114-E; arXiv:1005.2757v1 [hep-ex]. |
[PDG-Stat] | "Statistics", G. Cowan, in Review of Particle Physics, C. Amsler et al., PL B667, 1 (2008) and 2009 partial update for the 2010 edition ( http://pdg.lbl.gov/2009/reviews/rpp2009-rev-statistics.pdf ). |