ToyCalculator

class pyhf.infer.calculators.ToyCalculator(data, pdf, init_pars=None, par_bounds=None, fixed_params=None, test_stat='qtilde', ntoys=2000, track_progress=True)[source]

Bases: object

The Toy-based Calculator.

__init__(data, pdf, init_pars=None, par_bounds=None, fixed_params=None, test_stat='qtilde', ntoys=2000, track_progress=True)[source]

Toy-based Calculator.

Parameters

  • data (tensor) – The observed data.

  • pdf (Model) – The statistical model adhering to the schema model.json.

  • init_pars (tensor of float) – The starting values of the model parameters for minimization.

  • par_bounds (tensor) – The extrema of values the model parameters are allowed to reach in the fit. The shape should be (n, 2) for n model parameters.

  • fixed_params (tensor of bool) – The flag to set a parameter constant to its starting value during minimization.

  • test_stat (str) –

    The test statistic to use as a numerical summary of the data: 'qtilde', 'q', or 'q0'.

    • 'qtilde': (default) performs the calculation using the alternative test statistic, \(\tilde{q}_{\mu}\), as defined under the Wald approximation in Equation (62) of [1007.1727] (qmu_tilde()).

    • 'q': performs the calculation using the test statistic \(q_{\mu}\) (qmu()).

    • 'q0': performs the calculation using the discovery test statistic \(q_{0}\) (q0()).

  • ntoys (int) – Number of toys to use (how many times to sample the underlying distributions).

  • track_progress (bool) – Whether to display the tqdm progress bar or not (outputs to stderr).

Returns

The calculator for toy-based quantities.

Return type

ToyCalculator

Methods

distributions(poi_test, track_progress=None)[source]

Probability distributions of the test statistic value under the signal + background and background-only hypotheses.

These distributions are produced by generating pseudo-data (“toys”) with the nuisance parameters set to their conditional maximum likelihood estimators at the corresponding value of the parameter of interest for each hypothesis, following the joint recommendations of the ATLAS and CMS experiments in Procedure for the LHC Higgs boson search combination in Summer 2011.

Example

>>> import pyhf
>>> import numpy.random as random
>>> random.seed(0)
>>> pyhf.set_backend("numpy")
>>> model = pyhf.simplemodels.uncorrelated_background(
...     signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
... )
>>> observations = [51, 48]
>>> data = observations + model.config.auxdata
>>> mu_test = 1.0
>>> toy_calculator = pyhf.infer.calculators.ToyCalculator(
...     data, model, ntoys=100, track_progress=False
... )
>>> sig_plus_bkg_dist, bkg_dist = toy_calculator.distributions(mu_test)
>>> sig_plus_bkg_dist.pvalue(mu_test), bkg_dist.pvalue(mu_test)
(array(0.14), array(0.79))
Parameters

  • poi_test (float or tensor) – The value for the parameter of interest.

  • track_progress (bool) – Whether to display the tqdm progress bar or not (outputs to stderr)

Returns

The distributions under the hypotheses.

Return type

Tuple (EmpiricalDistribution)

expected_pvalues(sig_plus_bkg_distribution, bkg_only_distribution)[source]

Calculate the \(\mathrm{CL}_{s}\) values corresponding to the median significance of variations of the signal strength from the background only hypothesis \(\left(\mu=0\right)\) at \((-2,-1,0,1,2)\sigma\).

Example

>>> import pyhf
>>> import numpy.random as random
>>> random.seed(0)
>>> pyhf.set_backend("numpy")
>>> model = pyhf.simplemodels.uncorrelated_background(
...     signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
... )
>>> observations = [51, 48]
>>> data = observations + model.config.auxdata
>>> mu_test = 1.0
>>> toy_calculator = pyhf.infer.calculators.ToyCalculator(
...     data, model, ntoys=100, track_progress=False
... )
>>> sig_plus_bkg_dist, bkg_dist = toy_calculator.distributions(mu_test)
>>> CLsb_exp_band, CLb_exp_band, CLs_exp_band = toy_calculator.expected_pvalues(sig_plus_bkg_dist, bkg_dist)
>>> CLs_exp_band
[array(0.), array(0.), array(0.08403955), array(0.21892596), array(0.86072977)]
Parameters

  • sig_plus_bkg_distribution (EmpiricalDistribution) – The distribution for the signal + background hypothesis.

  • bkg_only_distribution (EmpiricalDistribution) – The distribution for the background-only hypothesis.

Returns

The \(p\)-values for the test statistic corresponding to the \(\mathrm{CL}_{s+b}\), \(\mathrm{CL}_{b}\), and \(\mathrm{CL}_{s}\).

Return type

Tuple (tensor)

pvalues(teststat, sig_plus_bkg_distribution, bkg_only_distribution)[source]

Calculate the \(p\)-values for the observed test statistic under the signal + background and background-only model hypotheses.

Example

>>> import pyhf
>>> import numpy.random as random
>>> random.seed(0)
>>> pyhf.set_backend("numpy")
>>> model = pyhf.simplemodels.uncorrelated_background(
...     signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
... )
>>> observations = [51, 48]
>>> data = observations + model.config.auxdata
>>> mu_test = 1.0
>>> toy_calculator = pyhf.infer.calculators.ToyCalculator(
...     data, model, ntoys=100, track_progress=False
... )
>>> q_tilde = toy_calculator.teststatistic(mu_test)
>>> sig_plus_bkg_dist, bkg_dist = toy_calculator.distributions(mu_test)
>>> CLsb, CLb, CLs = toy_calculator.pvalues(q_tilde, sig_plus_bkg_dist, bkg_dist)
>>> CLsb, CLb, CLs
(array(0.03), array(0.37), array(0.08108108))
Parameters

  • teststat (tensor) – The test statistic.

  • sig_plus_bkg_distribution (EmpiricalDistribution) – The distribution for the signal + background hypothesis.

  • bkg_only_distribution (EmpiricalDistribution) – The distribution for the background-only hypothesis.

Returns

The \(p\)-values for the test statistic corresponding to the \(\mathrm{CL}_{s+b}\), \(\mathrm{CL}_{b}\), and \(\mathrm{CL}_{s}\).

Return type

Tuple (tensor)

teststatistic(poi_test)[source]

Compute the test statistic for the observed data under the studied model.

Example

>>> import pyhf
>>> import numpy.random as random
>>> random.seed(0)
>>> pyhf.set_backend("numpy")
>>> model = pyhf.simplemodels.uncorrelated_background(
...     signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
... )
>>> observations = [51, 48]
>>> data = observations + model.config.auxdata
>>> mu_test = 1.0
>>> toy_calculator = pyhf.infer.calculators.ToyCalculator(
...     data, model, ntoys=100, track_progress=False
... )
>>> toy_calculator.teststatistic(mu_test)
array(3.93824492)
Parameters

poi_test (float or tensor) – The value for the parameter of interest.

Returns

The value of the test statistic.

Return type

Tensor