Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

split into two files for binned and unbinned #70

Draft
wants to merge 71 commits into
base: master
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
4449e46
split into two files for binned and unbinned
eunseongleee Jun 28, 2022
e3c50aa
split unbinned / binned cluster likelihoods
itrharrison Jul 13, 2022
b302208
added test yaml
itrharrison Jul 13, 2022
f19d9c3
merge conflicts
itrharrison Jul 13, 2022
d9722e7
Update clusters.py
borisbolliet Jul 13, 2022
203cb91
Update clusters.py
borisbolliet Jul 13, 2022
293d1b0
merging
borisbolliet Jul 14, 2022
46dadab
Update .gitignore
borisbolliet Jul 14, 2022
d2e3752
works with so files and print n(z) from unbinned lkl
borisbolliet Jul 14, 2022
e6488ac
fix to get the unbinned code run fast
borisbolliet Jul 14, 2022
c56aadb
merging in progreess: adding ccl
borisbolliet Sep 2, 2022
3e9cc12
Update test_binned_lkl_ccl.yaml
borisbolliet Sep 2, 2022
b5a779f
merging done
borisbolliet Sep 3, 2022
2afc946
Create DR5-compare-SOLikeT-mocks-sim-kit_NemoCCL_A10tSZ_DR5White_ACT-…
borisbolliet Sep 3, 2022
c199193
starting modifying unbinned lkl
borisbolliet Sep 3, 2022
47cb0c3
in progress (fails)
borisbolliet Sep 4, 2022
6ef757f
unbinned works with dr5 sims but minor bug with ccl unbinned
borisbolliet Sep 4, 2022
7dea653
wip
borisbolliet Sep 6, 2022
a1db65d
working
borisbolliet Sep 6, 2022
c6a8376
deleting unused files
borisbolliet Sep 6, 2022
d1c0b2c
sync in progress
borisbolliet Sep 6, 2022
305b57a
Update clusters.py
borisbolliet Sep 7, 2022
ff2bde4
tidying a bit
borisbolliet Sep 7, 2022
923fa90
fixed discrepancy between binned and unbinned
borisbolliet Sep 7, 2022
15e158d
added directions in yaml file
borisbolliet Sep 7, 2022
dfc798e
instructions
borisbolliet Sep 7, 2022
c7f3198
commented class_sz
borisbolliet Sep 8, 2022
8025b8e
Merge branch 'master' into dev-clusters-both
borisbolliet Sep 8, 2022
596b8ba
vectorized unbinned lkl
borisbolliet Sep 8, 2022
172e145
binned_lkl_camb
borisbolliet Sep 9, 2022
fa7ac43
merging
borisbolliet Sep 9, 2022
f2531bc
Update clusters.py
borisbolliet Sep 9, 2022
769e83b
Added scatter for injection completeness.
anicola Sep 12, 2022
53f80b7
Adding notebooks.
anicola Sep 12, 2022
23b96cd
Injection-based completeness in terms of observed y.
anicola Sep 13, 2022
d692aa2
Adding notebook for more mock comparisons.
anicola Sep 13, 2022
66074a6
Added scatter to Qfit completeness.
anicola Sep 13, 2022
03b8d87
Adapted config files.
anicola Sep 13, 2022
4f00066
Added mass conversion to unbinned likelihood.
anicola Sep 13, 2022
9bb089c
beginning of speeding up binned likelihood
eunseongleee Sep 21, 2022
9fa268f
removing obsolete options
eunseongleee Sep 22, 2022
78ecf4d
Update .gitignore
eunseongleee Sep 22, 2022
9f0ba06
Delete soliket/clusters/data/act directory
eunseongleee Sep 22, 2022
016b5da
speeding up binned lkl
eunseongleee Sep 28, 2022
a9468c5
correcting the average Q computation in an injection method
eunseongleee Oct 18, 2022
401b1bc
added nemo-sz as a requirement
itrharrison Oct 22, 2022
4494e52
Update test_unbinned_lkl_camb_dr5.yaml
borisbolliet Dec 6, 2022
5d8b0c8
synchronising binned and unbinned lkl and tidying up options regardin…
eunseongleee Dec 15, 2022
6e00212
Merge branch 'dev-clusters-both' of https://github.com/simonsobs/SOLi…
eunseongleee Dec 15, 2022
1b1f537
update pytest for both cluster lkl
eunseongleee Dec 20, 2022
035227e
tidying up
eunseongleee Dec 20, 2022
fb8c8ad
speeding up unbinned lkl (no scatter)
eunseongleee Dec 23, 2022
dc7ab7c
speeding up the unbinned lkl (no scatter)
eunseongleee Jan 4, 2023
6ac62a4
switching the error func in completeness to the step func
eunseongleee Jan 19, 2023
74aa92d
tidying up
eunseongleee Feb 13, 2023
41a45f8
tidying up unbinned lkl
eunseongleee Feb 16, 2023
85ebfab
updating the unbinned likelihood
eunseongleee Mar 20, 2023
0ad184d
Adding functionality to compute gof for Poisson statistics.
anicola Mar 21, 2023
f0b7d29
tidying up
eunseongleee Jun 9, 2023
5790c69
note added for cash statistics
eunseongleee Jun 9, 2023
2462855
porting class_sz
borisbolliet Sep 13, 2023
526d974
class_sz works for binned lkl
borisbolliet Sep 13, 2023
b7814bd
class_sz for unbinned
borisbolliet Sep 13, 2023
540f523
implemented debiasing at theory level in soliket/ccl binned lkl. Many…
borisbolliet Sep 13, 2023
a183bb7
added catalog for class_sz and skyfrac dependence in unbinned lkl
borisbolliet Sep 13, 2023
c1ba3b1
test yaml files
borisbolliet Sep 21, 2023
cb7bd50
push cib x lensing
borisbolliet Mar 12, 2024
1bf904a
provider/theory renaming
May 22, 2024
932e441
added new optimization correction factor to binned and unbinned
nbatta Jul 9, 2024
080c7a2
opt_bias_func checked for binned
eunseongleee Jul 15, 2024
cd3d4dd
version check
eunseongleee Sep 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
working
borisbolliet committed Sep 6, 2022
commit a1db65dd2776f05a0349d6ebab6bf90c7dafc1ab
883 changes: 299 additions & 584 deletions soliket/clusters/clusters.py

Large diffs are not rendered by default.

8 changes: 7 additions & 1 deletion soliket/clusters/input_files/test_unbinned_lkl_camb_dr5.yaml
Original file line number Diff line number Diff line change
@@ -17,12 +17,18 @@ likelihood:
tile_file: 'selFn/tileAreas.txt' # Path to tile file
rms_file: 'selFn/RMSTab.fits' # Path to RMS file

# Y-M relation
YM:
Mpivot: 2.9e14 # Mpivot in Y-M relation in [h^-1 Msun]


# Selection function
selfunc:
SNRcut: 6. # S/N cutoff in number counts
dwnsmpl_bins: 3
save_dwsmpld : True
mode : 'downsample'
average_Q: False # Use average Q function

# theorypred:
# choose_theory: 'camb' # Theory prediction mode, possibilities are camb, class, CCL (CCL is all CCL)
@@ -110,7 +116,7 @@ params:
# sigma8 : 0.81
tenToA0 : 4.35e-5
B0 : 0.08
scatter_sz : 0.2
scatter_sz : 0.0
bias_sz : 1.
m_nu : 0.0
C0 : 0. # doesnt matter
91 changes: 48 additions & 43 deletions soliket/clusters/sz_utils.py
Original file line number Diff line number Diff line change
@@ -40,13 +40,15 @@ def P_Yo(self, rms_bin_index,LgY, M, z, param_vals, Ez_fn, Da_fn):
H0 = param_vals["H0"]

Ma = np.outer(M, np.ones(len(LgY[0, :])))
print('Ma',np.exp(Ma))
# exit(0)

Ytilde, theta0, Qfilt = y0FromLogM500(
np.log10(param_vals["massbias"] * Ma / (H0 / 100.0)),
np.log10(param_vals["bias_sz"] * np.exp(Ma) / (H0 / 100.0)), # not sure about h
z,
self.lkl.allQ[:,rms_bin_index],
self.lkl.tt500,
sigma_int=param_vals["scat"],
sigma_int=param_vals["scatter_sz"],
B0=param_vals["B0"],
H0=param_vals["H0"],
Ez_fn=Ez_fn,
@@ -62,8 +64,8 @@ def P_Yo(self, rms_bin_index,LgY, M, z, param_vals, Ez_fn, Da_fn):

numer = -1.0 * (np.log(Y / Ytilde)) ** 2
ans = (
1.0 / (param_vals["scat"] * np.sqrt(2 * np.pi)) *
np.exp(numer / (2.0 * param_vals["scat"] ** 2))
1.0 / (param_vals["scatter_sz"] * np.sqrt(2 * np.pi)) *
np.exp(numer / (2.0 * param_vals["scatter_sz"] ** 2))
)
return ans

@@ -72,8 +74,11 @@ def P_Yo_vec(self, rms_index, LgY, M, z, param_vals, Ez_fn, Da_fn):
# Ma = np.outer(M, np.ones(len(LgY[0, :])))
# print('M',np.exp(M))
# exit(0)
# self._get_y0(mass, z, mass_500c, use_Q=True, **params_values_dict):


Ytilde, theta0, Qfilt = y0FromLogM500(
np.log10(param_vals["bias_sz"] * np.exp(M) / (H0 / 100.0)), # TBD: check h units here.
np.log10(param_vals["bias_sz"] * np.exp(M) / (H0 / 100.0)), # TBD: check h units here.
z,
self.lkl.allQ[:,rms_index],
self.lkl.tt500,
@@ -99,7 +104,7 @@ def P_Yo_vec(self, rms_index, LgY, M, z, param_vals, Ez_fn, Da_fn):
1.0 / (param_vals["scatter_sz"] * np.sqrt(2 * np.pi)) *
np.exp(numer / (2.0 * param_vals["scatter_sz"] ** 2))
)
if param_vals["scatter_sz"] == 0:
if param_vals["scatter_sz"] == 0: # not sure what to do yet...
ans[:,:,:] = 1
print('ans',ans)
print(np.shape(ans))
@@ -137,7 +142,7 @@ def P_of_gt_SN(self, rms_index, LgY, MM, zz, Ynoise, param_vals, Ez_fn, Da_fn):
# sig_thresh = np.transpose(sig_thresh, (0, 2, 1))
ans = np.trapz(P_Y * sig_thresh, x=LgY, axis=2) #* np.log(10)
print('ans',ans)
exit(0)
# exit(0)
return ans

def PfuncY(self, rms_index, YNoise, M, z_arr, param_vals, Ez_fn, Da_fn):
@@ -179,42 +184,42 @@ def Pfunc_per(self, rms_bin_index,MM, zz, Y_c, Y_err, param_vals, Ez_fn, Da_fn):
ans = np.trapz(P_Y * P_Y_sig, LgY, np.diff(LgY), axis=1)

return ans

def Pfunc_per_parallel(self, Marr, zarr, Y_c, Y_err, param_vals, Ez_fn, Da_fn):
# LgY = self.LgY
# LgYa = np.outer(np.ones(Marr.shape[0]), LgY)

# LgYa = np.outer(np.ones([Marr.shape[0], Marr.shape[1]]), LgY)
# LgYa2 = np.reshape(LgYa, (Marr.shape[0], Marr.shape[1], len(LgY)))

# Yc_arr = np.outer(np.ones(Marr.shape[0]), Y_c)
# Yerr_arr = np.outer(np.ones(Marr.shape[0]), Y_err)

# Yc_arr = np.repeat(Yc_arr[:, :, np.newaxis], len(LgY), axis=2)
# Yerr_arr = np.repeat(Yerr_arr[:, :, np.newaxis], len(LgY), axis=2)

# P_Y_sig = self.Y_prob(Yc_arr, LgYa2, Yerr_arr)
# P_Y = np.nan_to_num(self.P_Yo(LgYa2, Marr, zarr, param_vals, Ez_fn))

P_Y_sig = self.Y_prob(Y_c, self.LgY, Y_err)
P_Y = np.nan_to_num(self.P_Yo(rms_bin_index,self.LgY, Marr, zarr, param_vals, Ez_fn, Da_fn))

ans = np.trapz(P_Y * P_Y_sig, x=self.LgY, axis=2)

return ans

def Pfunc_per_zarr(self, MM, z_c, Y_c, Y_err, int_HMF, param_vals):
LgY = self.LgY

# old was z_arr
# P_func = np.outer(MM, np.zeros([len(z_arr)]))
# M_arr = np.outer(MM, np.ones([len(z_arr)]))
# M200 = np.outer(MM, np.zeros([len(z_arr)]))
# zarr = np.outer(np.ones([len(M)]), z_arr)

P_func = self.P_of_Y_per(LgY, MM, z_c, Y_c, Y_err, param_vals)

return P_func
# more parallelized implementation... maybe not needed...
# def Pfunc_per_parallel(self, Marr, zarr, Y_c, Y_err, param_vals, Ez_fn, Da_fn):
# # LgY = self.LgY
# # LgYa = np.outer(np.ones(Marr.shape[0]), LgY)
#
# # LgYa = np.outer(np.ones([Marr.shape[0], Marr.shape[1]]), LgY)
# # LgYa2 = np.reshape(LgYa, (Marr.shape[0], Marr.shape[1], len(LgY)))
#
# # Yc_arr = np.outer(np.ones(Marr.shape[0]), Y_c)
# # Yerr_arr = np.outer(np.ones(Marr.shape[0]), Y_err)
#
# # Yc_arr = np.repeat(Yc_arr[:, :, np.newaxis], len(LgY), axis=2)
# # Yerr_arr = np.repeat(Yerr_arr[:, :, np.newaxis], len(LgY), axis=2)
#
# # P_Y_sig = self.Y_prob(Yc_arr, LgYa2, Yerr_arr)
# # P_Y = np.nan_to_num(self.P_Yo(LgYa2, Marr, zarr, param_vals, Ez_fn))
#
# P_Y_sig = self.Y_prob(Y_c, self.LgY, Y_err)
# P_Y = np.nan_to_num(self.P_Yo(rms_bin_index,self.LgY, Marr, zarr, param_vals, Ez_fn, Da_fn))
#
# ans = np.trapz(P_Y * P_Y_sig, x=self.LgY, axis=2)
#
# return ans
#
# def Pfunc_per_zarr(self, MM, z_c, Y_c, Y_err, int_HMF, param_vals):
# LgY = self.LgY
#
# # old was z_arr
# # P_func = np.outer(MM, np.zeros([len(z_arr)]))
# # M_arr = np.outer(MM, np.ones([len(z_arr)]))
# # M200 = np.outer(MM, np.zeros([len(z_arr)]))
# # zarr = np.outer(np.ones([len(M)]), z_arr)
#
# P_func = self.P_of_Y_per(LgY, MM, z_c, Y_c, Y_err, param_vals)
#
# return P_func


###
9 changes: 2 additions & 7 deletions soliket/poisson.py
Original file line number Diff line number Diff line change
@@ -11,7 +11,7 @@ class PoissonLikelihood(Likelihood):
columns = None

def initialize(self):
print('initializing poisson')
# print('initializing poisson')
catalog = self._get_catalog()
if self.columns is None:
self.columns = catalog.columns
@@ -36,12 +36,7 @@ def _get_n_expected(self, **kwargs):

def logp(self, **params_values):
pk_intp = self.theory.get_Pk_interpolator(("delta_nonu", "delta_nonu"), nonlinear=False)
print('got pk_intp')
rate_fn = self._get_rate_fn(pk_intp,**params_values)
print('rate_fn',rate_fn)
n_expected = self._get_n_expected(pk_intp,**params_values)
print('n_expected:',n_expected)
# exit(0)
# nz_expected = self._get_nz_expected(**params_values)
# print('nz_expected:',nz_expected)

return self.data.loglike(rate_fn, n_expected)
2 changes: 1 addition & 1 deletion soliket/poisson_data.py
Original file line number Diff line number Diff line change
@@ -66,7 +66,7 @@ def loglike(self, rate_fn, n_expected, broadcastable=False):
[
rate_fn(**{c: self.catalog[c].values[i] for c in self.columns})
# for i in range(len(self))
for i in range(100) ## quick fix to make the code run fast
for i in range(10) ## quick fix to make the code run fast
]
)