Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

modelParameters is not working - interpolation issue #135

Open
daniellabardalezgagliuffi opened this issue Jul 23, 2018 · 25 comments
Open

modelParameters is not working - interpolation issue #135

daniellabardalezgagliuffi opened this issue Jul 23, 2018 · 25 comments
Labels
not verified

Comments

@daniellabardalezgagliuffi
Copy link
Collaborator

Hey Adam,

I think we can run modelParameters way faster if we use map. The problem comes when I try to unpack the map object, as seen below.

nbin = int(nsamp*0.20) #200,000 rows
t0 = time.time()
bind = np.sort(np.random.choice(np.arange(len(simdf)),size=int(nbin)))
binpropsdf = pd.DataFrame(index=bind,columns=['mass1','mass2','q','age','teff1','teff2','spt2','spt1',
'MJ1','MJ2','MH1','MH2','MK1','MK2'])
binpropsdf['age'] = simulate.simulateAges(distribution='uniform',age_range=[0.5,10],num=nbin)
binpropsdf['q'] = simulate.simulateMassRatios(distribution='allen',num=nbin)
binpropsdf['teff1'] = simdf.loc[bind,'true_sptn'].map(lambda x: empirical.typeToTeff(x,'dupuy')[0])
binpropsdf['mass1'] = map(lambda x,y: evolve.modelParameters('baraffe03',teff=x, age=y), binpropsdf['teff1'],binpropsdf['age'])
print(time.time() - t0, ' seconds')
21.901501893997192 seconds

binpropsdf['mass1'].head()
4 <map object at 0x1c14e94c88>
10 <map object at 0x1c14e94c88>
30 <map object at 0x1c14e94c88>
35 <map object at 0x1c14e94c88>
40 <map object at 0x1c14e94c88>
Name: mass1, dtype: object

This returns a map object for mass1. the only way to unpack this is to put a list() around it, and that's where the problem comes:

binpropsdf['mass1'] = list(map(lambda x,y: evolve.modelParameters('baraffe03',teff=x, age=y), binpropsdf['teff1'],binpropsdf['age']))

ValueError Traceback (most recent call last)
in ()
11 binpropsdf['q'] = simulate.simulateMassRatios(distribution='allen',num=nbin)
12 binpropsdf['teff1'] = simdf.loc[bind,'true_sptn'].map(lambda x: empirical.typeToTeff(x,'dupuy')[0])
---> 13 binpropsdf['mass1'] = list(map(lambda x,y: evolve.modelParameters('baraffe03',teff=x, age=y), binpropsdf['teff1'],binpropsdf['age']))
14 print(time.time() - t0)

in (x, y)
11 binpropsdf['q'] = simulate.simulateMassRatios(distribution='allen',num=nbin)
12 binpropsdf['teff1'] = simdf.loc[bind,'true_sptn'].map(lambda x: empirical.typeToTeff(x,'dupuy')[0])
---> 13 binpropsdf['mass1'] = list(map(lambda x,y: evolve.modelParameters('baraffe03',teff=x, age=y), binpropsdf['teff1'],binpropsdf['age']))
14 print(time.time() - t0)

~/Python/splat/splat/evolve.py in modelParameters(*model, **kwargs)
631 for p in pkeys:
632 inparams[p] = mkwargs[p][i]
--> 633 par = _modelParametersSingle(model,**inparams)
634 for p in list(EVOLUTIONARY_MODEL_PARAMETERS.keys()):
635 outparams[p].append(par[p])

~/Python/splat/splat/evolve.py in _modelParametersSingle(*args, **kwargs)
424 vals = [lmodel[P[0][0]][ai][i],lmodel[P[0][0]][ai+1][aj]]
425 f = interp1d(lmodel['age'][ai:ai+2],vals)
--> 426 Ge.append(f(numpy.log10(params['age'])))
427 try:
428 f = interp1d(Ge, Ma)

/anaconda/lib/python3.6/site-packages/scipy/interpolate/polyint.py in call(self, x)
77 """
78 x, x_shape = self._prepare_x(x)
---> 79 y = self._evaluate(x)
80 return self._finish_y(y, x_shape)
81

/anaconda/lib/python3.6/site-packages/scipy/interpolate/interpolate.py in _evaluate(self, x_new)
632 y_new = self._call(self, x_new)
633 if not self._extrapolate:
--> 634 below_bounds, above_bounds = self._check_bounds(x_new)
635 if len(y_new) > 0:
636 # Note fill_value must be broadcast up to the proper size

/anaconda/lib/python3.6/site-packages/scipy/interpolate/interpolate.py in _check_bounds(self, x_new)
664 "range.")
665 if self.bounds_error and above_bounds.any():
--> 666 raise ValueError("A value in x_new is above the interpolation "
667 "range.")
668

ValueError: A value in x_new is above the interpolation range.
@daniellabardalezgagliuffi
Copy link
Collaborator Author

A simple example of how map works:

list(map(lambda x,y: x+y,np.arange(3),np.arange(3)))

[0, 2, 4]

@daniellabardalezgagliuffi
Copy link
Collaborator Author

Actually, the problem is in modelParameters altogether... I can't run it in batch without the interpolation issue:
binpropsdf['mass1'] = evolve.modelParameters('baraffe03',teff=binpropsdf['teff1'], age=binpropsdf['age'])

ValueError Traceback (most recent call last)
in ()
10 binpropsdf['q'] = simulate.simulateMassRatios(distribution='allen',num=nsamp)
11 binpropsdf['teff1'] = simdf['true_sptn'].map(lambda x: empirical.typeToTeff(x,'dupuy')[0])
---> 12 binpropsdf['mass1'] = evolve.modelParameters('baraffe03',teff=binpropsdf['teff1'], age=binpropsdf['age'])
13 print(time.time() - t0)

~/Python/splat/splat/evolve.py in modelParameters(*model, **kwargs)
631 for p in pkeys:
632 inparams[p] = mkwargs[p][i]
--> 633 par = _modelParametersSingle(model,**inparams)
634 for p in list(EVOLUTIONARY_MODEL_PARAMETERS.keys()):
635 outparams[p].append(par[p])

~/Python/splat/splat/evolve.py in _modelParametersSingle(*args, **kwargs)
424 vals = [lmodel[P[0][0]][ai][i],lmodel[P[0][0]][ai+1][aj]]
425 f = interp1d(lmodel['age'][ai:ai+2],vals)
--> 426 Ge.append(f(numpy.log10(params['age'])))
427 try:
428 f = interp1d(Ge, Ma)

/anaconda/lib/python3.6/site-packages/scipy/interpolate/polyint.py in call(self, x)
77 """
78 x, x_shape = self._prepare_x(x)
---> 79 y = self._evaluate(x)
80 return self._finish_y(y, x_shape)
81

/anaconda/lib/python3.6/site-packages/scipy/interpolate/interpolate.py in _evaluate(self, x_new)
632 y_new = self._call(self, x_new)
633 if not self._extrapolate:
--> 634 below_bounds, above_bounds = self._check_bounds(x_new)
635 if len(y_new) > 0:
636 # Note fill_value must be broadcast up to the proper size

/anaconda/lib/python3.6/site-packages/scipy/interpolate/interpolate.py in _check_bounds(self, x_new)
664 "range.")
665 if self.bounds_error and above_bounds.any():
--> 666 raise ValueError("A value in x_new is above the interpolation "
667 "range.")
668

ValueError: A value in x_new is above the interpolation range.

@daniellabardalezgagliuffi
Copy link
Collaborator Author

Yup, I get this same error on the very first set of values:

evolve.modelParameters('baraffe03',teff=2419.93, age=6.56767)

@daniellabardalezgagliuffi daniellabardalezgagliuffi changed the title modelParameters could work way faster using map but there is an interpolation issue modelParameters is not working - interpolation issue Jul 23, 2018
@daniellabardalezgagliuffi
Copy link
Collaborator Author

I updated splat and now the error has changed to this for the line in the last comment:

ValueError Traceback (most recent call last)
in ()
----> 1 evolve.modelParameters('baraffe03',teff=2419.93, age=6.56767)

~/Python/splat/splat/evolve.py in modelParameters(*model, **kwargs)
631 numberValues = len(mkwargs[p])
632
--> 633 # now loop through each parameter set to determine remaining parameters
634 for i in range(numberValues):
635 for p in pkeys:

~/Python/splat/splat/evolve.py in _modelParametersSingle(*args, **kwargs)
408 print('{}: {}'.format(k,params[k]))
409 print(P)
--> 410 raise ValueError('\nProblem with one_param interpolation\n')
411
412 if numpy.log10(params['age']) < numpy.min(lmodel['age']) or \

/anaconda/lib/python3.6/site-packages/numpy/core/fromnumeric.py in amin(a, axis, out, keepdims)
2350
2351 return _methods._amin(a, axis=axis,
-> 2352 out=out, **kwargs)
2353
2354

/anaconda/lib/python3.6/site-packages/numpy/core/_methods.py in _amin(a, axis, out, keepdims)
27
28 def _amin(a, axis=None, out=None, keepdims=False):
---> 29 return umr_minimum(a, axis, None, out, keepdims)
30
31 def _sum(a, axis=None, dtype=None, out=None, keepdims=False):

ValueError: zero-size array to reduction operation minimum which has no identity

@aburgasser
Copy link
Owner

hmm, I'm not able to reproduce this last problem, and I'm confused as to why it is throwing an error on a comment line. Have you modified the code? I suggest restarting your kernel and then just trying the last (single parameter) interpolation.

What version of python are you running?

For reference, here is what I get for these parameters:

import splat
import splat.evolve as spev
spev.modelParameters('baraffe03',teff=2419.93, age=6.56767)

{'mass': <Quantity 0.08328793877705343 solMass>,
'age': <Quantity 6.56767 Gyr>,
'temperature': <Quantity 2419.93 K>,
'gravity': <Dex 5.331114546660216 dex(cm / s2)>,
'luminosity': <Dex -3.4904435980894015 dex(solLum)>,
'radius': <Quantity 0.103580618503181 solRad>}

@aburgasser
Copy link
Owner

For the earlier part, I suspect you may have nan's and/or are out of range on some parameters. I could figure out a way to stop the error messages.

@daniellabardalezgagliuffi
Copy link
Collaborator Author

@daniellabardalezgagliuffi
Copy link
Collaborator Author

@daniellabardalezgagliuffi
Copy link
Collaborator Author

@daniellabardalezgagliuffi
Copy link
Collaborator Author

Given these inputs:
modelParameters('saumon',teff=2419.93, age=6.5)

After a lot of unsuccessful debugging, I've isolated the problem to _modelParametersSingle on line 372:
Ge.append(f(numpy.log10(params['age'])))

where the interpolating function f cannot evaluate the x_new given:
params['age'] = 6.5 <-- input
numpy.log10(params['age']) = 0.812913356643 <-- x_new

The interpolating function is defined as:
f = interp1d(lmodel['age'][ai:ai+2],vals)

where:
x = lmodel['age'][ai:ai+2] = [0.77815125038364363, -0.52287874528033762]
y = vals = [2.4608978427565478, 2.8208579894397001]

which are defined from:
vals = [lmodel[P[0][0]][ai][i],lmodel[P[0][0]][ai+1][aj]]
P = [['temperature', 3.3838028035616627]]
ai = numpy.argmin(numpy.abs(adiff))
adiff = [numpy.log10(params['age'])-a for a in lmodel['age']]
aj = numpy.argmin(numpy.abs([a-m for a in lmodel['mass'][ai+1]]))

Now, I'm not 100% sure I understand the contents of lmodel, nor the indices that can be used inside it, but I think the problem may be that the high limit of lmodel for the interpolation, aj, is too low, or rather that the range of possible x's for the interpolation function is too tight. If we're looking for the temperature 3.38 within the lmodel['temperature'] vector, that is defined between the y vals, this temperature will never be found.

@aburgasser
Copy link
Owner

I'll have to spend some time on this; I would suggest not using map until I can replicate (since I can't see them) and resolve these problems. Since the code returns nans if a parameter range is outside the model range, it may be that map is choking on these. Sorry, you'll have to run a large set overnight.

@daniellabardalezgagliuffi
Copy link
Collaborator Author

@aburgasser
Copy link
Owner

As I said, until I can replicate, I don't know what to fix

@aburgasser
Copy link
Owner

what versions of:

astropy
numpy
scipy

are you running?

@daniellabardalezgagliuffi
Copy link
Collaborator Author

@aburgasser
Copy link
Owner

ha ha ha

@daniellabardalezgagliuffi
Copy link
Collaborator Author

@aburgasser
Copy link
Owner

??? what's with the ageism comment ???

@daniellabardalezgagliuffi
Copy link
Collaborator Author

@aburgasser
Copy link
Owner

numpy 1.11.3
scipy 0.18.1
astropy 3.0.3

@daniellabardalezgagliuffi
Copy link
Collaborator Author

@daniellabardalezgagliuffi
Copy link
Collaborator Author

@daniellabardalezgagliuffi
Copy link
Collaborator Author

@aburgasser
Copy link
Owner

just updated scipy & numpy and I can still run this line, so yup, it's not those packages. You're also running the same version of python3, so that's not it. hmmm.

@daniellabardalezgagliuffi
Copy link
Collaborator Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
not verified
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@aburgasser @daniellabardalezgagliuffi