adding a to_pandas() method to lmfit.Params

[mate377]

Hi,
the representation of the params object seems to me identical to a pandas DataFrame, however the params object doesn't have the same flexibility as a DataFrame.
With the params in a DataFrame one could for example:

• easily slice for function parameters, or for properties (value, standard error, ...)
• combine fit results with other data
• do math with the results as a whole

All this would be accessible from a method "to_pandas()" or "to_dataframe" in the same way as other libraries like xarray. I think it would simplify the work of those not stopping at the "fit report" but going ahead with more analysis!

[newville]

@mate377
I will admit to being somewhat "meh" with pandas and nearly allergic to using spreadsheets as computational devices. I recognize that I am not in the majority with this view. I have never been in a situation where I thought "what I need here is a Pandas DataFrame". Just to give some assurance that I'm not a complete lunatic, CSV is a fine way to store and transmit data.

And I get that you want to convert a Parameters object to (and perhaps from?) a pandas Dataframe, but I don't (yet?) know why anyone else wants that or why that should be done within lmfit.

the representation of the params object seems to me identical to a pandas DataFrame,

How do they seem identical to you? A DataFrame is a pretty generic table of values, typically with uniform datatypes for columns and a number of rows for "instances". Apparently, a lot of people like to use these.

A lmfit.Parameters object contains a number of individual Parameter objects, each of which has a name and value, and a handful of attributes, with a variety of data types (str, float, bool, dict), though some of these may not be set to a valid or clear value at all times. An individual Parameter must be mutable. The set of individual Parameter objects in a Parameters object can have an arbitrary order, but that order must be preserved throughout the lifetime of the Parameters object. A Parameters object also contains an instance of an _asteval.Interpreter that is used to evaluate constraint expressions.

One can certainly serialize a Parameters object (we do that using JSON), but I'm not certain how serialization to a DataFrame would work. For completeness, you need not only the attributes of the individual Parameter objects, but the set of user-defined symbols in the asteval Interpreter.

however the params object doesn't have the same flexibility as a DataFrame.

I don't know that this difference is important. How would having the same flexibility as a DataFrame help a Parameters object? What replaces the flexibility of being an inspectable Python object?

With the params in a DataFrame one could for example:

• easily slice for function parameters, or for properties (value, standard error, ...)

Perhaps something like [(p.name, p.value, p.stderr) for p in params.values()] would be useful?

• combine fit results with other data

I'm not sure I see how that would work.

• do math with the results as a whole

What math would that be?

All this would be accessible from a method "to_pandas()" or "to_dataframe" in the same way as other libraries like xarray. I think it would simplify the work of those not stopping at the "fit report" but going ahead with more analysis!

Well, you have a structured Parameters object with a set of accessible attributes.

I can certainly believe that "dump to" a DataFrame could work, but less sure about "load from" a DataFrame in a way that is reliable.

[mate377]

@newville
thank you for your response, and sorry for the delay. I want to address some of your points:

How do they seem identical to you?

It's just the representation, what's displayed, I know that underneath they are different objects!

One can certainly serialize a Parameters object (we do that using JSON), but I'm not certain how serialization to a DataFrame would work

I don't know the details of what's inside the parameter object, but it seems something will be left over in the process. I don't think it is an issue, they don't need to be equivalent, in the sense that we can go from params to DataFrame and viceversa loosing information in the process. For example, I can imagine the constraint on the parameters will be loss going from params to DataFrame and back to params.

Perhaps something like [(p.name, p.value, p.stderr) for p in params.values()] would be useful?

Of course. Here pandas would make it just a bit more convenient

I'm not sure I see how that would work.
What math would that be?

@Tillsten made an example describing his workflow, which incidentally is very similar to my use case!

[Anselmoo]

One can certainly serialize a Parameters object (we do that using JSON), but I'm not certain how serialization to a DataFrame would work.

It would be very nice, if the lmfit printout will get a change for a .json export. Especially, the goodness of fit can profit a lot from JSON or toml export.

[newville]

@Anselmoo I'm afraid I do not understand either of these sentences. Are you saying we should change the printout based on some JSON setting? I am completely lost by "the goodness of fit can profit a lot from JSON or toml export.". How would the goodness of fit profit from any output?

Are you trying to say that you want lmfit to be able to export some values as JSON?

[Anselmoo]

Are you trying to say that you want lmfit to be able to export some values as JSON?

Sorry @newville yes you are right. An JSON export of the statistic would be nice.

Is now clear? Sorry for inconvenience

[newville]

@Anselmoo well, this is sort of an out-of-the-blue comment in a conversation from 3 months ago labeled "a to_pandas() method to lmfit.Params". That would not include goodness-of-fit statistics after a fit, and would not be JSON. ;).

There already is a ModelResult.dumps(): that is in JSON, and it does include fit statistics. Is that insufficient?

And that is sort of keeping in what you will have read elsewhere in the conversation: It is not really clear what is wanted that should be added and maintained within lmfit.

[Tillsten]

I think the combination of lmfit with pandas can be rather useful. A common workflow of mine is where I have serveral spectra and fit them all with the same model. Then I put the results into a dataframe to compare the different fit results, since it is much more handy than just using dict. This way, it is easy to add depenend columns or additional info about the different spectra. Also plotting is simplyfies.

Here a reallife example i use to put the data into a dataframe:

# fit_dict is a dict containing the ModelResults of different fits. The key is the sample name.

fit_vals = defaultdict(list)
for r in fit_dict:
   fit_vals['name'].append(r)
   for p in fit_dict[r].params:
       fit_vals[p].append(fit_dict[r].params[p].value)
   for p in fit_dict[r].params:
       if fit_dict[r].ci_out is not None and p in fit_dict[r].ci_out:
           fit_vals[p + '_err'].append((fit_dict[r].ci_out[p][2][1] -
                                       fit_dict[r].ci_out[p][0][1])/2)
           fit_vals[p + '_err_u'].append(fit_dict[r].ci_out[p][2][1])
           fit_vals[p + '_err_l'].append(fit_dict[r].ci_out[p][0][1])
           fit_vals[p + '_err_std'].append(fit_dict[r].ci_out[p][0][1])
       else:
           fit_vals[p + '_err'].append(fit_dict[r].params[p].stderr)
for p in fit_vals:
   fit_vals[p] = np.array(fit_vals[p])
df = pd.DataFrame(fit_vals)

I am +0.5 to adding such a function to lmfit, since I think it rather useful in a large number of usecases. On the other hand, it simple to roll your own. I am definitly for adding an example, maybe I can find some time.

[newville]

@mate377 @Tillsten I would definitely be ok with adding methods to convert Parameters and/or ModelResult to a tabular format: A table is a fine way to visualize and easily compare results from multiple fits. I'm a little less sure about having a single Parameters or single ModelResult as a table - what sort of work would be done on that?

Is CSV or Pandas DataFrame a better choice for a "tabular" format?

[mate377]

I think pandas is the perfect choice for a "tabular" format. For a single parameter, I would say that the same applies, i.e. that it can still be represented as a DataFrame.

[newville]

@mate377
Wait,

the representation of the params object seems to me identical to a pandas DataFrame,

How do they seem identical to you?

It's just the representation, what's displayed, I know that underneath they are different objects!

OK, I generally associate "representation" with something more than "what is displayed". Because, like displayed where and how?

I'm not sure I see how that would work.
What math would that be?

@Tillsten made an example describing his workflow, which incidentally is very similar to my use case!

Well, that seems like a fine thing to do, but I don't really see any math there. And Till's code added in other results, say from conf_intervals. And the "combine with other results" was definitely specific to that selected structure of results from different fits.

This is kind of the heart of my concern here. You can certainly convert Parameters to DataFrames -- that's a fine choice, though certainly not the only one. Anyway, it is not very complicated, and might look something like:

import pandas as pd
def params_to_dataframe(params):
    "convert parametes to a dataframe"
    par_attrs = ('name', 'value', 'stderr', 'vary', 'expr', 'init_value',
                 'min', 'max', 'brute_step', 'correl')
    dat = {attr: [] for attr  in par_attrs}
    for par in params.values():
        for attr in par_attrs:
            dat[attr].append(getattr(par, attr, None))
    return pd.DataFrame(dat, columns=par_attrs)

I think you are saying that you want us to maintain that as a Parameters method. I am not 100% opposed to that, but I'm certain it is better for us to do that or just to expect that this happens in user code. Like, that is not what Till did ;).

You might have a list (or a dict or a sqlite database) of fit results using
case A: 50 fits to 10 datasets, with 5 different models or starting values per dataset.
case B: 50 fits to 50 datasets, all using the same model.

And you want to compare the various fits - perfectly reasonable . But what do want to display and sort on? Maybe you just want parameter name, value, stderr. Maybe you also need dataset name. Maybe you need a name or index for the different models and/or starting values. Do you always sneed the starting values? Do you care about brute_step? The discussion here is only about Parameters, but wouldn't you also want fit statistics from ModelResult or MinimizerResult? I know I would.

The code above is basically what Parameters.to_dataframe() would do (we would add tests for "is pandas installed?"): it could not do a lot more. Is that something people would actually use? Maybe it is just better to document "this is how you convert Parameters into a pandas DataFrame", and let the users do that on their own.

As it turns out, we're working to release version 1.1.0 very soon (like Monday or Tuesday). The above code could go in, but we need to hear a resounding yes very, very soon (approximately "right now") for that to happen.

[mate377]

@newville

And you want to compare the various fits - perfectly reasonable . But what do want to display and sort on? Maybe you just want parameter name, value, stderr. Maybe you also need dataset name. Maybe you need a name or index for the different models and/or starting values. Do you always sneed the starting values? Do you care about brute_step?

I think everything could actually go inside, since after is just an easy filtering with pandas

The discussion here is only about Parameters, but wouldn't you also want fit statistics from ModelResult or MinimizerResult? I know I would.

Ah yes, definitely yes. I'm sorry, I just started using the library quite recently, so I don't know all the features yet, but in principle I think that all the "tabular" results would benefit from a .to_pandas() method!

In the end, my biased answer is: yes, the method should be there.

My less biased answer is that, interoperability between python packages is very important, pandas has become a standard to work with "tabular" data, and a method to convert to pandas is a quite powerful link in the ecosystem.

[reneeotten]

@newville @mate377 this is something worth thinking about - even though I am not really sold on the idea. For sure it should not be pushed quickly without thinking it through so that it possibly could go into the next release. Our plan is (or should be) to release minor point releases more quickly so this is a perfect thing to go in later if we would decide to do so.

[newville]

Yep, I agree that this would be potentially for next release and benefit from design. I'm of the opinion that this is easy enough to do in downstream code. And, really what is wanted is a way to store, manage, inspect, and sort multiple fits.