Ideas for Saving Previous Queries

[BradyAJohnston]

I am looking to try and incoroprate webchem into another package, for doing calculations for buffer solutions and other mass/molarity calculations. Currently works great for this, but I was wondering if any of the developers had any ideas for saving previous queries, so that the second time the user searched "potassium chloride" etc, you wouldn't have to query the online databases and could instead match it to a local set of previously used options.

I have a couple of ideas of how I might implement it, but I was wondering if any of the developers had an idea that might use the webchem internals some more.

Thanks for all the hard work in making this great package so far!

[Aariq]

I think I briefly considered trying to work on this using something like the memoise package. I think it's a great idea, but I think we'd need to devote some time looking into how to best implement it. I don't personally have any experience with memoisation or caching, so I'd need to do some learning to even know how feasible this is.

[Aariq]

Definitely interested in hearing your ideas though!

[stitam]

Thanks @BradyAJohnston for opening this discussion, I think implementing a way to quickly recover results would add a lot to the package!

I'm currently using the package for querying info about thousands of chemicals and what I currently do to make my life easier is cache the results using save(). I create a .rda file in my working directory and I use that file to quickly access the data afterwards. I use this method with loops, so when I query 1000 chemicals I cache the output after each query. This way when the function is interrupted for whatever reason I still have quick access to all the data that has already been retrieved.

Thanks @Aariq for linking the memoise package, I think using a package that is already out there is probably better than finding our own way around this. Perhaps, we could have an argument in our functions where we specify whether we want to cache results or not and if we do we can cache them using the package?

[ethanbass]

I've been using memoise in conjunction with webchem (thanks to this post!). It seems like it might be kind of complicated to integrate memoisation into the package, but I wonder if it would be worth pointing out this possibility prominently in the documentation somewhere? It can really speed things up if you are trying run a list of queries where there a lot of repeats (and also reduce the number of calls to the server).

[stitam]

Thanks @ethanbass, I would prefer incorporating some kind of caching mechanism in the functions. The approach I described earlier was implemented in chembl_query(). Do you think implementing something like that for each function that interacts with databases could work?

[ethanbass]

I did a little digging and from what I can tell, this recommendation seems to be mainly a way to circumvent an R CMD check warning that arises from using memoise at build-time. See r-lib/memoise#76. I don't think my proposed function is likely to have this issue however, since memoise isn't actually run at buildtime, but only internally inside the function.

(Edit: It doesn't seem like there is a warning. I guess one issue with this implementation is that the cache is not being saved between multiple calls to the function, which may not be the desired behavior. However, if you specify the path to a cache file on the disk (as in @stitam's original implementation), the results will seemingly be saved between multiple calls to the function).

[Aariq]

Thanks for tracking that down! I never figured out why that was a recommendation.

[Aariq]

I think I don't quite understand what is meant by "build-time dependency" here. Why would memoising foo() inside an exported function not cause the same problem as memoising an exported function?

[ethanbass]

I don't claim to have a deep understanding of this, but I think the key distinction is between "build-time" and "run-time" dependencies. I think the warning is triggered by running memoise outside of a function, whereas if you run it inside of another function, it's a run-time dependency and there's no warning.

I guess the distinction is based on whether you need the dependency to build the package. In the former case, you will not be able to install the package at all without memoise, whereas in the latter case, you can install the package, but you may not be able to run the function.

[ethanbass]

I'm having trouble reproducing the alleged R CMD check warning at all though, even when running memoise outside of a function. I made this little test package that I think should trigger the warning (https://github.com/ethanbass/memoise_test), but it doesn't.