Memory refactor

[dgilperez]

Hi!

I was playing with derailed_benchmarks gem in one of my projects, looking to reduce its memory footprint, and I noticed rest-client was using quite a few, mainly because of it's dependency to domain_name.

I was looking at your code and noticed that we could improve that by using marshaling rather than a huge constant in lib/domain_name/etld_data.rb.

This PR is my take towards this. Please let me know what do you think.

The output of bundle exec derailed bundle:mem (locally adding derailed as a development dependency) was:

TOP: 2.4141 MiB
  domain_name: 2.3789 MiB
    domain_name/etld_data: 2.0859 MiB

And after the changes here is:

TOP: 0.7109 MiB
  domain_name: 0.668 MiB
    yaml: 0.4609 MiB
      psych: 0.457 MiB

Being half of the memory actually been used by yaml, which will be already a dependency quite a lot of apps already down there.

Thanks!

[tijmenb]

👍

[knu]

Cool, but I'd like to know how this would reduce memory footprint before anything. Is that the cost of running the code to build a large hash and then keep the byte code that is only run once? Just curious, but how much of it will stay on memory forever?

Yaml would be fine, but I'm not fond of distributing Marshal'd binary because it will enforce you to build a gem with the oldest ruby supported in order to make sure to generate a binary that is readable by any version of ruby.

[knu]

I think Yaml.dump(etld_data, yaml) would be better because it'd save the generation of a large string.

[dgilperez]

Done!

[dgilperez]

About the second part of your question, if I understand correctly how Marshalling works, there should be no problem with ruby >= 1.8.0, as the docs say:

The first two bytes of the stream contain the major and minor version, each as a single byte encoding a digit. The version implemented in Ruby is 4.8 (stored as “x04x08”) and is supported by ruby 1.8.0 and newer.

Different major versions of the Marshal format are not compatible and cannot be understood by other major versions. Lesser minor versions of the format can be understood by newer minor versions. Format 4.7 can be loaded by a 4.8 implementation but format 4.8 cannot be loaded by a 4.7 implementation.

So we might have problems with people using Marshal 4.7 (previous to ruby 1.8), but even if we want to support that (do we?), we may generate the file with that format ...

Anyway, the main reason for using Marshall is speed, and although that is a clear advantage for .dump, that is probably not the point here. Deserializing is also reportedly faster with Marshal, but not that much.

I may be missing a whole bunch of points here anyway. Let me know if you want me to use Yaml instead.

[dgilperez]

About the first part of the question, I'm not entirely sure. bundle exec derailed bundle:mem calculates the memory used at boot time. I assume that memory stays as a base memory footprint for any app depending on ruby-domain_name. If that's true, and considering rest-client is a common dependency, even if we are talking about just about 2MiB, that would be noticeable by many. I may be wrong.

May I politely ask @schneems about his opinion?

[dgilperez]

@knu I just pushed a non-marshall version that performs about as well.

[dgilperez]

Hey @knu, did you have time to review this? Thanks 😄

[dgilperez]

Hi @knu any news on this?

[knu]

So sorry for the delay. Could you rebase this on master?

[dgilperez]

There you go @knu ! I'm closing in favour of #8