Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory refactor - follow up #6 (rebased) #8

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Memory refactor - follow up #6 (rebased) #8

wants to merge 1 commit into from

Conversation

dgilperez
Copy link

This is a rebased PR following up #6.

Current results from bundle exec derailed bundle:mem:

TOP: 2.5 MiB
  domain_name: 2.4844 MiB
    yaml: 1.8906 MiB
      psych: 1.8867 MiB
        psych/visitors: 0.9063 MiB
          psych/visitors/yaml_tree: 0.5 MiB
        psych/nodes: 0.4414 MiB
    ipaddr: 0.3164 MiB

@dgilperez
marshalls etld_data hash for speed and memory footprint reduction
c409551 
refactor etld data generation to generate the yaml and cache file on rake, cleaning up some code in the process

ruby 1.8 does not have File.binread

remove derailed development dependency

fixes Rakefile to revert calling the data generation tool without any rake task and fix bug in data update

updates data to 2015-09-15 17:22:03 +0200 and change test since .cy domain changed

prefers YAML.dump(content, file) to file.write content.to_yaml

prefers plain YAML to Marshall binary
@dgilperez dgilperez mentioned this pull request Jun 16, 2016
Closed
@knu
Copy link
Owner

knu commented Jun 17, 2016

Hmm, so, using "yaml" makes this library consume as much memory as it does now, right? Is it that "yaml" is a popular library so it does not count in most apps?

@knu
Copy link
Owner

knu commented Jun 17, 2016

If the motivation for this PR is to reduce memory footprint when your app happens to load this library via dependencies but does not actually use DomainName at all, what if you just use autoload or a similar technique to simply delay the loading of domain_name/etld_data.rb, instead of using yaml or marshal? Is using a deserializer better than generating data in byte-code, as I asked in the previous PR? Can you get numbers of memory footprint after eTLD data is loaded, with and without this change?

@knu
Copy link
Owner

knu commented Jun 17, 2016

I'm finally getting to grasp what your PR is about, so pardon for asking so many questions. 😅

@dgilperez
Copy link
Author

@knu I'm sorry I'm not going to be able to devote time to this PR in a while ... I'm into a great load of work atm and I don't actually recall the details of this after the time passed ...

@tayler1
Copy link

tayler1 commented Feb 5, 2017 •
edited
Loading

I did some benchmarks. Did run it several times and this is average summary results.

ruby 2.3 / derailed bundle:mem

current master

TOP: 2.3281 MiB
  rake: 1.4063 MiB
    rake/rake_module: 0.9805 MiB
      rake/application: 0.9766 MiB (Also required by: rake)
        rake/file_list: 0.5703 MiB (Also required by: rake)
  domain_name: 0.875 MiB
    domain_name/etld_data: 0.7148 MiB

yaml branch

TOP: 2.5898 MiB
  rake: 1.6406 MiB
    rake/rake_module: 0.8828 MiB
      rake/application: 0.875 MiB (Also required by: rake)
        rake/file_list: 0.4531 MiB (Also required by: rake)
  domain_name: 0.8828 MiB
    yaml: 0.7461 MiB
      psych: 0.7422 MiB
        psych/visitors: 0.3359 MiB

ruby 2.3 / test.rb
contents only single init of DomainName.new with idn domain

current master

Total allocated: 90222 bytes (678 objects)
Total retained:  4976 bytes (27 objects)

allocated memory by gem
-----------------------------------
     60892  unf-0.1.4
     17484  other
      4805  ruby-domain_name-16bf27a2b3e7
      4522  unf_ext-0.0.7.2
      2519  ruby-2.3.0/lib

yaml branch

Total allocated: 9399597 bytes (94811 objects)
Total retained:  708561 bytes (7796 objects)

allocated memory by gem
-----------------------------------
   9310855  ruby-2.3.0/lib
     60892  unf-0.1.4
     17564  other
      5764  ruby-domain_name-c409551cd591
      4522  unf_ext-0.0.7.2

ruby 2.3 / bench.rb
10000 * new idn / 10000 * new from tld list / 10000 * new ascii domain / 100000 * decode_hostname

current master

                       user     system      total        real
new int            0.430000   0.000000   0.430000 (  0.432575)
new etld           0.070000   0.000000   0.070000 (  0.074813)
ascii              0.070000   0.000000   0.070000 (  0.065333)
decode             0.970000   0.010000   0.980000 (  0.978375)

yaml branch

                       user     system      total        real
new int            0.580000   0.010000   0.590000 (  0.578547)
new etld           0.060000   0.000000   0.060000 (  0.070711)
ascii              0.060000   0.000000   0.060000 (  0.067486)
decode             0.960000   0.000000   0.960000 (  0.962656)

ruby 2.4 / derailed bundle:mem

current master

TOP: 3.9609 MiB
  domain_name: 2.3008 MiB
    domain_name/etld_data: 2.2422 MiB
  rake: 1.6133 MiB
    rake/rake_module: 1.2852 MiB
      rake/application: 1.2773 MiB (Also required by: rake)
        rake/file_list: 0.9336 MiB (Also required by: rake)

yaml branch

TOP: 1.9453 MiB
  rake: 1.4375 MiB
    rake/rake_module: 1.2148 MiB
      rake/application: 1.2109 MiB (Also required by: rake)
        rake/file_list: 0.9258 MiB (Also required by: rake)
  domain_name: 0.4492 MiB
    yaml: 0.3633 MiB
      psych: 0.3633 MiB

ruby 2.4 / test.rb

current master

Total allocated: 87820 bytes (676 objects)
Total retained:  4289 bytes (27 objects)

allocated memory by gem
-----------------------------------
     59509  unf-0.1.4
     17038  other
      4800  ruby-domain_name-16bf27a2b3e7
      4401  unf_ext-0.0.7.2
      2072  ruby-2.4.0/lib

yaml branch

Total allocated: 9073821 bytes (94782 objects)
Total retained:  548373 bytes (7796 objects)

allocated memory by gem
-----------------------------------
   8987074  ruby-2.4.0/lib
     59509  unf-0.1.4
     17118  other
      5719  ruby-domain_name-c409551cd591
      4401  unf_ext-0.0.7.2

ruby 2.4 / bench.rb

current master

                       user     system      total        real
new int            0.430000   0.000000   0.430000 (  0.434277)
new etld           0.060000   0.000000   0.060000 (  0.058922)
ascii              0.050000   0.000000   0.050000 (  0.057676)
decode             0.920000   0.010000   0.930000 (  0.926379)

yaml branch

                       user     system      total        real
new int            0.580000   0.010000   0.590000 (  0.581749)
new etld           0.060000   0.000000   0.060000 (  0.060923)
ascii              0.060000   0.000000   0.060000 (  0.060432)
decode             0.910000   0.000000   0.910000 (  0.915235)

ruby 2.4 / test.rb process size

current master Activity: 22.7 MB
yaml branch Activity: 26.8 MB

ruby 2.4 is faster and better than 2.3
derailed bundle memory looks random, don't understand how it counts, I got different results just on every relaunch.

So. I don't know how it performs in real world.
@knu, looks like your custom code is quite good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dgilperez @knu @tayler1