Any plan to port ferrum into Crystal?

[zw963]

Hi, i am a Rubyist, but, in recent days, i use Crystal exclusive basically.

I port some gems into Crystal,

e.g.

https://github.com/crystal-china/procodile_cr
https://github.com/crystal-china/hashr

It not easy, but it is possible, From my experience, 80%+ code can be reused.

And, you will get a static compiled, portable binary (as go/rust), and far less memory footprint with great speed increase.se

I use ferrum in my own project, I expect to use it with Crystal, is there any plan to port to Crystal?

[route]

Hello!

No I don't have any plans in any foreseeable future, because I have a backlog of tasks even for these projects.
As for Crystal I think the main pain point would be Threads and different concurrency model.

[zw963]

Yes, Crystal concurrency model only support multi-thread experiential(not as good as golang), but that is not a big deal as far as i know.

most of part of Thread.new in ferrum can be replace by fiber + channel and work just well in only single thread, that not would be a hard work, the only issue for me is, i use ferrum, but i never discover the source code, and missing the knowledge of CDP.

BTW: the reason i want port this gem is, when i write a crawler use Crystal, deploy is so easy, only one binary is enough.

[zw963]

Hi, just want to know if any maintainer have time for give some help for this porting?

Base on my previous experiences, this should not a big deal anyway, except i really don't understand what is the CDP.

For the outer gems, it shouldn't be a issue.

concurrent-ruby, webrick and websocket-driver all can be replace by crystal std-lib,

i don't know what is the usage of addressable, but, it probably can be replace by one of https://crystalshards.org/shards/search?q=addressable

Anyway, i may not be the most suitable person to porting this, but, i really hope there is a ferrum fork in Crystal, to release as a high-performance, portable binary.

[route]

The issue is not only port but also maintain and add features. For the years of coming into existence Ferrum didn't have any constant maintainer except me, I don't even talk about core team. I try to find time to maintain all of my projects, but crystal is a bit too much.

[zw963]

The issue is not only port but also maintain and add features. For the years of coming into existence Ferrum didn't have any constant maintainer except me, I don't even talk about core team. I try to find time to maintain all of my projects, but crystal is a bit too much.

Yes, maintain is a big issue, but, porting from Ruby to Crystal, probably as easy as to do some replace only at some(most of) times, unless you use too many meta-programming, e.g. eval family or create method/class dynamically or run method use send.

Sure, this is not easy, at least, user need understand deeper for Crystal when porting, to avoid the pitfall regarding to Type,Macro,nonblock IO etc, But, according to my experience, it is possible even user only understand 50% of current ruby project, the essence of these two languages is so similar, user still can make code works on Crystal, even better, the code is much clearer than the previous Ruby version always because Crystal.

BTW: there is a working selenium.cr fork exists, i create a shard use it for translate EN->zh-CN in several website use headless firefox, but, the performance is quite bad, because i have to kill my firefox process after run once in terminal, otherwise, when i try to run translate next time, will get a error, more detail check this, could you please tell me if this is possible when use ferrum? (e.g. chrome headless is running in the backend, and reuse it when run translate instead).

Thanks

[route]

This should be doable with ferrum if you run the browser, and create/dispose pages. This reuses the session.

[zw963]

That a lot, there are so many spec! i consider this will make porting to Crystal brought great convenience.

But, there are a spec failed, any idea?

 ╰─ $ rspec ./spec/page_spec.rb:19

Google Chrome 107.0.5304.110 

Run options: include {:locations=>{"./spec/page_spec.rb"=>[19]}}
Starting Puma
* Version 4.3.12 , codename: Mysterious Traveller
* Min threads: 0, max threads: 4
* Listening on tcp://127.0.0.1:36227

Ferrum::Page
  #go_to
    with failing response
      handles navigation error (FAILED - 1)

Failures:

  1) Ferrum::Page#go_to with failing response handles navigation error
     Failure/Error:
       expect { page.go_to("http://nope:#{port}/") }.to raise_error(
         Ferrum::StatusError,
         %r{Request to http://nope:\d+/ failed \(net::ERR_NAME_NOT_RESOLVED\)}
       )
     
       expected Ferrum::StatusError with message matching /Request to http:\/\/nope:\d+\/ failed \(net::ERR_NAME_NOT_RESOLVED\)/, got #<Ferrum::PendingConnectionsError: Request to http://nope:36227/ reached server, but there are still pending connections: http://nope:36227/> with backtrace:
         # ./lib/ferrum/page.rb:123:in `rescue in go_to'
         # ./lib/ferrum/page.rb:113:in `go_to'
         # ./spec/page_spec.rb:20:in `block (5 levels) in <top (required)>'
         # ./spec/page_spec.rb:20:in `block (4 levels) in <top (required)>'
     # ./spec/page_spec.rb:20:in `block (4 levels) in <top (required)>'

Finished in 5.38 seconds (files took 0.37738 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/page_spec.rb:19 # Ferrum::Page#go_to with failing response handles navigation error

Ruby 3.2.1

Gems included by the bundle:
  * Ascii85 (1.1.0)
  * addressable (2.8.1)
  * afm (0.2.2)
  * ast (2.4.2)
  * byebug (11.1.3)
  * chunky_png (1.4.0)
  * concurrent-ruby (1.2.0)
  * diff-lcs (1.5.0)
  * ferrum (0.13)
  * hashery (2.1.2)
  * image_size (2.1.2)
  * json (2.6.3)
  * kramdown (2.4.0)
  * mustermann (2.0.2)
  * nio4r (2.5.8)
  * parallel (1.22.1)
  * parser (3.2.1.0)
  * pdf-reader (2.11.0)
  * public_suffix (5.0.1)
  * puma (4.3.12)
  * rack (2.2.6.2)
  * rack-protection (2.2.4)
  * rainbow (3.1.1)
  * rake (13.0.6)
  * redcarpet (3.6.0)
  * regexp_parser (2.7.0)
  * rexml (3.2.5)
  * rspec (3.12.0)
  * rspec-core (3.12.1)
  * rspec-expectations (3.12.2)
  * rspec-mocks (3.12.3)
  * rspec-support (3.12.0)
  * rubocop (1.45.1)
  * rubocop-ast (1.26.0)
  * rubocop-rake (0.6.0)
  * ruby-progressbar (1.11.0)
  * ruby-rc4 (0.1.5)
  * ruby2_keywords (0.0.5)
  * sinatra (2.2.4)
  * tilt (2.1.0)
  * ttfunk (1.7.0)
  * unicode-display_width (2.4.2)
  * webrick (1.7.0)
  * websocket-driver (0.7.5)
  * websocket-extensions (0.1.5)
  * yard (0.9.28)
Use `bundle info` to print more detailed information about a gem

[zw963]

Hi, just let you know, i am starting porting ferrum int Crystal. project is here.

It is still in a very very early stage, i only copy lib/*.rb into src/*.cr, and fixed many compile-time error only.

the left steps is figure out the type of important objects, and do API translation follow Ruby/Crystal document.

i believe spec will make this work quite easier, a minimum spec is a good start, could you please give a clue
about which spec is least dependent and relatively independent to as a start?

Thanks.

[route]

Don't fully understand your question. All specs are important, if not they are useless. Take one and make it work, it doesn't matter which will be first