Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Http driver config #273

Open
wants to merge 19 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 18 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 15 additions & 14 deletions .rubocop_todo.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# This configuration was generated by
# `rubocop --auto-gen-config`
# on 2016-03-28 20:14:45 +0900 using RuboCop version 0.39.0.
# on 2016-11-21 15:47:20 -0800 using RuboCop version 0.39.0.
# The point is for the user to remove these configuration records
# one by one as the offenses are removed from the code base.
# Note that changes in the inspected code, or installation of new
Expand Down Expand Up @@ -31,22 +31,26 @@ Lint/UnusedMethodArgument:
- 'lib/neo4j/relationship.rb'
- 'lib/neo4j/session.rb'

# Offense count: 16
# Offense count: 24
Metrics/AbcSize:
Max: 17

# Offense count: 9
# Offense count: 12
# Configuration parameters: CountComments.
Metrics/ClassLength:
Max: 191

# Offense count: 508
# Offense count: 1
Metrics/CyclomaticComplexity:
Max: 8

# Offense count: 657
# Configuration parameters: AllowHeredoc, AllowURI, URISchemes.
# URISchemes: http, https
Metrics/LineLength:
Max: 180

# Offense count: 20
# Offense count: 27
# Configuration parameters: CountComments.
Metrics/MethodLength:
Max: 14
Expand All @@ -56,6 +60,10 @@ Metrics/MethodLength:
Metrics/ModuleLength:
Max: 113

# Offense count: 1
Metrics/PerceivedComplexity:
Max: 8

# Offense count: 8
Style/AccessorMethodName:
Exclude:
Expand All @@ -72,7 +80,7 @@ Style/ClassVars:
Exclude:
- 'lib/neo4j/session.rb'

# Offense count: 72
# Offense count: 84
Style/Documentation:
Enabled: false

Expand All @@ -82,7 +90,7 @@ Style/GuardClause:
Exclude:
- 'lib/neo4j-core/query_clauses.rb'

# Offense count: 56
# Offense count: 65
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did these counts go up?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my guess was that no one had run rubocop --auto-gen-config in a while (since 2016-03-28 in fact). A lot of metrics changed that were unrelated to what I did here, I just needed to update the metrics because I was getting caught by AbcSize.

Since the rest of those rules were already being muted via this file, it just updated the counts of existing violations since the previous run.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, sorry, the offense count increase in fine, but the "Max" counts increasing is worrying. We try not to go backwards with our Rubocop todo values (but anything that we set in .rubocop.yml is our official agreed rule / max). If you're having trouble getting the code to fit in the metrics we can help (or increase the values if it really is too cumbersome)

Copy link
Contributor Author

@brucek brucek Dec 2, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. I had kinda been hoping to not take on a refactor job as well, for two reasons:

  1. more work ;-)
  2. not wanting to get in a long "style conversation" when I inevitably refactor something in a way you guys don't feel good about

Anyways, I just reverted the rubocop_todo.yml and made some changes that fixes the Rubocop issues, but to me the code looks worse. You can be the judge though.

# Cop supports --auto-correct.
Style/MutableConstant:
Enabled: false
Expand All @@ -92,10 +100,3 @@ Style/MutableConstant:
Style/RedundantSelf:
Exclude:
- 'lib/neo4j-core/query_clauses.rb'

# Offense count: 1
# Cop supports --auto-correct.
Style/RescueEnsureAlignment:
Exclude:
- 'spec/shared_examples/node_with_tx.rb'

2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ before_script:
- "echo 'dbms.memory.heap.max_size=1000' >> ./db/neo4j/development/conf/neo4j-wrapper.conf"
- "echo 'dbms.memory.heap.initial_size=1000' >> ./db/neo4j/development/conf/neo4j-wrapper.conf"
- "bin/rake neo4j:start --trace"
- "sleep 10"
- "sleep 11"
script:
- "bundle exec rspec $RSPEC_OPTS"
language: ruby
Expand Down
8 changes: 7 additions & 1 deletion Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,11 @@ group 'test' do
gem 'rspec', '~> 3.0'
gem 'rspec-its'
gem 'dotenv'
gem 'activesupport', '~> 4.0'
gem 'activesupport', RUBY_VERSION.to_f >= 2.2 ? '>= 4.0' : '~> 4'

gem 'em-http-request', '>= 1.1', require: 'em-http', platforms: :ruby
gem 'em-synchrony', '>= 1.0.3', require: ['em-synchrony', 'em-synchrony/em-http'], platforms: :ruby
gem 'excon', '>= 0.27.4'
gem 'patron', '>= 0.4.2', platforms: :ruby
gem 'typhoeus', '>= 0.3.3'
end
30 changes: 30 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,36 @@ To make a basic connection to Neo4j to execute Cypher queries, first choose an a

neo4j_adaptor = Neo4j::Core::CypherSession::Adaptors::Embedded.new('/file/path/to/graph.db')

The `http_adaptor` can also take `:faraday_options`. Multiple middlewares with the same key can be passed through using arrays:

http_adaptor = Neo4j::Core::CypherSession::Adaptors::HTTP.new('http://neo4j:pass@localhost:7474',
faraday_options: {
adapter: :typhoeus,
# This will pass 2 items to Faraday. You must use multidimensional arrays to pass single args separately
request: [
[:multipart],
[:url_encoded]
],
# This will only pass a single item
response: [:multi_json, symbolize_keys: true, content_type: 'application/json']
})

This will initialize Faraday like so:

Faraday.new(url: 'http://neo4j:pass@localhost:7474') do |faraday|
faraday.request :multipart
faraday.request :url_encoded

faraday.response :multi_json, symbolize_keys: true, content_type: 'application/json'
faraday.adapter: :typhoeus
end

The order that the keys are passed through to Faraday will remain hash ordered *except* that `adapter` is always last. Arrays within each key are passed in order.

Note you **must** install any required http adaptor gems yourself as per [Faraday](https://github.com/lostisland/faraday). Ex for `:typhoeus`, add to your Gemfile:

gem 'typhoeus'

Once you have an adaptor you can create a session like so:

neo4j_session = Neo4j::Core::CypherSession.new(http_adaptor)
Expand Down
36 changes: 23 additions & 13 deletions lib/neo4j-server/cypher_session.rb
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
require 'uri'
require 'neo4j/core/cypher_session/adaptors/faraday_helpers'

module Neo4j
module Server
Expand All @@ -7,7 +8,22 @@ module Server
end

class CypherSession < Neo4j::Session
module PrivateMethods
private

def extract_faraday_options(params, defaults = {})
verify_faraday_options(params.delete(:faraday_options) || params.delete('faraday_options') || {}, defaults)
end

def extract_basic_auth(url, params)
return unless url && URI(url).userinfo
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a huge deal, but you could perhaps create a uri object at the start and use it in all of the places you're using URI(url)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call

params[:basic_auth] = {username: URI(url).user, password: URI(url).password}
end
end

include Resource
extend Neo4j::Core::CypherSession::Adaptors::FaradayHelpers
extend Neo4j::Server::CypherSession::PrivateMethods

alias super_query query
attr_reader :connection
Expand All @@ -24,15 +40,16 @@ def initialize(data_url, connection)
# @see https://github.com/lostisland/faraday
def self.create_connection(params, url = nil)
init_params = params[:initialize] && params.delete(:initialize)

request = [:multi_json]
request.unshift([:basic_auth, params[:basic_auth][:username], params[:basic_auth][:password]]) if params[:basic_auth]

faraday_options = extract_faraday_options(
params, request: request, response: [:multi_json, symbolize_keys: true, content_type: 'application/json'])
conn = Faraday.new(url, init_params) do |b|
b.request :basic_auth, params[:basic_auth][:username], params[:basic_auth][:password] if params[:basic_auth]
b.request :multi_json
# b.response :logger, ::Logger.new(STDOUT), bodies: true

b.response :multi_json, symbolize_keys: true, content_type: 'application/json'
# b.use Faraday::Response::RaiseError
b.use Faraday::Adapter::NetHttpPersistent
# b.adapter Faraday.default_adapter
set_faraday_middleware b, faraday_options
end
conn.headers = {'Content-Type' => 'application/json', 'User-Agent' => ::Neo4j::Session.user_agent_string}
conn
Expand All @@ -58,13 +75,6 @@ def self.establish_session(root_data, connection)
CypherSession.new(data_url, connection)
end

def self.extract_basic_auth(url, params)
return unless url && URI(url).userinfo
params[:basic_auth] = {username: URI(url).user, password: URI(url).password}
end

private_class_method :extract_basic_auth

def db_type
:server_db
end
Expand Down
49 changes: 49 additions & 0 deletions lib/neo4j/core/cypher_session/adaptors/faraday_helpers.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
module Neo4j
module Core
class CypherSession
module Adaptors
module FaradayHelpers
private

def verify_faraday_options(faraday_options = {}, defaults = {})
faraday_options.symbolize_keys!.reverse_merge!(defaults)
faraday_options[:adapter] ||= :net_http_persistent
require 'typhoeus/adapters/faraday' if faraday_options[:adapter].to_sym == :typhoeus
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the user require the gem paths that they need depending on the adaptor that they are using? That way we wouldn't need to include the various gems in our Gemfile

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that is true... I was just bitten by typhoeus/typhoeus#226 (comment) which was a little hard to track down the solution, so I figured it would be a little easier to just handle that for them instead (plus it is only needed here, not in their project generally). Happy to take it out if you'd rather doc it / force folks to figure it out, but it seems like an easy win that doesn't really cost much here.

in the Gemfile I only include the adapter gems in :test (and not in the .gemspec) so tests here will fail if there's any issues vs letting people hit them when using whatever gems and fail and then report issues.

I can imagine if there is any issues with the adapter gem(s), it requires a little work to maintain the tests, but again, it seems like an easy thing we can stay on top of here vs just letting people get bitten and have to figure it out themselves..?

I'm of the understanding that only the .gemspec gems are exposed outwards, and so having gem in the Gemfile doesn't really impact anyone else - is that correct?

faraday_options
end

def set_faraday_middleware(faraday, options = {adapter: :net_http_persistent})
adapter = options.delete(:adapter)
send_all_faraday_options(faraday, options)
faraday.adapter adapter
end

def send_all_faraday_options(faraday, options)
options.each do |key, value|
next unless faraday.respond_to? key
if value.is_a? Array
if value.none? { |arg| arg.is_a? Array }
faraday.send key, *value
else
value.each do |args|
arg_safe_send faraday, key, args
end
end
else
faraday.send key, value
end
end
end
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really sure I get why all of this is necessary. Doesn't the second argument of Faraday.new take a hash of options which is an alternative to sending methods to the faraday object in the block syntax? Maybe that's where we're misunderstanding...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't look too hard, but I took these comments to mean no since I don't see anything mentioning :response as a key, for instance. But yeah, if that's true that makes life much easier - I'll give it a try


def arg_safe_send(object, msg, args)
if args.is_a? Array
object.send(msg, *args)
else
object.send(msg, args)
end
end
end
end
end
end
end
32 changes: 18 additions & 14 deletions lib/neo4j/core/cypher_session/adaptors/http.rb
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
require 'neo4j/core/cypher_session/adaptors'
require 'neo4j/core/cypher_session/adaptors/has_uri'
require 'neo4j/core/cypher_session/adaptors/faraday_helpers'
require 'neo4j/core/cypher_session/responses/http'

# TODO: Work with `Query` objects
Expand All @@ -16,7 +17,8 @@ def initialize(url, options = {})
end

def connect
@requestor = Requestor.new(@url, USER_AGENT_STRING, self.class.method(:instrument_request))
@requestor = Requestor.new(@url, USER_AGENT_STRING, self.class.method(:instrument_request),
@options[:faraday_options] || @options['faraday_options'] || {})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that you should just be able to do @options['faraday_options'] || {}, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not unless I stringify_keys! since coders will likely use :faraday_options but when config'ed via neo4j.yml (ie in Rails), the keys are strings. There is an explicit test for this.

rescue Faraday::ConnectionFailed => e
raise CypherSession::ConnectionFailedError, "#{e.class}: #{e.message}"
end
Expand Down Expand Up @@ -91,15 +93,16 @@ def connected?
# - Sets headers, including user agent string
class Requestor
include Adaptors::HasUri
include FaradayHelpers
default_url('http://neo4:neo4j@localhost:7474')
validate_uri { |uri| uri.is_a?(URI::HTTP) }

def initialize(url, user_agent_string, instrument_proc)
def initialize(url, user_agent_string, instrument_proc, faraday_options)
self.url = url
@user = user
@password = password
@user_agent_string = user_agent_string
@faraday = faraday_connection
@faraday = faraday_connection(faraday_options)
@instrument_proc = instrument_proc
end

Expand Down Expand Up @@ -134,22 +137,23 @@ def get(path, body = '', options = {})

private

def faraday_connection
def faraday_connection(faraday_options = {})
verify_faraday_options(faraday_options,
request: [
[:basic_auth, user, password],
:multi_json
],
response: [:multi_json, symbolize_keys: true, content_type: 'application/json']
)
require 'faraday'
require 'faraday_middleware/multi_json'

Faraday.new(url) do |c|
c.request :basic_auth, user, password
c.request :multi_json

c.response :multi_json, symbolize_keys: true, content_type: 'application/json'
c.use Faraday::Adapter::NetHttpPersistent

conn = Faraday.new(url) do |c|
# c.response :logger, ::Logger.new(STDOUT), bodies: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just verifying, if you leave these lines in, are they no longer defaults but rather they override the options?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is my understanding - all the middleware that gets called during initialization gets mounted in Faraday and these would then always get mounted no matter if people wanted them or not, and there would be no way to get rid of them.


c.headers['Content-Type'] = 'application/json'
c.headers['User-Agent'] = @user_agent_string
set_faraday_middleware c, faraday_options
end
conn.headers = {'Content-Type' => 'application/json', 'User-Agent' => @user_agent_string}
conn
end

def request_body(body)
Expand Down
29 changes: 29 additions & 0 deletions spec/neo4j-server/cypher_session/shared_examples/adaptor.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
RSpec.shared_examples 'Neo4j::Server::CypherSession::Adaptor' do
describe 'faraday_options' do
describe 'a faraday connection type adapter option' do
it 'can use a user supplied faraday connection for a new session' do
connection = Faraday.new do |faraday|
faraday.request :basic_auth, basic_auth_hash[:username], basic_auth_hash[:password]

faraday.request :multi_json
faraday.response :multi_json, symbolize_keys: true, content_type: 'application/json'
faraday.adapter Faraday.default_adapter
end
connection.headers = {'Content-Type' => 'application/json'}

expect(connection).to receive(:get).at_least(:once).and_call_original
create_server_session(connection: connection)
end

it 'will pass through a symbol key' do
expect_any_instance_of(Faraday::Connection).to receive(:adapter).with(:typhoeus).and_call_original
create_server_session(faraday_options: {adapter: :typhoeus})
end

it 'will pass through a string key' do
expect_any_instance_of(Faraday::Connection).to receive(:adapter).with(:typhoeus).and_call_original
create_server_session('faraday_options' => {'adapter' => :typhoeus})
end
end
end
end
18 changes: 3 additions & 15 deletions spec/neo4j-server/e2e/cypher_session_spec.rb
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
require 'spec_helper'
require './spec/neo4j-server/cypher_session/shared_examples/adaptor'

module Neo4j
module Server
Expand All @@ -18,27 +19,14 @@ def open_session
Neo4j::Session.set_current(@before_session)
end

it 'can use a user supplied faraday connection for a new session' do
connection = Faraday.new do |faraday|
faraday.request :basic_auth, basic_auth_hash[:username], basic_auth_hash[:password]

faraday.request :multi_json
faraday.response :multi_json, symbolize_keys: true, content_type: 'application/json'
faraday.adapter Faraday.default_adapter
end
connection.headers = {'Content-Type' => 'application/json'}

expect(connection).to receive(:get).at_least(:once).and_call_original
create_server_session(connection: connection)
end

it 'adds host and port to the connection object' do
connection = Neo4j::Session.current.connection
expect(connection.port).to eq ENV['NEO4J_URL'] ? URI(ENV['NEO4J_URL']).port : 7474
expect(connection.host).to eq 'localhost'
end
end

it_behaves_like 'Neo4j::Server::CypherSession::Adaptor'
end

describe 'named sessions' do
before { Neo4j::Session.current && Neo4j::Session.current.close }
Expand Down
Loading