Skip to content

Commit

Permalink
support rails 7.1, fix schema migration final #105, int data #68
Browse files Browse the repository at this point in the history
  • Loading branch information
@PNixx
PNixx committed Jan 10, 2024
1 parent 0b591b5 commit cc3ecfd
Show file tree
Hide file tree
Showing 13 changed files with 299 additions and 289 deletions.
72 changes: 31 additions & 41 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Clickhouse::Activerecord

A Ruby database ActiveRecord driver for ClickHouse. Support Rails >= 5.2.
A Ruby database ActiveRecord driver for ClickHouse. Support Rails >= 7.1.
Support ClickHouse version from 22.0 LTS.

## Installation
Expand Down Expand Up @@ -50,41 +50,31 @@ class ActionView < ActiveRecord::Base
end
```

## Usage in Rails 5
## Usage in Rails

Add your `database.yml` connection information with postfix `_clickhouse` for you environment:

```yml
development_clickhouse:
development:
adapter: clickhouse
database: database
```
Add to your model:
Your model example:
```ruby
class Action < ActiveRecord::Base
establish_connection "#{Rails.env}_clickhouse".to_sym
end
```

For materialized view model add:
```ruby
class ActionView < ActiveRecord::Base
establish_connection "#{Rails.env}_clickhouse".to_sym
self.is_view = true
end
```

Or global connection:

```yml
development:
adapter: clickhouse
database: database
```
## Usage in Rails 6 with second database
## Usage in Rails with second database

Add your `database.yml` connection information for you environment:

Expand All @@ -102,31 +92,31 @@ Connection [Multiple Databases with Active Record](https://guides.rubyonrails.or
```ruby
class Action < ActiveRecord::Base
connects_to database: { writing: :clickhouse, reading: :clickhouse }
establish_connection :clickhouse
end
```

### Rake tasks

**Note!** For Rails 6 you can use default rake tasks if you configure `migrations_paths` in your `database.yml`, for example: `rake db:migrate`

Create / drop / purge / reset database:

$ rake clickhouse:create
$ rake clickhouse:drop
$ rake clickhouse:purge
$ rake clickhouse:reset
$ rake db:create
$ rake db:drop
$ rake db:purge
$ rake db:reset

Prepare system tables for rails:
Or with multiple databases:

$ rake clickhouse:prepare_schema_migration_table
$ rake clickhouse:prepare_internal_metadata_table
$ rake db:create:clickhouse
$ rake db:drop:clickhouse
$ rake db:purge:clickhouse
$ rake db:reset:clickhouse

Migration:

$ rails g clickhouse_migration MIGRATION_NAME COLUMNS
$ rake clickhouse:migrate
$ rake clickhouse:rollback
$ rake db:migrate
$ rake db:rollback

### Dump / Load for multiple using databases

Expand Down Expand Up @@ -195,20 +185,20 @@ User.joins(:actions).using(:group_id)
Integer types are unsigned by default. Specify signed values with `:unsigned =>
false`. The default integer is `UInt32`

| Type (bit size) | Range | :limit (byte size) |
| :--- | :----: | ---: |
| Int8 | -128 to 127 | 1 |
| Int16 | -32768 to 32767 | 2 |
| Int32 | -2147483648 to 2,147,483,647 | 3,4 |
| Int64 | -9223372036854775808 to 9223372036854775807] | 5,6,7,8 |
| Int128 | ... | 9 - 15 |
| Int256 | ... | 16+ |
| UInt8 | 0 to 255 | 1 |
| UInt16 | 0 to 65,535 | 2 |
| UInt32 | 0 to 4,294,967,295 | 3,4 |
| UInt64 | 0 to 18446744073709551615 | 5,6,7,8 |
| UInt256 | 0 to ... | 8+ |
| Array | ... | ... |
| Type (bit size) | Range | :limit (byte size) |
|:----------------|:--------------------------------------------:|-------------------:|
| Int8 | -128 to 127 | 1 |
| Int16 | -32768 to 32767 | 2 |
| Int32 | -2147483648 to 2,147,483,647 | 3,4 |
| Int64 | -9223372036854775808 to 9223372036854775807] | 5,6,7,8 |
| Int128 | ... | 9 - 15 |
| Int256 | ... | 16+ |
| UInt8 | 0 to 255 | 1 |
| UInt16 | 0 to 65,535 | 2 |
| UInt32 | 0 to 4,294,967,295 | 3,4 |
| UInt64 | 0 to 18446744073709551615 | 5,6,7,8 |
| UInt256 | 0 to ... | 8+ |
| Array | ... | ... |

Example:

Expand Down
2 changes: 1 addition & 1 deletion clickhouse-activerecord.gemspec
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Gem::Specification.new do |spec|
spec.require_paths = ['lib']

spec.add_runtime_dependency 'bundler', '>= 1.13.4'
spec.add_runtime_dependency 'activerecord', '>= 5.2', '< 7'
spec.add_runtime_dependency 'activerecord', '>= 7.1'

spec.add_development_dependency 'rake', '~> 13.0'
spec.add_development_dependency 'rspec', '~> 3.4'
Expand Down
6 changes: 3 additions & 3 deletions lib/active_record/connection_adapters/clickhouse/schema_statements.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ def execute(sql, name = nil, settings: {})
do_execute(sql, name, settings: settings)
end

def exec_insert(sql, name, _binds, _pk = nil, _sequence_name = nil)
def exec_insert(sql, name, _binds, _pk = nil, _sequence_name = nil, returning: nil)
new_sql = sql.dup.sub(/ (DEFAULT )?VALUES/, " VALUES")
do_execute(new_sql, name, format: nil)
true
end

def exec_query(sql, name = nil, binds = [], prepare: false)
def internal_exec_query(sql, name = nil, binds = [], prepare: false, async: false)
result = do_execute(sql, name)
ActiveRecord::Result.new(result['meta'].map { |m| m['name'] }, result['data'], result['meta'].map { |m| [m['name'], type_map.lookup(m['type'])] }.to_h)
rescue ActiveRecord::ActiveRecordError => e
Expand Down Expand Up @@ -137,7 +137,7 @@ def create_table_definition(table_name, **options)
Clickhouse::TableDefinition.new(self, table_name, **options)
end

def new_column_from_field(table_name, field)
def new_column_from_field(table_name, field, _definitions)
sql_type = field[1]
type_metadata = fetch_type_metadata(sql_type)
default = field[3]
Expand Down
115 changes: 63 additions & 52 deletions lib/active_record/connection_adapters/clickhouse_adapter.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -136,15 +136,19 @@ def initialize(logger, connection_parameters, config, full_config)

# Support SchemaMigration from v5.2.2 to v6+
def schema_migration # :nodoc:
ClickhouseActiverecord::SchemaMigration
ClickhouseActiverecord::SchemaMigration.new(self)
end

def internal_metadata # :nodoc:
ClickhouseActiverecord::InternalMetadata.new(self)
end

def migrations_paths
@full_config[:migrations_paths] || 'db/migrate_clickhouse'
end

def migration_context # :nodoc:
ClickhouseActiverecord::MigrationContext.new(migrations_paths, schema_migration)
ClickhouseActiverecord::MigrationContext.new(migrations_paths, schema_migration, internal_metadata)
end

def arel_visitor # :nodoc:
Expand All @@ -159,66 +163,73 @@ def valid_type?(type)
!native_database_types[type].nil?
end

def extract_limit(sql_type) # :nodoc:
case sql_type
when /(Nullable)?\(?String\)?/
super('String')
when /(Nullable)?\(?U?Int8\)?/
1
when /(Nullable)?\(?U?Int16\)?/
2
when /(Nullable)?\(?U?Int32\)?/
nil
when /(Nullable)?\(?U?Int64\)?/
8
else
super
class << self
def extract_limit(sql_type) # :nodoc:
case sql_type
when /(Nullable)?\(?String\)?/
super('String')
when /(Nullable)?\(?U?Int8\)?/
1
when /(Nullable)?\(?U?Int16\)?/
2
when /(Nullable)?\(?U?Int32\)?/
nil
when /(Nullable)?\(?U?Int64\)?/
8
else
super
end
end
end

# `extract_scale` and `extract_precision` are the same as in the Rails abstract base class,
# except this permits a space after the comma
# `extract_scale` and `extract_precision` are the same as in the Rails abstract base class,
# except this permits a space after the comma

def extract_scale(sql_type)
case sql_type
when /\((\d+)\)/ then 0
when /\((\d+)(,\s?(\d+))\)/ then $3.to_i
def extract_scale(sql_type)
case sql_type
when /\((\d+)\)/ then 0
when /\((\d+)(,\s?(\d+))\)/ then $3.to_i
end
end
end

def extract_precision(sql_type)
$1.to_i if sql_type =~ /\((\d+)(,\s?\d+)?\)/
end

def initialize_type_map(m) # :nodoc:
super
register_class_with_limit m, %r(String), Type::String
register_class_with_limit m, 'Date', Clickhouse::OID::Date
register_class_with_precision m, %r(datetime)i, Clickhouse::OID::DateTime

register_class_with_limit m, %r(Int8), Type::Integer
register_class_with_limit m, %r(Int16), Type::Integer
register_class_with_limit m, %r(Int32), Type::Integer
register_class_with_limit m, %r(Int64), Type::Integer
register_class_with_limit m, %r(Int128), Type::Integer
register_class_with_limit m, %r(Int256), Type::Integer

register_class_with_limit m, %r(UInt8), Type::UnsignedInteger
register_class_with_limit m, %r(UInt16), Type::UnsignedInteger
register_class_with_limit m, %r(UInt32), Type::UnsignedInteger
register_class_with_limit m, %r(UInt64), Type::UnsignedInteger
#register_class_with_limit m, %r(UInt128), Type::UnsignedInteger #not implemnted in clickhouse
register_class_with_limit m, %r(UInt256), Type::UnsignedInteger
# register_class_with_limit m, %r(Array), Clickhouse::OID::Array
m.register_type(%r(Array)) do |sql_type|
Clickhouse::OID::Array.new(sql_type)
def extract_precision(sql_type)
$1.to_i if sql_type =~ /\((\d+)(,\s?\d+)?\)/
end

def initialize_type_map(m) # :nodoc:
super
register_class_with_limit m, %r(String), Type::String
register_class_with_limit m, 'Date', Clickhouse::OID::Date
register_class_with_precision m, %r(datetime)i, Clickhouse::OID::DateTime

register_class_with_limit m, %r(Int8), Type::Integer
register_class_with_limit m, %r(Int16), Type::Integer
register_class_with_limit m, %r(Int32), Type::Integer
register_class_with_limit m, %r(Int64), Type::Integer
register_class_with_limit m, %r(Int128), Type::Integer
register_class_with_limit m, %r(Int256), Type::Integer

register_class_with_limit m, %r(UInt8), Type::UnsignedInteger
register_class_with_limit m, %r(UInt16), Type::UnsignedInteger
register_class_with_limit m, %r(UInt32), Type::UnsignedInteger
register_class_with_limit m, %r(UInt64), Type::UnsignedInteger
#register_class_with_limit m, %r(UInt128), Type::UnsignedInteger #not implemnted in clickhouse
register_class_with_limit m, %r(UInt256), Type::UnsignedInteger
# register_class_with_limit m, %r(Array), Clickhouse::OID::Array
m.register_type(%r(Array)) do |sql_type|
Clickhouse::OID::Array.new(sql_type)
end
end
end

# In Rails 7 used constant TYPE_MAP, we need redefine method
def type_map
@type_map ||= Type::TypeMap.new.tap { |m| ClickhouseAdapter.initialize_type_map(m) }
end

def _quote(value)
def quote(value)
case value
when Array
'[' + value.map { |v| _quote(v) }.join(', ') + ']'
'[' + value.map { |v| quote(v) }.join(', ') + ']'
else
super
end
Expand Down
2 changes: 1 addition & 1 deletion lib/arel/visitors/clickhouse.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ def aggregate(name, o, collector)

def visit_Arel_Table o, collector
collector = super
collector << ' FINAL ' if o.final
collector << ' FINAL' if o.final
collector
end

Expand Down
Loading

0 comments on commit cc3ecfd

Please sign in to comment.