Kiq

Kiq is a robust and extensible job processing queue that aims for compatibility with Sidekiq, Sidekiq Pro and Sidekiq Enterprise.

Job queuing, processing and reporting are all built on GenStage. That means maximum parallelism with the safety of backpressure as jobs are processed.

Why Kiq?

Many features and architectural choices in Kiq were drawn from existing Elixir job processing packages like Exq, Verk and EctoJob. Each of those packages are great and have varying strenghts, but they lacked seamless interop with Sidekiq Pro or Sidekiq Enterprise jobs.

Sidekiq Pro and Enterprise are amazing pieces of commercial software and worth every cent—your organization should buy them! Kiq is intended as a bridge for your team to interop between Ruby and Elixir. As an organization embraces Elixir it becomes necessary to run some background jobs in Elixir, and it must be just as reliable as when jobs were ran through Sidekiq.

Sidekiq Pro & Enterprise Comaptible Feature Set

Kiq's feature set includes many marquee offerings from Sidekiq, Sidekiq Pro and Sidekiq Enterprise—plus some additional niceties made possible by running on the BEAM. Here is a table highlighting the Kiq's features compared to the various Sidekiq versions:

Feature	Kiq	Sidekiq	Sidekiq Pro	Sidekiq Ent
Max Size Queues	✅	❌	❌	❌
Structured Logging	✅	❌	❌	❌
Scheduled Jobs	✅	✅	✅	✅
Error Handling	✅	✅	✅	✅
Expiring Jobs	✅	❌	✅	✅
Worker Metrics	✅	❌	✅	✅
Reliable Client	✅	❌	✅	✅
Reliable Server	✅	❌	✅	✅
Rolling Restarts	✅	❌	❌	✅
Periodic Jobs	✅	❌	❌	✅
Unique Jobs	✅	❌	❌	✅
Leader Election	✅	❌	❌	✅
Multi Process	✅	❌	❌	✅
Web UI †	❌	✅	✅	✅
Batch Jobs ‡	❌	❌	✅	✅
Encryption ‡	❌	❌	❌	✅
Rate Limiting ‡	❌	❌	❌	✅

• † Kiq relies on Sidekiq's Web UI
• ‡ Planned, but not implemented yet

If a feature isn't supported or planned it is probably for one of these reasons:

1. We get it for free on the BEAM and it isn't necessary, i.e. (safe shutdown, multi-process, rolling restarts)
2. We enable developers to use custom reporters to do it themselves (stats, error reporting)

Design Decisions

• Avoid global and compile time configuration. All configuration can be defined programatically, eliminating the need for hacks like {:system, "REDIS_URL"}.
• Not an application, it is included in your application's supervision tree
• Testing focused, provide helpers and modes to aid testing
• Extensible job handling via GenStage consumers
• Simplified worker definitions to ease job definition and pipelining

Installation

Add kiq to your list of dependencies in mix.exs:

def deps do
  [
    {:kiq, "~> 0.7"}
  ]
end

Then run mix deps.get to install the dependency.

Finally, add the supervisor to your application's supervision tree:

{MyApp.Kiq, []}

Kiq itself is not an application and must be started within your application's supervision tree. All of your application's configuration and custom methods should be put into the supervisor.

Usage

Kiq isn't an application that must be started. Similarly to Ecto, you define one or more Kiq modules within your application. This allows multiple supervision trees with entirely different configurations.

Run the generator to define a Kiq supervisor for your application:

mix kiq.gen.supervisor MyApp.Kiq

Include the module in your application's supervision tree:

defmodule MyApp.Application do
  @moduledoc false

  use Application

  alias MyApp.{Endpoint, Kiq, Repo}

  def start(_type, _args) do
    children = [
      {Repo, []},
      {Endpoint, []},
      {Kiq, []}
    ]

    Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
  end
end

With the supervision tree in place you are ready to start creating workers! The simplest way to create a worker is through the generator:

mix kiq.gen.worker MyApp.Workers.Business

That will define a worker with a perform/1 function where all the magic will happen.

See mix help kiq.gen.worker for additional options.

Check the hexdocs for additional details, configuration options, how to test, defining workers and custom reporters.

Benchmarks

Kiq has a set of benchmarks to track the performance of important operations. Benchmarks are ran using the Benchee library and require Redis to be running.

To run all benchmarks:

mix run bench/bench_helper.exs

Contributing

Clone the repository and run $ mix test to make sure everything is working. For tests to pass, you must have a Redis server running on localhost, port 6379, database 3. You can configure a different host, port and database by setting the REDIS_URL environment variable before testing.

Note that tests will wipe the the configured database on the Redis server multiple times while testing. By default database 3 is used for testing.

License

Kiq is released under the MIT license. See the LICENSE.