This document summarizes information relevant to Storm committers and contributors. It includes information about the development processes and policies as well as the tools we use to facilitate those.
Table of Contents
If you are reading this document then you are interested in contributing to the Storm project -- many thanks for that! All contributions are welcome: ideas, documentation, code, patches, bug reports, feature requests, etc. And you do not need to be a programmer to speak up.
This section explains how to perform common activities such as reporting a bug or merging a pull request.
To report a bug you should open an issue in our issue tracker that summarizes the bug. Set the form field "Issue type" to "Bug". If you have not used the issue tracker before you will need to register an account (free), log in, and then click on the blue "Create Issue" button in the top navigation bar.
In order to help us understand and fix the bug it would be great if you could provide us with:
- The steps to reproduce the bug. This includes information about e.g. the Storm version you were using.
- The expected behavior.
- The actual, incorrect behavior.
Feel free to search the issue tracker for existing issues (aka tickets) that already describe the problem; if there is such a ticket please add your information as a comment.
If you want to provide a patch along with your bug report: That is great! In this case please send us a pull request as described in section Create a pull request below. You can also opt to attach a patch file to the issue ticket, but we prefer pull requests because they are easier to work with.
To request a new feature you should open an issue in our issue tracker and summarize the desired functionality. Set the form field "Issue type" to "New feature". If you have not used the issue tracker before you will need to register an account (free), log in, and then click on the blue "Create Issue" button in the top navigation bar.
You can also opt to send a message to the Storm Users mailing list.
Before you set out to contribute code we recommend that you familiarize yourself with the Storm codebase, notably by reading through the Implementation documentation.
If you are interested in contributing code to Storm but do not know where to begin: In this case you should browse our issue tracker for open issues and tasks. You may want to start with beginner-friendly, easier issues (newbie issues and trivial issues) because they require learning about only an isolated portion of the codebase and are a relatively small amount of work.
Please use idiomatic Clojure style, as explained in this Clojure style guide. Another useful reference is the Clojure Library Coding Standards. Perhaps the most important is consistenly writing a clear docstring for functions, explaining the return value and arguments. As of this writing, the Storm codebase would benefit from various style improvements.
Contributions to the Storm codebase should be sent as GitHub pull requests. See section Create a pull request below for details. If there is any problem with the pull request we can iterate on it using the commenting features of GitHub.
-
For small patches, feel free to submit pull requests directly for those patches.
-
For larger code contributions, please use the following process. The idea behind this process is to prevent any wasted work and catch design issues early on.
- Open an issue on our issue tracker if a similar issue does not exist already. If a similar issue does exist, then you may consider participating in the work on the existing issue.
- Comment on the issue with your plan for implementing the issue. Explain what pieces of the codebase you are going to touch and how everything is going to fit together.
- Storm committers will iterate with you on the design to make sure you are on the right track.
- Implement your issue, create a pull request (see below), and iterate from there.
Documentation contributions are very welcome!
You can contribute documentation by pull request, as same as code contribution.
Main directory is docs/, and you can refer to docs/README.md for how to build / test website.
Pull requests should be done against the read-only git repository at https://github.com/apache/storm.
Take a look at Creating a pull request. In a nutshell you need to:
- Fork the Storm GitHub repository at https://github.com/apache/storm/ to your personal GitHub account. See Fork a repo for detailed instructions.
- Commit any changes to your fork.
- Send a pull request to the Storm GitHub repository
that you forked in step 1. If your pull request is related to an existing Storm JIRA ticket -- for instance, because
you reported a bug report via JIRA earlier -- then prefix the title of your pull request with the corresponding JIRA
ticket number (e.g.
STORM-123: ...).
You may want to read Syncing a fork for instructions on how to keep
your fork up to date with the latest changes of the upstream (official) storm repository.
NOTE: The information in this section may need to be formalized via proper project bylaws.
Pull requests are approved with two +1s from committers and need to be up for at least 72 hours for all committers to have a chance to comment. In case it was a committer who sent the pull request than two different committers must +1 the request.
This section applies to committers only.
Important: A pull request must first be properly approved before you are allowed to merge it.
Committers that are integrating patches or pull requests should use the official Apache repository at https://git-wip-us.apache.org/repos/asf/storm.git.
To pull in a merge request you should generally follow the command line instructions sent out by GitHub.
-
Go to your local copy of the Apache git repo, switch to the
masterbranch, and make sure it is up to date.$ git checkout master $ git fetch origin $ git merge origin/master -
Create a local branch for integrating and testing the pull request. You may want to name the branch according to the Storm JIRA ticket associated with the pull request (example:
STORM-1234).$ git checkout -b <local_test_branch> # e.g. git checkout -b STORM-1234 -
Merge the pull request into your local test branch.
$ git pull <remote_repo_url> <remote_branch> -
Assuming that the pull request merges without any conflicts: Update the top-level
CHANGELOG.md, and add in the JIRA ticket number (example:STORM-1234) and ticket description to the change log. Make sure that you place the JIRA ticket number in the commit comments where applicable. -
Run any sanity tests that you think are needed.
-
Once you are confident that everything is ok, you can merge your local test branch into your local
masterbranch, and push the changes back to the official Apache repo.# Pull request looks ok, change log was updated, etc. We are ready for pushing. $ git checkout master $ git merge <local_test_branch> # e.g. git merge STORM-1234 # At this point our local master branch is ready, so now we will push the changes # to the official Apache repo. Note: The read-only mirror on GitHub will be updated # automatically a short while after you have pushed to the Apache repo. $ git push origin master -
The last step is updating the corresponding JIRA ticket. Go to JIRA and resolve the ticket. Be sure to set the
Fix Version/sfield to the version you pushed your changes to. It is usually good practice to thank the author of the pull request for their contribution if you have not done so already.
The following commands must be run from the top-level directory.
# Build the code and run the tests (requires nodejs, python and ruby installed)
# `mvn clean package` will fail because storm-core requires storm-maven-plugin.
# This plugin should be installed before compiling storm-core.
$ mvn clean install
# Build the code and run the tests, with specifying default test timeout (in millisecond)
$ export STORM_TEST_TIMEOUT_MS=10000
$ mvn clean install
# Build the code but skip the tests
$ mvn clean install -DskipTests=true
In case you modified storm.thrift, you have to regenerate thrift code as java and python code before compiling whole project.
cd storm-core/src
sh genthrift.shYou can also run tests selectively via the Clojure REPL. The following example runs the tests in
auth_test.clj, which has the namespace
backtype.storm.security.auth.auth-test.
First, start the REPL from within the relevant sub-project (here: storm-core):
$ cd storm-core/
$ mvn clojure:repl
Now we run the tests in auth_test.clj in the REPL:
;; You can use both absolute as well as relative paths to the .clj file.
(load-file "test/clj/backtype/storm/security/auth/auth_test.clj")
(ns backtype.storm.security.auth.auth-test)
(run-tests)Tip: IDEs such as IntelliJ IDEA support a built-in Clojure REPL, which you can also use to run tests selectively. Sometimes you may find that tests pass/fail depending on which REPL you use, which -- although frustrating -- can be helpful to narrow down errors.
You can create a distribution (like what you can download from Apache) as follows. Note that the instructions below do not use the Maven release plugin because creating an official release is the task of our release manager.
# First, build the code.
$ mvn clean install # you may skip tests with `-DskipTests=true` to save time
# Create the binary distribution.
$ cd storm-dist/binary && mvn package
The last command will create Storm binaries at:
storm-dist/binary/target/apache-storm-<version>.pom
storm-dist/binary/target/apache-storm-<version>.tar.gz
storm-dist/binary/target/apache-storm-<version>.zip
including corresponding *.asc digital signature files.
After running mvn package you may be asked to enter your GPG/PGP credentials (once for each binary file, in fact).
This happens because the packaging step will create *.asc digital signatures for all the binaries, and in the workflow
above your GPG private key will be used to create those signatures.
You can verify whether the digital signatures match their corresponding files:
# Example: Verify the signature of the `.tar.gz` binary.
$ gpg --verify storm-dist/binary/target/apache-storm-<version>.tar.gz.asc
Tests should never rely on timing in order to pass. In Storm can properly test functionality that depends on time by simulating time, which means we do not have to worry about e.g. random delays failing our tests indeterministically.
If you are testing topologies that do not do full tuple acking, then you should be testing using the "tracked
topologies" utilities in backtype.storm.testing.clj. For example,
test-acking (around line 213) tests the acking system in
Storm using tracked topologies. Here, the key is the tracked-wait function: it will only return when both that many
tuples have been emitted by the spouts and the topology is idle (i.e. no tuples have been emitted nor will be emitted
without further input). Note that you should not use tracked topologies for topologies that have tick tuples.
The source code of Storm is managed via git. For a number of reasons there is more than one git repository associated with Storm.
- Committers only: https://git-wip-us.apache.org/repos/asf/storm.git is the official and authoritative git repository for Storm, managed under the umbrella of the Apache Software Foundation. Only official Storm committers will interact with this repository. When you push the first time to this repository git will prompt you for your username and password. Use your Apache user ID and password, i.e. the credentials you configured via https://id.apache.org/ after you were onboarded as a committer.
- Everybody else: https://github.com/apache/storm/ is a read-only mirror of the official git repository. If you are not a Storm committer (most people) this is the repository you should work against. See Development workflow above on how you can create a pull request, for instance.
An automated bot (called ASF GitHub Bot in Storm JIRA) runs periodically to merge changes in the official Apache repo to the read-only GitHub mirror repository, and to merge comments in GitHub pull requests to the Storm JIRA.
Issue tracking includes tasks such as reporting bugs, requesting and collaborating on new features, and administrative activities for release management. As an Apache software project we use JIRA as our issue tracking tool.
The Storm JIRA is available at:
If you do not have a JIRA account yet, then you can create one via the link above (registration is free).
If you have any questions after reading this document, then please reach out to us via the Storm Developers mailing list.
And of course we also welcome any contributions to improve the information in this document!