From 13336630ce152a1a617a2653189c5b16edf61b08 Mon Sep 17 00:00:00 2001 From: jdmooney Date: Sun, 25 Jun 2023 20:57:04 -0400 Subject: [PATCH 01/14] Minor revisions to multitier doc --- userguide/aws/how-to-set-up-a-multi-tier-system.adoc | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/userguide/aws/how-to-set-up-a-multi-tier-system.adoc b/userguide/aws/how-to-set-up-a-multi-tier-system.adoc index b87ce5c38..cc3c3fb16 100644 --- a/userguide/aws/how-to-set-up-a-multi-tier-system.adoc +++ b/userguide/aws/how-to-set-up-a-multi-tier-system.adoc @@ -25,6 +25,7 @@ The setup procedure includes eight steps: . <> . <> +NOTE: AWS user interfaces change frequently. You may see screens slightly different from those shown below. [[step1]] == Step 1: Login to AWS @@ -118,7 +119,7 @@ First, you need to edit the database properties. should be re Second, you may optionally edit any Kill Bill properties that you need to change from the standard defaults. For more information see the https://docs.killbill.io/latest/userguide_configuration.html[Kill Bill Configuration Guide]. -Save this script in a text file. You will need it in the next step. +Save this script in a text file. You will need it at the end of Step 4.2, below. [[step4]] == Step 4: Launch EC2 Instances @@ -284,7 +285,7 @@ You can now login to Kaui from your browser using the URL `\http:// In addition, you can login to the Kill Bill server using the URL `\http://:8080`. This provides access to certain detailed reports that may be needed for maintenance, including metrics, event logs, and the Swagger API pages. -Repeat the tests for your other instance(s). If these logins succeed, your EC2 instances and your RDS databases are setup properly. +Repeat the tests for your other instance(s). You should also ensure that actions taken in Kaui from one instance are visible from another. If these logins and tests succeed, your EC2 instances and your RDS databases are setup properly. [[step7]] From d399aac6fc401392830d7ab5f583e35f44dc4bbc Mon Sep 17 00:00:00 2001 From: reshmabidikar Date: Thu, 20 Jul 2023 14:18:33 +0530 Subject: [PATCH 02/14] Additional config properties --- ...llbill-configuration-properties-table.adoc | 22 ++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/userguide/platform/includes/killbill-configuration-properties-table.adoc b/userguide/platform/includes/killbill-configuration-properties-table.adoc index df37fea67..92cbdcece 100644 --- a/userguide/platform/includes/killbill-configuration-properties-table.adoc +++ b/userguide/platform/includes/killbill-configuration-properties-table.adoc @@ -40,7 +40,13 @@ KB_org_killbill_payment_plugin_timeout |Timeout for each payment attempt |30 KB_org_killbill_payment_plugin_threads_nb |Number of threads for plugin executor dispatcher |10 |Config File/Environment Variable |Payment |org.killbill.payment.globalLock.retries/ -KB_org_killbill_payment_globalLock_retries |Maximum number of times the system will retry to grab global lock (with a 100ms wait each time) |50 |Config File/Environment Variable +KB_org_killbill_payment_globalLock_retries |Maximum number of times the system will retry to grab global lock (with a 100ms wait each time) |50 |Config File/Environment Variable +|Payment |org.killbill.payment.method.overwrite/ + +KB_org_killbill_payment_method_overwrite |Ability to overwrite an existing payment method from a control plugin |false |Config File/Environment Variable +|Payment |org.killbill.payment.allow.emptyInvoice/ + +KB_org_killbill_payment_allow_emptyinvoices |Ability to allow payments for zero amount invoices |false |Config File/Environment Variable |Invoice |org.killbill.invoice.maxNumberOfMonthsInFuture/ KB_org_killbill_invoice_maxNumberOfMonthsInFuture |Maximum target date to consider when generating an invoice |36 |Per-Tenant/Config File/Environment Variable @@ -91,7 +97,10 @@ KB_org_killbill_invoice_parkAccountsWithUnknownUsage |Whether to park accounts KB_org_killbill_invoice_rescheduleIntervalOnLock |Time delay to reschedule an invoice run when lock is held |0s |Per-Tenant/Config File/Environment Variable |Invoice |org.killbill.invoice.maxInvoiceLimit/ -KB_org_killbill_invoice_maxInvoiceLimit |How far back in time should invoice generation look at |P200Y |Per-Tenant/Config File/Environment Variable +KB_org_killbill_invoice_maxInvoiceLimit |How far back in time should invoice generation look at |P200Y |Per-Tenant/Config File/Environment Variable +|Invoice |org.killbill.rescheduleIntervalOnLock/ + +KB_org_killbill_rescheduleIntervalOnLock |Tme delay to reschedule an invoice run when lock is held |30s, 1m, 1m, 3m, 3m, 10m |Per-Tenant/Config File/Environment Variable |Database |org.killbill.dao.url/ KB_org_killbill_dao_url |The jdbc url for the database |jdbc:h2:file:/var/tmp/killbill;MODE=MYSQL; @@ -817,7 +826,14 @@ KB_org_killbill_security_skipAuthForPlugins |Specifies whether authentication |Security |killbill.server.ldap/- |Coming Soon |false |System property |Security |killbill.server.okta/- |Coming Soon |false |System property |Security |killbill.server.auth0/- |Coming Soon |false |System Property -|Security |killbill.server.rbac/- |Coming Soon |true |System Property +|Security |killbill.server.rbac/- |Coming Soon |true |System Property + +|Logging |killbill.server.log.obfuscate.keywords |Keywords to obfuscate |accountnumber,authenticationdata,bankaccountnumber,banknumber,bic,cardvalidationnum,cavv,ccFirstName,ccLastName,ccNumber,ccTrackData, ccVerificationValue,ccvv,cvNumber,cvc,cvv,email,iban,name,number,password, xid |System Property + +|Logging |killbill.server.log.obfuscate.patterns |Key value patterns to use |\s*=\s*'([^']+)',\s*=\s*"([^"]+)" |System Property + +|Logging |killbill.server.log.obfuscate.patterns.separator |Key value pattern separator |, |System Property + |Cache |org.killbill.catalog.frequentValuesCacheSize/- |Coming Soon |1000 |System property |Cache |org.killbill.cache.disabled/ From 44d6651a1e5b509cffa5db7538b84711bac86bcc Mon Sep 17 00:00:00 2001 From: vnandwana Date: Wed, 26 Jul 2023 19:57:25 +0530 Subject: [PATCH 03/14] Update how-to-set-up-a-multi-tier-system.adoc * Added the script that can be used to update KB and Kaui properties. * Added NOTE where needed. * Corrected orders of steps. --- .../how-to-set-up-a-multi-tier-system.adoc | 223 +++++++++--------- 1 file changed, 117 insertions(+), 106 deletions(-) diff --git a/userguide/aws/how-to-set-up-a-multi-tier-system.adoc b/userguide/aws/how-to-set-up-a-multi-tier-system.adoc index b87ce5c38..ea7dac37d 100644 --- a/userguide/aws/how-to-set-up-a-multi-tier-system.adoc +++ b/userguide/aws/how-to-set-up-a-multi-tier-system.adoc @@ -78,55 +78,123 @@ image::../assets/aws/multitier-connectivity-and-security.png[align=center] === 2.3. Setup the Security Rules -Lastly, on the *Connectivity and Security* panel, locate and click on the link for the default VPC security group. You will need to add an inbound security rule, because the database by default does not allow external access. In the panel for this group, click on *Inbound Rules* and select *Edit Inbound Rules*. Next click on *Add rule*. In the *Type* column select `MYSQL/Aurora`. The port will be set to 3306 automatically. In the *Source* column, click on the search icon and select `0.0.0.0/0`. Finally, click on *Save Rules* in the bottom right. Your database is ready to go. +Lastly, on the *Connectivity and Security* panel of the DB writer instance, locate and click on the link for the default VPC security group. You will need to add an inbound security rule because the database by default does not allow external access. In the panel for this group, click on *Inbound Rules* and select *Edit Inbound Rules*. Next, click on *Add rule*. In the *Type* column select `MYSQL/Aurora`. The port will be set to 3306 automatically. In the *Source* column, click on the search icon and select `0.0.0.0/0`. Finally, click on *Save Rules* in the bottom right. Your database is ready to go. + +[NOTE] +==== +The current security configuration allows open access to the database from any IP address, which can pose significant security risks. Exposing a database to the public internet without proper restrictions is generally not recommended as it may lead to potential vulnerabilities and make your data susceptible to attacks. + +To ensure robust protection, it is essential to promptly update the database security settings after completing the Multi-Tier setup. We recommend restricting access so that only EC2 instances running Kill Bill and Kaui are permitted to connect to the database. +==== [[step3]] -== Step 3: Edit the Configuration Script +== Step 3: Create the Databases + +Kill Bill requires two databases, with the names `killbill` and `kaui`. We provide predefined schemas for these databases. -To set up the EC2 instances you will need to provide them with information needed to connect to the databases. We provide a brief configuration script to simplify this process. The template for this script is as follows: +To create the databases, you will need to login to one of your instances as described above. Once you are logged in, you can use the `mysql` command to create the two databases `killbill` and `kaui`. The credentials for this command are the same ones you set up for the database and copied to the configuration file. Note that the should *not* include the port number. +The password will not be echoed when it is typed. [source,bash] ---- -#!/bin/bash +> mysql -h -u -p +Enter Password: +mysql> create database killbill; +mysql> create database kaui; +mysql> exit +---- +The next step is to install the schemas. These can be found at: + +* killbill schema: `https://docs.killbill.io/latest/ddl.sql` +* kaui schema: `https://github.com/killbill/killbill-admin-ui/blob/master/db/ddl.sql` -DB_PROPS="/var/tmp/db.props.$$" -KB_PROPS="/var/tmp/kb.props.$$" - -cat <<_EOF > $DB_PROPS -# -# EDIT THE FOLLOWING DB PROPERTIES AS NEEDED: -# -DB_SERVER=:3306 -DB_USER= -DB_PASSWORD= -KILLBILL_DB_NAME=killbill -KAUI_DB_NAME=kaui -_EOF - -cat <<_EOF > $KB_PROPS -# -# EDIT THE FOLLOWING KB PROPERTIES AS NEEDED: -# -org.killbill.dontexist=foo -_EOF - -su -l -c "cd /var/lib/tomcat/bin && /var/lib/tomcat/bin/updateProperties.sh $DB_PROPS $KB_PROPS" tomcat +One easy way to do this is to return to your local computer (type `exit`) and download the schemas (give them distinct names), then use the `sftp` command to upload them to your EC2 instance home directory with the commands: + +[source,bash] +---- +sftp -i PRIVATE_KEY.pem ubuntu@INSTANCE_IP +put killbill.ddl +put kaui.ddl +exit ---- -First, you need to edit the database properties. should be replaced by the full name of the DB endpoint, as given in the *Connectivity and Security* panel (see above). The port number 3306 is required. and should be set to the administrator credentials you have chosen for the RDS instance. +Once the files are successfully uploaded, login again to your instance using the `ssh` command. You can now install the schemas: + +[source,bash] +---- +> mysql -h -u -p < killbill.ddl + +> mysql -h -u -p < kaui.ddl +---- +To ensure that the databases are setup correctly, login to `mysql` again, then try the SHOW TABLES command: + +[source,bash] +---- +> mysql -h -u -p -Second, you may optionally edit any Kill Bill properties that you need to change from the standard defaults. For more information see the https://docs.killbill.io/latest/userguide_configuration.html[Kill Bill Configuration Guide]. +use killbill; +show tables; +use kaui; +show tables; +exit +---- -Save this script in a text file. You will need it in the next step. +Each `show tables` command should display a list of table names for the database. [[step4]] -== Step 4: Launch EC2 Instances +== Step 4: Edit the Configuration Script + +To configure the EC2 instances and establish their connection to the databases, you'll need to provide essential information. Fortunately, Kill Bill and Kaui are equipped to read environment variables, making the setup more straightforward. For your convenience, we have a concise configuration script available to streamline this process. Below is the template for the script: + + +[source,bash] +---- +#!/bin/bash + +db_host="" +db_port="" +db_user="" +db_password="" +kb_admin_password="" + +cat << EOF > /etc/environment +KB_org_killbill_dao_url=jdbc:mysql://$db_host:$db_port/killbill +KB_org_killbill_dao_user=$db_user +KB_org_killbill_dao_password=$db_password + +KB_org_killbill_billing_osgi_dao_url=jdbc:mysql://$db_host:$db_port/killbill +KB_org_killbill_billing_osgi_dao_user=$db_user +KB_org_killbill_billing_osgi_dao_password=$db_password + +KB_ADMIN_PASSWORD=$kb_admin_password + +KAUI_DB_ADAPTER=mysql2 +KAUI_DB_URL=jdbc:mysql://$db_host:$db_port/kaui +KAUI_DB_USERNAME=$db_user +KAUI_DB_PASSWORD=$db_password +KAUI_KILLBILL_URL=http://127.0.0.1:8080 +EOF +---- + +In the above script, replace the value of `db_host` with the full name of the DB writer endpoint obtained from the Connectivity and Security panel, as indicated earlier. Ensure to specify the appropriate database port number (`3306` for MySQL or `5432` for PostgreSQL) by setting `db_port`. The `kb_admin_password` value will be utilized as the admin password for Kaui login. + +Optionally, you may choose to customize other Kill Bill properties based on your specific needs. For more detailed information on available configuration options, please refer to the documentation provided at: https://docs.killbill.io/latest/userguide_configuration.html. + +[NOTE] +==== +The Kaui properties present in the provided template are required for proper functioning. Missing any of these properties may prevent the Kill Bill service from starting successfully. Ensure to set all the necessary properties correctly to ensure a smooth setup process. +==== + +Save this script to a file as it will be necessary during the launch of EC2 instances. + +[[step5]] +== Step 5: Launch EC2 Instances The next step is to launch the number of EC2 instances you want, all based on the Kill Bill single AMI. -=== 4.1. Subscribe to the AMI +=== 5.1. Subscribe to the AMI To start the installation process, point your browser to the +++ @@ -145,7 +213,7 @@ Click *Continue to Subscribe*. The next page will give the AWS Terms and Conditi Accept the terms if asked. You will then see a new message confirming that you have subscribed. Next, click *Continue to Configuration*. -=== 4.2. Configure the Instances +=== 5.2. Configure the Instances The next page will give several configuration options: @@ -169,22 +237,7 @@ image::../assets/aws/single-tier-keypair.png[align=center] The key pair provides the credentials you will need to login to your EC2 instance. For details about key pairs, see the https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html[AWS documentation]. We recommend that you create a new key pair. Click *Create Key Pair* to display a pane to be used for the creation. Give the key pair a simple, easy to remember name such as `My-Key-Pair`. Do not change the other options on this pane. Then click *Download Key Pair*. Important: You *must* save the private key that will be generated in this step. If you lose this key, you will *not* be able to login to your instance. -Finally, scroll to the bottom and open the section labeled *Advanced Details*. You will see a long list of settings. At the very bottom of this list is a box headed *User Settings*. Copy your configuration file into this box. - -=== 4.3. Launch your Instances - -When the key pair is generated, click *Launch Instances*. You should see the screen below: - -image::../assets/aws/multitier-launching.png[align=center] - -Your instances are finally launching! To follow what is happening on the EC2 Dashboard, scroll all the way down to the bottom, and click *View all instances* at the bottom right. This will take you to the *Instances* screen which is part of the EC2 Dashboard. - -image::../assets/aws/multitier-instances.png[align=center] - - -In a short time, the *Instance State* for each instance should indicate *Running*. You will need to scroll to the right to see all of the information available about your instances. In particular, make a note of the *Availability Zone* (such as `us-east-1a`) assigned to each instance. You will need this information later. - -=== 4.4. Setup Security Rules +=== 5.3. Setup Security Rules The next step is to scroll down in the menu on the left side to select *Security Groups*. You should see a list of two or more groups. Select the group whose name begins with `Kill Bill on AWS`, then scroll to the bottom and select the tab for *Inbound Rules*. You should see: @@ -203,79 +256,37 @@ Your Inbound Rules should now look like this: image::../assets/aws/multitier-inbound-new.png[align=center] +=== 5.4. Add user data script +Finally, scroll to the bottom and open the section labeled *Advanced Details*. You will see a long list of settings. At the very bottom of this list is a box headed *User data*. Paste the script created in Step 4 here. -=== 4.5. Login to an Instance - -Now that your instances are set up, you need to ensure that you can login to them for configuration and maintenance when needed. To login, use the secure shell command: - -`ssh -i .pem ubuntu@` - -Here is the pathname where you have stored the private key that was downloaded when you generated your key pair, and is the IPV4 address for any one of your instances as described earlier. The private key will not work unless its access controls are set to readable by the owner only. - -On Windows versions before Windows 10, you may need to download a program called PuTTY to enable `ssh`. On Windows 10 `ssh` is available but may need to be activated through the Settings screen. - -The first time you login, you will see a warning message asking if you want to add this host to your list of hosts. You should answer `yes`. - -You will now be able to explore your instance and perform various configuration and maintenance tasks. To exit from your login, type `exit`. +=== 5.5. Launch your Instances -NOTE: We recommend that you *remove* the SSH rule from your security group when you are *not* doing configuration or maintenance. +When the key pair is generated, click *Launch Instances*. You should see the screen below: +image::../assets/aws/multitier-launching.png[align=center] -[[step5]] -== Step 5: Create the Databases +Your instances are finally launching! To follow what is happening on the EC2 Dashboard, scroll all the way down to the bottom, and click *View all instances* at the bottom right. This will take you to the *Instances* screen which is part of the EC2 Dashboard. -Kill Bill requires two databases, with the names `killbill` and `kaui`. We provide predefined schemas for these databases. +image::../assets/aws/multitier-instances.png[align=center] -To create the databases, you will need to login to one of your instances as described above. Once you are logged in, you can use the `mysql` command to create the two databases `killbill` and `kaui`. The credentials for this command are the same ones you set up for the database and copied to the configuration file. Note that the should *not* include the port number. -The password will not be echoed when it is typed. +In a short time, the *Instance State* for each instance should indicate *Running*. You will need to scroll to the right to see all of the information available about your instances. In particular, make a note of the *Availability Zone* (such as `us-east-1a`) assigned to each instance. You will need this information later. -[source,bash] ----- -> mysql -h -u -p -Enter Password: -mysql> create database killbill; -mysql> create database kaui; -mysql> exit ----- -The next step is to install the schemas. These can be found at: +=== 5.6. Login to an Instance -* killbill schema: `https://docs.killbill.io/latest/ddl.sql` -* kaui schema: `https://github.com/killbill/killbill-admin-ui/blob/master/db/ddl.sql` +Now that your instances are set up, you need to ensure that you can login to them for configuration and maintenance when needed. To login, use the secure shell command: -One easy way to do this is to return to your local computer (type `exit`) and download the schemas (give them distinct names), then use the `sftp` command to upload them to your EC2 instance home directory with the commands: +`ssh -i .pem ubuntu@` -[source,bash] ----- -sftp -i PRIVATE_KEY.pem ubuntu@INSTANCE_IP -put killbill.ddl -put kaui.ddl -exit ----- +Here is the pathname where you have stored the private key that was downloaded when you generated your key pair, and is the IPV4 address for any one of your instances as described earlier. The private key will not work unless its access controls are set to readable by the owner only. -Once the files are successfully uploaded, login again to your instance using the `ssh` command. You can now install the schemas: +On Windows versions before Windows 10, you may need to download a program called PuTTY to enable `ssh`. On Windows 10 `ssh` is available but may need to be activated through the Settings screen. -[source,bash] ----- -> mysql -h DB-INSTANCE-NAME -u ADMIN-NAME -p killbill < killbill.ddl -Enter Password: -> mysql -h DB-INSTANCE-NAME -u ADMIN-NAME -p kaui < kaui.ddl -Enter Password: ----- -To ensure that the databases are setup correctly, login to `mysql` again, then try the SHOW TABLES command: +The first time you login, you will see a warning message asking if you want to add this host to your list of hosts. You should answer `yes`. -[source,bash] ----- -> mysql -h DB-INSTANCE-NAME -u ADMIN-NAME -p -Enter Password: -use killbill -show tables; -use kaui -show tables; -exit ----- +You will now be able to explore your instance and perform various configuration and maintenance tasks. To exit from your login, type `exit`. -Each `show tables` command should display a list of table names for the database. +NOTE: We recommend that you *remove* the SSH rule from your security group when you are *not* doing configuration or maintenance. [[step6]] == Step 6: Perform Initial Testing From f9dc9141b18d482e32d1419608a7d2818f7f6f97 Mon Sep 17 00:00:00 2001 From: KBbitsP <75751774+KBbitsP@users.noreply.github.com> Date: Fri, 28 Jul 2023 22:25:52 +0530 Subject: [PATCH 04/14] Changes to Ldap/Okta/Auth0 flow docs --- userguide/tutorials/user_management.adoc | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/userguide/tutorials/user_management.adoc b/userguide/tutorials/user_management.adoc index de27266f0..80bb01819 100644 --- a/userguide/tutorials/user_management.adoc +++ b/userguide/tutorials/user_management.adoc @@ -168,7 +168,7 @@ The following System Property should also be set: [source,properties] ---- -killbill.server.ldap=true +-Dkillbill.server.ldap=true ---- Notes: @@ -199,7 +199,7 @@ The following https://docs.killbill.io/latest/userguide_configuration.html#syste [source,properties] ---- -killbill.server.okta=true +-Dkillbill.server.okta=true ---- Notes: @@ -244,7 +244,7 @@ The following System Property should also be set: [source,properties] ---- -killbill.server.auth0=true +-Dkillbill.server.auth0=true ---- Notes: @@ -290,7 +290,11 @@ Roles can be created by going to `/role_definitions/new`. If you are delegating roles and users management to a third-party system, Kaui only needs to know the login of the users. Password and roles being managed outside of Kill Bill, you need to access your third-party system to set and update these. -Note that there is no auto-discovery of logins: you need to enter all logins from your system manually in Kaui before such users can access the UI (check the box "Managed externally" when adding these users). +Note that there is no auto-discovery of logins: you need to enter all logins from your system manually in Kaui before such users can access the UI (check the box "Managed externally" when adding these users). At the time of creating the user is Kaui, the username must match the one used in the authenticating system. + +When adding the user, the password field will be greyed out. Kaui does not require a password, as authentication will be handled by one of the above systems. + +At the time of login, the user will visit the Kaui login page and enter the username and password created in the authenticating system. Since Kaui is already aware of the username, a request will be sent to KillBill, which will then redirect the authentication request to the authenticating system. Upon receiving a successful response from Okta, Kaui will log in the user. === Users and tenants From 8d5b41d7688e96b127c3056bc871cb1cd0e0c21d Mon Sep 17 00:00:00 2001 From: KBbitsP <75751774+KBbitsP@users.noreply.github.com> Date: Fri, 28 Jul 2023 22:31:12 +0530 Subject: [PATCH 05/14] Update user_management.adoc --- userguide/tutorials/user_management.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/userguide/tutorials/user_management.adoc b/userguide/tutorials/user_management.adoc index 80bb01819..b267babac 100644 --- a/userguide/tutorials/user_management.adoc +++ b/userguide/tutorials/user_management.adoc @@ -294,7 +294,7 @@ Note that there is no auto-discovery of logins: you need to enter all logins fro When adding the user, the password field will be greyed out. Kaui does not require a password, as authentication will be handled by one of the above systems. -At the time of login, the user will visit the Kaui login page and enter the username and password created in the authenticating system. Since Kaui is already aware of the username, a request will be sent to KillBill, which will then redirect the authentication request to the authenticating system. Upon receiving a successful response from Okta, Kaui will log in the user. +At the time of login, the user will visit the Kaui login page and enter the username and password created in the authenticating system. Since Kaui is already aware of the username, a request will be sent to KillBill, which will then redirect the authentication request to the authenticating system. Upon receiving a successful response from the authenticating system, Kaui will log in the user. === Users and tenants From 1492efda4dda40db0f282dd0db757d71c3a8e848 Mon Sep 17 00:00:00 2001 From: Pierre-Alexandre Meyer Date: Fri, 28 Jul 2023 22:18:13 +0100 Subject: [PATCH 06/14] Update user_management.adoc --- userguide/tutorials/user_management.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/userguide/tutorials/user_management.adoc b/userguide/tutorials/user_management.adoc index b267babac..b4d3e11a6 100644 --- a/userguide/tutorials/user_management.adoc +++ b/userguide/tutorials/user_management.adoc @@ -294,7 +294,7 @@ Note that there is no auto-discovery of logins: you need to enter all logins fro When adding the user, the password field will be greyed out. Kaui does not require a password, as authentication will be handled by one of the above systems. -At the time of login, the user will visit the Kaui login page and enter the username and password created in the authenticating system. Since Kaui is already aware of the username, a request will be sent to KillBill, which will then redirect the authentication request to the authenticating system. Upon receiving a successful response from the authenticating system, Kaui will log in the user. +At the time of login, the user will visit the Kaui login page and enter the username and password created in the authenticating system. Since Kaui is already aware of the username, a request will be sent to Kill Bill, which will then redirect the authentication request to the authenticating system. Upon receiving a successful response from the authenticating system, Kaui will log in the user. === Users and tenants From 6359d10c4095563de97ce688199fd1dbf9b21e79 Mon Sep 17 00:00:00 2001 From: reshmabidikar Date: Fri, 4 Aug 2023 15:56:19 +0530 Subject: [PATCH 07/14] Database migration docs - first draft --- .../how-to-upgrade-the-database.adoc | 18 +++ .../how-to-upgrade-plugin-database.adoc | 126 ++++++++++++++++++ .../how-to-upgrade-the-killbill-database.adoc | 124 +++++++++++++++++ 3 files changed, 268 insertions(+) create mode 100644 userguide/database-migration/how-to-upgrade-the-database.adoc create mode 100644 userguide/database-migration/includes/how-to-upgrade-plugin-database.adoc create mode 100644 userguide/database-migration/includes/how-to-upgrade-the-killbill-database.adoc diff --git a/userguide/database-migration/how-to-upgrade-the-database.adoc b/userguide/database-migration/how-to-upgrade-the-database.adoc new file mode 100644 index 000000000..3490db698 --- /dev/null +++ b/userguide/database-migration/how-to-upgrade-the-database.adoc @@ -0,0 +1,18 @@ += How to upgrade the Kill Bill Database + +== Overview +Often, when you are upgrading to a higher Kill Bill version, you would also need to upgrade the Kill Bill/plugin database. This document is a step-by-step guide on how the database upgrade can be performed. + +== Prerequisites + +Ensure that you have https://github.com/killbill/killbill-cloud/tree/master/kpm#kpm-installation[kpm] installed. + +== Upgrading Kill Bill Database + +[[step_1]] +include::{sourcedir}/database-migration/includes/how-to-upgrade-the-killbill-database.adoc[] + +== Upgrading Plugin Database + +[[step_1]] +include::{sourcedir}/database-migration/includes/how-to-upgrade-plugin-database.adoc[] \ No newline at end of file diff --git a/userguide/database-migration/includes/how-to-upgrade-plugin-database.adoc b/userguide/database-migration/includes/how-to-upgrade-plugin-database.adoc new file mode 100644 index 000000000..fa2eacd11 --- /dev/null +++ b/userguide/database-migration/includes/how-to-upgrade-plugin-database.adoc @@ -0,0 +1,126 @@ +[[step_1]] +=== Step 1 - Obtain the migrations to run + +The `kpm migrate` command can be used to obtain database migrations. + +*Command* + +[source,bash] +---- +kpm migrations github_repository from_tag to_tag +---- + +*Example* + +To download the migration files to migrate the analytics plugin from `8.0.0` to `8.1.0`: + +[source,bash] +---- +kpm migration killbill-analytics-plugin analytics-plugin-8.0.0 analytics-plugin-8.1.0 +---- + +This command creates a temporary folder with the migration SQL statements and displays its path. + +[NOTE] +*Note:* Because the implementation relies on the GitHub API, unauthenticated requests are subject to rate limiting. To work around it, generate a token via https://github.com/settings/tokens (default public, read-only, permissions will work) and specify it to KPM via `kpm migrations killbill killbill-0.22.32 killbill-0.24.0 --token=TOKEN` + +=== Step 2 - Execute the migrations + +The migrations can be executed either manually or via https://flywaydb.org/[Flyway]. + +==== Executing migrations manually + +Execute the SQL scripts obtained above directly on your database. + +==== Executing migrations via the Kill Bill Flywar Jar in Dev environments + +We publish a https://repo1.maven.org/maven2/org/kill-bill/billing/killbill-util/0.24.4/killbill-util-0.24.4-flyway.jar[killbill-flyway.jar] jar which is a wrapper around the Flyway utility and can be used to execute the migrations. In order to execute the migrations via this jar, you can do the following: + +1. If Flyway is being used the first time, run the `baseline` command to create the `schema_history` table. ++ +*Command* ++ +[source, bash] +---- +java -jar killbill-flyway.jar -url=jdbc:mysql://127.0.0.1:3306/ -user= -password= -table=_schema_history baseline +---- ++ +*Example* ++ +[source, bash] +---- +java -jar killbill-flyway.jar -url=jdbc:mysql://127.0.0.1:3306/killbill -user=root -password=killbill -table=analytics_schema_history baseline +---- + ++ +2. Run the `migrate` command to execute the migrations. ++ +*Command* ++ +[source, bash] +---- +java -jar killbill-flyway.jar -url=jdbc:mysql://127.0.0.1:3306/ -user= -password= -locations=filesystem: -table=_schema_history migrate +---- ++ +*Example* ++ +[source, bash] +---- +java -jar killbill-flyway.jar -url=jdbc:mysql://127.0.0.1:3306/killbill -user=root -password=killbill -locations=filesystem:C:/var/migrations -table=analytics_schema_history migrate +---- + +==== Executing migrations via the Kill Bill Flywar Jar in Prod environments + +In production environments, database access is often restricted and developers don’t necessarily have rights to execute DDL commands (i.e. CREATE, ALTER, DROP, etc. statements). In such cases, migrations can be executed as follows: + +1. Create the `schema_history` table manually by running the following SQL statements: ++ +[source, sql] +---- +CREATE TABLE `_schema_history` ( + `installed_rank` int(11) NOT NULL, + `version` varchar(50) DEFAULT NULL, + `description` varchar(200) NOT NULL, + `type` varchar(20) NOT NULL, + `script` varchar(1000) NOT NULL, + `checksum` int(11) DEFAULT NULL, + `installed_by` varchar(100) NOT NULL, + `installed_on` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + `execution_time` int(11) NOT NULL, + `success` tinyint(1) NOT NULL, + PRIMARY KEY (`installed_rank`), + KEY `schema_history_s_idx` (`success`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +insert into _schema_history (installed_rank, version, description, type, script, installed_by, installed_on, execution_time, success) VALUES (1, 1, '<< Flyway Baseline >>', 'BASELINE', '<< Flyway Baseline >>', 'admin', NOW(), 0, 1); +---- ++ +Ensure that you replace `` with the name of a plugin like `analytics`. ++ +2. Run the `dryRunMigrate` command. This simply lists the DDL statements required for the database upgrade but does not actually run them. ++ +*Command* ++ +[source, bash] +---- +java -jar killbill-flyway.jar -url=jdbc:mysql://127.0.0.1:3306/ -user= -password= -locations=filesystem: -table=_schema_history dryRunMigrate +---- ++ +*Example* ++ +[source, bash] +---- +java -jar killbill-flyway.jar -url=jdbc:mysql://127.0.0.1:3306/killbill -user=root -password=killbill -locations=filesystem:C:/var/migrations -table=analytics_schema_history dryRunMigrate +---- + + +==== Executing migrations via the Flyway commandline tool + +The migrations can also be executed via the https://documentation.red-gate.com/fd/command-line-184127404.html[Flyway commandline tool] as follows: + +[source, bash] +---- +flyway -url=jdbc:mysql://127.0.0.1:3306/ -user= -password= -table=_schema_history baseline # first time only + +flyway -url=jdbc:mysql://127.0.0.1:3306/ -user= -password= -locations=filesystem: -table=_schema_history migrate +---- diff --git a/userguide/database-migration/includes/how-to-upgrade-the-killbill-database.adoc b/userguide/database-migration/includes/how-to-upgrade-the-killbill-database.adoc new file mode 100644 index 000000000..a0a8563ad --- /dev/null +++ b/userguide/database-migration/includes/how-to-upgrade-the-killbill-database.adoc @@ -0,0 +1,124 @@ +[[step_1]] +=== Step 1 - Obtain the migrations to run + +The `kpm migrate` command can be used to obtain database migrations. + +*Command* + +[source,bash] +---- +kpm migrations github_repository from_tag to_tag +---- + +*Example* + +To download the migration files to migrate from Kill Bill `0.22.32` to `0.24.0`: + +[source,bash] +---- +kpm migrations killbill killbill-0.22.32 killbill-0.24.0 +---- + +This command creates a temporary folder with the migration SQL statements and displays its path. + +[NOTE] +*Note:* Because the implementation relies on the GitHub API, unauthenticated requests are subject to rate limiting. To work around it, generate a token via https://github.com/settings/tokens (default public, read-only, permissions will work) and specify it to KPM via `kpm migrations killbill killbill-0.22.32 killbill-0.24.0 --token=TOKEN` + +=== Step 2 - Execute the migrations + +The migrations can be executed either manually or via https://flywaydb.org/[Flyway]. + +==== Executing migrations manually + +Execute the SQL scripts obtained above directly in database. + +==== Executing migrations via the Kill Bill Flywar Jar in Dev environments + +We publish a https://repo1.maven.org/maven2/org/kill-bill/billing/killbill-util/0.24.4/killbill-util-0.24.4-flyway.jar[killbill-flyway.jar] jar which is a wrapper around the Flyway utility and can be used to execute the migrations. In order to execute the migrations via this jar, you can do the following: + +1. If Flyway is being used the first time, run the `baseline` command to create the `schema_history` table. ++ +*Command* ++ +[source, bash] +---- +java -jar killbill-flyway.jar -url=jdbc:mysql://127.0.0.1:3306/ -user= -password= baseline +---- ++ +*Example* ++ +[source, bash] +---- +java -jar killbill-flyway.jar -url=jdbc:mysql://127.0.0.1:3306/killbill -user=root -password=killbill baseline +---- + ++ +2. Run the `migrate` command to execute the migrations. ++ +*Command* ++ +[source, bash] +---- +java -jar killbill-flyway.jar -url=jdbc:mysql://127.0.0.1:3306/ -user= -password= -locations=filesystem: migrate +---- ++ +*Example* ++ +[source, bash] +---- +java -jar killbill-flyway.jar -url=jdbc:mysql://127.0.0.1:3306/killbill -user=root -password=killbill -locations=filesystem:C:/var/migrations migrate +---- + +==== Executing migrations via the Kill Bill Flywar Jar in Prod environments + +In production environments, database access is often restricted and developers don’t necessarily have rights to execute DDL commands (i.e. CREATE, ALTER, DROP, etc. statements). In such cases, migrations can be executed as follows: + +1. Create the `schema_history` table manually by running the following SQL statements: ++ +[source, sql] +---- +CREATE TABLE `schema_history` ( + `installed_rank` int(11) NOT NULL, + `version` varchar(50) DEFAULT NULL, + `description` varchar(200) NOT NULL, + `type` varchar(20) NOT NULL, + `script` varchar(1000) NOT NULL, + `checksum` int(11) DEFAULT NULL, + `installed_by` varchar(100) NOT NULL, + `installed_on` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + `execution_time` int(11) NOT NULL, + `success` tinyint(1) NOT NULL, + PRIMARY KEY (`installed_rank`), + KEY `schema_history_s_idx` (`success`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +insert into schema_history (installed_rank, version, description, type, script, installed_by, installed_on, execution_time, success) VALUES (1, 1, '<< Flyway Baseline >>', 'BASELINE', '<< Flyway Baseline >>', 'admin', NOW(), 0, 1); +---- ++ +2. Run the `dryRunMigrate` command. This simply lists the DDL statements required for the database upgrade but does not actually run them. ++ +*Command* ++ +[source, bash] +---- +java -jar killbill-flyway.jar -url=jdbc:mysql://127.0.0.1:3306/ -user= -password= -locations=filesystem: dryRunMigrate +---- ++ +*Example* ++ +[source, bash] +---- +java -jar killbill-flyway.jar -url=jdbc:mysql://127.0.0.1:3306/killbill -user=root -password=password -locations=filesystem:C:/var/migrations dryRunMigrate +---- + + +==== Executing migrations via the Flyway commandline tool + +The migrations can also be executed via the https://documentation.red-gate.com/fd/command-line-184127404.html[Flyway commandline tool] as follows: + +[source, bash] +---- +flyway -url=jdbc:mysql://127.0.0.1:3306/ -user= -password= baseline # first time only + +flyway -url=jdbc:mysql://127.0.0.1:3306/ -user= -password= -locations=filesystem: migrate +---- From f80be3597c38d257b20497cd125e59eaada1bbf8 Mon Sep 17 00:00:00 2001 From: vnandwana Date: Wed, 9 Aug 2023 01:01:37 +0530 Subject: [PATCH 08/14] Update how-to-set-up-a-multi-tier-system.adoc Addressed review comments --- userguide/aws/how-to-set-up-a-multi-tier-system.adoc | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/userguide/aws/how-to-set-up-a-multi-tier-system.adoc b/userguide/aws/how-to-set-up-a-multi-tier-system.adoc index ea7dac37d..bdb22d446 100644 --- a/userguide/aws/how-to-set-up-a-multi-tier-system.adoc +++ b/userguide/aws/how-to-set-up-a-multi-tier-system.adoc @@ -78,7 +78,7 @@ image::../assets/aws/multitier-connectivity-and-security.png[align=center] === 2.3. Setup the Security Rules -Lastly, on the *Connectivity and Security* panel of the DB writer instance, locate and click on the link for the default VPC security group. You will need to add an inbound security rule because the database by default does not allow external access. In the panel for this group, click on *Inbound Rules* and select *Edit Inbound Rules*. Next, click on *Add rule*. In the *Type* column select `MYSQL/Aurora`. The port will be set to 3306 automatically. In the *Source* column, click on the search icon and select `0.0.0.0/0`. Finally, click on *Save Rules* in the bottom right. Your database is ready to go. +Lastly, on the *Connectivity and Security* panel, locate and click on the link for the default VPC security group. You will need to add an inbound security rule because the database by default does not allow external access. In the panel for this group, click on *Inbound Rules* and select *Edit Inbound Rules*. Next, click on *Add rule*. In the *Type* column select `MYSQL/Aurora`. The port will be set to 3306 automatically. In the *Source* column, click on the search icon and select `0.0.0.0/0`. Finally, click on *Save Rules* in the bottom right. Your database is ready to go. [NOTE] ==== @@ -92,7 +92,9 @@ To ensure robust protection, it is essential to promptly update the database sec Kill Bill requires two databases, with the names `killbill` and `kaui`. We provide predefined schemas for these databases. -To create the databases, you will need to login to one of your instances as described above. Once you are logged in, you can use the `mysql` command to create the two databases `killbill` and `kaui`. The credentials for this command are the same ones you set up for the database and copied to the configuration file. Note that the should *not* include the port number. +To create the databases, you will need to login to one of your instances as described above. Once you are logged in, you can use the `mysql` command to create the two databases `killbill` and `kaui`. The credentials required for this command are the same ones you set up for the database in step 2.1 above. + +Note that the host should *not* include the port number and there is no space after `-h` and `-u` options. The password will not be echoed when it is typed. @@ -177,7 +179,7 @@ KAUI_KILLBILL_URL=http://127.0.0.1:8080 EOF ---- -In the above script, replace the value of `db_host` with the full name of the DB writer endpoint obtained from the Connectivity and Security panel, as indicated earlier. Ensure to specify the appropriate database port number (`3306` for MySQL or `5432` for PostgreSQL) by setting `db_port`. The `kb_admin_password` value will be utilized as the admin password for Kaui login. +In the above script, replace the value of `db_host` with the full name of the DB writer endpoint obtained from the Connectivity and Security panel, as indicated earlier. Be sure to specify the appropriate database port number (`3306` for MySQL or `5432` for PostgreSQL) by setting `db_port`. The `kb_admin_password` value will be utilized as the admin password for Kaui login. Optionally, you may choose to customize other Kill Bill properties based on your specific needs. For more detailed information on available configuration options, please refer to the documentation provided at: https://docs.killbill.io/latest/userguide_configuration.html. From da7b862ef133b8f914d02037ff75ebb93edf665e Mon Sep 17 00:00:00 2001 From: jdmooney Date: Wed, 9 Aug 2023 18:18:45 -0400 Subject: [PATCH 09/14] FIrst draft usage_plugin.adoc --- userguide/tutorials/usage_plugin.adoc | 95 +++++++++++++++++++++++++++ 1 file changed, 95 insertions(+) create mode 100644 userguide/tutorials/usage_plugin.adoc diff --git a/userguide/tutorials/usage_plugin.adoc b/userguide/tutorials/usage_plugin.adoc new file mode 100644 index 000000000..5fe3ed3b3 --- /dev/null +++ b/userguide/tutorials/usage_plugin.adoc @@ -0,0 +1,95 @@ += Developing a Usage Plugin + +[[overview]] +== Overview + +The payment flows can be modified at runtime using a control plugin. + +Control plugins implement the https://github.com/killbill/killbill-plugin-api/blob/master/control/src/main/java/org/killbill/billing/control/plugin/api/PaymentControlPluginApi.java[PaymentControlPluginApi] and are invoked before as well as after the payment (either on success or on failure). These plugins can be used to abort a payment (e.g. fraud detection), schedule retries in case of failures, etc. + +To invoke a control plugin, simply pass the property *controlPluginName* when triggering the payment (multiple control plugins can be invoked in a pipeline). Kill Bill will create a payment attempt associated with that transaction (there is a 1-1 mapping between an attempt and a payment transaction), and store the plugin name(s) and properties which are used for retries (any plugin that sets a retry date is responsible to correctly remove sensitive information such as CVV, to avoid storing them on disk). + +For retries specifically, a payment will be associated to multiple transactions (all sharing the same transaction external key and typically in a `PAYMENT_FAILURE` status). Each of these transactions will be associated to an attempt in a `RETRIED` state. + +=== UsagePluginApi + +The https://github.com/killbill/killbill-plugin-api/blob/master/control/src/main/java/org/killbill/billing/control/plugin/api/UsagePluginApi.java[UsagePluginApi] exists to develop payment control plugins. Those plugins are called from the core payment system before payments get executed and after the payment operation has completed or failed. + +Implementers of these plugins need to implement the following APIs: + +* `getUsageForAccount`: This method gets usage data for a specified account. +* `getUsageForSubscription`: This method gets usage data for a particular subscription. + +Each of the APIs take the following parameters: + +* `startDate`: The beginning of the date range to be queried +* `endDate`: The end of the date range to be queried +* `context`: Provides the full `UsageContext` associated with this query (account, ...) +* `properties`: A list of `PluginProperty` items that are passed from the client making the API call to the plugin. + +In addition, the `getUsageForAccount` API includes a `subscriptionId` parameter. + +Both APIs return a `RawUsageResult` that contains: + + +== Use Cases + +There are many use cases associated with that plugin API. Let's describe a few below: + +=== Fraud Detection System + +A fraud detection system typically needs to run some real-time checks right before a payment is executed; often a score associated with the payment is computed, and based on the result different things can happen. Let's assume the following for the sake of an example. A score between 1-100 is returned from the plugin (or third-party system with which the plugin integrates): + +* If score < 25, payment should go through +* If 25 < score < 75, enable 3DSecure (use case of a credit card payment where the issuer supports such verification mechanism) +* If score score > 75, reject + +Implementing such a plugin becomes quite easy: One needs to implement the `priorCall` API, extract the score based on the `PaymentControlContext` (and potentially additional properties that could be used to tweak the behavior on a per call basis). Based on the score, the plugin would either: + +* Let the payment go through (case < 25) +* Add specific plugin properties that are required to activate the 3-DS (e.g client ip, browser info, ...) (25 < score < 75) +* Abort the payment by setting the `isAborted` from `PriorPaymentControlResult` otherwise + +Take a look at the https://github.com/killbill/killbill-accertify-plugin[Accertify] plugin for a concrete example. + +=== Payment Retries + +In some cases, we want to have the ability to control when a payment should be retried. The plugin needs to be able to control the following: + +* Rules on when to retry and when not to retry (e.g retry for `InsufficientFund` error and payment amount > $100) +* Date when the payment should be retried + +Implementing such a basic plugin is also quite easy. The developer needs to implement the business logic in the `onFailureCall` API to decide what to do upon failure, and provide the `nextRetryDate` from the `OnFailurePaymentControlResult if necessary. + +=== Payment Routing + +Payment routing is itself a vast topic and we could envision different types of payment routing: + +* Routing across different payment gateways/processors. The idea here is to address the following concerns: +** Resilience: If one gateway is down or if its latency becomes too high, re-route traffic to another one +** Cost optimization: Based on the contract negociated with the third party gateway start to move some traffic when some quota (e.g volume) was reached +** Acceptance Rate: A payment may be denied when going through a certain gateway and yet succeed when going through a different gateway (because of complicated set of rules that are implemented and which take into account all kinds of variables including the processor through which the payment flows) +* Routing across different payment methods. Assuming a given user has entered several payment methods (for example a CC and a PayPal account), the system could prompt the user to chose PayPal if the payment failed after using the CC. This obvioulsy requires some intricate integration from the plugin to the UI presented to the user, but this is certainly not impossible to achieve. + +The current `PaymentControlPluginApi` provides all the info and control to implement such logic because one can override the `paymentMethodId` in the `adjustedPaymentMethodId` of the `PriorPaymentControlResult` to modify at real time where to route the payment. + + +=== Disbursement + +Some merchants (organization) may accept some consumer payments for a service that is (partially) provided by an other entity (or entities). In such a scenario, it may become imperative to pass through the payment to that other entity. As an example we could envision a use case when the merchant accepting the payment would keep a fee that would be specified at the payment level. + +A possible implementation is to rely on the `PaymentControlPluginApi`. Any additional metadata associated with each payment (fee, details about the entity providing the service, ...) can easily be passed as a set of `PluginProperty`, that the plugin would extract to compute in real time what needs to be disbursed and which fees need to be kept. The logic associated with the disbursement could become quite complicated (rules on what to disburse to whom and when, batching, ...). There is a choice of implementing a full fledge plugin that takes cares of all this business logic (this is certainly possible since plugins can manage their own state, export new endpoints, ...) or having the plugin interact with a standalone service in charge of such operations. + +=== Currency Conversion + +In some cases, it may be necessary to offer a price in a currency and actually execute the payment using a different currency (this is based on a real use case where some brazilian customers would first pay in BRL for a service offered by a US based company, and then from one day to the next, the brazilian monetary policy changed and forbid payments in BRL outside of the country). In such scenario (among others) the choice is to lose those customers, or message them about changing the currency based on the current exchange rate, and implement the change. + +Fortunately implementing such a change is quite easy with the `PaymentControlPluginApi` because the the payment `amount` and `currency` can be overriden in the `PriorPaymentControlResult`. So, in such a scenario the plugin would implement the `priorCall` API to: + +* Ignore non `BRL` payments +* Perform the currency conversion for such `BRL` payments (by possibly integrating with a third party service for currency conversion), and return new `amount` and `currency` + + +=== Others + +There are many uses cases one could come up with, including some or a combination of the use cases presented above. Another dimension we have seen in the past is related to the Kill Bill integration with the rest of the stack. As an example, it could very well be that some pieces of the payment infrastrcuture already exist outside of Kill Bill (e.g access to the detail of a payment method), and in such case one could leverage this API in a clever way to make that integration possible. From e05b3c9056fad495177bc2419c7d438a18fb0d7c Mon Sep 17 00:00:00 2001 From: reshmabidikar Date: Fri, 11 Aug 2023 17:09:22 +0530 Subject: [PATCH 10/14] Deleted old migration doc and added link to the new doc to left nav --- html5/_main_toc.html.slim | 2 +- userguide/tutorials/database_migrations.adoc | 187 ------------------- 2 files changed, 1 insertion(+), 188 deletions(-) delete mode 100644 userguide/tutorials/database_migrations.adoc diff --git a/html5/_main_toc.html.slim b/html5/_main_toc.html.slim index 63f404d60..fe99d40b1 100644 --- a/html5/_main_toc.html.slim +++ b/html5/_main_toc.html.slim @@ -36,7 +36,7 @@ nav.sidebar-nav a.nav-link href="/latest/userguide_deployment.html" | Going to Production li - a.nav-link href="/latest/database_migrations.html" + a.nav-link href="/latest/how-to-upgrade-the-database.html" | Database Migrations li a.nav-link href="/latest/PostgreSQL.html" diff --git a/userguide/tutorials/database_migrations.adoc b/userguide/tutorials/database_migrations.adoc deleted file mode 100644 index 3f3bc7d8e..000000000 --- a/userguide/tutorials/database_migrations.adoc +++ /dev/null @@ -1,187 +0,0 @@ -= Kill Bill Database Migrations - -== Overview - -A typical Kill Bill deployment is comprised of several components: The core Kill Bill webapp and some plugins. Migrating the software to a higher version may require to migrate one, a few or all the different components and so potentially upgrading the database schema associated with one, a few or all of those components. The compatibility version between core Kill Bill and its plugins depends on the version of the https://github.com/killbill/killbill-plugin-api[plugin api] exported by a given Kill Bill version. The https://github.com/killbill/killbill-cloud/tree/master/kpm[KPM] tool allows to list compatible plugins associated with a given Kill Bill version: - -[source,bash] ----- -kpm info --version ----- - -Kill Bill and individual plugins assume specific tables, functions, etc. are installed in your database. The source of truth for the schema (DDL) files are: - -* In individual modules for Kill Bill, under `\*/src/main/resources/org/killbill/billing/*/ddl.sql` (for example https://github.com/killbill/killbill/tree/master/account/src/main/resources/org/killbill/billing/account/ddl.sql[account/src/main/resources/org/killbill/billing/account/ddl.sql]). -* Under `src/main/resources/ddl.sql` for Java plugins (https://github.com/killbill/killbill-adyen-plugin/tree/master/src/main/resources/ddl.sql[Adyen example]). - -These files always contain the latest version, matching the latest code in that specific branch/tag. - -To access a specific version: - -* For Kill Bill itself, we publish a full DDL file per major version (https://docs.killbill.io/0.22/ddl.sql[0.22 example]). The DDL for the latest stable version of Kill Bill is at https://docs.killbill.io/latest/ddl.sql[https://docs.killbill.io/latest/ddl.sql]. -* For individual plugins, you can get the DDL file of a given version on GitHub by looking-up the specific tag (for v4.0.2 of the CyberSource plugin, it would be at https://github.com/killbill/killbill-cybersource-plugin/tree/v4.0.2/db/ddl.sql). - -== Migrations with Docker images - -The `Migrations tooling` section below goes into details on how migrations are handled for individual components, and the tooling available to you to update your database schema when upgrading Kill Bill. - -If you are using our standard Docker images, the procedure is simplified and as follows: - -1. Update the `/var/lib/killbill/kpm.yml` file in one of your containers to reflect the new version of Kill Bill you want to upgrade to. -2. Run `$MIGRATIONS_CMD`. This will: - ** Configure the database for migrations if needed (i.e. create the `schema_version` table(s), see below) - ** Fetch all migrations needed to upgrade to the Kill Bill version specified in the `kpm.yml` - ** Display the SQL that would be run (for Kill Bill and each plugin) and prompt the user whether it should be applied - -Depending on your dataset size, you may or may not want the script to apply these directly (e.g. you may prefer to use an online-schema-change tool for production), in which case you can simply grab the SQL output instead. - -Caveats: - -* This is only supported since `killbill/killbill:0.20.10` -* Ruby migrations aren't supported (if you have any, see the manual steps below) - -== Migrations tooling - -In order to ease the database migrations, we decided to rely on standard tools for migrating from one given version to a higher version: - -* For Java componenents (Kill Bill itself, java plugins), we rely on https://flywaydb.org/[Flyway] -* For Ruby components (KAUI, ...), we rely on http://edgeguides.rubyonrails.org/active_record_migrations.html[Active Record migrations] - -In addition to these standard tools, we also enhanced our existing tooling to support the following use cases: - -* Ability to download all the migration files associated with a migration from version `N` to `M` (see section `KPM` below) -* Ability to output SQL (migration files) for production systems (see section `SQL output` below) - -=== KPM - -We extended (https://github.com/killbill/killbill-cloud/tree/master/kpm[KPM] with a new `migrations` verb to be able to **download migration files** that should be applied when migrating from one version to the next. - -Usage: - -[source,bash] ----- -kpm migrations github_repository from_tag to_tag ----- - -For example, to download all migration files for Kill Bill between 0.16.12 and 0.18.0: - -``` -kpm migrations killbill killbill-0.16.12 killbill-0.18.0 -``` - -Similar examples for Java plugins: - -[source,bash] ----- -kpm migrations killbill-analytics-plugin analytics-plugin-3.0.0 analytics-plugin-3.0.1 -kpm migrations killbill-cybersource-plugin v4.0.1 v4.0.2 ----- - -Because the implementation relies on the https://developer.github.com/[GitHub API], unauthenticated requests are subject to https://developer.github.com/v3/#rate-limiting[rate limiting]. To work around it, go to https://github.com/settings/tokens to generate a token (default public, read-only, permissions will work) and specify it to KPM: - -[source,bash] ----- -kpm migrations killbill killbill-0.16.3 killbill-0.16.4 --token=TOKEN ----- - -=== SQL output - -In production environments, database access is often restricted and developers don't necessarily have rights to execute DDL commands (i.e. `CREATE`, `ALTER`, `DROP`, etc. statements). To get around this, we provide scripts to get access to the SQL that needs to be run for the migrations, including the `INSERT` statements to update the metadata table(s). - -==== Java - -For Java code (Kill Bill and Java plugins), we publish a `killbill-flyway.jar` binary which is a wrapper around the `flyway` utility and exposes a `dryRunMigrate` command. Simply run it against your production database (read-only credentials are sufficient) to get the raw SQL. - -Usage example: - -[source,bash] ----- -# /var/folders/tmp/migrations is the directory where the migration files were downoaded (using kpm migrate) -java -jar killbill-flyway.jar -locations=filesystem:/var/folders/tmp/migrations -url='jdbc:mysql://DATABASE_IP:DATABASE_PORT/DATABASE_NAME' -user= -password= dryRunMigrate ----- - -==== Ruby - -For Ruby code, we provide a `killbill-migration` script with the `killbill` gem (JRuby is recommended, see our https://docs.killbill.io/latest/payment_plugin.html#_building_ruby_plugins[installation instructions]). - -Besides outputting the SQL, it can also run the migrations. - -Usage examples: - -[source,bash] ----- -killbill-migration current_version cybersource -killbill-migration sql_for_migration cybersource --path=XXX -killbill-migration migrate cybersource --path=XXX -killbill-migration ruby_dump cybersource -killbill-migration sql_dump cybersource ----- - -Options `--username`, `--password`, `--database` and `--host` are supported. - -If you are a plugin developer, these commands are also available as part of the default Rake tasks of the plugin itself, for example: - -[source,bash] ----- -PLUGIN_NAME=paypal_express USERNAME=root PASSWORD=root DB=killbill rake killbill:db:sql_for_migration -PLUGIN_NAME=paypal_express USERNAME=root PASSWORD=root DB=killbill rake killbill:db:migrate ----- - -Note that the `PLUGIN_NAME` variable needs to match the prefix of the `schema_migrations` table. - - -=== Kill Bill - -Starting with Kill Bill 0.16.4, SQL migrations files for Flyway are provided. They can be found on a per-module basis under `\*/src/main/resources/org/killbill/billing/*/migration/`. The versioning is based on the creation timestamp (i.e. `date +'%Y%m%d%H%M%S'`) and must be unique for each file so as to indicate the ordering (what Flyway calls the `version_rank`). - -==== Baseline - -For Flyway migrations to work correctly, there is an initial https://flywaydb.org/documentation/command/baseline[baseline] operation that needs to happens so as to create the metadata table, called `schema_version`. In the scenario where a developer can run its own migrations, the following command would be run: - -[source,bash] ----- -flyway -url='jdbc:mysql://DATABASE_IP:DATABASE_PORT/DATABASE_NAME' -user= -password= baseline ----- - -In the production scenario, the `schema_version` along with the initial version would have to be created manually: - -[source,sql] ----- -CREATE TABLE `schema_version` ( - `installed_rank` int(11) NOT NULL, - `version` varchar(50) DEFAULT NULL, - `description` varchar(200) NOT NULL, - `type` varchar(20) NOT NULL, - `script` varchar(1000) NOT NULL, - `checksum` int(11) DEFAULT NULL, - `installed_by` varchar(100) NOT NULL, - `installed_on` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, - `execution_time` int(11) NOT NULL, - `success` tinyint(1) NOT NULL, - PRIMARY KEY (`installed_rank`), - KEY `schema_version_s_idx` (`success`) -) ENGINE=InnoDB DEFAULT CHARSET=utf8; - -insert into schema_version (installed_rank, version, description, type, script, installed_by, installed_on, execution_time, success) VALUES (1, 1, '<< Flyway Baseline >>', 'BASELINE', '<< Flyway Baseline >>', 'admin', NOW(), 0, 1); ----- - -==== Migration from version `N` to `M` - -A typical migration from version `N` to `M` will require to first identify the set of migrations files that should be applied and then either apply them using Flyway commands or manually (production use case). - - -[source,bash] ----- -# Will download migration files in a temporary folder (e.g /var/folders/XXX) -kpm migrations killbill killbill-N killbill-M --token='GITHUB_TOKEN' - -# If using flyway is an option -flyway -url='jdbc:mysql://DATABASE_IP:DATABASE_PORT/DATABASE_NAME' -user= -password= -locations=filesystem:/var/folders/XXX migrate - ----- - -=== Java plugins - -Java plugins migrations also rely on https://flywaydb.org/[Flyway] and follow a similar workflow than Kill Bill itself. They can be found in each plugin under `src/main/resources/migration/`. - -The metadata table should be called `_schema_version`. Make sure to specify `-table=_schema_version` when invoking Flyway. \ No newline at end of file From f34ee9dd6d0730a41d26a3dbdb72a72637945694 Mon Sep 17 00:00:00 2001 From: vnandwana Date: Sat, 12 Aug 2023 16:14:35 +0530 Subject: [PATCH 11/14] Corrected multi-tier product listing URL --- userguide/tutorials/aws.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/userguide/tutorials/aws.adoc b/userguide/tutorials/aws.adoc index 7846c8645..dacdfbcca 100644 --- a/userguide/tutorials/aws.adoc +++ b/userguide/tutorials/aws.adoc @@ -24,7 +24,7 @@
Multi-Tier single AMI with external database

Fully customizable production environment (advanced users)

- Launch + Launch

Docs

From 1161017dae78753c3fb25e6e9e02a11059c324b3 Mon Sep 17 00:00:00 2001 From: jdmooney Date: Sun, 13 Aug 2023 16:28:49 -0400 Subject: [PATCH 12/14] usage_plugin.adoc first complete draft --- userguide/tutorials/usage_plugin.adoc | 74 ++------------------------- 1 file changed, 5 insertions(+), 69 deletions(-) diff --git a/userguide/tutorials/usage_plugin.adoc b/userguide/tutorials/usage_plugin.adoc index 5fe3ed3b3..28ab36d5a 100644 --- a/userguide/tutorials/usage_plugin.adoc +++ b/userguide/tutorials/usage_plugin.adoc @@ -3,19 +3,12 @@ [[overview]] == Overview -The payment flows can be modified at runtime using a control plugin. +A usage plugin implements the https://github.com/killbill/killbill-plugin-api/blob/master/usage/src/main/java/org/killbill/billing/usage/plugin/api/UsagePluginApi.java[UsagePluginApi]. This plugin can be used to collect usage statistics that are collected by a separate application. This application may have specialized knowledge of how to measure specific types of resource units, such as cellphone minutes. -Control plugins implement the https://github.com/killbill/killbill-plugin-api/blob/master/control/src/main/java/org/killbill/billing/control/plugin/api/PaymentControlPluginApi.java[PaymentControlPluginApi] and are invoked before as well as after the payment (either on success or on failure). These plugins can be used to abort a payment (e.g. fraud detection), schedule retries in case of failures, etc. - -To invoke a control plugin, simply pass the property *controlPluginName* when triggering the payment (multiple control plugins can be invoked in a pipeline). Kill Bill will create a payment attempt associated with that transaction (there is a 1-1 mapping between an attempt and a payment transaction), and store the plugin name(s) and properties which are used for retries (any plugin that sets a retry date is responsible to correctly remove sensitive information such as CVV, to avoid storing them on disk). - -For retries specifically, a payment will be associated to multiple transactions (all sharing the same transaction external key and typically in a `PAYMENT_FAILURE` status). Each of these transactions will be associated to an attempt in a `RETRIED` state. === UsagePluginApi -The https://github.com/killbill/killbill-plugin-api/blob/master/control/src/main/java/org/killbill/billing/control/plugin/api/UsagePluginApi.java[UsagePluginApi] exists to develop payment control plugins. Those plugins are called from the core payment system before payments get executed and after the payment operation has completed or failed. - -Implementers of these plugins need to implement the following APIs: +Developers of usage plugins need to implement the following APIs: * `getUsageForAccount`: This method gets usage data for a specified account. * `getUsageForSubscription`: This method gets usage data for a particular subscription. @@ -24,72 +17,15 @@ Each of the APIs take the following parameters: * `startDate`: The beginning of the date range to be queried * `endDate`: The end of the date range to be queried -* `context`: Provides the full `UsageContext` associated with this query (account, ...) +* `context`: Provides the full `UsageContext` associated with this query, including the account ID. * `properties`: A list of `PluginProperty` items that are passed from the client making the API call to the plugin. In addition, the `getUsageForAccount` API includes a `subscriptionId` parameter. -Both APIs return a `RawUsageResult` that contains: +Both APIs return a https://github.com/killbill/killbill-api/blob/master/src/main/java/org/killbill/billing/usage/api/RawUsageRecord.java[RawUsageRecord]. == Use Cases -There are many use cases associated with that plugin API. Let's describe a few below: - -=== Fraud Detection System - -A fraud detection system typically needs to run some real-time checks right before a payment is executed; often a score associated with the payment is computed, and based on the result different things can happen. Let's assume the following for the sake of an example. A score between 1-100 is returned from the plugin (or third-party system with which the plugin integrates): - -* If score < 25, payment should go through -* If 25 < score < 75, enable 3DSecure (use case of a credit card payment where the issuer supports such verification mechanism) -* If score score > 75, reject - -Implementing such a plugin becomes quite easy: One needs to implement the `priorCall` API, extract the score based on the `PaymentControlContext` (and potentially additional properties that could be used to tweak the behavior on a per call basis). Based on the score, the plugin would either: - -* Let the payment go through (case < 25) -* Add specific plugin properties that are required to activate the 3-DS (e.g client ip, browser info, ...) (25 < score < 75) -* Abort the payment by setting the `isAborted` from `PriorPaymentControlResult` otherwise - -Take a look at the https://github.com/killbill/killbill-accertify-plugin[Accertify] plugin for a concrete example. - -=== Payment Retries - -In some cases, we want to have the ability to control when a payment should be retried. The plugin needs to be able to control the following: - -* Rules on when to retry and when not to retry (e.g retry for `InsufficientFund` error and payment amount > $100) -* Date when the payment should be retried - -Implementing such a basic plugin is also quite easy. The developer needs to implement the business logic in the `onFailureCall` API to decide what to do upon failure, and provide the `nextRetryDate` from the `OnFailurePaymentControlResult if necessary. - -=== Payment Routing - -Payment routing is itself a vast topic and we could envision different types of payment routing: - -* Routing across different payment gateways/processors. The idea here is to address the following concerns: -** Resilience: If one gateway is down or if its latency becomes too high, re-route traffic to another one -** Cost optimization: Based on the contract negociated with the third party gateway start to move some traffic when some quota (e.g volume) was reached -** Acceptance Rate: A payment may be denied when going through a certain gateway and yet succeed when going through a different gateway (because of complicated set of rules that are implemented and which take into account all kinds of variables including the processor through which the payment flows) -* Routing across different payment methods. Assuming a given user has entered several payment methods (for example a CC and a PayPal account), the system could prompt the user to chose PayPal if the payment failed after using the CC. This obvioulsy requires some intricate integration from the plugin to the UI presented to the user, but this is certainly not impossible to achieve. - -The current `PaymentControlPluginApi` provides all the info and control to implement such logic because one can override the `paymentMethodId` in the `adjustedPaymentMethodId` of the `PriorPaymentControlResult` to modify at real time where to route the payment. - - -=== Disbursement - -Some merchants (organization) may accept some consumer payments for a service that is (partially) provided by an other entity (or entities). In such a scenario, it may become imperative to pass through the payment to that other entity. As an example we could envision a use case when the merchant accepting the payment would keep a fee that would be specified at the payment level. - -A possible implementation is to rely on the `PaymentControlPluginApi`. Any additional metadata associated with each payment (fee, details about the entity providing the service, ...) can easily be passed as a set of `PluginProperty`, that the plugin would extract to compute in real time what needs to be disbursed and which fees need to be kept. The logic associated with the disbursement could become quite complicated (rules on what to disburse to whom and when, batching, ...). There is a choice of implementing a full fledge plugin that takes cares of all this business logic (this is certainly possible since plugins can manage their own state, export new endpoints, ...) or having the plugin interact with a standalone service in charge of such operations. - -=== Currency Conversion - -In some cases, it may be necessary to offer a price in a currency and actually execute the payment using a different currency (this is based on a real use case where some brazilian customers would first pay in BRL for a service offered by a US based company, and then from one day to the next, the brazilian monetary policy changed and forbid payments in BRL outside of the country). In such scenario (among others) the choice is to lose those customers, or message them about changing the currency based on the current exchange rate, and implement the change. - -Fortunately implementing such a change is quite easy with the `PaymentControlPluginApi` because the the payment `amount` and `currency` can be overriden in the `PriorPaymentControlResult`. So, in such a scenario the plugin would implement the `priorCall` API to: - -* Ignore non `BRL` payments -* Perform the currency conversion for such `BRL` payments (by possibly integrating with a third party service for currency conversion), and return new `amount` and `currency` - - -=== Others +The main use case for this plugin is to collect information on resource usage that can be used for usage-based billing. -There are many uses cases one could come up with, including some or a combination of the use cases presented above. Another dimension we have seen in the past is related to the Kill Bill integration with the rest of the stack. As an example, it could very well be that some pieces of the payment infrastrcuture already exist outside of Kill Bill (e.g access to the detail of a payment method), and in such case one could leverage this API in a clever way to make that integration possible. From cd4843c9acf842c4154c13f8758b4ebaf63ff3a3 Mon Sep 17 00:00:00 2001 From: jdmooney Date: Tue, 15 Aug 2023 14:12:36 -0400 Subject: [PATCH 13/14] Some fixes to usage_plugin.adoc --- userguide/tutorials/usage_plugin.adoc | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/userguide/tutorials/usage_plugin.adoc b/userguide/tutorials/usage_plugin.adoc index 28ab36d5a..b36f4c52b 100644 --- a/userguide/tutorials/usage_plugin.adoc +++ b/userguide/tutorials/usage_plugin.adoc @@ -18,14 +18,19 @@ Each of the APIs take the following parameters: * `startDate`: The beginning of the date range to be queried * `endDate`: The end of the date range to be queried * `context`: Provides the full `UsageContext` associated with this query, including the account ID. -* `properties`: A list of `PluginProperty` items that are passed from the client making the API call to the plugin. +* `properties`: An optional list of `PluginProperty` items that are passed from the plogin to the caller. -In addition, the `getUsageForAccount` API includes a `subscriptionId` parameter. +In addition, the `getUsageForSubsceiption` API includes a `subscriptionId` parameter. -Both APIs return a https://github.com/killbill/killbill-api/blob/master/src/main/java/org/killbill/billing/usage/api/RawUsageRecord.java[RawUsageRecord]. +Both APIs return a https://github.com/killbill/killbill-api/blob/master/src/main/java/org/killbill/billing/usage/api/RawUsageRecord.java[RawUsageRecord]. This record includes: + +* `date`: The date of the record +* `unitType`: The type of resource used +* `amount`: The number of units used +* `trackingId`: Id for the record == Use Cases -The main use case for this plugin is to collect information on resource usage that can be used for usage-based billing. +The main use case for this plugin is to collect usage data that can be used for usage-based billing. From 7a340aa38398b0770ee47833f4295defaa44682b Mon Sep 17 00:00:00 2001 From: jdmooney Date: Thu, 17 Aug 2023 09:43:45 -0400 Subject: [PATCH 14/14] A few more corrections to usage_plugin.adoc --- userguide/tutorials/usage_plugin.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/userguide/tutorials/usage_plugin.adoc b/userguide/tutorials/usage_plugin.adoc index b36f4c52b..7ba8759a9 100644 --- a/userguide/tutorials/usage_plugin.adoc +++ b/userguide/tutorials/usage_plugin.adoc @@ -3,7 +3,7 @@ [[overview]] == Overview -A usage plugin implements the https://github.com/killbill/killbill-plugin-api/blob/master/usage/src/main/java/org/killbill/billing/usage/plugin/api/UsagePluginApi.java[UsagePluginApi]. This plugin can be used to collect usage statistics that are collected by a separate application. This application may have specialized knowledge of how to measure specific types of resource units, such as cellphone minutes. +A usage plugin implements the https://github.com/killbill/killbill-plugin-api/blob/master/usage/src/main/java/org/killbill/billing/usage/plugin/api/UsagePluginApi.java[UsagePluginApi]. This plugin can be used to collect usage data that is collected by a separate application. This application may have specialized knowledge of how to measure specific types of resource units, such as cellphone minutes. === UsagePluginApi @@ -22,10 +22,10 @@ Each of the APIs take the following parameters: In addition, the `getUsageForSubsceiption` API includes a `subscriptionId` parameter. -Both APIs return a https://github.com/killbill/killbill-api/blob/master/src/main/java/org/killbill/billing/usage/api/RawUsageRecord.java[RawUsageRecord]. This record includes: +Both APIs return a list of https://github.com/killbill/killbill-api/blob/master/src/main/java/org/killbill/billing/usage/api/RawUsageRecord.java[RawUsageRecords]. Each record includes: * `date`: The date of the record -* `unitType`: The type of resource used +* `unitType`: The type of resource used (as defined in the catalog) * `amount`: The number of units used * `trackingId`: Id for the record